zfs_pool
zfs_pool checks one ZFS pool on the node. zfs_pool is an
official extra module: a
Starlark script written by the MMWatchdog project. Every deploy with the
release's MMX workflows installs zfs_pool on every node. A check names the module
and the pool, and MMWatchdog runs the module once for each run of the check, on the
node.
The module asks the zpool program whether the pool is healthy, how full it
is, how fragmented its free space is, and how many days ago it was last
scrubbed. A scrub is ZFS's own check of a pool: a scrub reads every block
and repairs blocks that do not match the blocks' checksum from another copy.
The check fails while the pool is not healthy. A measurement limit can
make the check fail when the pool is too full or has gone too long without a scrub.
Choose zfs_pool for the health and the space of a whole pool. The core
module node_filesystem measures
one mounted filesystem, such as one ZFS dataset, and knows nothing of the
pool's devices.
Set up
- The programs
zpoolandzfs, which every node with ZFS has in the same package, such aszfsutils-linuxon Ubuntu. - No special account. The check runs as the unprivileged account
mmwatchdog, which may list a pool, read its status, and read when it was created. - Something that scrubs the pool, for a limit on
days_since_last_scrub. Ubuntu's and Debian's packagezfsutils-linuxscrubs every pool on the second Sunday of each month. Elsewhere, enable OpenZFS's monthly timer for each pool, such assystemctl enable --now zfs-scrub-monthly@tank.timer.
Inputs
| Input | Meaning | When omitted |
|---|---|---|
pool_name |
The name of the ZFS pool, as zpool list prints it, such as tank |
Required: mmwatchdog validate refuses the check |
Measurements
| Measurement | Unit | Meaning |
|---|---|---|
space_used_percent |
percent | The share of the pool's space that is in use, 0 to 100 |
free_space_fragmentation_percent |
percent | How fragmented the free space of the pool is, which zpool list prints in its column FRAG. The number says nothing about how fragmented the files are. A pool without a number in FRAG has no free_space_fragmentation_percent measurement |
days_since_last_scrub |
days | The time since the last scrub of the pool ended, such as 14.5. The next table says what days_since_last_scrub counts from while a scrub has not ended |
A check can set limits on space_used_percent,
free_space_fragmentation_percent, and days_since_last_scrub.
Write fail_when_measurement_above or fail_when_measurement_below beside
the module, and MMWatchdog applies the limits
(checks that measure a number).
A check with a limit on free_space_fragmentation_percent is unknown for a
pool that has no such number.
What days_since_last_scrub counts from
The module reads the line scan: of zpool status tank. When the line
says a scrub completed, MMWatchdog keeps the completed scrub's date between runs.
A resilver copies data onto a disk that was replaced or came back.
A resilver is not a scrub: it does not read every block of the whole pool.
ZFS replaces the previous scan line when another scan starts. MMWatchdog's saved
completed scrub date stays unchanged until another scrub completes.
| What MMWatchdog has observed | days_since_last_scrub counts from |
The message of a healthy run ends with |
|---|---|---|
The scan line says scrub repaired ... on Sun Oct 11 00:36:02 2026 |
The end of the scrub named in the scan line. MMWatchdog saves the date | ; its last scrub ended on Sun Oct 11 00:36:02 2026 |
| A scrub runs, is paused or canceled, or a resilver replaces the scan line | The saved date of the last completed scrub | ; its last completed scrub observed by MMWatchdog ended on Sun Oct 11 00:36:02 2026 |
| MMWatchdog has no saved completed scrub date, and the scan line shows none | The pool's creation | ; MMWatchdog has not observed a completed scrub, so days_since_last_scrub counts from the pool's creation |
Without a saved completed scrub date, the pool's creation is a conservative starting point. A completed scrub that MMWatchdog never observed cannot be recovered from a later scan line. zpool writes each date in the node's own time zone, and the module reads the date in that same time zone.
When the check fails, and when it cannot check
| The pool | The run | Message |
|---|---|---|
Is ONLINE and zpool status -x calls the pool healthy |
Healthy | the ZFS pool tank is ONLINE and healthy |
Is in another state, such as DEGRADED or FAULTED |
Failed | the ZFS pool tank is DEGRADED, and after a colon what zpool status says is wrong |
Is ONLINE and has a problem, such as a device with errors |
Failed | the ZFS pool tank is ONLINE and has a problem, and after a colon what zpool status says is wrong |
| Is unavailable to programs on the node because the node did not import the pool | Failed | this node has no ZFS pool named tank |
| Is above a limit on a measurement | Failed | space_used_percent is 83 (fails above 80); the module reported: the ZFS pool tank is ONLINE and healthy; its last scrub ended on Sun Oct 11 00:36:02 2026 |
| Has gone longer without a scrub than a limit allows | Failed | days_since_last_scrub is 40.5 (fails above 35); the module reported: the ZFS pool tank is ONLINE and healthy; its last scrub ended on Sun Sep 6 00:31:12 2026 |
Cannot be asked or listed, or zpool cannot say whether the pool has a problem |
Unknown | See errors and what they mean |
- A pool that is missing fails the check on purpose: a pool that was not imported at the start of the machine is not there for the programs that use it.
- What
zpool statussays is wrong is the text after its headingstatus:, whichzpoolwrites on several lines. The module joins the text afterstatus:into one line. - The module reports the measurements with every run that lists the pool, also with a failed one.
- MMWatchdog applies a limit only to a healthy run. A failed run stays failed
with the module's message. With a limit, the message of a healthy run
starts with the reading and its limit, such as
space_used_percent is 42 (fails above 80); the module reported: the ZFS pool tank is ONLINE and healthy; its last scrub ended on Sun Oct 11 00:36:02 2026. - A resilver that runs fails the check by itself:
zpool status -xcalls the pool's state a problem until the resilver ends, and the message saysthe ZFS pool tank is ONLINE and has a problem: One or more devices is currently being resilvered.
MMWatchdog sends the alert when the check has failed for alert_after_failing_for.
When the check has been unknown for alert_after_failing_for, MMWatchdog
opens a cannot-check incident
(the failing time).
Examples
A pool that is healthy and below 80 percent
# /etc/mmwatchdog/check_groups/storage.yml
check_groups:
storage:
check_every: 1m
time_limit_per_run: 20s
alert_after_failing_for: 0s
recovered_after_healthy_for: 1m
checks:
pool_tank_is_healthy_and_below_80_percent:
zfs_pool: # the module
pool_name: tank # the pool, as zpool list prints it
fail_when_measurement_above:
space_used_percent: 80
alert_severity: sos
Every minute MMWatchdog runs the module on the node, as the account mmwatchdog. The check
fails while the pool tank is not healthy, and when the pool is more than
80 percent full. A ZFS pool gets slow long before it is full, so a limit of
80 percent is a common choice. When one disk of a mirrored pool fails, the pool
turns DEGRADED, and MMWatchdog sends the sos alert at once, with what
zpool status says is wrong.
After the disk is replaced and the pool is healthy for a minute,
MMWatchdog sends the "ok again".
Two levels of space, and fragmentation
# /etc/mmwatchdog/check_groups/storage.yml
check_groups:
storage:
check_every: 5m
time_limit_per_run: 20s
alert_after_failing_for: 15m
recovered_after_healthy_for: 15m
checks:
pool_tank_is_below_80_percent:
zfs_pool:
pool_name: tank
fail_when_measurement_above:
space_used_percent: 80
free_space_fragmentation_percent: 50
alert_severity: warn
pool_tank_is_below_90_percent:
zfs_pool:
pool_name: tank
fail_when_measurement_above:
space_used_percent: 90
alert_severity: sos
The first check sends a warn alert when the pool has been more than 80
percent full, or its free space more than 50 percent fragmented, for 15
minutes. The second check sends an sos alert above 90 percent. Each check
also fails while the pool is not healthy, so a degraded pool sends the
warn and sos alerts.
A pool that goes too long without a scrub
# /etc/mmwatchdog/check_groups/storage.yml
check_groups:
storage:
check_every: 1h
time_limit_per_run: 20s
alert_after_failing_for: 0s
recovered_after_healthy_for: 0s
checks:
pool_tank_was_scrubbed_within_35_days:
zfs_pool:
pool_name: tank
fail_when_measurement_above:
days_since_last_scrub: 35 # a monthly scrub, with a few days to spare
alert_severity: warn
Every hour MMWatchdog runs the module on the node. The pool tank is scrubbed once
a month, by the scrub of Ubuntu's or Debian's package or by OpenZFS's
monthly timer. When no scrub has ended for 35 days, because the timer is
off or a scrub never ends, the check fails.
MMWatchdog then sends a warn alert.
The check also fails while the pool is not healthy. A scrub that runs or is canceled keeps the reading counting from the last completed scrub. A new pool counts from creation, so the new pool sends no alert in the first 35 days. After the next scrub ends, the check is healthy again and MMWatchdog sends the "ok again".
How it works
Each run of the check is one run of the script module.star, as the account
mmwatchdog, under the check's time_limit_per_run:
- The module runs
zpool list -H -p -o name,health,capacity,fragmentation tank.-Hwrites one line without a heading, with a tab between the columns, and-pwrites plain numbers:42and not42%. - The module runs
zpool status tank, for what is wrong with the pool and for its scan line. A completed scrub updates the date kept withwrite_state; other scan lines use the date fromread_state.date --file=- +%sconverts the completed date andnowinto seconds since 1970. With no known completed scrub, the module gets the pool's creation fromzfs get -H -p -o value creation tank. The module subtracts the last completed scrub's timestamp, or the pool's creation timestamp, from the current timestamp. Dividing the elapsed seconds by 86,400, the seconds in a day, givesdays_since_last_scrub. - A pool that is not
ONLINEfails, with whatzpool statussays is wrong. - For a pool that is
ONLINE, the module runszpool status -x tank, which saysis healthyfor a pool without a problem. Another answer fails, with whatzpool statussays is wrong.
Errors and what they mean
| Message | What the message means and what to do |
|---|---|
check_groups.storage.checks.pool_tank_is_healthy_and_below_80_percent.zfs_pool.pool_name: is required: ... |
mmwatchdog validate: the check leaves out pool_name |
check_groups.storage.checks.pool_tank_is_healthy_and_below_80_percent.fail_when_measurement_above.used_percent: this check has no such measurement; it measures ... |
mmwatchdog validate: a limit names a measurement that the module does not report. Use space_used_percent or free_space_fragmentation_percent |
this node has no ZFS pool named tank |
Failed. The pool is not imported on this node. Import the pool with zpool import tank, or correct pool_name |
the ZFS pool tank is ONLINE, and zpool could not say whether it has a problem: ... |
Unknown. zpool status -x ended with an error, and the text after the colon says why |
zpool could not list the pool tank: ... |
Unknown. zpool list failed for another reason than a missing pool, and the text after the colon is zpool's own words |
zpool gave an answer about the pool tank that the module does not understand: ... |
Unknown. The answer of zpool list has another form than the module reads, and the text after the colon is its first line |
MMWatchdog cannot run the zpool program: ... |
Unknown. The node has no zpool program, or the account mmwatchdog cannot run it. Install ZFS, or remove the check from a node without ZFS |
the result has no measurement named free_space_fragmentation_percent, which this check has a limit on |
Unknown. The pool has no fragmentation number. Remove the limit on free_space_fragmentation_percent for this pool |
the result has no measurement named days_since_last_scrub, which this check has a limit on |
Unknown. The module could not measure the time since the last scrub. Run sudo mmwatchdog check-now storage; the message of a check without a limit on days_since_last_scrub ends with one of the texts below |
...; MMWatchdog could not read when it was last scrubbed: zpool status ended with ... |
zpool status tank failed, and the text after ended with is zpool's own words |
...; MMWatchdog could not read the date Sun Oct 11 00:36:02 2026 with the date program: ... |
The date program did not read the date of the scan line. The text after the colon is the date program's own words |
...; MMWatchdog has not observed a completed scrub and could not read when the pool was created: zfs get creation ended with ... |
No completed scrub date is known, and zfs get creation failed, for example because the node has no zfs program |
The errors that every module that is not core can meet, such as a run past
time_limit_per_run, are in
checks with a module that is not core.
Tested on
-
On Ubuntu 24.04 and 26.04, against a real pool (
tests/integration/service_scenarios.py, in the services suite). The scenario installs Ubuntu's packagezfsutils-linux, and loads the ZFS kernel module that comes with Ubuntu's kernel. The pool mirrors two files. The check finds the pool healthy and measures its use and its fragmentation, fails with zpool's own reason while one of the two files used as mirrored devices is offline, and fails when the pool is exported. A pool that is not there fails.A check with a limit on
days_since_last_scrubreads the real scan line in each form: a new pool that was never scrubbed, a scrub that ended, a scrub that runs, is paused, and is canceled, and the resilver after a mirrored device came back. OpenZFS's settingzfs_scan_suspend_progresssuspends scrub progress while ZFS still reports the scrub as running.Ubuntu 24.04 has OpenZFS 2.2.2, and Ubuntu 26.04 has 2.4.1. OpenZFS 2.1 and 2.3, which target distributions outside Ubuntu can install, write the scan line with the words used by OpenZFS 2.2.2 and 2.4.1.
-
On every other target distribution, on a node without ZFS: Debian 12 and 13, Rocky Linux 9, AlmaLinux 10, Fedora 45, Arch Linux, and openSUSE Leap (every target distribution). The check is unknown, with
MMWatchdog cannot run the zpool program:.Ubuntu is the one target distribution whose own packages hold ZFS ready to use. Debian and the others build the kernel module on the node, and the Red Hat family needs the OpenZFS project's own package source. MMWatchdog's tests build no kernel module and add no package source. The module only reads what
zpoolprints, so its code is the same on every distribution. -
Go tests (
tests/official_extra_modules/check_modules_test.go,TestZFSPool) run the script throughmmwatchdog run-starlark-modulewith stand-ins forzpool,zfs, anddatethat write answers in their output formats. Pool health cases cover healthy, degraded with zpool's whole reason, online with a problem, a missing pool, azpoolthat cannot report health, and a missingzpoolprogram. Missing-measurement cases cover a missing fragmentation number and a date thatdatecannot read.Scrub-age cases cover each scan-line form in the scrub-age table, including a day of the month below 10. Separate module runs test retention of a completed scrub date through a running, paused, or canceled scrub and a resilver. The Go tests ran with
go test ./...on macOS on ARM64.
Why it works this way
- The module asks
zpooland reads its plain numbers.zpoolis the program that every node with ZFS has, and-H -pgives one line that a script reads without guessing. - For an
ONLINEpool, the module also runszpool status -x. A pool can beONLINEand still have a problem, such as a device with errors, and onlyzpool status -xsays so. - A missing pool fails, and is not unknown. The pool is not there for the programs that use it, which is a failure that someone must fix.
- The module measures the time since the last scrub. In the beta
validation with real-world Monit work, a company's production setup
checked ZFS pools with a shell script of 147 lines. The script also
alerted on a pool that had gone 90 days without a scrub. A limit on
days_since_last_scrubalerts on overdue scrubs without a script. - Only a completed scrub resets the reading. Starting or canceling a scrub and replacing a disk do not prove the whole pool was checked. Keeping the completed date prevents a started or canceled scrub, or a disk replacement's resilver, from hiding an overdue scrub.
- Without a known completed scrub, the reading counts from creation. ZFS keeps only its latest scan line, so an older completed scrub may no longer be visible when MMWatchdog first runs. Counting from creation is conservative and does not let a canceled scrub make an old pool healthy.
- The module asks the
dateprogram for the time. A script has no clock of its own. Thedateprogram reads zpool's date in the node's own time zone, in which zpool writes the date. - The measurement is named
free_space_fragmentation_percent.FRAGmeasures the free space only, and the name says so, so that nobody reads it as the fragmentation of the files. - The module needs no root. The account
mmwatchdogmay list a pool and read its status. zfs_poolis an official extra module, not a core module. ZFS is uncommon work on a Linux node. A script can followzpoolin a new release without a newmmwatchdogprogram.
This page is official_extra_modules/zfs_pool/README.md in the MMWatchdog repository.