node_filesystem
node_filesystem measures one filesystem of the node: how much of its space
and of its inodes is in use, and how much space is free. The value of the
check is the absolute path where the filesystem is mounted, such as / or
/srv/data. MMWatchdog reads the filesystem counts through the kernel's
statfs call and runs no program to read the counts.
An inode is the filesystem's record of one file. A filesystem with many small files can run out of inodes while space is left.
A node_filesystem check that states an alert_severity also fails while
nothing is mounted at its path, and while its filesystem is mounted
read-only, with or without a limit. So one check of a filesystem covers its
space, its inodes, and that it can be written.
To apply matching limits to every filesystem of the node, use
every_disk_and_network_filesystem.
You do not need to name each filesystem to apply the limits.
To measure the disk itself, how busy it is and how much is read and
written, use node_disk_device.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
node_filesystem(string; required): The absolute path where the filesystem is mounted, such as /.
Measurements
space_used_percent(percent): The share of the filesystem's space that is in use.inodes_used_percent(percent): The share of the filesystem's inodes that is in use. A filesystem without a fixed number of inodes has no such measurement.space_free_gib(GiB): The space that an ordinary account can still fill.
A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check without a limit can still fail, as When the check fails, and when it cannot check says, so it may state an alert_severity and no limit. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
A check with an alert_severity fails at that severity:
- when a reading passes a limit. A reading strictly above its
fail_when_measurement_abovenumber, or strictly below itsfail_when_measurement_belownumber, fails the check; an equal reading passes (checks that measure a number). - when nothing is mounted at the check's path, with the message
no filesystem is mounted at /mnt/san. The check reports no reading then. - when the filesystem at the check's path is mounted read-only. The
message starts with
/srv/data is mounted read-only;and goes on with the readings, which MMWatchdog still records.
A check with only_record_readings: true only records its readings
(checks that only record).
The check with only_record_readings: true notes a read-only mount in the
check's message and never fails or alerts. A message can be /srv/data is mounted read-only; space_used_percent is 40; inodes_used_percent is 3; space_free_gib is 12. While nothing is mounted at the path of a check with
only_record_readings: true, the check is unknown because no readings are
available. The check with only_record_readings: true reports no filesystem is mounted at /srv/data and opens no incident.
MMWatchdog cannot check, and the observation is unknown, when a condition in
the following list occurs. An unknown result of a check that can alert opens
a separate cannot-check incident after alert_after_failing_for
(scheduling and observed state).
- The filesystem does not answer. A filesystem on another machine can stop answering, for example when that machine is gone. See a filesystem that does not answer.
- The kernel answers with an error, such as a stale NFS file handle. The message is the kernel's error.
- The check has a limit on
inodes_used_percent, and the filesystem has no fixed inode count, such as btrfs. The message is/srv has no fixed inode count, so inodes_used_percent cannot be measured. Leave out theinodes_used_percentlimit on a filesystem without a fixed inode count. The filesystem's limits on space still work. - MMWatchdog cannot read the kernel's list of mounted filesystems, with a
message that starts with
MMWatchdog cannot read the kernel's list of mounted filesystems:.
Examples
Limits on space, on inodes, and on free space
# /etc/mmwatchdog/check_groups/root_filesystem.yml
check_groups:
root_filesystem:
check_every: 1m
time_limit_per_run: 5s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
root_filesystem_is_below_90_percent:
node_filesystem: /
fail_when_measurement_above:
space_used_percent: 90
inodes_used_percent: 90
alert_severity: warn
root_filesystem_has_10_gib_free:
node_filesystem: /
fail_when_measurement_below:
space_free_gib: 10
alert_severity: sos
Every minute, MMWatchdog on the node measures the filesystem at /.
root_filesystem_is_below_90_percent fails when more than 90 percent of the
space, or more than 90 percent of the inodes, is in use. The message of
root_filesystem_is_below_90_percent names the space and inode readings,
such as space_used_percent is 62.5 (fails above 90); inodes_used_percent is 12 (fails above 90). After 10 minutes of failing,
MMWatchdog sends a warn alert. root_filesystem_has_10_gib_free fails when less
than 10 GiB are free, and alerts at sos. A filesystem mounted read-only
fails root_filesystem_is_below_90_percent and
root_filesystem_has_10_gib_free. A recovery command
triggered by root_filesystem_is_below_90_percent or
root_filesystem_has_10_gib_free can clean up old files.
A filesystem that must be mounted
A check with an alert_severity and no limit says "this filesystem must be
mounted, and writable":
# /etc/mmwatchdog/check_groups/san_storage.yml
check_groups:
san_storage:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 5m
recovered_after_healthy_for: 2m
checks:
san_storage_is_mounted:
node_filesystem: /mnt/san
alert_severity: sos
While a filesystem is mounted at /mnt/san and can be written, the check is
healthy and records its readings. When the filesystem is unmounted, the
check fails with no filesystem is mounted at /mnt/san, and after 5 minutes
MMWatchdog sends an sos alert. A recovery command
triggered by san_storage_is_mounted can mount the filesystem again. Once
the filesystem at /mnt/san has been mounted for 2 minutes,
MMWatchdog sends the recovery alert.
A filesystem that must be writable
A filesystem that the kernel remounted read-only after I/O errors leaves services running while every write to the affected filesystem fails:
# /etc/mmwatchdog/check_groups/data_filesystem.yml
check_groups:
data_filesystem:
check_every: 1m
time_limit_per_run: 5s
alert_after_failing_for: 2m
recovered_after_healthy_for: 1m
node_filesystem: /srv/data
fail_when_measurement_above:
space_used_percent: 85
alert_severity: sos
The data_filesystem check group holds one direct check,
data_filesystem/node_filesystem.
data_filesystem/node_filesystem fails when the filesystem at /srv/data
is more than 85 percent full, and also when the filesystem is mounted
read-only, even below 85 percent.
After 2 minutes MMWatchdog sends an sos alert whose message starts with
/srv/data is mounted read-only;. Write filesystem checks with an
alert_severity for the filesystems that are meant to be writable.
How it works
Each run does three steps on the node:
- MMWatchdog reads the kernel's list of mounted filesystems,
/proc/self/mountinfo, and looks for a filesystem that is mounted exactly at the check's path. The path is the one where the filesystem is mounted, asfindmntlists it, not a directory inside the filesystem. - MMWatchdog asks the kernel for the counts of the filesystem mounted at
the check's path, as
dfdoes. - MMWatchdog computes the readings, applies the limits, and notes a read-only mount.
| Measurement | Meaning |
|---|---|
space_used_percent |
The share of the filesystem's space that ordinary accounts cannot fill any more, 0 to 100. Space that the filesystem keeps for root counts as used |
inodes_used_percent |
The share of the filesystem's inodes that is in use, 0 to 100. MMWatchdog provides no inode-use percentage for a filesystem without a fixed inode count, such as btrfs |
space_free_gib |
The space that an ordinary account can still fill, in GiB |
MMWatchdog stores the readings from each scheduled run in the node's measurement history, whether the check has a limit or only records.
A filesystem that does not answer
For a filesystem on another machine, the kernel holds the node's
statfs call without end while the remote machine is gone.
MMWatchdog cannot cancel the blocked system call. So:
- MMWatchdog waits for the kernel's answer no longer than
time_limit_per_run. The check is then unknown withthe filesystem does not answer: the filesystem at /mnt/san gave no answer within the time limit of the run. - MMWatchdog starts no second
statfscall for the check's path while the firststatfscall remains blocked. Later runs wait for the pending call's answer and say when MMWatchdog started waiting:the filesystem does not answer: MMWatchdog has waited for the filesystem at /mnt/san since 14:02:00 UTC. - Concurrent checks of the filesystem at the same path share one pending
statfscall and receive the result of the shared call. - Once the kernel returns a result, the next run starts a new
statfscall.
Errors and what they mean
Each validation error starts with the file, the line, and the path of the
setting, such as check_groups.root_filesystem.checks.root_filesystem_is_below_90_percent.node_filesystem:.
must be the absolute path where the filesystem is mounted, such as /. The path is relative.this check has no such measurement; it measures space_used_percent, inodes_used_percent, space_free_gib. A limit names another measurement.expected a number from 0 to 100for a limit on a_percentmeasurement, andexpected a number of 0 or morefor a limit onspace_free_gib.- A validation warning from
mmwatchdog validatewhen the filesystem of a check with a limit is missing from thefilesystems_to_ignoreof anevery_disk_and_network_filesystemcheck: one full filesystem would send two alerts. Theevery_disk_and_network_filesystempage shows the warning.
The messages of an unknown run are listed under
when MMWatchdog cannot check.
/srv/data: filesystem counts unavailable or invalid means that the kernel
gave counts that say nothing, such as a size of zero.
/srv/data: inode counts invalid makes a check with a limit on
inodes_used_percent unknown when the kernel reports more free inodes than
inodes.
Tested on
-
Go tests in
internal/checksandalerts:resources_test.go(TestSwapLoadAndFilesystemReadings,TestMeasurementLimitsAreOptionalAndStrict) covers the readings, a read-only filesystem with and without analert_severity, and a filesystem without a fixed inode count;node_measurements_test.gocovers the kernel's list of mounts, a check of a path with nothing mounted, a filesystem that does not answer, and two questions about one filesystem at once.resources_linux_test.goininternal/daemonreads the filesystem at/on Linux. -
On Linux virtual machines used for integration tests, the suite
resources.pyrecords the space, the inodes, and the free space of/. The suite remounts a small tmpfs read-only. Checks of the tmpfs that state analert_severityfail, with or without a limit, and keep the readings. The checks with analert_severityrecover once the tmpfs is writable. The check that only records notes the read-only mount and opens no incident.node_scenarios.pyunmounts a tmpfs. The check that states analert_severityfails and recovers when the tmpfs is mounted again. The check that only records is unknown and records no reading of the filesystem containing the directory exposed after the tmpfs is unmounted. -
The
resources.pyandnode_scenarios.pysuites and the Go tests passed on every target distribution in the latest run of the whole catalog (validation record). A filesystem that answers with an error, such as a stale NFS file handle, is covered by Go tests only.
Why it works this way
-
Recording readings is an explicit choice. State
only_record_readings: trueto record readings without alert notifications or recovery actions. A missing severity must not silently turn an intended alert into readings only. -
A check with an
alert_severityfails when nothing is mounted. Without checking for a filesystem mounted exactly at the check's path, MMWatchdog would read the filesystem that holds the empty directory, usually/. The check would then report the containing filesystem's numbers. A check of a missing filesystem never reports another filesystem's numbers. -
A check with an
alert_severityfails on a read-only mount, without a limit. A filesystem that is gone or cannot be written is a failure at any value. A full filesystem is a failure only above the limit that the operator sets (check group decisions). -
No new module for "is mounted". One check of a filesystem covers its space, its inodes, and that it is mounted and writable, at the severity that the check states.
-
A check that only records never alerts, also for a read-only mount. The check that only records notes the read-only mount in the check's message. The read-only mount remains visible to the reader of
mmwatchdog status. -
MMWatchdog keeps at most one question waiting for each path. A check that runs every minute against a filesystem that does not answer would otherwise leave one more question waiting at each run.
This page is docs/reference/modules/node_filesystem.md in the MMWatchdog repository.