file
A file check looks at one file or directory on the node: whether it
exists, who owns it, what its permissions are, and how old it is, and for a
regular file also what its content is and how large it is. MMWatchdog asks the
kernel about the path and, for a file's content, reads it, as root. The
check fails when nothing is at the path, unless it states
must_not_exist: true. With must_not_exist: true, an existing path fails
the check, and a missing path passes.
To watch the lines that a program adds to a log file, use
log_lines. A job can write a new file name on each
run, such as a backup with the date in the name. For jobs with changing
output file names, a heartbeat sent by the
job when the job finishes is better.
Inputs
path(string; required): The absolute path of the file or directory that the check looks at.must_not_exist(boolean; optional): With true, the check fails while a file or directory is at the path, and takes no other input and no limit.expected_owner(string; optional): The user account that must own the file or directory.expected_group(string; optional): The group that the file or directory must belong to.expected_permissions_mode(string; optional): The permissions that the file or directory must have, as chmod writes them, in quotes, such as "0600".expected_sha256_checksum(string; optional): The SHA-256 checksum that the file's content must have: 64 hex characters, as sha256sum prints them. Only for a regular file.
Measurements
file_age_minutes(minutes): The time since the file's content was last changed, in minutes. For a directory, the time since an entry was last added, removed, or renamed in it.file_age_hours(hours): The same time in hours.file_age_days(days): The same time in days.file_size_mib(MiB): The size of a regular file. A directory has none.
The module fails a check by itself, and it also reports these measurements. A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check with the module always states an alert_severity. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
- A missing file fails the check, with the message
/var/backups/database.sql.gz does not exist, unless the check statesmust_not_exist: true. Withmust_not_exist: true, the check fails while the checked file exists, with/srv/myapp/maintenance.flag exists, and is healthy with/srv/myapp/maintenance.flag does not exist. - The path names a regular file or a directory. Anything else at the
path fails the check, and the message says what is there, such as
/run/myapp/queue is a named pipe, not a regular file or a directory. A symbolic link stands for the file or directory that it names. - A directory has no content to compare and no size. A check of a
directory with
expected_sha256_checksumfails with/etc/cron.d is a directory, and expected_sha256_checksum checks the content of a regular file, and one with a limit onfile_size_mibfails with/var/backups is a directory, and the limit on file_size_mib in fail_when_measurement_below checks the size of a regular file. Directory checks with a checksum or size limit stay failed until corrected or until a regular file replaces the directory. - Each
expected_input that differs is named with what the file has and what was expected, such as/etc/ssh/sshd_config has permissions mode 0644, expected 0600. Several differences are joined in one message, separated by;. - A limit on a measurement fails the check as for every module with
measurements (limits).
The message names each reading that has a limit, in front of what the
check found:
file_age_hours is 30.5 (fails above 26); /var/backups/database.sql.gz exists. - A healthy check says
/etc/ssh/sshd_config is as expected, or/var/backups/database.sql.gz existsfor a check without anexpected_input, after the readings that have a limit. - MMWatchdog cannot check, and the run is unknown, when MMWatchdog
cannot look at the path or read the file, for example a very large file with
expected_sha256_checksumthat MMWatchdog cannot read withintime_limit_per_run. A file that was last changed in the future has no age, so a check with a limit on its age is then unknown, and its message gives the file's last modification time. - A file on a network filesystem whose server is gone cannot be
examined: the kernel holds the request without end. No time limit ends
the blocked request. The check is then unknown, and the run releases
the run's place so the node's remaining checks can run
(a run that does not end).
A
node_filesystemor anevery_disk_and_network_filesystemcheck says in plain words that such a filesystem does not answer.
Examples
A backup that is fresh and not empty
# /etc/mmwatchdog/check_groups/files.yml
check_groups:
files:
check_every: 5m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
database_backup_is_fresh_and_not_empty:
file:
path: /var/backups/database.sql.gz
fail_when_measurement_above:
file_age_hours: 26 # fails when the file was last changed more than 26 hours ago
fail_when_measurement_below:
file_size_mib: 1 # fails when the file is smaller than 1 MiB
alert_severity: sos
Every 5 minutes, MMWatchdog on the node looks at /var/backups/database.sql.gz. The
check fails when the file is missing, when it was last changed more than 26
hours ago, or when it is smaller than 1 MiB. When the check has failed for 10
minutes, MMWatchdog sends the sos alert. A nightly backup that runs late
by more than 2 hours therefore alerts, as does a backup that wrote an
almost empty file.
A settings file that must not change
# /etc/mmwatchdog/check_groups/files.yml
check_groups:
files:
check_every: 5m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
ssh_server_settings_are_unchanged:
file:
path: /etc/ssh/sshd_config
expected_owner: root
expected_group: root
expected_permissions_mode: "0600"
expected_sha256_checksum: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
alert_severity: warn
The check fails when the settings file of the SSH server has another owner,
another group, other permissions, or other content than stated. sha256sum /etc/ssh/sshd_config on the node prints the checksum to write. The
permissions are in quotes, because YAML would read 0600 without quotes as
a number.
A directory with its owner and permissions
# /etc/mmwatchdog/check_groups/files.yml
check_groups:
files:
check_every: 5m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
cron_jobs_directory_is_as_expected:
file:
path: /etc/cron.d
expected_owner: root
expected_group: root
expected_permissions_mode: "0755"
alert_severity: warn
The check fails when /etc/cron.d is missing, has another owner or group,
or other permissions, such as 0777, which lets every account add a job
that cron runs as root. The check reads only the directory itself, not the
files in it.
A file that must be gone
# /etc/mmwatchdog/check_groups/files.yml
check_groups:
files:
check_every: 5m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
maintenance_flag_is_gone:
file:
path: /srv/myapp/maintenance.flag
must_not_exist: true # the check fails while this file exists
alert_severity: warn
The check fails while the file /srv/myapp/maintenance.flag exists. When
someone forgets to remove /srv/myapp/maintenance.flag after maintenance,
MMWatchdog sends the warn alert after 10 minutes.
How it works
- MMWatchdog looks at the file as root, so MMWatchdog can check any file.
- For the checksum, MMWatchdog opens the file at
pathonce. MMWatchdog takes the owner, permissions, size, and content from the open file. Replacing the file at the path during the run cannot combine properties from the original file and the replacement file. MMWatchdog openspathso that a named pipe atpathcannot hold up the run. - The owner and the group are compared by numeric ID first, so
expected_owner: "0"also works. An owner or group absent from the node's own files, such as an account of a directory service, is found through the node'sgetentprogram. MMWatchdog runsgetentas the unprivileged accountmmwatchdog. An unknown ID is shown as a number. - The permissions include the set-user-ID, set-group-ID, and sticky
bits, as
chmodwrites them. Mode"4755"makes an executable run with the file owner's user ID. - The age is the time since the file's content was last changed. The age has three units so that a limit reads naturally. A directory's age is the time since an entry was last added, removed, or renamed in it; a change of the content of a file in it does not count.
Errors and what they mean
Validation errors name the check, such as
check_groups.files.checks.ssh_server_settings_are_unchanged.file.expected_permissions_mode: ...:
| Message | What the message means, and what to do |
|---|---|
path: is required: the absolute path of the file or directory that the check looks at |
path is missing, relative, or not in its shortest form, such as /var//log or /var/log/. |
expected_owner: a check with must_not_exist: true looks only at whether the file exists; remove expected_owner |
A check with must_not_exist: true rejects every expected_ input, including expected_owner. |
fail_when_measurement_above: a file check with must_not_exist: true looks only at whether the file exists; remove fail_when_measurement_above |
A missing file has no age or size, so must_not_exist: true rejects fail_when_measurement_above and fail_when_measurement_below. |
expected_group: cannot be empty and holds no white space |
An expected_ input is empty or holds a space. |
expected_permissions_mode: expected the permissions as chmod writes them, in quotes, such as "0600" |
Write three or four octal digits in quotes. |
expected_sha256_checksum: expected a SHA-256 checksum: 64 hex characters in lower case, as sha256sum prints them |
Copy the checksum from sha256sum. |
Messages of a run:
| Message | What the message means, and what to do |
|---|---|
/etc/ssh/sshd_config is owned by the user account myapp, expected root |
The owner differs from expected_owner. Messages for group, permissions, and checksum differences also give the actual and expected values: belongs to the group ..., has permissions mode 0644, expected 0600, has the SHA-256 checksum ..., expected .... |
/etc/cron.d is a directory, and expected_sha256_checksum checks the content of a regular file |
A checksum needs a regular file. Remove expected_sha256_checksum, or point path at a file. |
/var/backups is a directory, and the limit on file_size_mib in fail_when_measurement_below checks the size of a regular file |
A directory has no size. Remove the limit, or point path at a file. |
/run/myapp/queue is a named pipe, not a regular file or a directory |
Something else than a file or a directory is at the path. |
MMWatchdog could not read all of /srv/big.img within the time limit of the run |
The file is too large to read for its checksum within time_limit_per_run. Give the check group a longer time_limit_per_run. |
MMWatchdog cannot look at PATH: ..., MMWatchdog cannot read PATH: ... |
The kernel refused, and the message gives its error. |
PATH was last changed at 2027-01-01 00:00 UTC, which is in the future, so its age cannot be measured |
The file's time of its last change is wrong. touch the file, or fix the program that wrote it. |
Tested on
- Go tests in
node_checks_test.gocover the following cases:TestFileCheckcovers a missing file with and withoutmust_not_exist, and age and size with limits above and below. The test checks owner, group, permissions with a set-user-ID bit, and checksum. The test checks that each difference is named and multiple differences are joined. Further cases cover numeric account IDs, a file unreadable within the time limit, a file from the future, and a symbolic link. The test also covers a named pipe at the path and validation of every input.TestFileCheckOfADirectorycovers a directory's owner, group, permissions, and age, and an entry added to the directory. The test checks that a checksum and a size limit fail, and coversmust_not_exist.TestAccountNamesOfADirectoryServicecovers names fromgetent.
- The node suite (
node_scenarios.py), on every target distribution: a backup file that is missing, fresh, too small, and too old; a settings file whose permissions and content change; a directory whose permissions change; a flag file that must not exist; and eight files on an NFS filesystem whose server stops, whose checks say that their run did not end and are healthy again when the server is back. - The latest run passed on every target distribution (validation record). Go tests only, not on a real node: an owner or a group of a directory service, and a filesystem that answers with an error such as a stale NFS handle. A stale NFS handle means the NFS server no longer recognizes the client's reference to a file.
Why it works this way
fileis a core module. A file that must exist or must not exist, its owner and permissions, its checksum, its size, and its age are among the most common checks of real monitoring configurations. Checks of file existence, owner, permissions, checksum, size, and age are useful on any Linux server.- The limits on age and size are ordinary limits on measurements. One way to write a limit serves every module, and the readings reach the measurement history.
- The age has three units.
file_age_hours: 26reads more naturally thanfile_age_minutes: 1560. must_not_exist: truetakes no other input and no limit. A missing file has no owner, age, or size.- A directory is accepted. Of the Monit checks collected for beta
validation, 167 looked at a directory, mostly at the directory's owner,
group, and permissions. Without the
filecheck, each directory check would need a shell check. A checksum or a size limit on a directory fails at the run and is not refused when the configuration loads, because only the node knows what is at the path. Afilecheck fails when the path names a named pipe, a socket, or a device. The collection of Monit checks used for the comparison had 4 checks of a named pipe, and a socket has itsunix_socketcheck. - The checksum check reads one open file. A file that another file replaces during the run then shows one file only, never the owner of one and the content of the other.
This page is docs/reference/modules/file.md in the MMWatchdog repository.