Module pages
Each page describes one module of MMWatchdog: what it checks or where it delivers alerts, its inputs and measurements, examples, how it works, its errors, where it is tested, and why it works this way. mmwatchdog describe-module NAME prints the same page in a terminal, on a node or on the machine where the configuration is written.
A core module is built into the mmwatchdog program. The core modules are listed by topic, and after them the official extra modules.
Services and protocols
http: Sends one HTTP or HTTPS request and checks the status code and, when asked, the content of the answer.tcp: Opens a TCP connection. The check is healthy when the connection opens. It can also send a text and expect a reply.udp: Sends a text in one UDP packet and is healthy when a reply arrives. It can also expect a text in the reply.unix_socket: Opens a connection to a Unix socket, a file through which programs on the same machine talk to each other. The check is healthy when the connection opens.ping: Asks another machine to answer, with the kind of request that the ping program sends. The check sends up to three requests and is healthy when one answer arrives.tls: Opens a TLS connection, verifies the server's certificate, and fails when the certificate expires soon.ssh: Asks an SSH server for its greeting, the line with which every SSH server starts. The check logs in nowhere.smtp: Talks to a mail server that accepts mail, by SMTP: waits for the server's greeting, greets it back, and leaves. The check sends no mail.imap: Talks to a mail server that serves mailboxes, by IMAP: waits for the server's greeting and logs out. The check logs in to no mailbox.pop3: Talks to a mail server that serves mailboxes, by POP3: waits for the server's greeting and quits. The check logs in to no mailbox.dns: Asks one DNS server for the records of one kind for one name. The check is healthy when the server answers with a record, and can also compare the answer with the values that are expected.
Databases, caches, and containers
memcached: Asks a memcached server for its statistics, and is healthy when the server gives them. It measures the server's connections, items, and memory, and its hits, misses, and evictions since the run before.redis: Asks a Redis server for an answer: the check sends PING and expects PONG. It can log in first.postgresql: Asks a PostgreSQL server a question through its client program, psql, and measures how far a replica is behind, how many connections are in use, how many block reads its shared buffers served, how long its write-ahead log waits to be archived, and how far its transaction IDs are toward wraparound. Without inputs, the check asks the server on this node.pgbackrest: Asks pgBackRest about the backups of one stanza, and measures how many hours ago its latest backup without errors finished. The check fails when pgBackRest reports a problem with the stanza or its repository.mysql: Asks a MySQL or a MariaDB server a question through its client program, and measures how far a replica is behind, how many connections are in use, and how long InnoDB's history list is. Without inputs, the check asks the server on this node.sqlite_database: Looks at the SQLite database file of another program: runs a small query on it, and reports the size of the database and of its write-ahead log. The check opens the file for reading only.docker_container: Asks Docker whether a named container runs. A container whose own health check fails counts as failed too.
Systemd units
systemd_unit_state: Asks systemd for the state of one unit of any kind, such as a service, a timer, or a path unit. The check is healthy while systemd reports the unit as active or reloading; a unit that is stopped, failed, or in another state fails the check.systemd_unit_commands: Watches one program of a systemd unit that runs several. The check is healthy while one of the commands that "systemctl status" lists for the unit contains a text.systemd_unit_crash_loop: Counts automatic restarts of a systemd unit. The check fails when the restart count exceeds automatic_restarts_above within automatic_restarts_within.every_systemd_unit: Watches every systemd unit of the node at once. Each unit that systemd reports as failed gets its own failure alert and its own recovery alert.systemd_unit_resource: Measures one resource of one systemd unit: its CPU, its memory, or the number of its processes.
The node's resources, storage, and hardware
node_resource: Measures one resource of the node: its CPU, its memory, its swap space, its load, the time that programs wait for a disk, the memory pressure, or its open files.node_filesystem: Measures the space and the inodes of one filesystem. With an alert severity, the check also fails while nothing is mounted at the path, and while the filesystem is mounted read-only.every_disk_and_network_filesystem: Applies its limits to every filesystem of the node that is mounted from a disk device or from another machine. Each filesystem that passes a limit, is mounted read-only, or does not answer gets its own failure alert and its own recovery alert.node_disk_device: Measures one disk device: how busy it is, and how much data is read from it and written to it.node_network_interface: Measures one network interface: the speed of its link, and the data that it receives and sends. With an alert severity, the check also fails while the interface has no link.node_temperature: Measures the temperature of one kind of hardware of the machine: the highest reading among the sensors of one name.mdadm_array: Watches one software RAID array of Linux, the kind that the mdadm program manages. The check fails while the array misses a device, and while the node has not put the array together.
The node's software, clock, and kernel
apt_security_updates: Fails while a security update waits to be installed on the node. The check reads what the node's own package manager already knows, and downloads nothing.dnf_security_updates: Fails while a security update waits to be installed on a node of the Red Hat family, such as Rocky Linux, AlmaLinux, and Fedora. The check reads what dnf already knows, and downloads nothing.zypper_security_patches: Fails while a security patch waits to be installed on a SUSE node, such as openSUSE Leap. The check reads what zypper already knows, and downloads nothing.ubuntu_pro_security_fixes: Fails while the node misses a security fix of an installed package because only an Ubuntu Pro service that the node does not get delivers the fix. The check reads what the Ubuntu Pro client on the node already knows, and downloads nothing.reboot_after_updates: Fails while the node needs a reboot, for example after a new kernel was installed. The check reads what the node's own distribution says about it.distribution_release_security_support: Fails when the security support that the check names ends soon for the node's Debian, Ubuntu, Red Hat family, or SUSE release, or has ended. The check reads the release dates of the package distro-info-data on Debian and Ubuntu, the SUPPORT_END of /etc/os-release on the Red Hat family, and the product file of the release on SUSE.known_exploited_vulnerabilities_in_installed_packages: Fails while a flaw on CISA's list of known exploited vulnerabilities affects an installed Debian or Ubuntu package. The check downloads CISA's list, asks OSV.dev for the flaws of the installed source packages by name only, and compares the versions on the node.node_clock: Asks the kernel whether the machine's clock follows a time server, whichever time service the machine runs.out_of_memory_kills: Fails in a run when the kernel ended at least one process to free memory since the check's previous run. The first run that sees a new kill opens its incident; the alert names each ended command.
Files, logs, and commands
file: Looks at one file or directory: whether it exists, who owns it, what its permissions are, how old it is, and for a regular file its content and how large it is. The check fails when nothing is at the path.log_lines: Reads the lines that a program adds to a log, from a file or from the journal of a systemd unit. A run reads only the lines that are new since the run before. The check states one source and one condition.shell: Runs a Bash script. The check is healthy when the script ends with an exit code that counts as success.
Alert destinations
webhook: Posts each alert event as JSON to a web address.email: Sends each alert as one email through a mail server.system_log: Writes one line to the system log of the machine, the systemd journal, for each alert: the title with the alert's severity in front.mmhotline: Sends each alert to MMHotline, the on-call program that a team runs on its own server.
Official extra modules
These modules are Starlark scripts that the MMWatchdog project writes and ships in official_extra_modules/ of every release. A deploy installs them on every node. Each one's page is the README.md in its directory.
apache_status(check): Reads the status page of an Apache web server, which mod_status serves, and measures how many of Apache's workers are busy.discord(alert destination): Delivers each alert to a Discord channel through a webhook.mattermost(alert destination): Delivers each alert to a Mattermost channel through an incoming webhook.mongodb(check): Checks a MongoDB server through MongoDB's own shell, mongosh. The check fails while the server does not answer, and for a member of a replica set while the member is neither primary, secondary, nor arbiter, or cannot reach another member.pushover(alert destination): Delivers each alert to a Pushover user or delivery group, and repeats each sos alert until somebody acknowledges it.slack(alert destination): Delivers each alert to a Slack channel through an incoming webhook.telegram(alert destination): Delivers each alert to a Telegram chat, as a bot.zfs_pool(check): Checks one ZFS pool through the zpool program. The check fails while the pool is not healthy, and measures how full and how fragmented the pool is and how long ago it was last scrubbed.
This page is docs/reference/modules/README.md in the MMWatchdog repository.