every_systemd_unit

every_systemd_unit watches every systemd unit on the node at once: services, timers, mounts, and every other kind of unit. The check is healthy while systemd lists no failed unit. Each unit that systemd reports as failed is its own incident, with its own alert and its own recovery. MMWatchdog asks systemd with systemctl list-units --state=failed.

The check needs no list of the units to watch, so it also covers a unit that nobody thought of. every_systemd_unit also covers a unit that is stopped most of the time and must never fail, such as the unit of a scheduled job. A scheduled job's unit is inactive between runs, so a systemd_unit_state check cannot watch the scheduled job's unit. Use systemd_unit_state for a unit that must run all the time. Write the continuously running unit into the every_systemd_unit check's systemd_units_to_ignore.

Inputs

  • fail_when (a_systemd_unit_has_failed; required): The check fails while a systemd unit has failed; write a_systemd_unit_has_failed, the only value.
  • systemd_units_to_ignore (list_of_strings; required): The units whose failure the check leaves out, each with its kind, such as myapp.service or backup.timer. An empty list leaves none out.

When the check fails, and when it cannot check

The check reports a healthy run when systemd lists no failed unit outside systemd_units_to_ignore. When systemd lists no failed unit at all, the message is no failed systemd units. A failed unit in systemd_units_to_ignore does not fail the check, and the message says so: no failed systemd units (not counting 1 in systemd_units_to_ignore: myapp.service).

Each failed unit that the check does not ignore is one finding of the check, and each finding is its own incident:

  • The check's message counts and names the failed units, such as 2 systemd units failed: backup.service, certbot.service. A message names at most 20 units and counts the rest, such as , and 3 more.

  • The message of one unit is systemd unit backup.service failed (systemd result: exit-code), with the Result that systemd gives, or systemd unit backup.service failed when systemd gives none.

  • When systemd no longer lists the unit as failed, the unit's message is systemd unit backup.service is no longer failed.

  • Each unit follows the check's settings by itself. Its failing time starts at the first run that lists the unit as failed, and it ends when systemd has not listed the unit as failed for recovered_after_healthy_for. The failed unit's incident opens after alert_after_failing_for, so a second failed unit is announced while the first failed unit's incident is open. The check's alert_severity, raise_to_sos_after_failing_for, send_alerts_only_to_destinations, and the reminders of a destination apply to each unit's incident on its own.

  • The check itself is failed while a unit has an open incident, and it has no failure incident of its own.

MMWatchdog cannot check, and the run is unknown, when it cannot ask systemd. The message starts with systemd inspection: MMWatchdog could not ask systemd for its failed units:. An unknown run belongs to the check, rather than to a unit. After alert_after_failing_for, MMWatchdog sends one cannot-check alert for the check, rather than one for each unit. Each unit's open incident stays open. Time when MMWatchdog cannot ask systemd counts as healthy time for no unit (the failing time).

Examples

Alert for every failed unit

# /etc/mmwatchdog/check_groups/failed_systemd_units.yml
check_groups:
  failed_systemd_units:
    check_every: 1m                               # ask systemd every minute
    time_limit_per_run: 5s
    alert_after_failing_for: 2m                   # alert for a unit that has been failed for 2 minutes
    recovered_after_healthy_for: 1m               # "recovered" when systemd has not listed the unit for 1 minute
    checks:
      machine_has_no_failed_systemd_unit:
        every_systemd_unit:
          fail_when: a_systemd_unit_has_failed
          systemd_units_to_ignore: [myapp.service]   # this unit has a check of its own
        alert_severity: warn

On the node app-1:

  1. At 02:00 the nightly backup unit backup.service ends with exit status 1, and systemd reports it as failed.
  2. At 02:02 MMWatchdog sends a warn alert for backup.service. The failure alert's title is app-1: failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed. The failure alert's body starts with systemd unit backup.service failed (systemd result: exit-code).
  3. At 03:00 the unit certbot.service fails too. At 03:02 MMWatchdog sends a second alert, for certbot.service.
  4. At 09:10 an operator repairs the backup and runs backup.service again. At 09:11 MMWatchdog sends the recovery notification for backup.service. The recovery notification's title is app-1: failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service is ok again. The recovery notification's message is systemd unit backup.service is no longer failed. The incident of certbot.service stays open.

When myapp.service fails, every_systemd_unit sends nothing, because the unit is in systemd_units_to_ignore. The unit's own systemd_unit_state check sends its alert.

What the commands show

mmwatchdog status shows the check with one line for each failed unit:

failed_systemd_units/machine_has_no_failed_systemd_unit failed       2 systemd units failed: backup.service, certbot.service
  backup.service             failed       systemd unit backup.service failed (systemd result: exit-code)
  certbot.service            confirming_failure systemd unit certbot.service failed (systemd result: exit-code)
  • In mmwatchdog status --json the check has one_incident_for_each_finding: true, and its own failure_incident never holds an incident. The check object's findings maps each unit to an entry with the fields of a check: state, last_observation, failing_since, and failure_incident. The summary has an attention entry for each unit with an open incident. Each attention entry's finding field holds the unit name.
  • mmwatchdog check-now prints the failed units of that moment, one line for each, such as Finding backup.service: systemd unit backup.service failed (systemd result: exit-code). In JSON, mmwatchdog check-now lists the currently failed units as findings. Each finding has an id field containing the unit name and a message field explaining the unit's failure.
  • mmwatchdog fleet NODE on the hub shows each failed unit under its check.

The validation warning for a unit with its own check

A unit with its own systemd_unit_state check belongs in the every_systemd_unit check's systemd_units_to_ignore. Otherwise, a failure of the unit with a systemd_unit_state check sends two alerts. mmwatchdog validate and mmwatchdog reload name each unit with a systemd_unit_state check missing from an every_systemd_unit check's systemd_units_to_ignore. The configuration stays valid:

$ mmwatchdog validate
Valid configuration: 2 check groups, 1 alert destination
An every_systemd_unit check alerts for every failed systemd unit outside the check's systemd_units_to_ignore list. The units listed below also have a systemd_unit_state check, so a unit failure would send two alerts. Add each listed unit to systemd_units_to_ignore:
  failed_systemd_units/machine_has_no_failed_systemd_unit: myapp.service is missing from systemd_units_to_ignore; myapp/myapp_service_is_running also checks the systemd unit

mmwatchdog validate --json lists the units missing from each every_systemd_unit check's ignore list under systemd_units_missing_from_systemd_units_to_ignore (validation warnings). The warning looks only at units with a systemd_unit_state check, not at units with another check.

How it works

  1. At each run, MMWatchdog runs systemctl list-units --state=failed --all --plain --full --no-legend --no-pager on the node, as MMWatchdog itself. The systemctl list-units command lists failed units of every kind.
  2. MMWatchdog reads the ActiveState and the Result of each listed unit with one systemctl show command. MMWatchdog leaves out a unit that stopped being failed after systemctl list-units and before systemctl show. When systemctl show gives no answer for a unit, the unit counts as failed, and the unit's message has no result.
  3. MMWatchdog leaves out the units in systemd_units_to_ignore and reports each other unit as a finding, with the unit's name as the finding's id.

MMWatchdog changes nothing about a unit and never clears a failed state. A unit stays failed in systemd until it runs again or someone runs systemctl reset-failed.

systemd_units_to_ignore is required, so a reader sees at once whether any unit is left out. [] leaves no unit out. Each entry is the full name of a unit with its kind, such as myapp.service or backup.timer. An entry is an exact name, not a pattern. The order of the entries does not matter.

A changed check starts afresh. systemd_units_to_ignore is part of the check. A reload that changes the ignore list or removes the check closes each failed unit's incident. The incidents close without a recovery (incident closed without recovery), like the incident of every changed check. A unit that is still failed and not ignored gets a new incident after alert_after_failing_for. A reload that only changes the order of the list changes nothing.

Alerts name the unit. Every alert about a unit puts the unit name after the check name in the alert's title. The alert's message also names the unit. The alert's event has the unit name as finding, as in webhook JSON (title and body of an alert).

mmwatchdog validate cannot know the unit's name, so its warning about long alert titles counts the every_systemd_unit check's title without a unit. When adding the unit name makes a title longer than 120 characters, MMWatchdog shortens the title as MMWatchdog shortens other names in a title.

In a fleet, the hub groups the alerts of each unit by themselves. backup.service failing on seven nodes becomes failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed on 7 nodes. A different unit that fails in the minute of the backup.service failure has separate messages (alerts in a fleet). Each unit has its own MMHotline incident key, such as failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed, so two failed units are two MMHotline incidents, and the recovery of one resolves only its own (mmhotline).

One cause, one alert. When an every_systemd_unit check is kept quiet because a check named by every_systemd_unit fails, each failed unit is kept quiet too.

When another check names the every_systemd_unit check as a cause, the alert of each failed unit names the checks kept quiet (one cause, one alert).

Recovery actions. A restart or a recovery command that names the every_systemd_unit check in triggered_by_checks counts the check's own failing time, which lasts while systemd lists any unit as failed. The check has no incident of its own, so a recovery command gets an empty MMWATCHDOG_INCIDENT_ID.

Errors and what they mean

Message What to do
every_systemd_unit.fail_when: is required: The check fails while a systemd unit has failed; write a_systemd_unit_has_failed, the only value. Write fail_when: a_systemd_unit_has_failed below every_systemd_unit.
every_systemd_unit.fail_when: expected one of a_systemd_unit_has_failed Use a_systemd_unit_has_failed, the only allowed value.
check_groups.failed_systemd_units.checks.machine_has_no_failed_systemd_unit.every_systemd_unit.systemd_units_to_ignore: is required: the systemd units whose failure this check leaves out, such as [myapp.service], or [] to leave none out Write the list. [] leaves no unit out.
...every_systemd_unit.systemd_units_to_ignore: expected distinct systemd unit names with their kind, such as [myapp.service, backup.timer] Write each unit once, with its kind after the last dot. A unit name may contain only letters, digits, and the characters _@.:\-, at most 255 characters.
An every_systemd_unit check alerts for every failed systemd unit outside the check's systemd_units_to_ignore list. ... (a validation warning) Add each named unit to systemd_units_to_ignore.
systemd inspection: MMWatchdog could not ask systemd for its failed units: ... (unknown) systemctl failed. The message contains the failed systemctl command's exit status and output.

Tested on

  • Go tests: In failed_units_test.go, TestFailedSystemdUnitsCheckAsksSystemd checks units of every kind and omits a unit that stops being failed between systemctl list-units and systemctl show. The test also checks that ignored units are left out and named in the message. When systemd cannot be inspected, the test expects an unknown run. TestEachFailedSystemdUnitIsItsOwnIncident checks the example above minute by minute.

    More tests in failed_units_test.go cover each unit's lifecycle and delivery: TestAFailedUnitFollowsTheChecksTimingsByItself, TestFailedUnitsWhileSystemdCannotBeAsked, TestFailedUnitsInMaintenance, TestFailedUnitsAreRaisedAndRemindedOfByThemselves, TestSystemdUnitsToIgnore, TestTheRecoveryOfAFailedUnitSaysThatTheUnitIsNoLongerFailed, TestFailedUnitsInStatusAcrossRestartAndReload, TestFailedUnitsTriggerARecoveryCommand, TestTheHubGroupsFailedUnitsByUnit, TestFailedUnitsHaveTheirOwnMMHotlineIncidents, TestFailedUnitsThroughTheHub, and TestRemindersOfTwoUnitsDoNotReplaceEachOther.

    Quiet checks are covered by onecause_test.go: TestAQuietCheckWithOneIncidentForEachFailedUnit and TestTheAlertOfEachFailedUnitNamesTheQuietChecks. TestFailedSystemdUnitsOnTheCommandLine in cmd/mmwatchdog/main_test.go checks the text and JSON of the mmwatchdog validate warning. The command test also checks for one line per unit in mmwatchdog status and mmwatchdog fleet NODE.

  • Suites on real nodes: the resources suite (lifecycle.py): real one-time units that fail. The suite verifies that an ignored failed unit does not fail the check. Two units fail in succession and get separate incidents. Each unit appears in the title, body, mmwatchdog status, attention list, and mmwatchdog check-now --json.

    A repaired unit recovers while the second unit's incident stays open, including across a restart of MMWatchdog. Adding a failed unit to systemd_units_to_ignore closes the unit's incident without a recovery. mmwatchdog validate warns about a unit with a dedicated systemd_unit_state check.

    The fleet suite (fleet.py): a failed unit on a joined node sends its alert and its recovery through the hub with the unit in the title, and the hub shows the unit under its check.

  • The resources and fleet suites passed on every target distribution. The validation record has the latest run.

  • Not tested on real nodes: maintenance windows, reminders, increases of incident severity to sos, and the MMHotline keys of a unit's incident, which the Go tests cover; an every_systemd_unit check as a quiet check or as a cause of one cause, one alert, which the Go tests cover; a joined node that sends a unit's incident straight to MMHotline while the hub is down, which the Go tests cover; and many failed units at once. Each failed unit is its own alert, with no upper limit.

Why it works this way

  • Each failed unit is its own incident. A second failed unit is announced at once. Each repaired unit reports a separate recovery. One incident for the whole list would stay open from the first failure to the last repair and announce nothing in between.
  • systemd_units_to_ignore is required, and a unit with its own check is written into it. A reader sees at once whether any unit is left out. Automatically ignoring units with dedicated checks would make systemd_units_to_ignore: [] misleading, because an empty list means no unit is ignored.
  • A failed unit takes the alert severity of its check. A reader sees the severity on the check itself, as for every alert. Two severities are two checks.
  • An unknown run is the check's, not a unit's. MMWatchdog learned nothing about any unit, so it sends one cannot-check alert and leaves each open incident as it is.
  • MMWatchdog never clears a failed state. systemd supervises the units, and MMWatchdog only observes them. A unit that stays failed stays visible.
  • One mechanism for findings. The every_systemd_unit module and every module that reports several findings at once use one mechanism for findings. Findings behave alike in status, alerts, and the fleet.

This page is docs/reference/modules/every_systemd_unit.md in the MMWatchdog repository.