every_systemd_unit
every_systemd_unit watches every systemd unit on the node at once:
services, timers, mounts, and every other kind of unit. The check is healthy
while systemd lists no failed unit. Each unit that systemd reports as
failed is its own incident, with its own alert and its own recovery. MMWatchdog
asks systemd with systemctl list-units --state=failed.
The check needs no list of the units to watch, so it also covers a unit that
nobody thought of. every_systemd_unit also covers a unit that is stopped
most of the time and must never fail, such as the unit of a scheduled job.
A scheduled job's unit is inactive between runs, so a
systemd_unit_state check cannot watch the
scheduled job's unit. Use systemd_unit_state for a unit
that must run all the time. Write the continuously running unit into the
every_systemd_unit check's systemd_units_to_ignore.
Inputs
fail_when(a_systemd_unit_has_failed; required): The check fails while a systemd unit has failed; write a_systemd_unit_has_failed, the only value.systemd_units_to_ignore(list_of_strings; required): The units whose failure the check leaves out, each with its kind, such as myapp.service or backup.timer. An empty list leaves none out.
When the check fails, and when it cannot check
The check reports a healthy run when systemd lists no failed unit outside
systemd_units_to_ignore. When systemd lists no failed unit at all, the
message is no failed systemd units. A failed unit in
systemd_units_to_ignore does not fail the check, and the message says so:
no failed systemd units (not counting 1 in systemd_units_to_ignore: myapp.service).
Each failed unit that the check does not ignore is one finding of the check, and each finding is its own incident:
-
The check's message counts and names the failed units, such as
2 systemd units failed: backup.service, certbot.service. A message names at most 20 units and counts the rest, such as, and 3 more. -
The message of one unit is
systemd unit backup.service failed (systemd result: exit-code), with theResultthat systemd gives, orsystemd unit backup.service failedwhen systemd gives none. -
When systemd no longer lists the unit as failed, the unit's message is
systemd unit backup.service is no longer failed. -
Each unit follows the check's settings by itself. Its failing time starts at the first run that lists the unit as failed, and it ends when systemd has not listed the unit as failed for
recovered_after_healthy_for. The failed unit's incident opens afteralert_after_failing_for, so a second failed unit is announced while the first failed unit's incident is open. The check'salert_severity,raise_to_sos_after_failing_for,send_alerts_only_to_destinations, and the reminders of a destination apply to each unit's incident on its own. -
The check itself is failed while a unit has an open incident, and it has no failure incident of its own.
MMWatchdog cannot check, and the run is unknown, when it cannot ask systemd. The
message starts with systemd inspection: MMWatchdog could not ask systemd for its failed units:. An unknown run belongs to the check, rather than to a unit.
After alert_after_failing_for, MMWatchdog sends one cannot-check alert
for the check, rather than one for each unit. Each unit's open incident
stays open. Time when MMWatchdog cannot ask systemd counts as healthy
time for no unit
(the failing time).
Examples
Alert for every failed unit
# /etc/mmwatchdog/check_groups/failed_systemd_units.yml
check_groups:
failed_systemd_units:
check_every: 1m # ask systemd every minute
time_limit_per_run: 5s
alert_after_failing_for: 2m # alert for a unit that has been failed for 2 minutes
recovered_after_healthy_for: 1m # "recovered" when systemd has not listed the unit for 1 minute
checks:
machine_has_no_failed_systemd_unit:
every_systemd_unit:
fail_when: a_systemd_unit_has_failed
systemd_units_to_ignore: [myapp.service] # this unit has a check of its own
alert_severity: warn
On the node app-1:
- At 02:00 the nightly backup unit
backup.serviceends with exit status 1, and systemd reports it asfailed. - At 02:02 MMWatchdog sends a
warnalert forbackup.service. The failure alert's title isapp-1: failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed. The failure alert's body starts withsystemd unit backup.service failed (systemd result: exit-code). - At 03:00 the unit
certbot.servicefails too. At 03:02 MMWatchdog sends a second alert, forcertbot.service. - At 09:10 an operator repairs the backup and runs
backup.serviceagain. At 09:11 MMWatchdog sends the recovery notification forbackup.service. The recovery notification's title isapp-1: failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service is ok again. The recovery notification's message issystemd unit backup.service is no longer failed. The incident ofcertbot.servicestays open.
When myapp.service fails, every_systemd_unit sends nothing, because the
unit is in systemd_units_to_ignore. The unit's own systemd_unit_state check
sends its alert.
What the commands show
mmwatchdog status shows the check with one line for each failed unit:
failed_systemd_units/machine_has_no_failed_systemd_unit failed 2 systemd units failed: backup.service, certbot.service
backup.service failed systemd unit backup.service failed (systemd result: exit-code)
certbot.service confirming_failure systemd unit certbot.service failed (systemd result: exit-code)
- In
mmwatchdog status --jsonthe check hasone_incident_for_each_finding: true, and its ownfailure_incidentnever holds an incident. The check object'sfindingsmaps each unit to an entry with the fields of a check:state,last_observation,failing_since, andfailure_incident. Thesummaryhas anattentionentry for each unit with an open incident. Each attention entry'sfindingfield holds the unit name. mmwatchdog check-nowprints the failed units of that moment, one line for each, such asFinding backup.service: systemd unit backup.service failed (systemd result: exit-code). In JSON,mmwatchdog check-nowlists the currently failed units asfindings. Each finding has anidfield containing the unit name and amessagefield explaining the unit's failure.mmwatchdog fleet NODEon the hub shows each failed unit under its check.
The validation warning for a unit with its own check
A unit with its own systemd_unit_state check belongs in
the every_systemd_unit check's systemd_units_to_ignore. Otherwise, a
failure of the unit with a systemd_unit_state check sends two alerts.
mmwatchdog validate and mmwatchdog reload name each unit with a
systemd_unit_state check missing from an every_systemd_unit check's
systemd_units_to_ignore. The configuration stays valid:
$ mmwatchdog validate
Valid configuration: 2 check groups, 1 alert destination
An every_systemd_unit check alerts for every failed systemd unit outside the check's systemd_units_to_ignore list. The units listed below also have a systemd_unit_state check, so a unit failure would send two alerts. Add each listed unit to systemd_units_to_ignore:
failed_systemd_units/machine_has_no_failed_systemd_unit: myapp.service is missing from systemd_units_to_ignore; myapp/myapp_service_is_running also checks the systemd unit
mmwatchdog validate --json lists the units missing from each
every_systemd_unit check's ignore list under
systemd_units_missing_from_systemd_units_to_ignore
(validation warnings). The warning
looks only at units with a systemd_unit_state check, not at units with
another check.
How it works
- At each run, MMWatchdog runs
systemctl list-units --state=failed --all --plain --full --no-legend --no-pageron the node, as MMWatchdog itself. Thesystemctl list-unitscommand lists failed units of every kind. - MMWatchdog reads the
ActiveStateand theResultof each listed unit with onesystemctl showcommand. MMWatchdog leaves out a unit that stopped being failed aftersystemctl list-unitsand beforesystemctl show. Whensystemctl showgives no answer for a unit, the unit counts as failed, and the unit's message has no result. - MMWatchdog leaves out the units in
systemd_units_to_ignoreand reports each other unit as a finding, with the unit's name as the finding'sid.
MMWatchdog changes nothing about a unit and never clears a failed state. A unit
stays failed in systemd until it runs again or someone runs systemctl reset-failed.
systemd_units_to_ignore is required, so a reader sees at once whether
any unit is left out. [] leaves no unit out. Each entry is the full name of
a unit with its kind, such as myapp.service or backup.timer. An entry is
an exact name, not a pattern. The order of the entries does not matter.
A changed check starts afresh. systemd_units_to_ignore is part of the
check. A reload that changes the ignore list or removes the check closes
each failed unit's incident.
The incidents close without a recovery (incident closed without recovery), like the incident of every changed check. A unit that is still
failed and not ignored gets a new incident after alert_after_failing_for.
A reload that only changes the order of the list changes nothing.
Alerts name the unit. Every alert about a unit puts the unit name after
the check name in the alert's title. The alert's message also names the
unit. The alert's event has the unit name as finding, as in webhook JSON
(title and body of an alert).
mmwatchdog validate cannot know the unit's name, so its warning about long alert
titles counts the every_systemd_unit check's title without a unit.
When adding the unit name makes a title longer than 120 characters,
MMWatchdog shortens the title as MMWatchdog shortens other names in a title.
In a fleet, the hub groups the alerts of each unit by themselves.
backup.service failing on seven nodes becomes
failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed on 7 nodes. A different unit that fails in the minute of the
backup.service failure has separate messages
(alerts in a fleet).
Each unit has its own MMHotline incident key, such as
failed_systemd_units/machine_has_no_failed_systemd_unit: backup.service failed, so two failed units are two MMHotline incidents, and the recovery
of one resolves only its own (mmhotline).
One cause, one alert. When an every_systemd_unit check is kept quiet
because a check named by every_systemd_unit fails, each failed unit is
kept quiet too.
When another check names the every_systemd_unit check as a cause, the
alert of each failed unit names the checks kept quiet
(one cause, one alert).
Recovery actions. A
restart or a
recovery command that names the
every_systemd_unit check in triggered_by_checks counts the check's own
failing time, which lasts while systemd lists any unit as failed.
The check has no incident of its own, so a recovery command gets an empty
MMWATCHDOG_INCIDENT_ID.
Errors and what they mean
| Message | What to do |
|---|---|
every_systemd_unit.fail_when: is required: The check fails while a systemd unit has failed; write a_systemd_unit_has_failed, the only value. |
Write fail_when: a_systemd_unit_has_failed below every_systemd_unit. |
every_systemd_unit.fail_when: expected one of a_systemd_unit_has_failed |
Use a_systemd_unit_has_failed, the only allowed value. |
check_groups.failed_systemd_units.checks.machine_has_no_failed_systemd_unit.every_systemd_unit.systemd_units_to_ignore: is required: the systemd units whose failure this check leaves out, such as [myapp.service], or [] to leave none out |
Write the list. [] leaves no unit out. |
...every_systemd_unit.systemd_units_to_ignore: expected distinct systemd unit names with their kind, such as [myapp.service, backup.timer] |
Write each unit once, with its kind after the last dot. A unit name may contain only letters, digits, and the characters _@.:\-, at most 255 characters. |
An every_systemd_unit check alerts for every failed systemd unit outside the check's systemd_units_to_ignore list. ... (a validation warning) |
Add each named unit to systemd_units_to_ignore. |
systemd inspection: MMWatchdog could not ask systemd for its failed units: ... (unknown) |
systemctl failed. The message contains the failed systemctl command's exit status and output. |
Tested on
-
Go tests: In
failed_units_test.go,TestFailedSystemdUnitsCheckAsksSystemdchecks units of every kind and omits a unit that stops being failed betweensystemctl list-unitsandsystemctl show. The test also checks that ignored units are left out and named in the message. When systemd cannot be inspected, the test expects an unknown run.TestEachFailedSystemdUnitIsItsOwnIncidentchecks the example above minute by minute.More tests in
failed_units_test.gocover each unit's lifecycle and delivery:TestAFailedUnitFollowsTheChecksTimingsByItself,TestFailedUnitsWhileSystemdCannotBeAsked,TestFailedUnitsInMaintenance,TestFailedUnitsAreRaisedAndRemindedOfByThemselves,TestSystemdUnitsToIgnore,TestTheRecoveryOfAFailedUnitSaysThatTheUnitIsNoLongerFailed,TestFailedUnitsInStatusAcrossRestartAndReload,TestFailedUnitsTriggerARecoveryCommand,TestTheHubGroupsFailedUnitsByUnit,TestFailedUnitsHaveTheirOwnMMHotlineIncidents,TestFailedUnitsThroughTheHub, andTestRemindersOfTwoUnitsDoNotReplaceEachOther.Quiet checks are covered by
onecause_test.go:TestAQuietCheckWithOneIncidentForEachFailedUnitandTestTheAlertOfEachFailedUnitNamesTheQuietChecks.TestFailedSystemdUnitsOnTheCommandLineincmd/mmwatchdog/main_test.gochecks the text and JSON of themmwatchdog validatewarning. The command test also checks for one line per unit inmmwatchdog statusandmmwatchdog fleet NODE. -
Suites on real nodes: the
resourcessuite (lifecycle.py): real one-time units that fail. The suite verifies that an ignored failed unit does not fail the check. Two units fail in succession and get separate incidents. Each unit appears in the title, body,mmwatchdog status,attentionlist, andmmwatchdog check-now --json.A repaired unit recovers while the second unit's incident stays open, including across a restart of MMWatchdog. Adding a failed unit to
systemd_units_to_ignorecloses the unit's incident without a recovery.mmwatchdog validatewarns about a unit with a dedicatedsystemd_unit_statecheck.The
fleetsuite (fleet.py): a failed unit on a joined node sends its alert and its recovery through the hub with the unit in the title, and the hub shows the unit under its check. -
The
resourcesandfleetsuites passed on every target distribution. The validation record has the latest run. -
Not tested on real nodes: maintenance windows, reminders, increases of incident severity to
sos, and the MMHotline keys of a unit's incident, which the Go tests cover; anevery_systemd_unitcheck as a quiet check or as a cause of one cause, one alert, which the Go tests cover; a joined node that sends a unit's incident straight to MMHotline while the hub is down, which the Go tests cover; and many failed units at once. Each failed unit is its own alert, with no upper limit.
Why it works this way
- Each failed unit is its own incident. A second failed unit is announced at once. Each repaired unit reports a separate recovery. One incident for the whole list would stay open from the first failure to the last repair and announce nothing in between.
systemd_units_to_ignoreis required, and a unit with its own check is written into it. A reader sees at once whether any unit is left out. Automatically ignoring units with dedicated checks would makesystemd_units_to_ignore: []misleading, because an empty list means no unit is ignored.- A failed unit takes the alert severity of its check. A reader sees the severity on the check itself, as for every alert. Two severities are two checks.
- An unknown run is the check's, not a unit's. MMWatchdog learned nothing about any unit, so it sends one cannot-check alert and leaves each open incident as it is.
- MMWatchdog never clears a failed state. systemd supervises the units, and MMWatchdog only observes them. A unit that stays failed stays visible.
- One mechanism for findings. The
every_systemd_unitmodule and every module that reports several findings at once use one mechanism for findings. Findings behave alike in status, alerts, and the fleet.
This page is docs/reference/modules/every_systemd_unit.md in the MMWatchdog repository.