systemd_unit_state
systemd_unit_state asks systemd for the state of one unit on the node: a
service, or a unit of another kind, such as a timer, a path unit, or a
socket. The check is healthy while systemd reports the unit as active or
reloading. A unit in any state except active or reloading, including
stopped or failed, fails the check. Set fail_when to
the_unit_is_not_active to state when the check
fails; "active" there includes reloading, because a reload does not stop
the unit's program. MMWatchdog asks with systemctl show and changes nothing
about the unit.
Choose a related module when knowing whether a unit is active is not enough:
systemd_unit_crash_loopfor a unit that crashes and that systemd starts again by itsRestart=setting. Such a unit isactiveat almost every run, sosystemd_unit_statestays healthy.systemd_unit_commandsfor one program of a unit that runs several. systemd reports the unit asactivewhile one of its programs has ended.every_systemd_unitfor every unit of the node at once, of every kind, and for a unit that is stopped most of the time, such as the unit of a scheduled job. Such a unit isinactivebetween its runs, so asystemd_unit_statecheck cannot watch it. Asystemd_unit_statecheck can watch the job's timer, which isactivewhile it waits for its next time.systemd_unit_resourcefor the CPU, the memory, or the number of processes of a unit.
A systemd_unit_state check is an ordinary check under checks, with a
name of your choice and its own alert_severity. MMWatchdog creates no check on its
own. One check group can hold checks of one unit, of several units, and of
the node.
Inputs
systemd_unit(string; required): The unit that the check watches, with its kind, such as myapp.service, backup.timer, or myapp.path.fail_when(the_unit_is_not_active; required): The check fails while systemd reports the unit as neither active nor reloading; write the_unit_is_not_active, the only value.
When the check fails, and when it cannot check
The check is healthy when systemd reports the unit's ActiveState as
active or reloading. For active, the message is systemd unit myapp.service is active. For reloading, the message is systemd unit myapp.service is reloading. A reload does not stop the unit's program.
Every check of a systemd unit treats active and reloading as running,
as checks of a systemd unit
explains. Every state except active and reloading fails the check, and
the message says which state systemd reports:
| systemd reports | Message of the failed check |
|---|---|
inactive |
systemd unit myapp.service is stopped. systemd reports no failure: the unit was stopped on request, or its program ended normally. MMWatchdog did not start the unit. For a unit that runs no program, such as a timer or a path unit: systemd unit myapp-reload.path is stopped. systemd reports no failure: the unit was stopped on request, or was never started. MMWatchdog did not start the unit. |
failed |
systemd unit myapp.service failed (systemd result: signal), with the Result that systemd gives, such as signal or exit-code. Without a result, systemd unit myapp.service failed |
Another state, such as activating while systemd restarts the unit, or deactivating while systemd stops it |
systemd unit myapp.service is activating |
A stopped unit is a failed check. The check says that the unit should run,
so a unit that someone stopped sends its alert after
alert_after_failing_for, unless a
maintenance window covers the
check group. MMWatchdog does not start the unit again:
a stopped unit stays stopped.
MMWatchdog cannot check, and the run is unknown, when:
- systemd does not know the unit:
systemd unit myapp.service not found, the message of every check of a systemd unit; systemctlfails, such as when systemd does not answer: the message starts withsystemd inspection:;- systemd gives no
ActiveState:systemd returned no ActiveState for myapp.service.
An unknown run opens no failure incident. The check follows the check
group's timings on its own track and, after alert_after_failing_for,
opens a cannot-check incident and sends a cannot-check alert, as for every check
(the failing time).
Examples
Alert when the unit stops
# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 1m
recovered_after_healthy_for: 30s
checks:
myapp_service_is_running:
systemd_unit_state:
systemd_unit: myapp.service
fail_when: the_unit_is_not_active
alert_severity: sos
On the node, MMWatchdog asks systemd every 30 seconds whether myapp.service is
active. When the unit stops or fails, the check fails at its next run. When
the check has failed for 1 minute, MMWatchdog opens an incident and sends an sos
alert with the message from the table above. When the unit has been active
again for 30 seconds, MMWatchdog sends the recovery.
Alert when a timer or a path unit stops
# /etc/mmwatchdog/check_groups/myapp_triggers.yml
check_groups:
myapp_triggers:
check_every: 1m
time_limit_per_run: 5s
alert_after_failing_for: 5m
recovered_after_healthy_for: 1m
checks:
backup_timer_is_waiting:
systemd_unit_state:
systemd_unit: myapp-backup.timer
fail_when: the_unit_is_not_active
alert_severity: warn
reload_path_unit_is_watching:
systemd_unit_state:
systemd_unit: myapp-reload.path
fail_when: the_unit_is_not_active
alert_severity: warn
A timer is active while it waits for its next time, and a path unit while
it watches its file or directory. When the timer or path unit stops, the
service that the stopped unit starts is no longer started, and nothing
reports a failure:
every_systemd_unit alerts for a timer or
path unit that fails, not for one that is stopped. The
backup_timer_is_waiting check sends a warn alert when
myapp-backup.timer has not been active for 5 minutes. The
reload_path_unit_is_watching check sends a warn alert when
myapp-reload.path has not been active for 5 minutes.
Restart the unit when it fails
# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s # run each check every 30 seconds
time_limit_per_run: 5s # one run of a check may take 5 seconds
alert_after_failing_for: 5m # send the alert after 5 minutes of failing
recovered_after_healthy_for: 30s # the failure is over after 30 healthy seconds
checks:
myapp_service_is_running:
systemd_unit_state:
systemd_unit: myapp.service
fail_when: the_unit_is_not_active
alert_severity: sos
myapp_answers_health_request:
http:
url: http://127.0.0.1:8080/health
alert_severity: sos
restart_systemd_unit:
systemd_unit: myapp.service # the unit that MMWatchdog restarts
triggered_by_checks: [myapp_service_is_running, myapp_answers_health_request]
restart_after_failing_for: 1m # restart after 1 minute of failing
max_attempts: 3
min_time_between_attempts: 1m
reset_attempts_after_healthy_for: 10m
On the node, when the program of myapp.service crashes and systemd
reports the unit as failed, the check myapp_service_is_running fails.
After 1 minute of failing, MMWatchdog restarts myapp.service. If the unit is active
again at the next run, the failing time ends after 30 healthy seconds and no
alert goes out. If the unit was stopped on request (inactive), MMWatchdog does not
restart it, and the alert goes out after 5 minutes.
Restart a systemd unit says
how restarts work for every check. Only a systemd_unit_state check of the
unit named by the restart follows the next rule. When systemd reports the
checked unit as active again, MMWatchdog does not restart the unit
(when a restart is due).
Several instances of one program
Several instances of one program, such as the workers worker@1.service and
worker@2.service started from one unit file, get one check group for each
instance. MMX writes the file from the list of instances.
# /etc/mmwatchdog/check_groups/workers.yml
check_groups:
worker_1:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 5m
recovered_after_healthy_for: 30s
checks:
worker_1_service_is_running:
systemd_unit_state:
systemd_unit: worker@1.service
fail_when: the_unit_is_not_active
alert_severity: warn
worker_2:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 5m
recovered_after_healthy_for: 30s
checks:
worker_2_service_is_running:
systemd_unit_state:
systemd_unit: worker@2.service
fail_when: the_unit_is_not_active
alert_severity: warn
On the node, each instance has its own check and alerts under its own name.
Each check group can carry a
restart of the check group's
instance, so only the instance that failed is restarted. One
maintenance command can name
several check groups, which covers all instances during a deploy. A check
cannot name a pattern such as worker@*.service.
How it works
- At each run, MMWatchdog runs
systemctl show --property=LoadState,ActiveState,Result -- myapp.serviceon the node, as MMWatchdog itself, within the check group'stime_limit_per_run. - A
LoadStateofnot-foundmakes the run unknown. Otherwise theActiveStatedecides, as the section above explains. TheResultgoes into the message of a failed unit.
systemd reports inactive both for a unit that was stopped on request and
for a program that ended on its own with exit status 0. A program that must
always run therefore needs Restart=always in its unit file, so that systemd
starts it again after every exit. Then inactive can only mean that the unit
was stopped on request.
Run your own program as a systemd unit
shows such a unit file.
An operator who stops a unit on purpose and wants no alert turns on a maintenance window before the stop. A unit that should stay stopped permanently is removed from the node's check groups.
Errors and what they mean
| Message | What to do |
|---|---|
check_groups.myapp.checks.myapp_service_is_running.systemd_unit_state.systemd_unit: is required: the systemd unit this check is about, such as myapp.service |
Name the unit with systemd_unit below systemd_unit_state. |
...systemd_unit_state.systemd_unit: must be the name of a systemd unit with its kind, such as myapp.service, backup.timer, or myapp.path |
Write the full name of the unit, with its kind at the end, such as .service, .timer, or .path. The name holds only letters, digits, and the characters _@.:\-. |
...systemd_unit_state.fail_when: is required: The check fails while systemd reports the unit as neither active nor reloading; write the_unit_is_not_active, the only value. |
Write fail_when: the_unit_is_not_active below systemd_unit_state. |
...systemd_unit_state.fail_when: expected one of the_unit_is_not_active |
Use the_unit_is_not_active, the only allowed value. |
systemd unit myapp.service not found (the check is unknown) |
systemd has no unit named myapp.service on the node running the check. Check the name with systemctl status myapp.service, or install the unit. |
A message that starts with systemd inspection: (the check is unknown) |
systemctl failed. The message holds the exit status of systemctl and what systemctl wrote. |
Tested on
- Go tests:
check_test.go,TestUnitStateMessages: verifies messages for active, reloading, stopped, failed, activating, and deactivating units, and for active, stopped, and failed path units. The test also accepts service, timer, path, and socket unit names.check_test.go,TestEveryCheckOfAnUnknownSystemdUnitSaysNotFound: verifies the not-found message for every check of an unknown systemd unit.config_test.go,TestSystemdUnitChecks: refuses a check without a unit name or with a name without the unit's kind. The test also accepts checks of two units and of the node in one check group.node_checks_test.go,TestNodeChecksRequireTheirFailureRule: verifies thatfail_whenis required and has only one allowed value.daemon_test.go,TestAStoppedUnitStaysStopped: verifies that MMWatchdog leaves a stopped unit stopped.daemon_test.go,TestRecoveryDoesNotRestartSystemdUnitThatJustBecameActive: verifies that a failed state check does not trigger a restart of a unit that became active again.
- Suites on real nodes:
acceptance(acceptance.py): verifies restarts and recovery of a real unit.diagnostics(diagnostics.py): verifies systemd's start limits and a slow start reported assystemd unit mmwatchdog-starting.service is activating.resources(lifecycle.py): verifies that a reloading unit is healthy and a deactivating unit fails without a restart while systemd stops the unit. For a unit stopped withsystemctl stop, the suite verifies that MMWatchdog leaves the unit stopped and keeps the incident open. When a unit's program is killed, the suite verifies that MMWatchdog reportsfailedwith systemd resultsignaland restarts the unit.
- The
acceptance,diagnostics, andresourcessuites passed on every target distribution. The validation record has the latest run.
Why it works this way
- The check names its unit on its own line, and MMWatchdog generates no check. Each line can be read alone, a check group can hold checks of the node and of several units, and every check is visible where it is configured.
- A stopped unit fails the check and alerts. The check says that the unit should run, and a unit stopped for a test and forgotten is an outage.
- The message says whether systemd reports a failure.
systemctl showtells a failure from no failure, but not a stop on request from a normal exit. Only the journal could, and reading it would give MMWatchdog a second source of truth.Restart=alwaysin the unit file covers every exit of the program instead. - A unit of every kind. A timer or path unit that stopped starts
nothing, so its service is no longer started, without any failure. systemd
exposes an
ActiveStateproperty for every kind of unit, so thesystemd_unit_statemodule can check every kind. Checks of one unit besidessystemd_unit_state, such assystemd_unit_commands, stay with services, because they ask about the programs that a service runs. - A reloading unit is healthy. A reload does not stop the unit's
program, and
systemctl is-activesucceeds foractiveandreloadingalone, so MMWatchdog and systemd agree on what runs. Every check of a systemd unit treatsactiveandreloadingas running (checks of a systemd unit). - A unit that systemd does not know is unknown, not failed. The check cannot establish health for a unit that does not exist.
- Several instances get one check group each, with no pattern over unit names. systemd forgets a stopped instance of a unit, so a pattern would find only the remaining instances and report them as healthy. And every unit that MMWatchdog watches is named in the configuration file.
This page is docs/reference/modules/systemd_unit_state.md in the MMWatchdog repository.