systemd_unit_state

systemd_unit_state asks systemd for the state of one unit on the node: a service, or a unit of another kind, such as a timer, a path unit, or a socket. The check is healthy while systemd reports the unit as active or reloading. A unit in any state except active or reloading, including stopped or failed, fails the check. Set fail_when to the_unit_is_not_active to state when the check fails; "active" there includes reloading, because a reload does not stop the unit's program. MMWatchdog asks with systemctl show and changes nothing about the unit.

Choose a related module when knowing whether a unit is active is not enough:

  • systemd_unit_crash_loop for a unit that crashes and that systemd starts again by its Restart= setting. Such a unit is active at almost every run, so systemd_unit_state stays healthy.
  • systemd_unit_commands for one program of a unit that runs several. systemd reports the unit as active while one of its programs has ended.
  • every_systemd_unit for every unit of the node at once, of every kind, and for a unit that is stopped most of the time, such as the unit of a scheduled job. Such a unit is inactive between its runs, so a systemd_unit_state check cannot watch it. A systemd_unit_state check can watch the job's timer, which is active while it waits for its next time.
  • systemd_unit_resource for the CPU, the memory, or the number of processes of a unit.

A systemd_unit_state check is an ordinary check under checks, with a name of your choice and its own alert_severity. MMWatchdog creates no check on its own. One check group can hold checks of one unit, of several units, and of the node.

Inputs

  • systemd_unit (string; required): The unit that the check watches, with its kind, such as myapp.service, backup.timer, or myapp.path.
  • fail_when (the_unit_is_not_active; required): The check fails while systemd reports the unit as neither active nor reloading; write the_unit_is_not_active, the only value.

When the check fails, and when it cannot check

The check is healthy when systemd reports the unit's ActiveState as active or reloading. For active, the message is systemd unit myapp.service is active. For reloading, the message is systemd unit myapp.service is reloading. A reload does not stop the unit's program. Every check of a systemd unit treats active and reloading as running, as checks of a systemd unit explains. Every state except active and reloading fails the check, and the message says which state systemd reports:

systemd reports Message of the failed check
inactive systemd unit myapp.service is stopped. systemd reports no failure: the unit was stopped on request, or its program ended normally. MMWatchdog did not start the unit. For a unit that runs no program, such as a timer or a path unit: systemd unit myapp-reload.path is stopped. systemd reports no failure: the unit was stopped on request, or was never started. MMWatchdog did not start the unit.
failed systemd unit myapp.service failed (systemd result: signal), with the Result that systemd gives, such as signal or exit-code. Without a result, systemd unit myapp.service failed
Another state, such as activating while systemd restarts the unit, or deactivating while systemd stops it systemd unit myapp.service is activating

A stopped unit is a failed check. The check says that the unit should run, so a unit that someone stopped sends its alert after alert_after_failing_for, unless a maintenance window covers the check group. MMWatchdog does not start the unit again: a stopped unit stays stopped.

MMWatchdog cannot check, and the run is unknown, when:

  • systemd does not know the unit: systemd unit myapp.service not found, the message of every check of a systemd unit;
  • systemctl fails, such as when systemd does not answer: the message starts with systemd inspection:;
  • systemd gives no ActiveState: systemd returned no ActiveState for myapp.service.

An unknown run opens no failure incident. The check follows the check group's timings on its own track and, after alert_after_failing_for, opens a cannot-check incident and sends a cannot-check alert, as for every check (the failing time).

Examples

Alert when the unit stops

# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
  myapp:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 1m
    recovered_after_healthy_for: 30s
    checks:
      myapp_service_is_running:
        systemd_unit_state:
          systemd_unit: myapp.service
          fail_when: the_unit_is_not_active
        alert_severity: sos

On the node, MMWatchdog asks systemd every 30 seconds whether myapp.service is active. When the unit stops or fails, the check fails at its next run. When the check has failed for 1 minute, MMWatchdog opens an incident and sends an sos alert with the message from the table above. When the unit has been active again for 30 seconds, MMWatchdog sends the recovery.

Alert when a timer or a path unit stops

# /etc/mmwatchdog/check_groups/myapp_triggers.yml
check_groups:
  myapp_triggers:
    check_every: 1m
    time_limit_per_run: 5s
    alert_after_failing_for: 5m
    recovered_after_healthy_for: 1m
    checks:
      backup_timer_is_waiting:
        systemd_unit_state:
          systemd_unit: myapp-backup.timer
          fail_when: the_unit_is_not_active
        alert_severity: warn
      reload_path_unit_is_watching:
        systemd_unit_state:
          systemd_unit: myapp-reload.path
          fail_when: the_unit_is_not_active
        alert_severity: warn

A timer is active while it waits for its next time, and a path unit while it watches its file or directory. When the timer or path unit stops, the service that the stopped unit starts is no longer started, and nothing reports a failure: every_systemd_unit alerts for a timer or path unit that fails, not for one that is stopped. The backup_timer_is_waiting check sends a warn alert when myapp-backup.timer has not been active for 5 minutes. The reload_path_unit_is_watching check sends a warn alert when myapp-reload.path has not been active for 5 minutes.

Restart the unit when it fails

# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
  myapp:
    check_every: 30s                    # run each check every 30 seconds
    time_limit_per_run: 5s              # one run of a check may take 5 seconds
    alert_after_failing_for: 5m         # send the alert after 5 minutes of failing
    recovered_after_healthy_for: 30s    # the failure is over after 30 healthy seconds
    checks:
      myapp_service_is_running:
        systemd_unit_state:
          systemd_unit: myapp.service
          fail_when: the_unit_is_not_active
        alert_severity: sos
      myapp_answers_health_request:
        http:
          url: http://127.0.0.1:8080/health
        alert_severity: sos
    restart_systemd_unit:
      systemd_unit: myapp.service       # the unit that MMWatchdog restarts
      triggered_by_checks: [myapp_service_is_running, myapp_answers_health_request]
      restart_after_failing_for: 1m     # restart after 1 minute of failing
      max_attempts: 3
      min_time_between_attempts: 1m
      reset_attempts_after_healthy_for: 10m

On the node, when the program of myapp.service crashes and systemd reports the unit as failed, the check myapp_service_is_running fails. After 1 minute of failing, MMWatchdog restarts myapp.service. If the unit is active again at the next run, the failing time ends after 30 healthy seconds and no alert goes out. If the unit was stopped on request (inactive), MMWatchdog does not restart it, and the alert goes out after 5 minutes.

Restart a systemd unit says how restarts work for every check. Only a systemd_unit_state check of the unit named by the restart follows the next rule. When systemd reports the checked unit as active again, MMWatchdog does not restart the unit (when a restart is due).

Several instances of one program

Several instances of one program, such as the workers worker@1.service and worker@2.service started from one unit file, get one check group for each instance. MMX writes the file from the list of instances.

# /etc/mmwatchdog/check_groups/workers.yml
check_groups:
  worker_1:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 5m
    recovered_after_healthy_for: 30s
    checks:
      worker_1_service_is_running:
        systemd_unit_state:
          systemd_unit: worker@1.service
          fail_when: the_unit_is_not_active
        alert_severity: warn
  worker_2:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 5m
    recovered_after_healthy_for: 30s
    checks:
      worker_2_service_is_running:
        systemd_unit_state:
          systemd_unit: worker@2.service
          fail_when: the_unit_is_not_active
        alert_severity: warn

On the node, each instance has its own check and alerts under its own name. Each check group can carry a restart of the check group's instance, so only the instance that failed is restarted. One maintenance command can name several check groups, which covers all instances during a deploy. A check cannot name a pattern such as worker@*.service.

How it works

  1. At each run, MMWatchdog runs systemctl show --property=LoadState,ActiveState,Result -- myapp.service on the node, as MMWatchdog itself, within the check group's time_limit_per_run.
  2. A LoadState of not-found makes the run unknown. Otherwise the ActiveState decides, as the section above explains. The Result goes into the message of a failed unit.

systemd reports inactive both for a unit that was stopped on request and for a program that ended on its own with exit status 0. A program that must always run therefore needs Restart=always in its unit file, so that systemd starts it again after every exit. Then inactive can only mean that the unit was stopped on request. Run your own program as a systemd unit shows such a unit file.

An operator who stops a unit on purpose and wants no alert turns on a maintenance window before the stop. A unit that should stay stopped permanently is removed from the node's check groups.

Errors and what they mean

Message What to do
check_groups.myapp.checks.myapp_service_is_running.systemd_unit_state.systemd_unit: is required: the systemd unit this check is about, such as myapp.service Name the unit with systemd_unit below systemd_unit_state.
...systemd_unit_state.systemd_unit: must be the name of a systemd unit with its kind, such as myapp.service, backup.timer, or myapp.path Write the full name of the unit, with its kind at the end, such as .service, .timer, or .path. The name holds only letters, digits, and the characters _@.:\-.
...systemd_unit_state.fail_when: is required: The check fails while systemd reports the unit as neither active nor reloading; write the_unit_is_not_active, the only value. Write fail_when: the_unit_is_not_active below systemd_unit_state.
...systemd_unit_state.fail_when: expected one of the_unit_is_not_active Use the_unit_is_not_active, the only allowed value.
systemd unit myapp.service not found (the check is unknown) systemd has no unit named myapp.service on the node running the check. Check the name with systemctl status myapp.service, or install the unit.
A message that starts with systemd inspection: (the check is unknown) systemctl failed. The message holds the exit status of systemctl and what systemctl wrote.

Tested on

  • Go tests:
    • check_test.go, TestUnitStateMessages: verifies messages for active, reloading, stopped, failed, activating, and deactivating units, and for active, stopped, and failed path units. The test also accepts service, timer, path, and socket unit names.
    • check_test.go, TestEveryCheckOfAnUnknownSystemdUnitSaysNotFound: verifies the not-found message for every check of an unknown systemd unit.
    • config_test.go, TestSystemdUnitChecks: refuses a check without a unit name or with a name without the unit's kind. The test also accepts checks of two units and of the node in one check group.
    • node_checks_test.go, TestNodeChecksRequireTheirFailureRule: verifies that fail_when is required and has only one allowed value.
    • daemon_test.go, TestAStoppedUnitStaysStopped: verifies that MMWatchdog leaves a stopped unit stopped.
    • daemon_test.go, TestRecoveryDoesNotRestartSystemdUnitThatJustBecameActive: verifies that a failed state check does not trigger a restart of a unit that became active again.
  • Suites on real nodes:
    • acceptance (acceptance.py): verifies restarts and recovery of a real unit.
    • diagnostics (diagnostics.py): verifies systemd's start limits and a slow start reported as systemd unit mmwatchdog-starting.service is activating.
    • resources (lifecycle.py): verifies that a reloading unit is healthy and a deactivating unit fails without a restart while systemd stops the unit. For a unit stopped with systemctl stop, the suite verifies that MMWatchdog leaves the unit stopped and keeps the incident open. When a unit's program is killed, the suite verifies that MMWatchdog reports failed with systemd result signal and restarts the unit.
  • The acceptance, diagnostics, and resources suites passed on every target distribution. The validation record has the latest run.

Why it works this way

  • The check names its unit on its own line, and MMWatchdog generates no check. Each line can be read alone, a check group can hold checks of the node and of several units, and every check is visible where it is configured.
  • A stopped unit fails the check and alerts. The check says that the unit should run, and a unit stopped for a test and forgotten is an outage.
  • The message says whether systemd reports a failure. systemctl show tells a failure from no failure, but not a stop on request from a normal exit. Only the journal could, and reading it would give MMWatchdog a second source of truth. Restart=always in the unit file covers every exit of the program instead.
  • A unit of every kind. A timer or path unit that stopped starts nothing, so its service is no longer started, without any failure. systemd exposes an ActiveState property for every kind of unit, so the systemd_unit_state module can check every kind. Checks of one unit besides systemd_unit_state, such as systemd_unit_commands, stay with services, because they ask about the programs that a service runs.
  • A reloading unit is healthy. A reload does not stop the unit's program, and systemctl is-active succeeds for active and reloading alone, so MMWatchdog and systemd agree on what runs. Every check of a systemd unit treats active and reloading as running (checks of a systemd unit).
  • A unit that systemd does not know is unknown, not failed. The check cannot establish health for a unit that does not exist.
  • Several instances get one check group each, with no pattern over unit names. systemd forgets a stopped instance of a unit, so a pattern would find only the remaining instances and report them as healthy. And every unit that MMWatchdog watches is named in the configuration file.

This page is docs/reference/modules/systemd_unit_state.md in the MMWatchdog repository.