systemd_unit_commands
systemd_unit_commands watches one program of a systemd unit that runs
several. Some units start several programs, and when one of them ends,
systemd still reports the unit as active, so a
systemd_unit_state check stays healthy.
A systemd_unit_commands check is healthy while a command of the unit
contains the text in
fail_when_no_command_contains.
The matching text can be the program's name. The check fails when no command
of the unit contains the value of fail_when_no_command_contains.
systemctl status myapp.service lists the commands that the unit runs:
├─1201 /usr/local/bin/myapp-web --port 8080
└─1215 /usr/local/bin/myapp-worker --queue mail
The check looks at exactly this list: the commands of the named unit, and of no other program on the node. The text can be any part of a command: the program's name, or one of its arguments.
Use systemd_unit_state for the unit as a whole. Give each program that
matters a check of its own with systemd_unit_commands. To count the
processes of a unit, use
systemd_unit_resource with
resource: processes, which reads the processes of the named unit.
Inputs
systemd_unit(string; required): The .service unit that runs the program, such as myapp.service.fail_when_no_command_contains(string; required): The check fails when no command of the unit contains this text: the name of the program, or one of its arguments.
When the check fails, and when it cannot check
The check is healthy while one command of the unit contains the text, with
the message systemd unit myapp.service runs /usr/local/bin/myapp-worker --queue mail (process 1215). The message shows
the command with the lowest process ID that contains the text, cut after 120
characters.
The check fails when:
- the program ends and the rest of the unit keeps running:
systemd unit myapp.service runs no command that contains myapp-worker. The body of the alert lists the commands that the unit does run, under the linethe commands that the unit runs:. Comparefail_when_no_command_containswith the listed commands to find typing mistakes. The alert body lists at most 11 commands, each cut after 120 bytes and followed by its process ID; - systemd reports a state other than
activeorreloading. A unit that does not run runs no command, so MMWatchdog fails the check with the unit's state:systemd unit myapp.service runs no command that contains myapp-worker: systemd unit myapp.service is inactive, oris failed,is activating, oris deactivating. MMWatchdog counts a unit as running while systemd reportsactiveorreloading, and reads the commands (checks of a systemd unit); - the unit is active and runs no program at all. systemd keeps such a
unit active when the unit says so (
RemainAfterExit=yes), as a unit made from an old init script does.
MMWatchdog cannot check, and the run is unknown, when systemd does not know the unit
(systemd unit myapp.service not found, the message of every check of a
systemd unit), when systemctl fails (a message
that starts with systemd inspection:), when systemd names a control group
with .. in its path, or when the node has no control group file system at
/sys/fs/cgroup. An unknown run opens a cannot-check
incident after alert_after_failing_for, as for every check
(the failing time).
Examples
Watch the worker program of a unit
# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 5m
recovered_after_healthy_for: 30s
checks:
myapp_service_is_running:
systemd_unit_state:
systemd_unit: myapp.service
fail_when: the_unit_is_not_active
alert_severity: sos
myapp_worker_is_running:
systemd_unit_commands:
systemd_unit: myapp.service
fail_when_no_command_contains: myapp-worker
alert_severity: sos
no_alerts_or_recovery_actions_while_these_checks_fail: [myapp/myapp_service_is_running]
On the node, every 30 seconds, MMWatchdog reads the commands of myapp.service.
While /usr/local/bin/myapp-worker --queue mail runs, the check
myapp_worker_is_running is healthy.
- The worker program ends, and
myapp-webkeeps running. At its next run the check fails withsystemd unit myapp.service runs no command that contains myapp-worker. - After 5 minutes of failing, MMWatchdog sends an
sosalert. Thesosalert's body lists/usr/local/bin/myapp-web --port 8080 (process 1201). - When someone stops
myapp.service,myapp_service_is_runningandmyapp_worker_is_runningfail. The example's last line suppresses alerts and recovery actions formyapp_worker_is_runningwhilemyapp_service_is_runningfails. MMWatchdog sends one alert for the stopped unit, frommyapp_service_is_running(one cause, one alert).
A second program of the unit gets a second systemd_unit_commands check.
To restart the unit when the worker ends, name myapp_worker_is_running
in the triggered_by_checks of the check group's
restart_systemd_unit. While
the unit is stopped in the example, MMWatchdog suppresses alerts and
recovery actions for myapp_worker_is_running. MMWatchdog therefore does
not restart the unit because of myapp_worker_is_running.
How it works
- At each run, MMWatchdog asks systemd for the unit's
ActiveStateand itsControlGroupwithsystemctl show, as MMWatchdog itself. - systemd keeps the processes of a unit in one control group of the
kernel. MMWatchdog reads the process IDs of the unit's control group and
every nested control group, without a depth limit, from the files
cgroup.procsbelow/sys/fs/cgroup. On a node with the older arrangement of control groups, MMWatchdog reads systemd's tree in/sys/fs/cgroup/unifiedor/sys/fs/cgroup/systemd. - MMWatchdog reads the command line of each process from
/proc. A process without a command line, such as a thread of the kernel, has its name in brackets, assystemctl statusshows it. A process that ended in the meantime is left out. - The check is healthy when one command contains the text.
An active unit without a control group runs no command. If the kernel removes the control group with the unit's last process after systemd answers but before MMWatchdog reads the control group, the run is unknown.
Choose matching text unique to the program you want to watch. The check
is healthy while any command of the unit contains the text. A unit that
starts its programs from a shell command, such as
sh -c 'myapp-web & myapp-worker & wait', also runs
that shell, and the shell's own command line holds both names.
Errors and what they mean
| Message | What to do |
|---|---|
check_groups.myapp.checks.myapp_worker_is_running.systemd_unit_commands.systemd_unit: is required: the systemd unit this check is about, such as myapp.service |
Name the unit that runs the program. |
...systemd_unit_commands.systemd_unit: must be a systemd .service unit name, such as myapp.service |
Write the full name of a .service unit. The name holds only letters, digits, and the characters _@.:\-. |
...systemd_unit_commands.fail_when_no_command_contains: is required: a text that one of the unit's commands contains, such as the name of the program |
Write the text. A text of only spaces counts as none. |
systemd unit myapp.service runs no command that contains myapp-worker, with the commands of the unit in the body |
The program does not run. If the program runs, compare fail_when_no_command_contains with the commands in the body and with systemctl status myapp.service. |
systemd gave a control group for myapp.service that MMWatchdog does not read: ... (unknown) |
systemd named a control group with .. in its path, which MMWatchdog refuses to follow. |
MMWatchdog cannot read the control group of myapp.service below /sys/fs/cgroup (unknown) |
The node has no control group file system that MMWatchdog can read at /sys/fs/cgroup. |
Tested on
- Go tests:
processes_test.goTestTheProcessesOfASystemdUnit(the processes of the unit's control group and of the groups inside it and of no other unit, a command found by its name and by an argument, the commands listed when none matches, a reloading unit read, a stopped and a deactivating unit failed, the older arrangement of control groups, a unit that systemd keeps active without a process, and a control group that is gone) andTestTheInputsOfASystemdUnitCommandsCheck;check_test.goTestEveryCheckOfAnUnknownSystemdUnitSaysNotFound(an unknown unit). - Suites on real nodes: the
nodesuite (node_scenarios.py): a real unit with two programs. The check fails when one program is killed and names the surviving program in the alert, and recovers when the killed program is started again. When the unit stops, the program check fails, but MMWatchdog suppresses alerts and recovery actions for the program check. The unit'ssystemd_unit_statecheck reports the stop. - The
nodesuite passed on every target distribution. The validation record has the latest run.
Why it works this way
-
An incomplete process list is unknown. A command in an unreadable or deeply nested control group must not be reported as absent.
-
The check looks only among the commands of the named unit. The text cannot match a program of another unit.
systemctl statusshows the operator the commands of the named unit. A search of the whole node, aspgrepdoes, would find a program with the watched program's name elsewhere. -
The check compares a text with the whole command, not a program name. Some services run every program as the same interpreter, such as
perl, each with a different script. Matching text in a script argument distinguishes the interpreter processes. -
systemd_unit_commandsandfail_when_no_command_containsuse "command", the word thatsystemctl statusshows. An operator does not need to know the kernel's words, process and command line, to write the check. -
A failed check lists the commands of the unit. Compare
fail_when_no_command_containswith the commands in the alert to find typing mistakes. -
A unit that does not run fails the check, not unknown. A unit that systemd reports in another state than
activeorreloading, such as a stopped one or one that systemd is stopping, runs no command, which is a fact that MMWatchdog observed. The unit's ownsystemd_unit_statecheck is the cause, andno_alerts_or_recovery_actions_while_these_checks_failkeeps thesystemd_unit_commandscheck quiet while thesystemd_unit_statecheck fails. Every check of a systemd unit counts the same states as running (checks of a systemd unit).
This page is docs/reference/modules/systemd_unit_commands.md in the MMWatchdog repository.