shell
A shell check runs a Bash script on the node with pipefail.
With pipefail, a pipeline succeeds only when every command in the pipeline
succeeds. The check is healthy when the script ends with an exit code that
counts as success. A one-line command and a script of many lines run alike, and pipes, redirection, variables, and ordinary Bash syntax work.
No wrapper script, JSON, or module written by the team is needed:
shell: test -f /var/lib/myapp/ready is a whole check.
A shell check passes or fails and reports no number.
A check of a team's own application may need a number with a limit,
application-specific inputs, or several findings. Such a check uses a
module of the project's own.
Inputs
A check writes the module's inputs as a mapping below the module's name. In place of the mapping, it can write one value after the name:
- the value of
shell(string): The script alone, in place of the mapping of inputs.
The inputs of the mapping:
script(string; required): The Bash script. A one-line command and a script of many lines run in the same way.success_exit_codes(list_of_integers; optional): The exit codes that count as healthy, [0] by default.run_as_user_account(string; optional): The account that the script runs as, mmwatchdog by default. The mmwatchdog account is unprivileged.run_from_directory(string; optional): The absolute path of the directory that the script starts in, / by default.env(mapping_of_strings; optional): Environment variables for the script, beside the standard ones that MMWatchdog gives it.
When the check fails, and when it cannot check
-
The check is healthy when the script ends with an exit code in
success_exit_codes, which is[0]without the input. An explicit list replaces the default. -
The check fails when the script ends by itself with any other exit code. The message is
command exited 1, with the exit code. -
MMWatchdog cannot check, and the run is unknown, when Bash cannot find or run a command of the script.
MMWatchdog identifies Bash's error by the script's exit code and the last line of the script's standard error. Exit code 127 means Bash cannot find the command or script. Examples are
bash: line 1: /usr/local/bin/check-queue: No such file or directoryandbash: line 1: myapp-cli: command not found.Exit code 126 means Bash found the command or script but cannot execute the command or script. For example,
...: Permission deniedcan mean the script is not executable.The message of the run adds Bash's line:
command exited 127, which Bash uses when it cannot find a command: bash: line 1: myapp-cli: command not found. An exit code insuccess_exit_codesstays healthy. -
A program that ends with 127 or 126 by itself fails the check unless the exit code is in
success_exit_codes, like any other exit code. The dynamic loader ends a program whose shared library is missing with 127 and writeserror while loading shared libraries: the program cannot work, which is what the check is for. -
MMWatchdog also reports an unknown run when the script cannot start, runs past
time_limit_per_run, or ends because of a signal. A script that writes more than 64 KiB to its standard output also makes the run unknown. The accepted exit codes cannot make a run healthy after a start error, timeout, signal, or standard output above 64 KiB. -
The exit code, the standard output, and the standard error are kept in the check's latest observation and in the evidence of the check's incident. The standard error keeps 16 KiB, followed by
[stderr truncated]when the script wrote more (built-in limits).
Examples
A one-line command
# /etc/mmwatchdog/check_groups/myapp_scripts.yml
check_groups:
myapp_selftest:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 5m
recovered_after_healthy_for: 1m
shell: myapp-cli selftest | grep 'all tests passed' > /dev/null
alert_severity: warn
Every minute, MMWatchdog on the node runs the command as the unprivileged account
mmwatchdog, in /. grep ends with exit code 0 when the self-test printed
all tests passed, and the check is healthy. When myapp-cli itself fails,
pipefail makes the whole pipe fail, even though grep ran. The example
uses grep ... > /dev/null and not grep -q: grep -q stops reading at
the first match, so a self-test that prints more after it can end with a
broken pipe, and pipefail would then fail a healthy run. When the check
has failed for 5 minutes, MMWatchdog sends the warn alert. The check is written
directly on the check group, so status shows it as myapp_selftest/shell.
A script with its own account, directory, and variables
# /etc/mmwatchdog/check_groups/myapp_scripts.yml
check_groups:
queue:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 5m
recovered_after_healthy_for: 1m
shell:
script: /usr/local/bin/check-queue
success_exit_codes: [0, 2]
run_as_user_account: myapp
run_from_directory: /srv/myapp
env: {QUEUE: default}
alert_severity: warn
MMWatchdog on the node runs /usr/local/bin/check-queue every minute, as the
account myapp, in the directory /srv/myapp, with the variable
QUEUE=default beside the standard ones. The script's exit codes 0 and 2
are both healthy, so a script that uses 2 for "the queue is busy, but fine"
does not alert. Any other exit code fails the check.
How it works
-
MMWatchdog runs
/bin/bash --noprofile --norc -o pipefail -c SCRIPT. Bash runs noninteractively, without profile or rc files. MMWatchdog does not change the script; Bash interprets it when the check runs.set -eandset -uare not added: a script that wants them sets them itself. -
The script runs in a separate process. With
run_as_user_account, MMWatchdog runs the script as the named account. Withoutrun_as_user_account, the script runs as the unprivileged accountmmwatchdog. Withrun_from_directory, the script starts in the named directory. Withoutrun_from_directory, the script starts in/.run_as_user_account: rootgives the script all the access of root. The account is the only boundary, and MMWatchdog gives the script no sandbox. The settingsrun_as_user_account,run_from_directory, andenvalso serve a module that is not core (check groups and checks, required settings and defaults). -
The script inherits no environment of the daemon. MMWatchdog gives the script
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, the account'sHOMEandUSER,LANG=C.UTF-8, and the variables inenv.BASH_ENVandENVcannot be set, so no file runs before the script. -
The run ends at
time_limit_per_run: MMWatchdog ends the script and every program that it started. The one exception is a program that makes itself independent of the script, as a daemon does when it starts. -
mmwatchdog check-now CHECK_GROUPruns the script once and shows its exit code, its standard output, and its standard error (operate). -
A recovery command runs in the same way as a shell check.
Errors and what they mean
Validation errors name the check, such as check_groups.queue.shell.success_exit_codes: ...:
| Message | What the message means, and what to do |
|---|---|
script: cannot be empty |
Write the script, or the command alone as the value of shell. |
success_exit_codes: cannot be empty |
Leave the input out for [0], or list the codes. |
success_exit_codes: expected distinct integers 0..255 |
An exit code is a whole number from 0 to 255, and each is listed once. |
run_from_directory: must be absolute |
Write the directory's absolute path. |
env.BASH_ENV: unsupported environment variable |
In the displayed error, BASH_ENV is forbidden. MMWatchdog also refuses ENV, MMWATCHDOG_BIN, and MMWATCHDOG_MODULE_HELPER. For another variable, the error can mean an invalid variable name or a value containing a zero byte. |
the module shell reports no measurement, so its check takes no limit |
A shell check takes no fail_when_measurement_above or fail_when_measurement_below. |
sudo mmwatchdog validate on the node also checks that the account and
the directory exist (validation), with
messages such as run_as_user_account: user: unknown user myapp and
run_from_directory: /srv/myapp is not a directory.
Messages of a run:
| Message | What the message means, and what to do |
|---|---|
command exited 3 |
The script ended with an exit code that is not in success_exit_codes: the check failed. The script's standard output and standard error are in the evidence. |
command exited 127, which Bash uses when it cannot find a command: bash: line 1: myapp-cli: command not found |
The script names a command or a file that is not on the node, or not in the script's PATH. MMWatchdog cannot check, and runs no recovery action. Install the program, or correct the name or the path. |
command exited 127 |
A program of the script ended with 127 by itself, such as one whose shared library is missing. The program's standard error in the evidence says why. The check failed. |
command exited 126, which Bash uses when it cannot run a command: bash: line 1: /usr/local/bin/check-queue: Permission denied |
The script names a file that the check's account cannot run, often because the file is not executable. Make /usr/local/bin/check-queue executable (chmod 0755), or let the check's account read and run /usr/local/bin/check-queue. |
the run took longer than 5s (time_limit_per_run), so MMWatchdog ended it |
The script ran past time_limit_per_run (here 5 seconds) and MMWatchdog ended it. The check is unknown, and the evidence shows no exit code, because the script wrote none. Make the script faster, or give the check group a longer time_limit_per_run. |
check process was terminated by a signal |
A signal ended the script, for example the kernel's out-of-memory killer. |
check standard output exceeds 65536 bytes |
The script wrote more than 64 KiB to its standard output. Write less, for example by sending the output of a command that the check only tests to /dev/null. |
run_as_user_account "myapp": user: unknown user myapp |
The account does not exist on the node. |
running as the user account myapp requires root |
MMWatchdog runs without root, as with mmwatchdog check-now --without-daemon by a normal user, and cannot change to another account. |
Tested on
-
Go tests:
TestShellContractincheck_test.gochecks a pipe that fails withpipefail, exit codes inside and outsidesuccess_exit_codes, and an unset variable withoutset -u.TestShellCheckThatCannotFindOrRunACommandIsUnknownincheck_test.gochecks unknown results for a missing script, a command not installed, and a script not executable. The test checks failure when a program ends with 127 for a missing shared library, or a script exits with 127 or 126 itself. The test also checks that 127 listed insuccess_exit_codesis healthy.TestTimeoutAndOutputLimitsincheck_test.gochecks unknown results for a script exceeding the time limit, endless output, and more than 64 KiB of standard output.TestRunCommandRunsAsMMWatchdogItselfOrAsACheckinprimitives_test.gochecks the account, directory, and use of only the environment that MMWatchdog gives the script.TestConfigStrictandTestValidateRuntimeChecksExecutioninconfig_test.gocover the validation of every input,BASH_ENVrefused, and a missing account or directory. -
The acceptance suite (
acceptance.py), on every target distribution:false | catfails with exit code 1,exit 2withsuccess_exit_codes: [0, 2]is healthy and keeps its exit code, andid -unprintsmmwatchdog.The acceptance, diagnostics, and one-cause scenarios use shell checks for failures, incidents, recovery commands, and alerts throughout, and the soak suite runs 24 shell checks that flip every 500 ms.
-
The latest run passed on every target distribution (validation record).
Why it works this way
-
shellis Bash only. One shell to learn, and Bash can call any interpreter. -
A shell check passes or fails and reports no number. A number with a limit is the job of a module. A second way to report a number would do one job twice.
-
The script runs as the unprivileged account
mmwatchdogby default. The account's home is apart from the daemon's state, which only root can read, so a check cannot read or change incidents, credentials, or the alert queue. -
The script inherits no environment of the daemon. The script's result then does not depend on the daemon's environment. No file runs before the script through
BASH_ENV. -
MMWatchdog does not change the script. Configuration is data, with no expressions or templates, so Bash runs the script text from the YAML configuration without changes.
-
A command that Bash cannot find or run is unknown, not failed. Such a script did not look at the service. Reporting the run as failed would blame a service that may be fine. If the check triggers a unit's restart, reporting the run as failed would restart that healthy unit. An unknown run never starts a recovery action.
MMWatchdog goes by Bash's own message and not by the exit code alone, because a program uses 127 for other faults too. The dynamic loader ends a program whose shared library is missing with 127. A program with a missing shared library is broken, so the program's check fails.
This page is docs/reference/modules/shell.md in the MMWatchdog repository.