Decide when a check alerts
When to use this page: you are writing or changing a check, and you want to choose how soon a failure alerts you, how serious the alert is, and when the problem counts as over.
Who acts: you, or your coding agent. Each step runs on your controller
machine, the machine where you run mmx, except where the step says "on the
node".
What this page does: you set the four timings of a check group, the
waiting times of a check that needs other values than its check group, and
the alert_severity of each check. Then you install the check group with
MMX and confirm on the node that MMWatchdog uses your values.
Terms: a run is one execution of a check; its result is ok,
failed, or unknown. A check's failing time starts at its first
failed run and ends when the check has been healthy for
recovered_after_healthy_for. Every waiting time of a check, such as
alert_after_failing_for, counts from the start of the failing time. The
alert severity of a check is info, warn, or sos; it decides which
alert destinations get the check's alerts.
Before you start
- The check group exists in the MMX project that installed MMWatchdog. If not, follow Add a check first.
- Open a terminal in the MMX project: the directory that holds
mmx_project_settings.yml. - Open the check group's file in
files/hub_and_standalone/check_groups/. This page usesmyapp.yml, the check group of Add a check, whose checks watchmyapp.serviceand its health page.
Steps
-
Set how often the checks run. On the check group,
check_everysays how often each check runs, andtime_limit_per_runsays how long one run may take:check_every: 30s # run each check every 30 seconds time_limit_per_run: 5s # one run of a check may take 5 seconds- A duration is a number with a unit:
100ms,30s,5m,26h, or7dfor whole days. check_everyis at least100ms. Every 30 seconds or every minute suits most checks.- Set
time_limit_per_runto a little more than a healthy run takes. Reason: a run that reaches the limit is ended. For a check that asks a server, such ashttp, no answer within the limit is a failure. For a check that runs a program, such asshell, the result isunknown. - A check cannot have its own
check_everyortime_limit_per_run. A check that must run at another pace goes in a check group of its own.
Check: both values are on the check group, and
time_limit_per_runis not longer thancheck_every. - A duration is a number with a unit:
-
Set how long a failure lasts before the alert.
alert_after_failing_foris the time from the first failed run to the alert:alert_after_failing_for: 1m # alert after 1 minute of failing- A short value alerts you sooner, and also for short problems that end on their own, such as a slow minute during a backup.
0salerts at the first failed run.- With a value above
0s, one failed run followed by healthy runs sends no alert. - The alert goes out at a failed run, never at a healthy one: at the
first failed run after
alert_after_failing_forhas passed.
Check: the value is on the check group.
-
Set when a problem is over.
recovered_after_healthy_foris how long a check must be healthy before its failing time ends. When an alert went out, the recovery message goes out then:recovered_after_healthy_for: 30s # the problem is over after 30 healthy secondsReason: a check that fails on and off is treated as failing the whole time. A healthy period shorter than
recovered_after_healthy_fordoes not end the failing time, and each failed run starts the count of healthy time again. So a longer value turns a check that fails on and off into one alert, instead of an alert and a recovery each time.0sends the failing time at the first healthy run.Check: the four timings
check_every,time_limit_per_run,alert_after_failing_for, andrecovered_after_healthy_forare all on the check group. Each one is required. -
Give a check its own waiting times, where it needs them. A check under
checksmay state its ownalert_after_failing_for, its ownrecovered_after_healthy_for, or both. A waiting time that the check does not state is the check group's. This memory check waits 10 minutes, because memory use above the limit for one minute is often harmless:memory_is_below_512_mib: systemd_unit_resource: systemd_unit: myapp.service resource: memory fail_when_measurement_above: memory_used_mib: 512 alert_severity: warn alert_after_failing_for: 10m # this check waits 10 minutes, not 1 recovered_after_healthy_for: 5m # and is over after 5 healthy minutesCheck: each check that needs other waiting times states them; every other check uses the check group's.
-
Set the alert severity of each check. Each check that can alert states its own
alert_severity; there is no default. Each alert destination, infiles/hub_and_standalone/alert_destinations.yml, takes the severities that itsalert_severitiesnames. So the severity decides who gets the alert:Severity Use it for Example Typical destination sosA person must act now, at any hour. The application is down. A destination that wakes a person. Only sosopens an incident in MMHotline.warnA person should look today. The disk is 85 percent full. The team's chat channel. infoA record; nobody needs to act. A notice that one of many web servers rebooted. The team's chat channel, or a quiet channel. A check whose problem grows more urgent with time can start at
warnand rise tosos.raise_to_sos_after_failing_forraises the check's open alert tososwhen the check has been failing that long, counted from the start of the failing time:alert_severity: warn raise_to_sos_after_failing_for: 1hFor a problem that grows by amount, such as a fuller disk, use a second check with a higher limit and a higher severity, as the starter check group does:
warnabove 85 percent andsosabove 95 percent.Check: every check has an
alert_severityofinfo,warn, orsos, written on the check and not on the check group. A check withonly_record_readings: truenever alerts and needs none. -
Preview the deploy:
mmx dryrun hub_and_standaloneCheck: the output lists your file as a change to
/etc/mmwatchdog:install / node / install_monitoring/activate/start_maintenance: succeeded, would_change — would run: tracked input /etc/mmwatchdog/check_groups/myapp.yml would changeThe dryrun of this project exits with code 2: something would change. It does not check your check group:
validate_configchecks only the copy of the configuration that the last run left on the node. Step 7 checks your check group before the run changes anything. -
Install the change:
mmx run hub_and_standaloneCheck: the run ends with
Run ...: succeeded. If it ends withfailed, see If something is different. -
Confirm the timings on the node:
ssh deploy@app01 sudo mmwatchdog status myappEach check shows the timings and the severity that MMWatchdog uses:
myapp/health_page_answers ok HTTP 200 and configured content match Config: /etc/mmwatchdog/check_groups/myapp.yml:13:7 Rule: check every 30s, time limit per run 5s, alert after failing for 1m, recovered after healthy for 30s Alert severity: sos Observed: 2026-11-15T04:47:01.824325463Z myapp/memory_is_below_512_mib ok memory_used_mib is 9.86 (fails above 512) Config: /etc/mmwatchdog/check_groups/myapp.yml:18:7 Rule: check every 30s, time limit per run 5s, alert after failing for 10m, recovered after healthy for 5m Alert severity: warn Raise to SOS after failing for: 1h Observed: 2026-11-15T04:47:01.359331501Z memory_used_mib: 9.86328125 MiB ...Check: the
Ruleline of each check shows your values, including the check's own waiting times.
The check group, whole
# files/hub_and_standalone/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s # run each check every 30 seconds
time_limit_per_run: 5s # one run of a check may take 5 seconds
alert_after_failing_for: 1m # alert after 1 minute of failing
recovered_after_healthy_for: 30s # the problem is over after 30 healthy seconds
checks:
service_is_running:
systemd_unit_state:
systemd_unit: myapp.service
fail_when: the_unit_is_not_active
alert_severity: sos
health_page_answers:
http:
url: http://127.0.0.1:8080/health
expected_http_status_codes: [200]
alert_severity: sos
memory_is_below_512_mib:
systemd_unit_resource:
systemd_unit: myapp.service
resource: memory
fail_when_measurement_above:
memory_used_mib: 512
alert_severity: warn
alert_after_failing_for: 10m # this check waits 10 minutes, not 1
recovered_after_healthy_for: 5m # and is over after 5 healthy minutes
raise_to_sos_after_failing_for: 1h
restart_systemd_unit: # restart myapp when the health page fails
systemd_unit: myapp.service
triggered_by_checks: [health_page_answers]
restart_after_failing_for: 1m
min_time_between_attempts: 1m
max_attempts: 3
reset_attempts_after_healthy_for: 10m
The restart_systemd_unit block is explained in
Restart a service automatically. Its
restart_after_failing_for counts from the same start of the failing time
as alert_after_failing_for, so you choose whether the restart or the
alert comes first.
If something is different
-
The run failed at
validate_config. The output names the file, the line, and the fault, such as:mmwatchdog: /usr/local/lib/mmwatchdog/candidate_config/check_groups/myapp.yml:4:5: check_groups.myapp.time_limit_per_run: 10s exceeds check_every 5sThe path is the copy that the run checks on the node; the line and column are those of your file. Other faults of this page's settings read:
check_groups.myapp.check_every: is required: how often each check of this check group runs, such as 30s check_groups.myapp.check_every: must be at least 100ms check_groups.myapp.alert_severity: set alert_severity on each check under checks, not on the check group check_groups.myapp.checks.health_page_answers.alert_severity: required: info, warn, or sosFix the file and run step 7 again. Nothing changed on the node.
-
A check fails on and off. It stays in one failing time, so it sends one alert after
alert_after_failing_for, and its alert says that it failed on and off:Warning: app01: myapp/health_page_answers failed expected HTTP 200, received 404 failing on and off since 04:32 UTC; 4 of the last 7 runs failedWhile its latest run is healthy,
statusshowsconfirming_failurebefore the alert, orconfirming_recoveryafter it. To get fewer such alerts, raiserecovered_after_healthy_for. MMWatchdog has no rule that counts failed runs, such as "4 failed runs out of 6": the two waiting times do that job. -
An alert rises to
sos. The destinations get a second message with the reason:SOS: app01: myapp/health_page_answers still failing, now SOS myapp/health_page_answers has been failing for 1m, so this alert is now SOS (raise_to_sos_after_failing_for: 1m)A
raise_to_sos_after_failing_forat or below the check'salert_after_failing_formakes the first alertsos. A maintenance window holds the raise until the window ends. The severity of an open alert only rises, never falls, until the problem is over. A change of a check'salert_severitykeeps its open alert; a higher severity raises the open alert soon after the change is installed. -
A check is
unknown: MMWatchdog cannot run it. An unknown run never opens a failure, but it keeps the failing time running and starts the count of healthy time again. A check that stays unknown foralert_after_failing_forsends a cannot-check alert, at the check's severity:Info: app01: MMWatchdog cannot check slow/sleeps_too_long the run took longer than 5s (time_limit_per_run), so MMWatchdog ended it MMWatchdog has not been able to run the check since 04:32 UTCThis one is a
shellcheck whose run took longer thantime_limit_per_run. Withraise_to_sos_after_failing_for, a cannot-check alert rises tososafter that long, counted from the first unknown run. -
A run does not end even after its time limit. Some runs stay blocked in the kernel, such as reading a file on a network filesystem whose server is gone. MMWatchdog waits 5 seconds after
time_limit_per_run, then marks the rununknownwiththe check run did not finish within the time limit, and starts no new run of that check until the old run ends. The other checks go on. Afteralert_after_failing_for, the cannot-check alert goes out. -
The check reports events, such as
out_of_memory_kills, orlog_lineswithfail_when_a_new_line_contains. Such an event check alerts at the first new event. It ignores the check group'salert_after_failing_forand refuses its own:check_groups.myapp.checks.no_oom_kills.alert_after_failing_for: an event check opens its incident at the first new event; remove alert_after_failing_for from this check. Set recovered_after_healthy_for to keep the incident open after the last eventIts
recovered_after_healthy_forcounts from the first run that finds no new event. Alog_linescheck withfail_when_no_new_line_foris not an event check: it uses all four timings. -
You deploy, or MMWatchdog restarts, while a check is failing. A restart of MMWatchdog, and the start or end of a maintenance window, end a failing time that has not alerted yet: the next failed run starts a new one. An alert that went out stays open until the check is healthy for
recovered_after_healthy_for. Eachmmx runthat changes the configuration restarts MMWatchdog inside a maintenance window. -
A failure goes on for hours. MMWatchdog sends no second failure alert. A destination's
remind_everysends a reminder of each open problem; see Delivery, reminders, and evidence. -
A script or a coding agent needs each check's effective waiting times. On the node,
sudo mmwatchdog config --jsongives them, with where each one is stated:"alert_after_failing_for": "10m", "alert_after_failing_for_stated_on": "check", ... "recovered_after_healthy_for": "5m", "recovered_after_healthy_for_stated_on": "check",stated_onischeckorcheck_group. -
You change a node that MMX did not install. Edit the file in
/etc/mmwatchdog/check_groups/on the node, then runsudo mmwatchdog validateandsudo mmwatchdog reload.
Related pages
- Add a check
- Restart a service automatically
- See what is happening: what each state in
statusmeans. - Which destination gets an alert
- Stop alerts that follow another failure
This page is docs/how-to/decide-when-a-check-alerts.md in the MMWatchdog repository.