See what is happening
When to use this page: an alert arrived, a check looks wrong, or you want to know what MMWatchdog sees on a node right now and what it did earlier.
Who acts: you, or your coding agent, on the node, logged in with an
account that can use sudo. The commands only read, except
check-now --count-as-scheduled-run, which step 4 explains.
What this page does: you read the state of each check, the history of what happened, and a fresh run of the checks.
Choose the command
| You want to know | Command |
|---|---|
| The state of every check now, and why | sudo mmwatchdog status |
| The state of the checks of one check group | sudo mmwatchdog status myapp |
| What happened and when: failures, restarts, recoveries | sudo mmwatchdog events myapp |
| What a check finds right now, without waiting for its next run | sudo mmwatchdog check-now myapp |
| The same, as data for a script or a coding agent | add --json to any of the commands |
Steps
-
Read the state of every check:
sudo mmwatchdog statusEach line is one check: its name as
CHECK_GROUP/CHECK, its state, and the result of its latest run:machine/cpu_use_is_recorded ok cpu_used_percent is 0.17 over 30.323s machine/memory_is_below_90_percent ok memory_used_percent is 17.73 (fails above 90) machine/root_filesystem_is_below_85_percent ok space_used_percent is 9.13 (fails above 85) machine/root_filesystem_is_below_95_percent ok space_used_percent is 9.13 (fails above 95) myapp/health_page_answers failed expected HTTP 200, received 404 myapp/service_is_running ok systemd unit myapp.service is active Alert deliveries: 0 waiting, 0 dropped This node sends alerts to nobody.The last lines say whether alerts reach their destinations. Here the node has no destination yet, so it sends alerts to nobody.
Check: the command exits with 0 when every check is healthy, 1 when a check has failed, and 3 when no check has failed but MMWatchdog cannot tell, such as a check that is
unknownor a destination that does not accept alerts. A failing destination shows in the output in both cases. A check inconfirming_failurecounts as healthy here, because it has not failed foralert_after_failing_foryet. Read the states in step 2. -
Read the checks of one check group, with the details:
sudo mmwatchdog status myappmyapp/health_page_answers failed expected HTTP 200, received 404 Config: /etc/mmwatchdog/check_groups/myapp.yml:13:7 Rule: check every 30s, time limit per run 5s, alert after failing for 1m, recovered after healthy for 30s Alert severity: sos Alert severity of the open failure incident: sos Observed: 2026-11-15T03:52:04.073604074Z Failure evidence: Confirmed incident: 5ebcbd3addb90ab46b602cbceac3c139 Config at failure: /etc/mmwatchdog/check_groups/myapp.yml:13:7 Waiting times: alert after failing for 1m, recovered after healthy for 30s Result: failed — "expected HTTP 200, received 404" Observed: 2026-11-15T03:52:04.073604074Z myapp/service_is_running ok systemd unit myapp.service is active ...Configis the file and the line of the check.Failure evidenceis the run that opened the failure: it stays the same while the check keeps failing, so you can see what the check found first.What to do in each state:
State What it means What to do okThe check is healthy. Nothing. confirming_failureThe check fails, but not yet for alert_after_failing_for. No alert has gone out. A check that fails on and off also showsconfirming_failurewhile its latest run is healthy.Wait, or fix the cause now. failedThe check has failed for alert_after_failing_for. The failure is open. Its alert went to each destination that takes its severity, unless a maintenance window or a failing cause holds it back.Fix the cause. Step 3 shows the history. confirming_recoveryThe failure is still open, but the latest run was healthy. The failure closes after recovered_after_healthy_forof healthy runs.Wait. unknownMMWatchdog could not tell whether the check is healthy, such as a shellcheck whose script ran longer thantime_limit_per_run. (Anhttpcheck that gets no answer in time has failed: the line saysno answer within 5s (time_limit_per_run).) The line says why.Fix what the line names. waiting for a new runafter a restart or a configuration change only means the check has not run since: waitcheck_every.How MMWatchdog works explains each state, and the two states of heartbeat and fleet checks, which wait for their first message.
Check: you know the state of each check and the reason in its line.
-
Read what happened and when:
sudo mmwatchdog events myappThe newest event comes first. A failure and its recovery each keep the evidence of the failure:
2026-11-15T03:53:08Z recovered myapp/health_page_answers HTTP 200 and configured content match Incident: 5ebcbd3addb90ab46b602cbceac3c139 ... 2026-11-15T03:52:04Z failed myapp/health_page_answers expected HTTP 200, received 404 Incident: 5ebcbd3addb90ab46b602cbceac3c139 ...Without a check group,
eventslists the events of the whole node, such as the maintenance window of each deploy.--last 2hshows only the events of the past two hours:sudo mmwatchdog events --last 2h2026-11-15T04:00:43Z maintenance_off every_check_group maintenance turned off by root: mmwatchdog deploy 2026-11-15T04:00:42Z maintenance_on every_check_group maintenance until 2026-11-15 04:15 UTC, turned on by root: mmwatchdog deploy 2026-11-15T03:59:10Z recovered myapp/health_page_answers HTTP 200 and configured content match 2026-11-15T03:58:10Z failed myapp/health_page_answers expected HTTP 200, received 404--since TIMEand--until TIMEselect a span of time instead.A problem that began while alerts were paused by a maintenance window shows two lines for one incident. The first ends with
(no alert sent: alerts were paused by a maintenance window). The second comes at the first run after the window and sends the alert.Check: the events show when the failure began and, if MMWatchdog acted, what it did, such as
systemd_unit_restart_requested. -
Run the checks now:
sudo mmwatchdog check-now myappmyapp/health_page_answers failed expected HTTP 200, received 404 Observed: 2026-11-15T03:51:31.502932446Z myapp/service_is_running ok systemd unit myapp.service is active Observed: 2026-11-15T03:51:31.509719375Zcheck-nowruns each check of the group once and shows what it found, with the output of a script or a module where it has one. It sends no alert, opens or closes no failure, restarts nothing, and runs no recovery command. So it is safe to run at any time.After you fixed the cause, count the runs as scheduled runs, so an open failure can close without waiting for the next scheduled run:
sudo mmwatchdog check-now myapp --count-as-scheduled-runmyapp/health_page_answers ok HTTP 200 and configured content match Observed: 2026-11-15T03:52:38.400184316Z myapp/service_is_running ok systemd unit myapp.service is active Observed: 2026-11-15T03:52:38.405070239Z The runs counted as scheduled runs for check group myapp; run mmwatchdog status myapp to see the resulting check status.These runs count like scheduled runs: a failure can close, and a failing check can restart its service. The failure still closes only after
recovered_after_healthy_forof healthy runs, sostatusshowsconfirming_recoveryuntil then:myapp/health_page_answers confirming_recovery the latest run is healthy; the alert closes once the check has been healthy for 30s (recovered_after_healthy_for): HTTP 200 and configured content matchCheck:
check-nowexits with 0 when every check of the group is healthy, 1 when one failed, and 3 when one is unknown.
If something is different
- A coding agent or a script reads the results. Add
--jsontostatus,events, orcheck-now.status --jsongives, for each check,state,last_observation,failing_since,failure_incident, andneeds_fresh_observation, among others. The command reference lists every field and exit code. - The node is not reachable, or you want every node at once. On a fleet
hub,
sudo mmwatchdog fleetshows every node and its failing checks; see Watch a fleet. - A check is
unknownand the line does not explain enough.check-nowshows the standard output and standard error of ashellcheck or of a module, and its exit code. When something fails lists the common causes. - You want to try a configuration before the node uses it. On the
node,
sudo mmwatchdog check-now myapp --without-daemon --config-dir DIRruns the checks of a configuration inDIRwithout the running MMWatchdog. Each run starts with nothing kept from earlier runs. statussaysThis node sends alerts to nobody. The node has no alert destination. Send alerts somewhere adds one.- A check stays quiet although it fails. Its line says that another check, which it names, fails, and MMWatchdog keeps the second check quiet while the first fails. Stop alerts that follow another failure explains the setting.
Related pages
This page is docs/how-to/see-what-is-happening.md in the MMWatchdog repository.