Report from a scheduled job
When to use this page: a backup, or another job that a systemd timer or cron starts, must tell you when it stops succeeding: when it fails, and also when it stops running at all, such as after someone removed its timer.
Who acts: you, or your coding agent. Each step runs on your controller
machine, the machine where you run mmx, except where the step says "on the
node". A coding agent can do every step: openssl on the controller
machine makes the values that go into the vault, so nobody types a secret.
What this page does: you add a heartbeat check group to the MMX project that installed MMWatchdog. After each success, the job sends MMWatchdog a short message, a heartbeat. When no heartbeat arrives in the time that you choose, MMWatchdog alerts, whatever the reason: the job failed, ran too long, or did not run. The next heartbeat ends the problem.
Terms: the heartbeat socket is the file
/run/mmwatchdog/heartbeat.sock on the node, a Unix socket: a job on the
same node sends its heartbeats to MMWatchdog through it, with no
certificate, port, or address. The token is a random text that the job
sends with each heartbeat, so that MMWatchdog knows that the heartbeat
comes from your job. The deadline is the time by which the next
heartbeat must arrive.
Choose the route
| Your situation | Section |
|---|---|
| The job runs on the node, and it is the first job there that sends heartbeats | Steps |
| The node already has a heartbeat check group, and you add another job | Add another job |
| The job runs as an account other than root | Steps, then If something is different |
| The job runs on another machine | Report from another machine |
Before you start
- MMWatchdog runs on the node, installed with MMX from the MMX project on your controller machine, with an alert destination such as a Slack channel. If not, follow Install MMWatchdog on one node first.
- The job exits with code 0 only when it succeeded. Reason: the heartbeat goes out after an exit code of 0, so a job that ends with 0 after a failure still reports success.
opensslis installed on the controller machine:openssl versionprints its version.- Open a terminal in the MMX project: the directory that holds
mmx_project_settings.yml.
Steps
This page uses a nightly backup of the application myapp, run on the
node app01 by the systemd service backup-myapp.service and its timer,
and a check group named nightly_backup.
-
Make the job's token:
openssl rand -hex 3258ba5ed9832159481c44241f0fe1d3fc923362d7e7ea0f507c2bbbad09785b93Reason: anyone who has the token can report success for this job, so the token is random and long. Each heartbeat check group needs a token of its own.
Check: the output is one line of 64 characters. MMWatchdog takes a token of 16 to 4096 bytes on one line.
-
Put the token in MMX's vault. MMX keeps secrets in the encrypted file
secrets_vault.agein the project, and writes them to the node during each install. Open the vault in your editor:mmx secrets editUnder
mmwatchdog_secret_files, beside the entries that are there, add your token:mmwatchdog_secret_files: slack-webhook-team: "https://hooks.slack.com/services/YOUR/WEBHOOK/ADDRESS" nightly-backup-heartbeat: "58ba5ed9832159481c44241f0fe1d3fc923362d7e7ea0f507c2bbbad09785b93"Save, and close the editor.
Check:
mmx secrets editprintsSaved encrypted vault: secrets_vault.age. -
List the token's file in the inventory. In
inventories/hub_and_standalone.yml, undervars:vars: mmwatchdog_config_source_dir: files/hub_and_standalone mmwatchdog_secret_files: [slack-webhook-team, nightly-backup-heartbeat]Each install writes each listed file to
/etc/mmwatchdog/secrets/on the node, readable only by root.Check: each name in the list is a name under
mmwatchdog_secret_filesin the vault. -
Write the heartbeat check group. Create
files/hub_and_standalone/check_groups/nightly_backup.yml:# files/hub_and_standalone/check_groups/nightly_backup.yml check_groups: nightly_backup: heartbeat: max_time_between_heartbeats: 26h # alert when no heartbeat came for 26 hours expected_bearer_token_file: /etc/mmwatchdog/secrets/nightly-backup-heartbeat alert_severity: sosmax_time_between_heartbeatsis the time from one heartbeat to the deadline of the next. Choose the time between two runs of the job, plus the longest that a run may take, plus some margin. For a nightly job,26hleaves 2 hours for a slow night.expected_bearer_token_filenames the token's file on the node, from step 3.alert_severitystands on the check group, besideheartbeat.- A heartbeat check group holds only the heartbeat. It takes none of the
four timings of other check groups, such as
check_every: the deadline is its only rule. Alerts go out at the deadline and end at the next heartbeat.
With this check group, the node gets the heartbeat socket. The socket has no setting: it exists whenever the configuration has at least one heartbeat check group.
Check: the file has
heartbeatwith its two settings, and analert_severityofinfo,warn, orsos. -
Preview the deploy:
mmx dryrun hub_and_standaloneCheck: the output has a line ending in
would create "/etc/mmwatchdog/secrets/nightly-backup-heartbeat", and:install / node / install_monitoring/activate/start_maintenance: succeeded, would_change — would run: tracked input /etc/mmwatchdog/check_groups/nightly_backup.yml would changeThe dryrun exits with code 2: something would change. It does not check the new configuration: step 6 does.
-
Install the change:
mmx run hub_and_standaloneCheck: the run ends with
Run ...: succeeded. If it ends withfailed, see If something is different. Then, on the node:ssh deploy@app01 sudo mmwatchdog status nightly_backupnightly_backup/heartbeat waiting_for_first_heartbeat no heartbeat has arrived yet Config: /etc/mmwatchdog/check_groups/nightly_backup.yml:4:5 Rule: next heartbeat due by 2026-11-16T10:39:09Z Alert severity: sos Alert deliveries: 0 waiting, 0 droppedThe first deadline counts from the install: if no heartbeat arrives within
max_time_between_heartbeats, here by 10:39 UTC the next day, MMWatchdog alerts.The heartbeat socket is there too:
ssh deploy@app01 ls -l /run/mmwatchdog/total 0 srw-rw-rw- 1 root root 0 Nov 15 08:39 heartbeat.sock srw------- 1 root root 0 Nov 15 08:39 mmwatchdog.sockEvery account on the node may connect to
heartbeat.sock, and every heartbeat still needs the token. -
Make the job send a heartbeat after each success. On the node, add an
ExecStartPostline to the job's service, here/etc/systemd/system/backup-myapp.service:[Unit] Description=Back up myapp [Service] Type=oneshot ExecStart=/usr/local/bin/backup-myapp ExecStartPost=/usr/local/bin/mmwatchdog heartbeat nightly_backup --bearer-token-file /etc/mmwatchdog/secrets/nightly-backup-heartbeat-
With
Type=oneshot, systemd runsExecStartPostonly when everyExecStartcommand exited with code 0. So a failed backup sends no heartbeat. -
Without
--receiver-url, the command sends the heartbeat through the heartbeat socket. Before it sends the token, it checks that root owns the socket and its directory. Reason: then no other account can have put its own socket there to collect the token. -
The service runs as root, so the command can read the token's file in
/etc/mmwatchdog/secrets/. A job of another account needs its own copy: see If something is different. -
The job's timer, here
backup-myapp.timer, needs no change. A job that has no timer yet gets one, such as/etc/systemd/system/backup-myapp.timerfor 02:00 each night; turn it on withsudo systemctl daemon-reloadandsudo systemctl enable --now backup-myapp.timer:[Unit] Description=Run backup-myapp each night [Timer] OnCalendar=*-*-* 02:00:00 Persistent=true [Install] WantedBy=timers.target
For a cron job, join the two commands with
&&, so that the heartbeat follows only a success, such as in/etc/cron.d/backup-myapp:0 2 * * * root /usr/local/bin/backup-myapp && /usr/local/bin/mmwatchdog heartbeat nightly_backup --bearer-token-file /etc/mmwatchdog/secrets/nightly-backup-heartbeatThen load the changed service:
sudo systemctl daemon-reloadCheck:
sudo systemd-analyze verify /etc/systemd/system/backup-myapp.serviceprints nothing. -
-
Run the job once, and see the heartbeat arrive. On the node:
sudo systemctl start backup-myapp.service sudo journalctl -u backup-myapp.service -n 5Nov 15 08:39:19 app01 systemd[1]: Starting backup-myapp.service - Back up myapp... Nov 15 08:39:19 app01 backup-myapp[16508]: backup done Nov 15 08:39:19 app01 mmwatchdog[16509]: Heartbeat accepted: nightly_backup Nov 15 08:39:19 app01 systemd[1]: backup-myapp.service: Deactivated successfully. Nov 15 08:39:19 app01 systemd[1]: Finished backup-myapp.service - Back up myapp.Then:
sudo mmwatchdog status nightly_backupnightly_backup/heartbeat ok heartbeat received Config: /etc/mmwatchdog/check_groups/nightly_backup.yml:4:5 Rule: next heartbeat due by 2026-11-16T10:39:19Z Last heartbeat: 2026-11-15T08:39:19Z Alert severity: sos ...Check: the journal shows
Heartbeat accepted: nightly_backup, andstatusshowsok, with the next deadline 26 hours after the last heartbeat. If the journal shows an error instead, see If something is different.
What happens when the job stops succeeding
At the deadline, MMWatchdog alerts at once, at the check group's
alert_severity. The alert names the last heartbeat and the deadline. To
see the alert without waiting a day, set max_time_between_heartbeats: 3m,
install the change, run the job once, and wait three minutes without
running it again. This alert comes from such a test:
SOS: app01: nightly_backup/heartbeat failed
heartbeat overdue: last heartbeat 2026-11-15T08:43:13Z; expected within 3m; deadline 2026-11-15T08:46:13Z
failing since 08:46 UTC
If no heartbeat came since the install, the alert says so:
heartbeat overdue: no heartbeat since monitoring started 2026-11-15T08:39:48Z; expected within 3m; deadline 2026-11-15T08:42:48Z.
The next heartbeat ends the problem at once, and each destination that got the alert gets:
Resolved SOS: app01: nightly_backup/heartbeat is ok again
heartbeat received
While the problem lasts, each destination's remind_every sends
reminders, as for any other check. On the node,
sudo mmwatchdog events nightly_backup lists each missed deadline and each
recovery, newest first:
2026-11-15T08:46:13Z failed nightly_backup/heartbeat heartbeat overdue: last heartbeat 2026-11-15T08:43:13Z; expected within 3m; deadline 2026-11-15T08:46:13Z
Incident: ec73a244a7399a5dbca926455de7c7a2
...
2026-11-15T08:43:13Z recovered nightly_backup/heartbeat heartbeat received
Incident: 9ee98951e1a91669a84aba14af3a736a
...
Put the real max_time_between_heartbeats back after the test. A changed
time makes a new check, so an alert of the test that is still open closes
with this message instead of Resolved:
Closed SOS: app01: nightly_backup/heartbeat incident closed, maybe not fixed
the check was changed in the configuration; MMWatchdog does not know whether the problem is fixed
Add another job
The node already has the heartbeat socket. Each job gets its own check group and its own token:
- Make the token:
openssl rand -hex 32. Check: one line of 64 characters. - Put the token in the vault with
mmx secrets edit, undermmwatchdog_secret_files, such asweekly-report-heartbeat: "...". Check:Saved encrypted vault: secrets_vault.age. - Add the file's name to
mmwatchdog_secret_filesininventories/hub_and_standalone.yml. Check: each name in the list is a name in the vault. - Write the job's check group, such as
files/hub_and_standalone/check_groups/weekly_report.yml, as in step 4, with its ownmax_time_between_heartbeatsandexpected_bearer_token_file: /etc/mmwatchdog/secrets/weekly-report-heartbeat. Check: no other check group has the name. - Preview and install:
mmx dryrun hub_and_standalone, thenmmx run hub_and_standalone. Check: the run ends withRun ...: succeeded, andsudo mmwatchdog status weekly_reporton the node showswaiting_for_first_heartbeat. - Make the job send its heartbeat, as in step 7, with the
new check group's name and token file.
Check: after you run the job once,
statusshowsok.
Report from another machine
A job on another machine cannot reach the heartbeat socket. It sends its heartbeats over HTTPS to the heartbeat receiver: the part of MMWatchdog on the node that takes heartbeats from the network. HTTPS needs a certificate and its key. Reason for HTTPS: nobody on the way can read the token.
First do steps 1 to 4 of Steps on your controller machine: the
token, the vault, the inventory, and the check group of the node that
receives the heartbeats. This section uses that node's address
203.0.113.11 and port 8444. Port 8444 is an example: any free port works.
-
Make the receiver's certificate and key:
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 3650 \ -subj "/CN=app01" -addext "subjectAltName=IP:203.0.113.11" \ -keyout heartbeat-receiver.key -out heartbeat-receiver.crtThe command writes two files: the key
heartbeat-receiver.key, which is secret, and the certificateheartbeat-receiver.crt, which is not. The certificate names the address where the other machine reaches the node, and is valid for 10 years (-days 3650). For a host name, writeDNS:app01.example.cominstead ofIP:203.0.113.11; for several names, join them with commas.Check:
openssl x509 -in heartbeat-receiver.crt -noout -ext subjectAltNameX509v3 Subject Alternative Name: IP Address:203.0.113.11 -
Put the certificate and the key in the vault. Run
mmx secrets edit, and undermmwatchdog_secret_filesadd the contents of the two files. Indent each line of a file by four spaces under its name:mmwatchdog_secret_files: slack-webhook-team: "https://hooks.slack.com/services/YOUR/WEBHOOK/ADDRESS" nightly-backup-heartbeat: "58ba5ed9832159481c44241f0fe1d3fc923362d7e7ea0f507c2bbbad09785b93" heartbeat-receiver.crt: | -----BEGIN CERTIFICATE----- MIIBhTCCASygAwIBAgIUeyhk7iNJEoOM17v9d9PO8PYxBR8wCgYIKoZI... ... -----END CERTIFICATE----- heartbeat-receiver.key: | -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----The certificate is not secret, but it goes in the vault too. Reason: each install writes the vault's files to the node before it checks the configuration, and the check opens the certificate.
Check:
mmx secrets editprintsSaved encrypted vault: secrets_vault.age. Then delete the key from the project directory, because the vault holds it now:rm heartbeat-receiver.keyKeep
heartbeat-receiver.crt: the other machine needs a copy in step 7. -
List the two files in the inventory. In
inventories/hub_and_standalone.yml:mmwatchdog_secret_files: [slack-webhook-team, nightly-backup-heartbeat, heartbeat-receiver.crt, heartbeat-receiver.key]Check: each name in the list is a name in the vault.
-
Turn on the receiver. Add this block at the end of
files/hub_and_standalone/mmwatchdog_node_settings.yml:# Where jobs on other machines send their heartbeats, over HTTPS. heartbeat_receiver: listen_on_address: 203.0.113.11:8444 tls_certificate_file: /etc/mmwatchdog/secrets/heartbeat-receiver.crt tls_private_key_file: /etc/mmwatchdog/secrets/heartbeat-receiver.keylisten_on_addressis the node's address that the other machine reaches, or0.0.0.0:8444for every IPv4 interface. An address without a host, such as:8444, is refused.Check:
mmx validateprintsValid: mmwatchdog. -
Open the port in the node's firewall for the other machine only. Reason: the token protects the receiver, but a closed port is one thing fewer that strangers can try. The command depends on the node's firewall.
Check: from a machine that is not the other machine, a connection to port 8444 fails.
-
Install the change:
mmx dryrun hub_and_standalone, thenmmx run hub_and_standalone. The install restarts MMWatchdog, because a new receiver needs a restart.Check: the run ends with
Run ...: succeeded, and on the nodesudo ss -ltnp | grep 8444shows the receiver:LISTEN 0 4096 203.0.113.11:8444 0.0.0.0:* users:(("mmwatchdog",pid=17872,fd=14)) -
Put the token, the certificate, and the program on the other machine. Put the token in a file of the job's account, with mode
0600, and the certificate beside it. Copy themmwatchdogprogram, the filemmwatchdogin the MMX project, to/usr/local/bin/there, or usecurlas below.Check:
ls -lin that directory shows the token's file with-rw-------. -
Make the job send its heartbeat over HTTPS, as in step 7, with
--receiver-urland--trusted-certificate-authority-file. In the directory that holds the two files:mmwatchdog heartbeat nightly_backup --receiver-url https://203.0.113.11:8444 --bearer-token-file nightly-backup-heartbeat --trusted-certificate-authority-file heartbeat-receiver.crtHeartbeat accepted: nightly_backup--trusted-certificate-authority-filemakes the command trust your certificate from step 1. In a service or a cron line, write the files' full paths.Check: the command prints
Heartbeat accepted: nightly_backup, andsudo mmwatchdog status nightly_backupon the node showsok.
The other machine has no mmwatchdog program. Any HTTPS client works:
the job sends POST /v1/heartbeats/CHECK_GROUP with the token as a bearer
token, and the receiver answers 204 once it has saved the heartbeat. With
curl, in the directory that holds the two files:
curl --fail --silent --show-error --cacert heartbeat-receiver.crt -X POST \
-H "Authorization: Bearer $(cat nightly-backup-heartbeat)" \
https://203.0.113.11:8444/v1/heartbeats/nightly_backup
curl prints nothing and exits with code 0 on success. The token appears
in the machine's process list while curl runs; on a machine where other
people have accounts, use the mmwatchdog program instead.
If something is different
-
The run failed at
validate_config. The output names the file, the line, and the fault, such as:mmwatchdog: /usr/local/lib/mmwatchdog/candidate_config/check_groups/nightly_backup.yml:8:5: check_groups.nightly_backup.check_every: does not apply to a heartbeat checkThe faults of this page's settings, and their fixes:
Error Fix check_groups.nightly_backup.check_every: does not apply to a heartbeat checkRemove the timing from the heartbeat check group check_groups.weekly_report.heartbeat.expected_bearer_token_file: must use a different token for each heartbeat check groupMake a new token for the second check group check_groups.nightly_backup.heartbeat.expected_bearer_token_file: ...: expected a single token of 16..4096 bytesPut a token of 16 to 4096 bytes, on one line, in the vault heartbeat_receiver: open /etc/mmwatchdog/secrets/heartbeat-receiver.crt: no such file or directoryOnly with a receiver: put the certificate and the key in the vault and the inventory, as in steps 2 and 3 of Report from another machine Fix the files and run
mmx run hub_and_standaloneagain. Nothing changed on the node. -
The job's heartbeat fails. When the command cannot connect, it tries three times in about 3 seconds. Then, and after any other error, it exits with code 4. A refused heartbeat (a wrong token, or no heartbeat check group of that name) exits with code 1 at once, so the job's log tells a setup fault from an outage. Nobody keeps the heartbeat to send later: the next success sends the next one. With
ExecStartPost, systemd then marks the job's servicefailed, although the backup itself succeeded. The command's errors:Error Fix there is no heartbeat socket /run/mmwatchdog/heartbeat.sock: MMWatchdog is not running on this node, or its configuration has no heartbeat check groupStart MMWatchdog, or install a heartbeat check group, as in steps 4 to 6. A job on another machine needs --receiver-urlthe heartbeat was not sent, because the heartbeat socket /run/mmwatchdog/heartbeat.sock is not safe for the token: its directory /run/mmwatchdog can be written by accounts other than root. ...Someone changed the socket or its directory. The message names what is wrong; for this example, run sudo chmod 755 /run/mmwatchdog. The command sends nothing until root owns the socket and its directory, neither is a symbolic link, and only root can write to the directoryheartbeat --trusted-certificate-authority-file applies only with --receiver-url; a heartbeat without --receiver-url goes through the heartbeat socket, which needs no certificateFor a job on the node, remove --trusted-certificate-authority-file. For a job on another machine, add--receiver-urlMMWatchdog refused the heartbeat (HTTP status code 401): the token does not match, or this node has no heartbeat check group named NAMEThe token, or the check group's name, does not match the node's configuration. MMWatchdog gives the same answer for both, so a stranger who guesses learns nothing. Over HTTPS the message begins the receiver refused the heartbeatthe account that runs this job cannot read /etc/mmwatchdog/secrets/nightly-backup-heartbeat: give the job its own copy of the token that its account can read, ...Give the account its own copy, as in the next item expected a single token of 16..4096 bytesThe token's file is empty or wrong: copy the token again tls: failed to verify certificate: x509: certificate signed by unknown authorityOver HTTPS: add --trusted-certificate-authority-filewith the certificate from step 1 of Report from another machinex509: certificate is valid for 203.0.113.11, not 127.0.0.1Over HTTPS: use the address that the certificate names, here https://203.0.113.11:8444dial tcp 203.0.113.11:8445: connect: connection refusedOver HTTPS: MMWatchdog does not run, or the receiver listens on another address or port: compare listen_on_addressheartbeat --receiver-url must start with https://Write https:// -
The job runs as an account other than root. That account cannot read
/etc/mmwatchdog/secrets/, but it can use the heartbeat socket. Give the account its own copy of the token, owned by the account with mode0600. On the node, for the accountdeploy:sudo install -o deploy -g deploy -m 0600 /etc/mmwatchdog/secrets/nightly-backup-heartbeat /home/deploy/nightly-backup-heartbeatThe job then uses
--bearer-token-file /home/deploy/nightly-backup-heartbeat. A token's file that other accounts can read still works, but the command warns:warning: /home/deploy/nightly-backup-heartbeat has permissions mode 0644: every account and the file's group can read the token; run chmod 600 /home/deploy/nightly-backup-heartbeat so that only the account of this job can read itRun the
chmod. When you change the token in the vault, update the copy too. -
The job runs more often than once a day. Choose
max_time_between_heartbeatsfrom its timer: a job every 15 minutes that takes up to 5 minutes needs about25m. -
You change
max_time_between_heartbeats. After the install, the check group waits for a first heartbeat again, as in step 6, and the deadline counts from the install. A deploy that does not change the check group, and a restart of MMWatchdog, keep the last heartbeat and the deadline. -
You want to know whether the job ran.
sudo mmwatchdog status nightly_backupshows the last heartbeat and the deadline.sudo mmwatchdog check-now nightly_backuprefuses withheartbeat checks receive heartbeats instead of running on demand. Use mmwatchdog status to inspect the latest receipt and deadline. -
A maintenance window covers the check group. A deadline missed during the window alerts when the window ends, without waiting. Pause alerts during planned work explains the windows.
-
You change the token. Change it in the vault with
mmx secrets editand runmmx run hub_and_standalone. A job that reads the file in/etc/mmwatchdog/secrets/sends the new token at its next run; update every copy on other accounts and machines. -
You change a node that MMX did not install. Put the files in
/etc/mmwatchdog/secrets/, owned by root with mode0600, change the configuration in/etc/mmwatchdog/, then runsudo mmwatchdog validateandsudo mmwatchdog reload. A reload that adds the first heartbeat check group opens the heartbeat socket. A change ofheartbeat_receiverneedssudo systemctl restart mmwatchdoginstead:sudo mmwatchdog reloadrefuses it.
Related pages
- Add a check
- Send alerts where your team sees them
- See what is happening
- Install MMWatchdog on one node: the vault and its secret files.
This page is docs/how-to/report-from-a-scheduled-job.md in the MMWatchdog repository.