Restart a service automatically
When to use this page: a check of your application fails, and a restart of the application's systemd unit usually fixes the problem, such as an application that hangs. Or a command of yours fixes it, such as deleting old cache files when the disk is full.
Who acts: you, or your coding agent. Each step runs on your controller
machine, the machine where you run mmx, except where the step says "on the
node". The test in step 7 makes your application hang for about two
minutes: on a production node, a coding agent asks the person first.
What this page does: you add a restart_systemd_unit block, or a
recovery_command block, to a check group, and install it with MMX. Then
you make the application hang and watch MMWatchdog restart it. MMWatchdog
makes a limited number of attempts, so it does not restart a broken service
forever.
Terms: a recovery action is a restart of a systemd unit or a run of
a recovery command. Each recovery action is an attempt: it does not
prove that the problem is fixed. The problem counts as over only when the
check has been healthy for its recovered_after_healthy_for. A check's
failing time starts at its first failed run; every waiting time on this
page counts from that start.
Choose the action
| The fix | Block in the check group | Follow |
|---|---|---|
| Restart the application's systemd unit | restart_systemd_unit |
Steps to restart a systemd unit |
| Run a command of yours, such as deleting old files | recovery_command |
Steps to run a command instead |
A check group can have both blocks.
Before you start
- The check group exists in the MMX project that installed MMWatchdog, with a check that fails when the application is broken, such as a check of its health page. If not, follow Add a check first.
- For a restart: the application runs as a systemd unit, such as
myapp.service. - Open a terminal in the MMX project: the directory that holds
mmx_project_settings.yml.
Steps to restart a systemd unit
-
Choose the checks that call for the restart. List them in
triggered_by_checks. This page restartsmyapp.servicewhen the checkhealth_page_answersfails:restart_systemd_unit: systemd_unit: myapp.service # the unit that MMWatchdog restarts triggered_by_checks: [health_page_answers]- Each name is a check of the same check group. For a check group with
one check written directly on the group, the name is the module's
name, such as
tcp. - Any check that can fail works, also a check that is not about the
unit: a limit on the node's memory can call for a restart of
nginx.service. - A check with
only_record_readings: truenever fails, so it is refused. - A check of the unit's state, such as
service_is_running, is often not needed here. Reason: MMWatchdog never starts a stopped unit (see If something is different), and systemd restarts a crashed unit itself when the unit hasRestart=on-failure. A health page that stops answering is what systemd cannot see.
Check: each name in
triggered_by_checksis a check of the check group. - Each name is a check of the same check group. For a check group with
one check written directly on the group, the name is the module's
name, such as
-
Choose when the restart comes.
restart_after_failing_foris the time from the first failed run to the restart. It counts from the same start as the check group'salert_after_failing_for, so you choose what comes first:restart_after_failing_for: 1m # restart after 1 minute of failing-
Shorter than
alert_after_failing_for: MMWatchdog restarts first, and alerts only if the failure lasts. A restart that fixes the problem sends no alert; it appears only inmmwatchdog eventsandstatus:2026-11-15T04:38:52Z systemd_unit_restart_result myapp/health_page_answers systemd unit restart finished. The next check runs will show whether the check has recovered 2026-11-15T04:38:52Z systemd_unit_restart_requested myapp/health_page_answers restart systemd unit myapp.service, attempt 1 -
Equal: the restart and the alert come at the same failed run. Choose this when a person should know about every hang.
-
Longer: the alert goes out first, and the restart follows if the failure is still there.
-
0srestarts at the first failed run.
Check: you know which comes first. This page uses
1mfor both. -
-
Choose the limits on attempts. Three settings keep restarts in bounds:
min_time_between_attempts: 1m # wait 1 minute after a restart before the next max_attempts: 3 # restart at most 3 times reset_attempts_after_healthy_for: 10m # count from 0 again after 10 healthy minutesmin_time_between_attemptscounts from the end of one restart, whensystemctl restartreturns, to the start of the next. Reason: the application gets time to start and answer before MMWatchdog judges the restart.max_attemptsis a whole number from 1, orunlimited. When the attempts are used, MMWatchdog stops restarting and keeps running the checks; a person then fixes the cause. Withunlimited, MMWatchdog never stops restarting, andmin_time_between_attemptsstill spaces the restarts.reset_attempts_after_healthy_foris how long every check intriggered_by_checksmust be healthy, after their alerts have closed, before the count of attempts starts from 0 again. Reason: a hang next week gets its full attempts, while a service that keeps failing does not.
Check: all six settings of
restart_systemd_unitare written: each one is required. -
Add the block to the check group. In
files/hub_and_standalone/check_groups/myapp.yml, addrestart_systemd_unitat the end, indented undermyapp:, besidechecksand not inside a check:restart_systemd_unit: # restart myapp when the health page fails systemd_unit: myapp.service triggered_by_checks: [health_page_answers] restart_after_failing_for: 1m min_time_between_attempts: 1m max_attempts: 3 reset_attempts_after_healthy_for: 10mThe check group, whole shows the file.
Check:
restart_systemd_unithas the same indentation aschecks. -
Preview the deploy:
mmx dryrun hub_and_standaloneCheck: the output lists your file as a change to
/etc/mmwatchdog:install / node / install_monitoring/activate/start_maintenance: succeeded, would_change — would run: tracked input /etc/mmwatchdog/check_groups/myapp.yml would changeThe dryrun exits with code 2: something would change. It does not check the check group itself: its
validate_configline checks only the copy of the configuration that the last run left on the node. Step 6 checks your file before the run changes anything. -
Install the change:
mmx run hub_and_standaloneCheck: the run ends with
Run ...: succeeded. If it ends withfailed, see If something is different. -
Make the application hang, and watch the restart. On the node, stop the service's main process without ending it:
ssh deploy@app01 'sudo kill -STOP "$(systemctl show --property MainPID --value myapp.service)"'The health page stops answering. After
restart_after_failing_for, one minute, plus up to onecheck_every, MMWatchdog restarts the unit. Two minutes later, read the events on the node:ssh deploy@app01 sudo mmwatchdog events myapp --last 5m2026-11-15T04:50:27Z recovered myapp/health_page_answers HTTP 200 and configured content match ... 2026-11-15T04:49:27Z systemd_unit_restart_result myapp/health_page_answers systemd unit restart finished. The next check runs will show whether the check has recovered 2026-11-15T04:49:27Z systemd_unit_restart_requested myapp/health_page_answers restart systemd unit myapp.service, attempt 1 2026-11-15T04:49:27Z failed myapp/health_page_answers Get "http://127.0.0.1:8080/health": no answer within 5s (time_limit_per_run) ...The restart events name the check that called for the restart. A finished restart does not close the alert: the check must then be healthy for
recovered_after_healthy_for, here 30 seconds. Because this page restarts and alerts at the same time, the destinations get theSOSmessage and then theResolved SOSmessage:SOS: app01: myapp/health_page_answers failed Get "http://127.0.0.1:8080/health": no answer within 5s (time_limit_per_run) failing since 04:48 UTCResolved SOS: app01: myapp/health_page_answers is ok again HTTP 200 and configured content matchsudo mmwatchdog statuson the node counts the attempts:Systemd unit restarts myapp.service: 1 of 3 attempts used; systemd unit restart finished. The next check runs will show whether the check has recoveredCheck: the events show
systemd_unit_restart_requestedand thenrecovered. If they showsystemd_unit_restart_withheld, or no restart, see If something is different.
The check group, whole
# files/hub_and_standalone/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s # run each check every 30 seconds
time_limit_per_run: 5s # one run of a check may take 5 seconds
alert_after_failing_for: 1m # alert after 1 minute of failing
recovered_after_healthy_for: 30s # the problem is over after 30 healthy seconds
checks:
service_is_running:
systemd_unit_state:
systemd_unit: myapp.service
fail_when: the_unit_is_not_active
alert_severity: sos
health_page_answers:
http:
url: http://127.0.0.1:8080/health
expected_http_status_codes: [200]
alert_severity: sos
restart_systemd_unit: # restart myapp when the health page fails
systemd_unit: myapp.service
triggered_by_checks: [health_page_answers]
restart_after_failing_for: 1m
min_time_between_attempts: 1m
max_attempts: 3
reset_attempts_after_healthy_for: 10m
Decide when a check alerts explains the four timings at the top of the check group.
Steps to run a command instead
-
Write the command. The command runs on the node like a
shellcheck: Bash withpipefail, written inline or as the path of an executable. This example deletes cache files older than one day:find /var/cache/myapp -type f -mtime +1 -print -delete- The command runs as the account
mmwatchdog, unlessrun_as_user_accountnames another account.rootmust be named. - It runs in
/, unlessrun_from_directorynames another directory, with the variables ofenvand no other environment. - MMWatchdog adds
MMWATCHDOG_CHECK_GROUPandMMWATCHDOG_CHECK, the failing check that called for the command, andMMWATCHDOG_INCIDENT_IDwhen that check's alert has gone out. Before the alert,MMWATCHDOG_INCIDENT_IDis empty. - What the command prints is kept in the event history, so
-printrecords which files it deleted. The check's own output does not reach the command.
Check: on the node, the command does its job when you run it yourself as that account.
- The command runs as the account
-
Write the check group with
recovery_command. Create a file infiles/hub_and_standalone/check_groups/, or add the block to an existing check group, besidechecks:# files/hub_and_standalone/check_groups/root_filesystem.yml check_groups: root_filesystem: check_every: 1m time_limit_per_run: 5s alert_after_failing_for: 10m recovered_after_healthy_for: 5m checks: root_filesystem_is_below_90_percent: node_filesystem: / fail_when_measurement_above: space_used_percent: 90 alert_severity: warn recovery_command: # delete old cache files when the disk is too full triggered_by_checks: [root_filesystem_is_below_90_percent] script: find /var/cache/myapp -type f -mtime +1 -print -delete run_as_user_account: root run_after_failing_for: 5m time_limit_per_attempt: 2m max_attempts: 2 min_time_between_attempts: 10m reset_attempts_after_healthy_for: 1hWhen the disk has been more than 90 percent full for 5 minutes, MMWatchdog runs the command. If the command frees enough space, no alert goes out. If the disk is still too full 10 minutes into the failing time, the
warnalert goes out. If it is still too full 10 minutes after the first command ended, MMWatchdog runs the command once more, and then stops.run_after_failing_forworks likerestart_after_failing_forin step 2 of the restart: shorter thanalert_after_failing_forruns the command before the alert.time_limit_per_attemptis how long one run of the command may take before MMWatchdog ends it.max_attempts,min_time_between_attempts, andreset_attempts_after_healthy_forwork as for a restart, in step 3.
Check: all seven settings without a default are written:
triggered_by_checks,script,run_after_failing_for,time_limit_per_attempt,max_attempts,min_time_between_attempts, andreset_attempts_after_healthy_for. -
Preview the deploy:
mmx dryrun hub_and_standaloneCheck: the output lists your file as a change, such as
tracked input /etc/mmwatchdog/check_groups/root_filesystem.yml would change. As in the restart steps, the dryrun exits with code 2, andvalidate_configdoes not check your new file. -
Install the change:
mmx run hub_and_standaloneCheck: the run ends with
Run ...: succeeded. -
Read what the command did, after its first run. On the node:
ssh deploy@app01 sudo mmwatchdog events root_filesystem2026-11-15T04:44:08Z recovery_command_result root_filesystem/root_filesystem_is_below_90_percent recovery command finished with exit code 0 Observed: 2026-11-15T04:44:08.211087112Z Exit code: 0 stdout: "/var/cache/myapp/old-export.bin\n" 2026-11-15T04:44:08Z recovery_command_started root_filesystem/root_filesystem_is_below_90_percent run recovery command, attempt 1sudo mmwatchdog statuscounts the attempts:Recovery command root_filesystem: 1 of 2 attempts used; exit 0A recovery command sends nothing to the alert destinations. As with a restart, an exit code of 0 does not close the alert: the check must be healthy for
recovered_after_healthy_for. Another attempt needs a failed run after the command ended.Check: the events show
recovery_command_startedandrecovery_command_resultwith the exit code and the output. Until the check first fails, the events show neither.
If something is different
-
The run failed at
validate_config. The output names the file, the line, and the fault, such as:mmwatchdog: /usr/local/lib/mmwatchdog/candidate_config/check_groups/myapp.yml:18:5: check_groups.myapp.restart_systemd_unit.restart_after_failing_for: is required: how long a check in triggered_by_checks must be failing before MMWatchdog restarts the unit, such as 1mThe line and column are those of your file. Other faults of this page's settings read:
check_groups.myapp.restart_systemd_unit.max_attempts: expected a whole number of at least 1, or unlimited check_groups.myapp.restart_systemd_unit.triggered_by_checks.0: unknown check: health_page check_groups.myapp.restart_systemd_unit.triggered_by_checks.0: check memory_is_recorded only records its readings and never fails; give it a limit to use it for recovery check_groups.myapp.recovery_command.run_after_failing_for: is required: how long a check in triggered_by_checks must be failing before MMWatchdog runs the command, such as 1mFix the file and run
mmx run hub_and_standaloneagain. Nothing changed on the node. -
The restarts did not help, and every attempt is used.
statussays so, such asSystemd unit restarts myapp.service: 2 of 2 attempts usedwithmax_attempts: 2. MMWatchdog keeps running the checks, and the alert goes out atalert_after_failing_foras usual. Fix the cause, then give MMWatchdog its attempts back. On the node:sudo mmwatchdog reset-recovery-attempts myappThe recovery attempts of check group myapp are reset. MMWatchdog can attempt recovery actions again.The command also clears the wait of
min_time_between_attempts, so the next failed run can restart the unit at once. The events showrecovery_attempts_reset myapp operator reset recovery attempts. The command resets both the restarts and the recovery command of the check group. For a check group with neither, it sayscheck group has no restart_systemd_unit and no recovery_commandand exits with 4. Without the command, the attempts start from 0 again afterreset_attempts_after_healthy_forof health. -
The unit is stopped, and MMWatchdog did not start it. MMWatchdog never starts a stopped unit. Before each restart, it asks systemd for the unit's state:
systemd reports What it means What MMWatchdog does activeThe unit runs, but a check fails, such as the health page. Restarts the unit. failedThe program crashed or ended with an error, and systemd did not restart it. Restarts the unit. inactiveSomeone or something stopped the unit on request, or its program ended normally. Leaves the unit alone and uses no attempt. Reason: whoever stopped the unit decides when it runs again, such as a person repairing a database by hand, a package upgrade, or a failover tool. The events show one line for the failing time:
2026-11-15T04:42:30Z systemd_unit_restart_withheld myapp/health_page_answers no restart: systemd unit myapp.service is stopped, and MMWatchdog never starts a stopped unitThe alert still goes out, from a check of the unit's state:
SOS: app01: myapp/service_is_running failed systemd unit myapp.service is stopped. systemd reports no failure: the unit was stopped on request, or its program ended normally. MMWatchdog did not start the unit.Start the unit with
sudo systemctl start myapp.service. To stop a unit on purpose without alerts, turn on a maintenance window first. A program that must always run needsRestart=alwaysin its unit file. Reason: systemd reports a program that ended with exit status 0 asinactive, the same as a stop on request; withRestart=always, systemd starts it again after every exit. -
systemd refuses to start the unit. MMWatchdog never clears systemd's start limits, so the unit stays
failedand its check keeps alerting.systemctl status myapp.serviceon the node says why. -
No restart happened although a check failed. MMWatchdog restarts only when all of these hold: the latest run of a check in
triggered_by_checksfailed, at leastrestart_after_failing_forinto its failing time; attempts remain, andmin_time_between_attemptshas passed; no maintenance window covers a check group that restarts the unit; and the check is not kept quiet because a check that it names fails (Stop alerts that follow another failure). A check that isunknownnever calls for a restart. Another restart needs a failed run after the previous restart ended. While systemd is starting, stopping, or reloading the unit, MMWatchdog waits. When the failing check is asystemd_unit_statecheck of the same unit, and systemd reports the unitactiveagain, MMWatchdog does not restart it. -
A restart takes long. MMWatchdog waits until
systemctl restartreturns; the unit'sTimeoutStopSecandTimeoutStartSecset how long that may take, and MMWatchdog adds no limit of its own.min_time_between_attemptscounts from the return, so a slow restart still gets the full wait. If MMWatchdog stopped during the restart, such as at a power loss, the wait counts from the start of that restart. -
Two check groups restart the same unit. They share one count of attempts, and MMWatchdog runs their restarts one at a time. Both must state the same
max_attempts,min_time_between_attempts, andreset_attempts_after_healthy_for; otherwise the run fails with a line such as:check_groups.myapp_port.restart_systemd_unit.max_attempts: differs from check_groups.myapp.restart_systemd_unit.max_attempts at ...:17:7: restart actions for myapp.service share an attempt budget. Set max_attempts to the same value in the restart_systemd_unit of both check groupsEach check group's own
restart_after_failing_forapplies to its own checks.reset-recovery-attemptswith either check group resets the unit's attempts. -
A check group has both a recovery command and a restart. When both are due at the same failed run, the command runs first. The restart then becomes due only at a failed run after the command ended. While a restart of the check group's unit runs, the command waits.
-
The recovery command failed or ran too long. An exit code other than 0, a run past
time_limit_per_attempt, or a command that could not start uses up the attempt. The eventrecovery_command_resultgives the exit code and the output, shortened to 16 KiB withtruncated: true. Long output is no failure: MMWatchdog reads up to 64 KiB of the command's standard output, and the rest does not change the attempt's result. MMWatchdog records each attempt before it starts the command or the restart, so a restart of MMWatchdog cannot forget an attempt. If it cannot record the attempt, it does not run the command. -
A script or a coding agent reads the attempts.
sudo mmwatchdog status --jsongivessystemd_unit_restart_attempts, by unit, andrecovery_command_attempts, by check group:"systemd_unit_restart_attempts": { "myapp.service": { "attempts_used": 2, "max_attempts": 2, "last_attempt_finished_at": "2026-11-15T04:40:03.739548956Z", "last_attempt_at": "2026-11-15T04:40:03.680851305Z", "last_result": "systemd unit restart finished. The next check runs will show whether the check has recovered" } }, "recovery_command_attempts": {}max_attemptsis a number, or the textunlimited. -
You change a node that MMX did not install. Edit the file in
/etc/mmwatchdog/check_groups/on the node, then runsudo mmwatchdog validateandsudo mmwatchdog reload.
Related pages
- Decide when a check alerts
- See what is happening
- Pause alerts during planned work
- Stop alerts that follow another failure
This page is docs/how-to/restart-a-service-automatically.md in the MMWatchdog repository.