Pause alerts during planned work
When to use this page: you plan work that makes checks fail on a node, such as moving an application's database, restarting several services, or rebooting the node, and you want no alerts and no automatic restarts while you work.
Who acts: you, or your coding agent, on the node, logged in with an
account that can use sudo. A deploy with mmx run needs no action from
you: it pauses alerts by itself, as
Deploys through MMX says.
What this page does: you open a maintenance window for one or more check groups, for a time that you state. During the window, MMWatchdog keeps running the checks of those check groups and shows their results, but sends no alert for them, restarts no systemd unit, and runs no recovery command. The window ends when you turn it off, or by itself when its time is over, so a window that you forget cannot keep the node quiet for good.
Terms: a check group is a named set of checks, such as myapp, the
checks of one application. sudo mmwatchdog status names each check as
CHECK_GROUP/CHECK. The whole-node window covers every check group of
the node.
Choose the window
| Your planned work | Window | Section |
|---|---|---|
| Work on one application, such as a database move | The application's check group | Pause one check group |
| Work on several applications at once, such as a deploy of several workers | Several check groups, with one command | Pause several check groups |
| Work on the whole node, such as a kernel update and a reboot | Every check group | Pause every check group |
A change to MMWatchdog's configuration with mmx run |
None: the run opens its own | Deploys through MMX |
| Routine work that your team does not need to hear about | Any of the above, without the two messages | If something is different |
| MMWatchdog itself must not run for a while | None: stop the service | Stop MMWatchdog for a while |
Before you start
- MMWatchdog runs on the node.
sudo mmwatchdog statuslists the checks. If not, follow Install MMWatchdog on one node first. - You know about how long the work takes. Every window needs
--for, the time after which it ends by itself. Choose a little more than you expect: you can end the window early, and you can make it longer, as If something is different says. - Your team sees each window. Each alert destination that takes
infoalerts gets one message when a window opens and one when it ends, with the reason and the account that opened it. A destination withoutinfoin itsalert_severitiesgets neither message.
Pause one check group
-
Find the name of the check group. On the node:
sudo mmwatchdog statusmachine/cpu_use_is_recorded ok cpu_used_percent is 4.08 over 0.737s ... myapp/health_page_answers ok HTTP 200 and configured content match myapp/service_is_running ok systemd unit myapp.service is active Alert deliveries: 0 waiting, 0 droppedCheck: the name before the
/, heremyapp, is the check group of the application that you work on. -
Open the window, with the longest time that the work may take and the reason:
sudo mmwatchdog maintenance on myapp --for 1h --reason "moving the myapp database"myapp is in maintenance until 2026-11-15 05:40 UTC (1h). MMWatchdog keeps running the checks of myapp and shows the results in mmwatchdog status. During maintenance, MMWatchdog sends no alert and attempts no systemd unit restart or recovery command for myapp. To end the maintenance early: sudo mmwatchdog maintenance off myappReason for
--reason: your team sees it, and knows that the work is planned. The reason is one line of at most 200 bytes.Check: your Slack channel, or each other destination that takes
info, shows the message:Info: app01: maintenance on for myapp maintenance until 2026-11-15 05:40 UTC, turned on by deploy: moving the myapp databasedeployis the account that ransudo. -
Do the work. The checks keep running. While you work:
sudo mmwatchdog status myappmyapp/health_page_answers failed expected HTTP 200, received 404 ... myapp/service_is_running ok systemd unit myapp.service is active ... Maintenance on: myapp until 2026-11-15 05:40 UTC, turned on by deploy: moving the myapp databaseCheck: the
Maintenance online names the check group. The checks still run and show what they find: a failing check showsconfirming_failure, orfailedonce it has failed foralert_after_failing_for, whichsudo mmwatchdog events myappthen records. MMWatchdog sends no alert and restarts nothing. To prove the window holds, break the check on purpose for longer than itsalert_after_failing_forand itsrestart_after_failing_for: the destinations get nothing but themaintenance onmessage, andeventsshows nosystemd_unit_restart_requested. The reminders of a problem that was open before the window wait too, and come again oneremind_everyafter the window, if the problem lasts. -
End the window when the work is done:
sudo mmwatchdog maintenance off myappmyapp is out of maintenance. Alerts, restarts and recovery commands resume after fresh checks.Check: the destinations get
Info: app01: maintenance off for myapp, andsudo mmwatchdog status myapphas noMaintenance online. Each check waits for its next run before it alerts or restarts anything, so a check may showunknownwithwaiting for a new runfor up to itscheck_every. Then the runs decide: a check that still fails alerts, and a failure that ended during the window sends nothing, neither the failure nor its recovery. MMWatchdog never sends later what the window held back.
If you forget step 4, the window ends by itself after --for, and the
destinations get the message with the word expired:
Info: app01: maintenance off for myapp
maintenance expired: short test
Pause several check groups
-
Open one window for all of them, with their names:
sudo mmwatchdog maintenance on myapp machine --for 10m --reason "deploy v1.4"myapp, machine are in maintenance until 2026-11-15 04:52 UTC (10m). MMWatchdog keeps running the checks of the check groups in maintenance and shows the results in mmwatchdog status. During maintenance, MMWatchdog sends no alert and attempts no systemd unit restart or recovery command for the check groups in maintenance. To end the maintenance early: sudo mmwatchdog maintenance off myapp machineCheck: the destinations get one message for the command, which names every check group:
Info: app01: maintenance on for myapp, machine. -
Do the work.
-
End the window:
sudo mmwatchdog maintenance off myapp machinemyapp, machine are out of maintenance. Alerts, restarts and recovery commands resume after fresh checks.Check: the destinations get one message,
Info: app01: maintenance off for myapp, machine.
Pause every check group
Use this for work on the node itself, such as a kernel update and a reboot.
-
Open the whole-node window:
sudo mmwatchdog maintenance on --every-check-group --for 15m --reason "kernel update and reboot"Every check group on this node is in maintenance until 2026-11-15 04:57 UTC (15m). MMWatchdog keeps running the checks of every check group and shows the results in mmwatchdog status. During maintenance, MMWatchdog sends no alert and attempts no systemd unit restart or recovery command for the check groups in maintenance. To end the maintenance early: sudo mmwatchdog maintenance off --every-check-groupThe window also covers a check group that a deploy adds during the window. A reboot during the window sends no reboot notice, also when the window ends while the node is down. Without the window, each reboot sends the notice
app01 rebooted; a window of a check group by name does not hold that notice back.Check: the destinations get
Info: app01: maintenance on for the whole node. -
Do the work, such as
sudo apt upgradeandsudo reboot. The window survives the reboot and a restart of MMWatchdog. -
End the window:
sudo mmwatchdog maintenance off --every-check-groupThe maintenance for every check group on this node is off. Alerts, restarts and recovery commands resume after fresh checks. A check group that has its own maintenance window stays in maintenance until that window ends.Check: the destinations get
Info: app01: maintenance off for the whole node. As the output says, a check group that you paused with its own name stays paused until its own window ends: end it withsudo mmwatchdog maintenance off myapp.
Deploys through MMX
mmx run hub_and_standalone pauses alerts by itself. A run that changes
the node's MMWatchdog configuration, modules, secret files, or program
opens a whole-node window for 15 minutes, restarts MMWatchdog, and ends the
window. Reason: the restart makes every check wait for a new run, and the
window keeps the restart from sending alerts. The destinations get the two
info messages, and sudo mmwatchdog events shows the window:
2026-11-15T04:40:00Z maintenance_off every_check_group maintenance turned off by root: mmwatchdog deploy
2026-11-15T04:39:59Z maintenance_on every_check_group maintenance until 2026-11-15 04:54 UTC, turned on by root: mmwatchdog deploy
- A run that changes nothing opens no window and sends nothing.
- A run that fails after the window opened leaves the window to end by itself, within 15 minutes.
- A run during a whole-node window that you opened yourself keeps your window, with its end time and reason, and opens none of its own, so the run's restart is already inside your window. A window of a check group by name stays too.
- A project that deploys often, such as from a scheduled job, can keep
the two messages out of the destinations: in each playbook, add
--without-maintenance-on-and-off-alertsto themmwatchdog maintenance online of the node taskstart_maintenance.sudo mmwatchdog eventsstill shows each window. An update brings new playbooks, so make the change again after each update.
Stop MMWatchdog for a while
Prefer a maintenance window: the checks keep running, so you see the results, and the window ends by itself. Stop MMWatchdog only when it must not run at all, such as while you test the node's disk in a way that a check would disturb.
-
Stop the service:
sudo systemctl stop mmwatchdogMMWatchdog sends nothing when it stops, and runs no check, sends no alert, and restarts nothing until it starts again. Nothing reminds you that it is stopped.
An
mmx runof the playbook starts MMWatchdog again, and enables it at boot, also aftersystemctl disable. Do not run the playbook for this node while MMWatchdog must stay stopped.Check:
systemctl is-active mmwatchdogprintsinactive. -
Start it again when the work is done:
sudo systemctl start mmwatchdogCheck:
sudo mmwatchdog statuslists the checks. A check that has not run since the start showsunknownwithwaiting for a new run. A start within the same boot sends no message. A stopped MMWatchdog also starts again when the node boots, because the service stays enabled.
On a node that sends check-ins to MMHotline, or that has joined a fleet
hub, a stopped MMWatchdog stops reporting, and MMHotline or the hub alerts.
On a joined node, open a whole-node window before you stop the service, as
If something is different says. To keep
MMWatchdog stopped across reboots too, use sudo systemctl disable --now mmwatchdog, and sudo systemctl enable --now mmwatchdog to undo it.
Remove MMWatchdog from a node covers removing
MMWatchdog.
If something is different
-
The work takes longer than
--for. Runmaintenance onagain for the same check group. The new window replaces the old one, and the destinations get a new message with the new end:sudo mmwatchdog maintenance on myapp --for 2h --reason "moving the myapp database, second try"--forhas no upper limit, anddmeans days, so--for 30dis valid. -
Routine work that the team does not need to hear about, such as a nightly data import. Add
--without-maintenance-on-and-off-alerts:sudo mmwatchdog maintenance on myapp --for 10m --reason "nightly data import" --without-maintenance-on-and-off-alertsmyapp is in maintenance until 2026-11-15 04:52 UTC (10m). ... Maintenance start and end are not announced.The window works as any other, and its start and end are still in
sudo mmwatchdog events; only the two messages are not sent, also when the window expires.sudo mmwatchdog statussaysmaintenance start and end are not announcedon theMaintenance online. -
The command refuses the window. The command opens no window at all and exits with code 4:
Error Fix mmwatchdog: unknown check group name "myap"Use a name that sudo mmwatchdog statuslistsmmwatchdog: maintenance on requires --for with a positive duration, such as 1h or 30dAdd --for, such as--for 1hmmwatchdog: maintenance takes check group names or --every-check-group, not bothWrite the names, or --every-check-groupmmwatchdog: maintenance reason must be single-line text of at most 200 bytesShorten --reasonto one line of at most 200 bytes -
You stop a service by hand. MMWatchdog never starts a systemd unit that is stopped, so the unit stays stopped, and its
systemd_unit_statecheck alerts afteralert_after_failing_for. Open a window for its check group first. -
A check group has a heartbeat check. When the window ends, MMWatchdog sends the alert of a heartbeat that was missed during the window, without waiting for a new run.
-
The node has joined a fleet hub. Run the command on the node, as above.
maintenance onandoffwait until the hub has the change, at most 25 seconds, so a reboot right aftermaintenance ondoes not look like a node that stopped reporting. If the hub does not accept the change, the window is still on or off on the node, and the command says so and exits with code 1, so a script can stop before it reboots the node. A window covers only the node where you run the command: for work on several nodes, run the command on each. -
The node is the fleet hub. The whole-node window on the hub also covers
node_reports, the check of the joined nodes, so the hub sends no alert that a joined node stopped reporting. To pause only those alerts, for example while you move several joined nodes, runsudo mmwatchdog maintenance on node_reports --for 1hon the hub. Watch a fleet explains the fleet. -
A deploy removes a check group that has a window. The window ends with the check group, with its end message.
Related pages
- Send alerts where your team sees them
- See what is happening
- Add a check
- The command reference, with every command of MMWatchdog.
This page is docs/how-to/pause-alerts-during-planned-work.md in the MMWatchdog repository.