Change a fleet
When to use this page: your nodes report to a fleet hub, and you want to add a node, remove a node for good, put nodes in node groups, give the fleet a new join code, or move the hub to another server.
Who acts: you, or your coding agent. Most steps run on your
controller machine, the machine where you run mmx, in the MMX project
that set up the fleet. Some steps run a command on the hub or on a node
through ssh, as the step shows. A coding agent shows the person the
output of each mmx dryrun and waits for their yes before the mmx run
that follows it, and asks the person to put each secret into the vault
themselves.
What this page does: each section makes one change to the fleet with
MMX or with a mmwatchdog fleet command on the hub, and checks the result
with sudo mmwatchdog fleet on the hub.
Terms: the fleet hub is the node that the other nodes report to.
A joined node is a node that reports to the hub. The join code is
the secret that lets a node join the hub; the vault of the MMX project
holds it as mmwatchdog_join_code. A node group is a name, such as
web_servers, that a node gives itself, so that the hub can show the
nodes of that name together.
Choose the change
| You want to | Section |
|---|---|
| Watch one more server with the fleet | Add a node |
| Stop watching a server, such as one that you shut down for good | Remove a node |
| See the web servers, or the nodes of one data center, together | Put nodes in node groups |
| Make the join code useless to someone who may have seen it | Rotate the join code |
| Make another server the hub, with the same fleet | Move the hub to another server |
Before you start
- The fleet was set up with MMX, as Set up a fleet
says. You work in that MMX project: the hub is in
inventories/hub_and_standalone.yml, the joined nodes are ininventories/hub_members.yml, and the vault holdsmmwatchdog_join_code. - You can log in to the hub, here
hub01at 203.0.113.10, and runsudo mmwatchdog fleetthere. The joined nodes areweb01andweb02, at 203.0.113.11 and 203.0.113.12.
Add a node
-
Name the node in the inventory of the joined nodes. In
inventories/hub_members.yml, undernodes, add the node:nodes: web01: address: 203.0.113.11 ssh_user: deploy web02: address: 203.0.113.12 # the new node ssh_user: deployThe node needs Linux with systemd, and an account that can use
sudowithout a password, as for the other nodes.Check: you can log in, and the project is valid:
ssh deploy@203.0.113.12 true && echo login works mmx validate join_hub --inventory hub_memberslogin works Valid: mmwatchdog -
Preview the change:
mmx dryrun join_hub --inventory hub_membersCheck: the lines of the new node say
would_change, and the lines of the other nodes sayunchanged. The exit code is 3, because the new node has no MMWatchdog yet, so some steps cannot be checked in advance. -
Install MMWatchdog on the node, and join it to the hub:
mmx run join_hub --inventory hub_membersRun b7da8267: succeeded 2 finishedThe run changes nothing on the nodes that already run this configuration.
-
Check that the node joined. On the hub:
sudo mmwatchdog fleet web02web02: reporting, exit 0 Contact: last report 2026-11-15T08:41:00Z (21s ago) Joined: 2026-11-15T08:41:00Z from 203.0.113.12 Node groups: web_servers ...Check: the node is
reporting. A node joins with its first report, a few seconds after MMWatchdog starts on it. If the hub answersunknown node "web02"after a minute, the last line ofsudo mmwatchdog statuson the node gives the reason, such asjoining the hub: hub returned HTTP 403: invalid join code.
Remove a node
Remove a node from the fleet when you shut it down for good, or when it should no longer be watched. Without these steps, the hub alerts that the node stopped reporting. Do the steps in this order: first the node, then the inventory, then the hub.
-
Remove MMWatchdog from the node, if the node still runs. Follow Remove MMWatchdog from a node, steps 1 to 8, then come back here for step 2; step 2 below takes the place of step 9 there. A joined node is in
inventories/hub_members.yml, so step 8 there names that inventory:mmx remove-mmx-from-node web02 --inventory hub_membersTo keep MMWatchdog's files on the node, do only step 2 there,
sudo systemctl disable --now mmwatchdog. If the server is already shut down for good, skip this step.Reason: a node whose MMWatchdog still runs joins the hub again with its next report, and
mmx remove-mmx-from-nodefinds the node only while the node is still in the inventory.Check:
ssh deploy@203.0.113.12 systemctl is-active mmwatchdogprintsinactive. -
Remove the node from the inventory. Delete the node's entry from
inventories/hub_members.yml, so that the nextmmx run join_hub --inventory hub_membersdoes not install it again.Check:
mmx validate join_hub --inventory hub_membersprintsValid: mmwatchdog, andmmx nodes join_hub --inventory hub_membersno longer lists the node. -
Remove the node from the hub. On the hub:
sudo mmwatchdog fleet forget web02Removed node web02 from the fleet; if MMWatchdog still runs on the removed node, the node joins again with the current join code.The hub deletes the node's entry, its last report, its events, its readings in the measurement history on the hub, and its credential. If the hub had alerted that the node stopped reporting, it closes that alert with
web02 was removed from the fleet; MMWatchdog does not know whether the problem is fixed.Check:
sudo mmwatchdog fleetcounts one node less, and does not list the node:Fleet at 203.0.113.10:8445: 2 nodes, 2 healthy Hub-outage alerter nodes: web01If the node was a hub-outage alerter node, the hub chooses another one with the next report of a joined node. The hub cannot remove itself:
sudo mmwatchdog fleet forget hub01saysthe hub cannot forget itself.
Put nodes in node groups
A node names its node groups in its own settings. The hub has no list of
node groups: a node group exists while a node names it. A node can be in
several node groups, such as web_servers and frankfurt.
-
Name the node group in the nodes' settings. For every node that uses
files/join_hub/, put this line infiles/join_hub/mmwatchdog_node_settings.yml:this_node_is_in_node_groups: [web_servers]If the file already has a
this_node_is_in_node_groupsline, change that line instead of adding a second one: replace the old name, or add the new name to the list, such as[web_servers, frankfurt]. The setting is always a list, also for one name. A name has at most 64 letters, digits, hyphens, and underscores. For nodes of another job, write the line in their own configuration directory, as Set up a fleet says. The hub can be in node groups too, with the same line infiles/hub_and_standalone/mmwatchdog_node_settings.yml.Check:
mmx validate join_hub --inventory hub_membersprintsValid: mmwatchdog. -
Preview and install the change:
mmx dryrun join_hub --inventory hub_members mmx run join_hub --inventory hub_membersRun e662682d: succeeded 2 finishedCheck: the dryrun exits with code 2, and lists
install_config: succeeded, would_changefor each node. The run ends withsucceeded. A fault in the line, such asthis_node_is_in_node_groups: expected a list, found the string "web_servers", fails the run atvalidate_config, and changes nothing on the nodes. -
See the node group on the hub. Each node sends its node groups with its next report:
sudo mmwatchdog fleet --node-group web_serversNode group web_servers of the fleet at 203.0.113.10:8445: 2 nodes, 2 healthyCheck: the count is the number of nodes that you put in the node group.
sudo mmwatchdog fleet web01showsNode groups: web_servers. Watch a fleet says how to read the view.
Rotate the join code
Rotate the join code when someone who should not have it may have seen it, such as after it was pasted into a chat. A node that joins with the code gets the fleet settings, with the secrets of your alert destinations.
-
Make a new join code. On the hub:
sudo mmwatchdog fleet rotate-join-codemmwatchdog-join 203.0.113.10:8445 ebba574e17670386674eeb04912fdd7555fb626c4f124addab10c6a5888168ee 2f543d908806e06776c120ae807508dbf104a13a95e9b806a2cc8cd9b418a34fThe command prints the new code. Only its last part, the join secret, is new. From now on the old code joins no node. The joined nodes keep reporting: they joined with the old code, and report with a credential of their own.
Check:
sudo mmwatchdog fleet join-codeprints the same new code. -
Put the new code in the vault. Open the vault with
mmx secrets edit, and replace the value ofmmwatchdog_join_codewith the new code:mmwatchdog_join_code: "mmwatchdog-join 203.0.113.10:8445 ebba574e...168ee 2f543d90...8a34f" # the whole line that rotate-join-code printedCheck:
mmx secrets editends withSaved encrypted vault: secrets_vault.age. -
Give every node the new code:
mmx dryrun join_hub --inventory hub_members mmx run join_hub --inventory hub_membersReason: a node uses the code when it has to join again, such as after its host name changes or after it was removed from the hub. With the old code, such a node cannot join, and the hub shows only
Refused joins: 1 with a wrong join code, latest from 203.0.113.12 at 2026-11-15T08:43:00Z.Check: the dryrun lists
join_code: succeeded, would_changefor each node, and the run ends withsucceeded. The run restarts MMWatchdog on each node, because the join code changed.sudo mmwatchdog fleeton the hub then counts every node as healthy.
Move the hub to another server
Move the hub when its server must go, such as to a larger server. The
fleet keeps its identity: the hub's key, certificate, join secret, nodes,
events, and the readings of the nodes are in /var/lib/mmwatchdog/ on the
hub, and you copy that directory to the new server. The steps move the hub
from hub01 at 203.0.113.10 to hub02 at 203.0.113.13, a server where
MMWatchdog has never run. The nodes cannot reach a hub for a few minutes
during the move; they keep their checks, and send alerts themselves.
-
See which nodes are the hub-outage alerter nodes. On the old hub:
sudo mmwatchdog fleetFleet at 203.0.113.10:8445: 3 nodes, 3 healthy Hub-outage alerter nodes: web01, web02Check: the line
Hub-outage alerter nodes:names one or two nodes. -
Hold the alerts about the hub on each of those nodes, for a little longer than the move takes:
ssh deploy@203.0.113.11 sudo mmwatchdog maintenance on node_reports --for 30m --reason "moving the fleet hub" ssh deploy@203.0.113.12 sudo mmwatchdog maintenance on node_reports --for 30m --reason "moving the fleet hub"node_reports is in maintenance until 2026-11-15 10:14 UTC (30m). ... The MMWatchdog fleet hub received the maintenance change.Reason: each hub-outage alerter node alerts when it has not reached the hub for 3 minutes, and the move may take longer.
Check: each command ends with
The MMWatchdog fleet hub received the maintenance change.Slack gets one start message, for the first node: the hub groups windows with the same reason and account, and names every node of the group in the one message when the windows end. -
Stop MMWatchdog on the old hub for good:
ssh deploy@203.0.113.10 sudo systemctl disable --now mmwatchdogReason: two hubs with the same state must never run at the same time, and the state must not change during the copy.
Check:
ssh deploy@203.0.113.10 systemctl is-active mmwatchdogprintsinactive. -
Copy the hub's state to the new server, straight from server to server:
ssh deploy@203.0.113.10 sudo tar -C /var/lib -czf - mmwatchdog | ssh deploy@203.0.113.13 sudo tar -C /var/lib -xzf -The copy holds the hub's private key and the join secret: do not keep it anywhere else.
Check: the new server has the state:
ssh deploy@203.0.113.13 sudo ls /var/lib/mmwatchdogdaemon.lock fleet.db measurement_history.db measurement_history_of_nodes.db state.db -
Name the new server as the hub. In
inventories/hub_and_standalone.yml, replace the old hub with the new one:nodes: hub02: address: 203.0.113.13 ssh_user: deployIn
files/hub_and_standalone/mmwatchdog_node_settings.yml, set the new address inaddress_for_nodes, and keep the other settings:fleet_hub_role: listen_on_address: 0.0.0.0:8445 address_for_nodes: 203.0.113.13:8445 # the new hub's address report_every: 30s alert_after_hub_unreachable_for: 3m send_alert_without_hub_after_waiting_for: 30s undelivered_alerts_of_nodes_content_size_limit: 32MiB keep_measurement_history_of_nodes_for: 7d measurement_history_of_nodes_size_limit: 20GiBCheck:
mmx validateprintsValid: mmwatchdog. -
Preview the install of the new hub:
mmx dryrun hub_and_standaloneCheck: the dryrun lists the changes on
hub02, and exits with code 3, as for a first install. -
Install the new hub:
mmx run hub_and_standaloneRun 2496d798: succeeded 1 finished Saved: /home/dev/monitoring/mmwatchdog-0.1.0-linux-amd64/.mmx/runs/20261115T094439.106346175-2496d798b467fb47The new hub sends no
rebootednotice: MMWatchdog compares the machine ID in/etc/machine-idwith the one saved in the copied state, sees that the state comes from another machine, and treats the start as a first start on this machine, not as a reboot. If both servers were made from one disk image and have the same/etc/machine-id, the new hub sendsWarning: hub02 rebootedinstead; no action is needed.Check: the last line of
ssh deploy@203.0.113.13 sudo mmwatchdog statusisFleet hub: nodes report to 203.0.113.13:8445; see mmwatchdog fleet, and norebootedmessage arrives. -
Put the new join code in the vault. On the new hub:
ssh deploy@203.0.113.13 sudo mmwatchdog fleet join-codemmwatchdog-join 203.0.113.13:8445 ebba574e17670386674eeb04912fdd7555fb626c4f124addab10c6a5888168ee 6e86939f0a274f03001be8c90ba26997315840fe4b969cb161a531702db4a3b7Only the address is new: the certificate's fingerprint and the join secret came with the state. Replace the value of
mmwatchdog_join_codewith this code, withmmx secrets edit.Check:
mmx secrets editends withSaved encrypted vault: secrets_vault.age. -
Give every node the new address:
mmx run join_hub --inventory hub_membersRun 0fb8b5c5: succeeded 2 finished Saved: /home/dev/monitoring/mmwatchdog-0.1.0-linux-amd64/.mmx/runs/20261115T094454.067688493-0fb8b5c5b074a443Each node reports to the new hub with its credential; no node joins again.
Check: on the new hub,
sudo mmwatchdog fleetcounts every node, and shows the windows of step 2:Fleet at 203.0.113.13:8445: 3 nodes, 3 healthy Hub-outage alerter nodes: web01, web02 Maintenance: web01 node_reports until 2026-11-15 10:14 UTC, turned on by deploy: moving the fleet hub Maintenance: web02 node_reports until 2026-11-15 10:14 UTC, turned on by deploy: moving the fleet hubThe new hub lists itself as
hub02; the old hub's name is gone from the fleet. -
End the windows of step 2:
ssh deploy@203.0.113.11 sudo mmwatchdog maintenance off node_reports ssh deploy@203.0.113.12 sudo mmwatchdog maintenance off node_reportsnode_reports is out of maintenance. Alerts, restarts and recovery commands resume after fresh checks. The MMWatchdog fleet hub received the maintenance change.Check: the last line of
sudo mmwatchdog statuson each node isFleet hub: 203.0.113.13:8445 as web01; last report ..., with a recent time. -
Remove MMWatchdog from the old server, or shut the server down. Remove MMWatchdog from a node says how to remove MMWatchdog's files. Do not start MMWatchdog there again with the old state.
If something is different
- The node to add already reported to the hub before, such as a
server that you removed and now add again. It joins as a new node, and
sudo mmwatchdog fleet web02shows the new join time. Its old events and readings are gone from the hub, if you removed it withfleet forget. - A node is replaced by a new server with the same host name. Shut
the old server down, and add the new one. While the old node is
reportingorwaiting_for_first_report, the hub refuses the new one withnode name "web02" is in use. A new node can use the name only after the fleet hub marks the current node as silent, andsudo mmwatchdog fleetshowsName conflict:. Once the old node is silent, the new one joins in its place, and the silence alert of the old node closes withweb02 joined the fleet again as a new node; MMWatchdog does not know whether the problem is fixed. - You add many nodes, or nodes of another job, such as Git servers with checks of their own. Give them a configuration directory of their own, as Set up a fleet says.
- The silence of a whole node group must wake someone, such as when
both servers of a background job stop. Add
raise_to_sos_when_every_node_of_these_node_groups_is_silent: [jobs_servers]to the settings of those nodes, next to theirthis_node_is_in_node_groups. Silent nodes gives the rules. - The join code was seen by someone, and a node may have joined with
it. Rotate the code, then compare the nodes that
sudo mmwatchdog fleet --jsonlists with your inventory, and remove each node that you do not know. Then change the secrets of your alert destinations, because a joined node has them: change each one in its service, put it in the vault withmmx secrets edit, and runmmx run hub_and_standalone. - The hub's address is a host name that stays the same, such as
hub.example.com:8445inaddress_for_nodes. Point the name at the new server before step 9. The join code holds the host name, so it does not change: skip step 8, and keep step 9 only if you changed other files. - The old hub's state is lost, such as with a dead disk. Install the
new hub as in Set up a fleet, steps 4 to 10: it
has a new identity and a new join code. Then run
mmx run join_hub --inventory hub_members, and each node joins the new hub. Until then, the nodes refuse the new hub's certificate, send their alerts themselves, and the hub-outage alerter nodes alert. The readings and events that the old hub held are gone; each node still has its own. - The fleet was not set up with MMX. The commands on the hub are the
same. Write the new join code to
/etc/mmwatchdog/secrets/fleet-hub-join-codeon each node yourself, and runsudo mmwatchdog reloadthere. Forgetting nodes, rotating the join code, and moving the hub gives the rules.
Related pages
- Set up a fleet
- Watch a fleet
- Pause alerts during planned work
- Remove MMWatchdog from a node
- Update a fleet
This page is docs/how-to/change-a-fleet.md in the MMWatchdog repository.