Set up a fleet
When to use this page: you have several Linux servers, and you want one of them, the fleet hub, to collect the state of all of them and to send the alerts of all of them, so that you see every server in one place. If MMWatchdog already runs on one of the servers, that server can become the hub: see If something is different.
Who acts: you, or your coding agent. Each step runs on your
controller machine: the machine where you run mmx, such as your
laptop. Some checks run a command on a node through ssh. A coding agent
shows the person the output of each mmx dryrun and waits for their yes
before the mmx run that follows it, and asks the person to put each
secret into the vault themselves, so that the secret never passes through
the chat.
What this page does: MMX, the Monkeymaxxing tool that sets up servers, installs MMWatchdog on one node as the fleet hub, then installs MMWatchdog on the other nodes and joins them to the hub. Then you prove that an alert of a joined node reaches Slack through the hub, and that the hub shows every node.
Terms: a node is a server that MMWatchdog watches. Each node runs its own checks and restarts its own services, also without the hub. The fleet hub is one node that also has the fleet hub role: it receives a node status report from each other node every 30 seconds, sends the alerts of every node, and alerts you when a node stops reporting. A joined node is a node that reports to the hub. The join code is the secret that lets a node join the hub. The fleet settings are what each joined node copies from the hub: the alert destinations, the content of each secret file that a destination names, and the fleet timings.
Before you start
- MMX is installed on the controller machine, which runs macOS or
Linux.
mmx versionprints a version. MMX's install section says how. - Each node runs Linux with systemd, on x86-64 (amd64) or ARM64. MMWatchdog is tested on Debian 12 and 13, Ubuntu 24.04 and 26.04, Rocky Linux 9, AlmaLinux 10, Fedora 45, Arch Linux, and openSUSE Leap 16.1.
- You log in to each node with
ssh, as root or as an account that can run any command withsudo. MMX uses your SSH settings, keys, and agent, and connects only to a node whose host key your SSH already trusts. - You chose the hub. Choose a server that stays for a long time, such as a utility server, not one that you replace often. The hub keeps the fleet's identity and the readings of every node, so it needs a reliable disk with room for them: this page lets the readings of the nodes take up to 20 GiB.
- Every node reaches the hub on one TCP port, here 8445, over HTTPS. The hub listens on that port on every network interface, so let only your nodes through the hub's firewall.
- You know where alerts go. This page uses a Slack channel with an incoming webhook. Send alerts where your team sees them covers email, a webhook of your own, and the other destinations. The hub must reach the destination, and so must every joined node: a joined node sends an alert itself when it cannot reach the hub.
This page uses three nodes: the hub hub01 at 203.0.113.10, and web01
and web02 at 203.0.113.11 and 203.0.113.12. The hub knows each node by
the node's host name, which hostname prints on the node.
Steps
-
Download the release for your nodes. Choose the archive for the nodes' processor, not the controller machine's:
linux-amd64for x86-64 nodes,linux-arm64for ARM64 nodes.uname -mon a node printsx86_64oraarch64.version=0.1.0 platform=linux-amd64 # linux-arm64 for ARM64 nodes release=https://github.com/monkeymaxxing/mmwatchdog/releases/download/v$version curl -fLO "$release/mmwatchdog-$version-$platform.tar.gz" curl -fLO "$release/SHA256SUMS"Check: both files are in the current directory.
-
Check the download:
shasum -a 256 --check --ignore-missing SHA256SUMS # on macOS sha256sum --check --ignore-missing SHA256SUMS # on Linuxmmwatchdog-0.1.0-linux-amd64.tar.gz: OKCheck: your archive's line ends with
OK. If not, download it again; do not unpack an archive that fails the check. -
Unpack the archive, and go into it:
tar -xzf "mmwatchdog-$version-$platform.tar.gz" cd "mmwatchdog-$version-$platform"The directory is an MMX project with two playbooks.
hub_and_standaloneinstalls the hub, with the configuration infiles/hub_and_standalone/, on the node ofinventories/hub_and_standalone.yml.join_hubinstalls every other node, with the configuration infiles/join_hub/, on the nodes ofinventories/hub_members.yml. Keep this directory, with the vaultsecrets_vault.agethat step 5 creates and the directory.mmx/, where MMX saves each run: every later change to the fleet is made from it. A version-control system such as Git suits it.Check:
ls playbookslistshub_and_standalone.ymlandjoin_hub.yml. -
Name the hub in the inventory. In
inventories/hub_and_standalone.yml, replace the starter node with the hub. Give each node its host name, herehub01: the fleet lists each node by its host name, so the inventory and the fleet then use the same names:nodes: hub01: address: 203.0.113.10 # the hub's address, or a host name from your SSH settings ssh_user: deploy # root, or an account that can use sudo without a passwordCheck: you can log in, and the project is valid:
ssh deploy@203.0.113.10 true && echo login works mmx validatelogin works Valid: mmwatchdogIf
sshasks whether to trust the node's host key, compare the fingerprint with the one your hosting provider shows and answeryes. Reason: MMX never accepts a new host key itself. -
Put the Slack address in MMX's vault. The address is a secret: anyone who has it can post to your channel. MMX keeps secrets in the encrypted file
secrets_vault.agein the project. Open the vault in your editor:mmx secrets editThe first time, MMX asks for a new passphrase twice. Add these two lines with your webhook's address, save, and close the editor:
mmwatchdog_secret_files: slack-webhook-team: "https://hooks.slack.com/services/YOUR/WEBHOOK/ADDRESS"Then list the file in
inventories/hub_and_standalone.yml, undervars, in place ofmmwatchdog_secret_files: []:vars: mmwatchdog_config_source_dir: files/hub_and_standalone mmwatchdog_secret_files: [slack-webhook-team]The install writes the file to
/etc/mmwatchdog/secrets/on the hub, readable only by root. The joined nodes need no such file: they get the address from the hub with the fleet settings.Check:
mmx secrets editends withSaved encrypted vault: secrets_vault.age, andmmx secrets list-keysprintsmmwatchdog_secret_files. -
Say where alerts go. Replace
destinations: {}infiles/hub_and_standalone/alert_destinations.ymlwith:destinations: team_slack_channel: slack: webhook_url_file: /etc/mmwatchdog/secrets/slack-webhook-team alert_severities: [info, warn, sos] # this destination takes alerts of every severity remind_every: 4h # remind of each open problem every 4 hoursThis file on the hub holds the destinations of the whole fleet.
Check:
mmx validateprintsValid: mmwatchdog. -
Give the hub the fleet hub role. At the end of
files/hub_and_standalone/mmwatchdog_node_settings.yml, replace the commented-out example, from the line# On the MMWatchdog fleet hub, add the fleet_hub_role blockto the end of the file, with this block, with the hub's address inaddress_for_nodes. All eight settings are required:fleet_hub_role: listen_on_address: 0.0.0.0:8445 # the hub listens on port 8445 of every interface address_for_nodes: 203.0.113.10:8445 # the address and port that the nodes connect to report_every: 30s # each node reports every 30 seconds; 3 missed reports make it silent alert_after_hub_unreachable_for: 3m # a node alerts that it cannot reach the hub after 3 minutes send_alert_without_hub_after_waiting_for: 30s # a node sends an alert itself when the hub has not taken it for 30 seconds undelivered_alerts_of_nodes_content_size_limit: 32MiB # the hub keeps at most 32 MiB of the nodes' alerts that wait for delivery keep_measurement_history_of_nodes_for: 7d # the hub keeps the readings that the nodes send for 7 days measurement_history_of_nodes_size_limit: 20GiB # and lets them take at most 20 GiB of its diskaddress_for_nodescan be a host name, such ashub01.example.com:8445. It goes into the join code, and every node connects to it.Check:
mmx validateprintsValid: mmwatchdog. -
Preview the install of the hub:
mmx dryrun hub_and_standaloneThe output lists each step of the playbook on the hub, such as:
install / hub01 / install_monitoring/changes/install_binary: would create "/usr/local/bin/mmwatchdog" install / hub01 / install_monitoring/changes/install_unit: would create "/etc/systemd/system/mmwatchdog.service"Check: the dryrun connected to the hub and lists the changes. On a first install its exit code is 3: before anything is installed, some steps cannot be checked in advance, and their lines say
not_checked. Exit code 1, or a line that saysfailed, is a fault. -
Install the hub:
mmx run hub_and_standaloneRun 38f83cc6: succeeded 1 finished Saved: /home/dev/monitoring/mmwatchdog-0.1.0-linux-amd64/.mmx/runs/20261115T092520.420798015-38f83cc6936b45e9On its first start, the hub creates its TLS key, a certificate, and the join secret, in
/var/lib/mmwatchdog/fleet.db.Check: the run ends with
succeeded, and the last line of the hub's status names the fleet address:ssh deploy@203.0.113.10 sudo mmwatchdog statusmachine/cpu_use_is_recorded ok cpu_used_percent is 0.51 over 29.484s ... Alert deliveries: 0 waiting, 0 dropped Fleet hub: nodes report to 203.0.113.10:8445; see mmwatchdog fleet -
Put the join code in the vault. Print the join code on the hub:
ssh deploy@203.0.113.10 sudo mmwatchdog fleet join-codemmwatchdog-join 203.0.113.10:8445 ebba574e17670386674eeb04912fdd7555fb626c4f124addab10c6a5888168ee 6e86939f0a274f03001be8c90ba26997315840fe4b969cb161a531702db4a3b7The code holds the hub's address, the fingerprint of the hub's certificate, and the join secret. It is a secret: a node that joins with it gets the fleet settings, with your Slack address. Open the vault with
mmx secrets editand add one line with the whole code. A coding agent asks the person to print the code and type it intommx secrets editthemselves, so the code never passes through the chat:mmwatchdog_join_code: "mmwatchdog-join 203.0.113.10:8445 ebba574e...168ee 6e86939f...4a3b7" # the whole line that join-code printedCheck:
mmx secrets list-keysprintsmmwatchdog_join_codeandmmwatchdog_secret_files. -
Name the other nodes in their inventory. Replace the starter nodes
app-1andapp-2ininventories/hub_members.ymlwith your nodes:nodes: web01: address: 203.0.113.11 ssh_user: deploy web02: address: 203.0.113.12 ssh_user: deployKeep
mmwatchdog_secret_files: []in this inventory: the joined nodes get the destinations' secrets from the hub. The checks of the joined nodes are infiles/join_hub/check_groups/; the starter check groupmachinealerts when memory or the root filesystem runs short.Check: you can log in to each node, and the project is valid:
ssh deploy@203.0.113.11 true && ssh deploy@203.0.113.12 true && echo login works mmx validate join_hub --inventory hub_memberslogin works Valid: mmwatchdog -
Preview the install of the nodes:
mmx dryrun join_hub --inventory hub_membersinstall / web01 / install_monitoring_on_a_joined_node/changes/join_code: would create "/etc/mmwatchdog/secrets/fleet-hub-join-code" install / web01 / install_monitoring_on_a_joined_node/changes/install_binary: would create "/usr/local/bin/mmwatchdog"Check: the dryrun connected to every node and lists the changes, and exits with code 3, as for the hub in step 8. The playbook writes the join code from the vault to
/etc/mmwatchdog/secrets/fleet-hub-join-codeon each node, readable only by root. -
Install MMWatchdog on the nodes, and join them to the hub:
mmx run join_hub --inventory hub_membersRun 17da248d: succeeded 2 finished Saved: /home/dev/monitoring/mmwatchdog-0.1.0-linux-amd64/.mmx/runs/20261115T092615.418622364-17da248dfc89be69Each node joins with its first report, a few seconds after MMWatchdog starts.
Check: the run ends with
succeeded. On a joined node, the last line of the status names the hub and the node's last report:ssh deploy@203.0.113.11 sudo mmwatchdog status... node_reports/to_hub ok reports reach the hub Alert deliveries: 0 waiting, 0 dropped Fleet hub: 203.0.113.10:8445 as web01; last report 2026-11-15T09:26:57Z -
See every node on the hub:
ssh deploy@203.0.113.10 sudo mmwatchdog fleetFleet at 203.0.113.10:8445: 3 nodes, 3 healthy Hub-outage alerter nodes: web01, web02The count includes the hub. A healthy node gets no line of its own; a node that needs attention is listed with its problem. Hub-outage alerter nodes are the two joined nodes that alert you when they cannot reach the hub.
Check: the count is the number of your nodes, hub included, and all are healthy. Right after an install, a node may be listed with
unknown: waiting for a new rununtil each check has run once: run the command again after 30 seconds. A node that is missing has not joined: see If something is different. Watch a fleet says how to read the fleet. -
Prove that an alert of a joined node reaches you through the hub. Add this test check group for the joined nodes:
# files/join_hub/check_groups/alert_path_test.yml # The check fails while the file /var/tmp/mmwatchdog-alert-test exists. check_groups: alert_path_test: check_every: 5s time_limit_per_run: 2s alert_after_failing_for: 0s # alert at the first failed run recovered_after_healthy_for: 0s # the failure is over at the first healthy run shell: test ! -e /var/tmp/mmwatchdog-alert-test alert_severity: warnInstall it, then make it fail on
web01, and after the alert arrives, recover:mmx run join_hub --inventory hub_members ssh deploy@203.0.113.11 touch /var/tmp/mmwatchdog-alert-test # the failure message arrives in Slack within about 10 seconds ssh deploy@203.0.113.11 rm /var/tmp/mmwatchdog-alert-test # the recovery message arrives within about 10 secondsSlack shows two messages, sent by the hub:
Warning: web01: alert_path_test/shell failed command exited 1 failing since 09:28 UTC exit code: 1Resolved warning: web01: alert_path_test/shell is ok again command exited 0Then delete
files/join_hub/check_groups/alert_path_test.ymland runmmx run join_hub --inventory hub_membersagain, which removes the test check group from the nodes.A run that changes the configuration restarts MMWatchdog on each node inside a whole-node maintenance window, so the restart sends no check alerts. Slack also shows the windows of the two runs. The hub sends one message when the first window opens, before the failure message:
Info: maintenance on for 2 nodes maintenance started on 2 nodes: web01, web02 (by root: mmwatchdog deploy)and one message when no node has been in a window for 10 minutes, about 10 minutes after the second run. The windows of both runs fall into that one pair of messages:
Info: maintenance off for 2 nodes maintenance turned off on 2 nodes: web02, web01 (by root: mmwatchdog deploy) maintenance ended on the last node in the alert at 09:29 UTC. Alerts on each node resumed when the node's maintenance ended. ...In a fleet with one joined node, the hub sends that node's own messages instead, such as
Info: web01: maintenance on for the whole node.Check: the failure and the recovery message arrived. If not, see If something is different.
The files you changed, whole
# inventories/hub_and_standalone.yml
vars:
mmwatchdog_config_source_dir: files/hub_and_standalone
mmwatchdog_secret_files: [slack-webhook-team]
nodes:
hub01:
address: 203.0.113.10
ssh_user: deploy
# files/hub_and_standalone/alert_destinations.yml
destinations:
team_slack_channel:
slack:
webhook_url_file: /etc/mmwatchdog/secrets/slack-webhook-team
alert_severities: [info, warn, sos] # this destination takes alerts of every severity
remind_every: 4h # remind of each open problem every 4 hours
# files/hub_and_standalone/mmwatchdog_node_settings.yml
keep_undelivered_alerts_for: 24h
undelivered_alerts_content_size_limit: 8MiB
alert_severity_when_this_node_goes_silent: sos
alert_severity_when_this_node_has_rebooted: warn
keep_measurement_history_for: 7d
measurement_history_size_limit: 256MiB
fleet_hub_role:
listen_on_address: 0.0.0.0:8445
address_for_nodes: 203.0.113.10:8445
report_every: 30s
alert_after_hub_unreachable_for: 3m
send_alert_without_hub_after_waiting_for: 30s
undelivered_alerts_of_nodes_content_size_limit: 32MiB
keep_measurement_history_of_nodes_for: 7d
measurement_history_of_nodes_size_limit: 20GiB
# inventories/hub_members.yml
vars:
mmwatchdog_config_source_dir: files/join_hub
mmwatchdog_secret_files: []
nodes:
web01:
address: 203.0.113.11
ssh_user: deploy
web02:
address: 203.0.113.12
ssh_user: deploy
The vault holds mmwatchdog_secret_files with the Slack address, and
mmwatchdog_join_code.
MMWatchdog now watches every node, and the hub shows them all. Next,
add a check for each service on the nodes: put the
check groups of the joined nodes in files/join_hub/check_groups/, and
install them with mmx run join_hub --inventory hub_members.
Where alerts go in a fleet
-
The hub sends every alert. The destinations are only in the hub's
alert_destinations.yml. A joined node sends each alert to the hub at once, and the hub delivers it. When the same check fails on several nodes within a minute, the hub sends the first alert at once and the others together in one message, such asmyapp/http failed on 7 nodes: app-2, app-3, …, and 2 more. Alerts in a fleet gives every rule. -
A joined node has no destinations of its own. MMWatchdog on a joined node refuses to start while
/etc/mmwatchdog/alert_destinations.ymlexists, andjoin_hubremoves that file from each node. -
Each joined node keeps a copy of the fleet settings, with the destinations and their secrets, in
/var/lib/mmwatchdog/fleet-settings-from-the-hub.json, readable only by root. A changed destination reaches every node with its next report. -
A node sends an alert itself when the hub does not take it. When a delivery to the hub has failed, the node sends each alert that the hub has not taken 30 seconds after the alert happened (
send_alert_without_hub_after_waiting_for), from its copy of the fleet settings. The node's log says so:the failed alert of myapp/health_page_answers goes directly to the alert destinations, because the latest attempt to deliver an alert to the hub failed: Post "https://203.0.113.10:8445/v1/alerts": dial tcp 203.0.113.10:8445: connect: connection refusedFor that reason each node must reach each destination, and
join_hubinstalls every destination module on every node. -
Two nodes alert you when the hub is down. Each hub-outage alerter node alerts after it has not reached the hub for 3 minutes (
alert_after_hub_unreachable_for), withSOS: web01 cannot reach the MMWatchdog fleet hub. Watch a fleet says more.
If something is different
-
MMWatchdog already runs on one node, from Install MMWatchdog on one node. That node can be the hub, if it stays for a long time: use the same project, and skip steps 1 to 6. Do steps 7 to 15:
mmx run hub_and_standalonein step 9 adds the role, and the node keeps its checks and destinations. To make another server the hub instead, see Move the hub. -
The nodes reach the hub at another address, such as through port forwarding or a proxy.
listen_on_addressis the interface and port on the hub;address_for_nodesis what the nodes connect to. Both go into effect whenmmx run hub_and_standalonerestarts MMWatchdog. A changedaddress_for_nodesneeds a new join code on every node, as Move the hub says. -
The nodes have different jobs, such as web servers and Git servers, each with its own checks. Copy
files/join_hub/to another directory, such asfiles/git_servers/, and change the check groups in it. Then choose the directory for those nodes with an MMX inventory group ininventories/hub_members.yml:# inventories/hub_members.yml vars: mmwatchdog_config_source_dir: files/join_hub # the configuration of each node without its own mmwatchdog_secret_files: [] nodes: web01: address: 203.0.113.11 ssh_user: deploy git01: address: 203.0.113.20 ssh_user: deploy groups: git_servers: nodes: [git01] vars: mmwatchdog_config_source_dir: files/git_servers # the whole configuration of these nodesA node's value replaces its group's, and a group's replaces the inventory's. An MMX inventory group only chooses files; a node group, which the hub shows, is another thing: Put nodes in node groups.
-
A node matters more than the others. In
files/join_hub/mmwatchdog_node_settings.yml,alert_severity_when_this_node_goes_silent: warnmakes one silent node a warning, andalert_severity_when_this_node_has_rebooted: infomakes a reboot news. For a node that matters, such as the database server, give it a configuration directory of its own, as above, withsos. -
Some nodes are ARM64 and some x86-64. One project serves nodes of one processor kind. Unpack both archives as two projects, give both vaults the same
mmwatchdog_join_code, and install the hub before any joined node. -
A node is missing from
mmwatchdog fleet. The last line ofsudo mmwatchdog statuson the node gives the latest error:Error on the node Fix dial tcp 203.0.113.10:8445: connect: connection refusedMMWatchdog does not run on the hub, or a firewall blocks the port. Check systemctl is-active mmwatchdogon the hub, and the firewalljoining the hub: hub returned HTTP 403: invalid join codeThe vault holds an old join code. Put the code that sudo mmwatchdog fleet join-codeprints on the hub in the vault, and runmmx run join_hub --inventory hub_membersTwo nodes with the same host name cannot both join: the hub refuses the second, and
sudo mmwatchdog fleetshows the name conflict. Give each node its own host name. Watch a fleet says when theName conflict:line goes away. -
The run fails with
duplicate service configuration path /etc/mmwatchdog/alert_destinations.yml, such as:install / web01 / install_monitoring_on_a_joined_node: failed, unchanged — duplicate service configuration path /etc/mmwatchdog/alert_destinations.yml: two node tasks that may run on the node write to the same path: node task install_monitoring_on_a_joined_node/install_config (playbooks/join_hub.yml:243), content from files/join_hub and node task install_monitoring_on_a_joined_node/remove_local_alert_destinations (playbooks/join_hub.yml:252)files/join_hub/holds analert_destinations.yml, whichinstall_configwould install, whileremove_local_alert_destinationsremoves that file from every joined node. Deletefiles/join_hub/alert_destinations.yml: the destinations belong infiles/hub_and_standalone/. The run stopped before it installed anything, so MMWatchdog on the nodes runs as before. Runmmx run join_hub --inventory hub_membersagain. -
The fleet is large. In a load test, a hub with 10,000 nodes used about one CPU core and 300 MiB of memory, with a 26 MiB
fleet.db, without readings. The readings add one write to the hub's disk for each report, and take up tomeasurement_history_of_nodes_size_limit. The hub saves every report before it answers, so give it a reliable disk. -
You update MMWatchdog later. Update the hub first, then the joined nodes: Update a fleet.
-
The nodes must not be installed by MMX. The how MMWatchdog works gives every setting of
fleet_hub_roleandjoin_fleet_hub, which you write in/etc/mmwatchdog/mmwatchdog_node_settings.ymlon each node.
Related pages
- Watch a fleet
- Change a fleet
- Install MMWatchdog on one node
- Send alerts where your team sees them
- Look back at measurements, with the readings of every node on the hub.
This page is docs/how-to/set-up-a-fleet.md in the MMWatchdog repository.