Install MMWatchdog on one node
When to use this page: you want MMWatchdog to watch a Linux server, a node, that does not run MMWatchdog yet. For several nodes that report to one fleet hub, install the first node with this page, then follow Set up a fleet.
Who acts: you, or your coding agent. Steps 1 to 9 run on your
controller machine: the machine where you run mmx, such as your
laptop. Steps 10 and 11 also run commands on the node. Show the person the
output of step 8 and wait for their yes before step 9, because step 9
changes the node.
What this page does: MMX, the
Monkeymaxxing tool that sets up servers, copies the mmwatchdog program,
its configuration, and its modules to the node, and starts MMWatchdog as the
systemd service mmwatchdog.service. Then you prove that an alert reaches
you.
Before you start
- MMX is installed on the controller machine, which runs macOS or
Linux.
mmx versionprints a version. MMX's install section says how. - The node runs Linux with systemd, on x86-64 (amd64) or ARM64. MMWatchdog is tested on Debian 12 and 13, Ubuntu 24.04 and 26.04, Rocky Linux 9, AlmaLinux 10, Fedora 45, Arch Linux, and openSUSE Leap 16.1. Limits on each distribution says what MMWatchdog cannot check on some of them.
- You log in to the node with
ssh, as root or as an account that can run any command withsudo. MMX uses your SSH settings, keys, and agent, and connects only to a node whose host key your SSH already trusts. - You know where alerts go. This page uses a Slack channel with an incoming webhook. For email, a webhook of your own, Telegram, Discord, Mattermost, Pushover, or MMHotline, see If something is different. The node must be able to reach the destination, for Slack over HTTPS.
Steps
-
Download the release for your node. Choose the archive for the node's processor, not the controller machine's:
linux-amd64for x86-64 nodes,linux-arm64for ARM64 nodes.uname -mon the node printsx86_64oraarch64.version=0.1.0 platform=linux-amd64 # linux-arm64 for ARM64 nodes release=https://github.com/monkeymaxxing/mmwatchdog/releases/download/v$version curl -fLO "$release/mmwatchdog-$version-$platform.tar.gz" curl -fLO "$release/SHA256SUMS"Check: both files are in the current directory.
If GitHub is down, build the same two files from the mirror at
gitlab.com/monkeymaxxing/mmwatchdog, which has the code and the release tags but no release archives. You need Git, Python 3, and Go 1.21 or newer; Go downloads the exact Go version that MMWatchdog needs:git clone https://gitlab.com/monkeymaxxing/mmwatchdog.git mmwatchdog-source git -C mmwatchdog-source checkout "v$version" python3 mmwatchdog-source/scripts/release.py "$version" --output-dir .The script builds both archives and
SHA256SUMSinto the current directory. Then continue with step 2. -
Check the download:
shasum -a 256 --check --ignore-missing SHA256SUMS # on macOS sha256sum --check --ignore-missing SHA256SUMS # on Linuxmmwatchdog-0.1.0-linux-amd64.tar.gz: OKCheck: your archive's line ends with
OK. If not, download it again; do not unpack an archive that fails the check.--ignore-missingskips the archive of the other processor, which you did not download. -
Unpack the archive, and go into it:
tar -xzf "mmwatchdog-$version-$platform.tar.gz" cd "mmwatchdog-$version-$platform"The directory is an MMX project. It holds the
mmwatchdogprogram for the node, the starter configuration infiles/hub_and_standalone/, the playbookhub_and_standalonethat installs MMWatchdog, and the inventoryinventories/hub_and_standalone.ymlthat names the node. Keep this directory, with the vaultsecrets_vault.agethat step 6 creates and the directory.mmx/, where MMX saves each run: every later change to the node is made from it. A version-control system such as Git suits it.Check:
lslistsmmx_project_settings.ymlandmmwatchdog. -
Name the node in the inventory. In
inventories/hub_and_standalone.yml, replace the starter valuesnode.example.comanddeploywith your node's address and account:nodes: node: address: 203.0.113.11 # the node's address, or a host name from your SSH settings ssh_user: deploy # root, or an account that can use sudo without a passwordCheck: you can log in, and the project is valid:
ssh deploy@203.0.113.11 true && echo login works mmx validatelogin works Valid: mmwatchdogIf
sshasks whether to trust the node's host key, compare the fingerprint with the one your hosting provider shows and answeryes. Reason: MMX never accepts a new host key itself. A coding agent, which cannot answer the prompt, shows the person the fingerprint and, after their yes, logs in once withssh -o StrictHostKeyChecking=accept-new deploy@203.0.113.11 true. Ifsshasks for a password, that works too: the firstmmx dryrunasks for it, and MMX then adds a key of its own to the node, as MMX's get started describes. -
Read the node settings.
files/hub_and_standalone/mmwatchdog_node_settings.ymlstates the six settings that every node needs. The starter values suit most nodes:keep_undelivered_alerts_for: 24h # keep an alert that could not be delivered, and try again, for 24 hours undelivered_alerts_content_size_limit: 8MiB # and keep at most 8 MiB of such alerts alert_severity_when_this_node_goes_silent: sos # used by a fleet hub when this node stops reporting alert_severity_when_this_node_has_rebooted: warn # the severity of the notice after a reboot of this node keep_measurement_history_for: 7d # keep every reading of the checks for 7 days measurement_history_size_limit: 256MiB # and never more than 256 MiB of readingsCheck: none; change a value only if you have a reason, such as
infofor the reboot notice of one of many web servers. -
Put the Slack address in MMX's vault. The address is a secret: anyone who has it can post to your channel. MMX keeps secrets in the encrypted file
secrets_vault.agein the project, and writes them to the node during the install. Open the vault in your editor:mmx secrets editThe first time, MMX asks for a new passphrase twice, and the editor shows an empty vault:
Vault passphrase: Confirm vault passphrase:Add these two lines with your webhook's address, save, and close the editor. A coding agent asks the person to run
mmx secrets editand type the address themselves, so the address never passes through the chat:mmwatchdog_secret_files: slack-webhook-team: "https://hooks.slack.com/services/YOUR/WEBHOOK/ADDRESS"Then list the file in
inventories/hub_and_standalone.yml, undervars, in place ofmmwatchdog_secret_files: []:vars: mmwatchdog_config_source_dir: files/hub_and_standalone mmwatchdog_secret_files: [slack-webhook-team]Each install writes each listed file to
/etc/mmwatchdog/secrets/on the node, readable only by root, before it checks the configuration.Check:
mmx secrets list-keysprintsmmwatchdog_secret_files. Ifmmx secrets editreports an error instead ofSaved encrypted vault: secrets_vault.age, it kept the vault as it was: run it again and fix the two lines. -
Say where alerts go. Replace
destinations: {}infiles/hub_and_standalone/alert_destinations.ymlwith:destinations: team_slack_channel: slack: webhook_url_file: /etc/mmwatchdog/secrets/slack-webhook-team alert_severities: [info, warn, sos] # this destination takes alerts of every severity remind_every: 4h # remind of each open problem every 4 hoursCheck:
mmx validateprintsValid: mmwatchdog. -
Preview what the install changes:
mmx dryrun hub_and_standaloneThe output lists each step of the playbook on the node, such as:
install / node / install_monitoring/changes/install_binary: would create "/usr/local/bin/mmwatchdog" absent -> 17318048 bytes, permissions_mode 0755, uid 0, gid 0 install / node / install_monitoring/changes/install_unit: would create "/etc/systemd/system/mmwatchdog.service"Check: the dryrun connected to the node and lists the changes. On a first install its exit code is 3: before anything is installed, some steps cannot be checked in advance, and their lines say
not_checkedorskipped, unknown. Exit code 1, or a line that saysfailed, is a fault; see If something is different. -
Install MMWatchdog:
mmx run hub_and_standaloneRun 2b792738: succeeded 1 finishedThe run checks the whole configuration on the node before it installs anything. The starter check group
machine, infiles/hub_and_standalone/check_groups/node.yml, records CPU use and alerts atwarnwhen memory use exceeds 90 percent or the root filesystem is more than 85 percent full, and atsoswhen the root filesystem is more than 95 percent full.Check: the run ends with
succeeded. -
Confirm that MMWatchdog runs. On the node:
ssh deploy@203.0.113.11 sudo mmwatchdog version ssh deploy@203.0.113.11 sudo mmwatchdog statusmmwatchdog 0.1.0 (ad75e5909a0af9b03215d2b2ac4c5bdfa9a56777, linux/amd64, modified=false)machine/cpu_use_is_recorded ok cpu_used_percent is 2.56 over 0.201s machine/memory_is_below_90_percent ok memory_used_percent is 17.41 (fails above 90) machine/root_filesystem_is_below_85_percent ok space_used_percent is 9.12 (fails above 85) machine/root_filesystem_is_below_95_percent ok space_used_percent is 9.28 (fails above 95) Alert deliveries: 0 waiting, 0 droppedCheck:
versionshows the release you installed and the node's processor, andstatuslists every check, with no line that says that a destination fails. A destination that fails has a line such asDestination team_slack_channel: 1 alert is not delivered, with the reason. -
Prove that an alert reaches you. A check that runs well proves nothing about the alert path, so make one check fail and recover. Add this test check group to the project:
# files/hub_and_standalone/check_groups/alert_path_test.yml # The check fails while the file /var/tmp/mmwatchdog-alert-test exists. check_groups: alert_path_test: check_every: 5s time_limit_per_run: 2s alert_after_failing_for: 0s # alert at the first failed run recovered_after_healthy_for: 0s # the failure is over at the first healthy run shell: test ! -e /var/tmp/mmwatchdog-alert-test alert_severity: warnInstall it, then make it fail, and after the alert arrives, recover:
mmx run hub_and_standalone ssh deploy@203.0.113.11 touch /var/tmp/mmwatchdog-alert-test # wait for the failure message in Slack ssh deploy@203.0.113.11 rm /var/tmp/mmwatchdog-alert-test # wait for the recovery messageSlack shows two messages:
Warning: app01: alert_path_test/shell failed command exited 1 failing since 04:04 UTCResolved warning: app01: alert_path_test/shell is ok again command exited 0Then delete
files/hub_and_standalone/check_groups/alert_path_test.ymland runmmx run hub_and_standaloneagain, which removes the test check group from the node.Each run that changes the configuration also sends two
Infomessages,maintenance onandmaintenance off: MMWatchdog restarts inside a short maintenance window, so the restart sends no check alerts.Check: both messages arrived. If not, see If something is different.
The two files you changed, whole
# inventories/hub_and_standalone.yml
# The directory of this project whose files each node gets as its
# MMWatchdog configuration in /etc/mmwatchdog. A node or a group of
# another role sets its own directory under files/ in its vars.
vars:
mmwatchdog_config_source_dir: files/hub_and_standalone
# The secret files that each node gets in /etc/mmwatchdog/secrets, with
# mode 0600. List the file names here, and put each file's content in the
# MMX vault under the secret mmwatchdog_secret_files with mmx secrets edit:
# mmwatchdog_secret_files:
# slack-webhook-team: "https://hooks.slack.com/services/..."
# A node of this inventory gets the whole mmwatchdog_secret_files secret,
# and writes the files listed here.
mmwatchdog_secret_files: [slack-webhook-team]
nodes:
node:
address: 203.0.113.11
ssh_user: deploy
# files/hub_and_standalone/alert_destinations.yml
destinations:
team_slack_channel:
slack:
webhook_url_file: /etc/mmwatchdog/secrets/slack-webhook-team
alert_severities: [info, warn, sos] # this destination takes alerts of every severity
remind_every: 4h # remind of each open problem every 4 hours
MMWatchdog now watches the node, and the starter check group alerts you when memory or the disk runs short. Next, add a check for each service or application on the node.
If something is different
- The node is ARM64. Download
linux-arm64in step 1. One project serves nodes of one processor kind: for x86-64 and ARM64 nodes, unpack both archives and use two projects. - Alerts go elsewhere than Slack. Email (
email), a JSON webhook of your own (webhook), and MMHotline (mmhotline) are built in. Telegram, Discord, Mattermost, and Pushover are modules that ship with MMWatchdog, named the same way. Each module's page, printed bymmwatchdog describe-module NAMEand listed in the module pages, gives its inputs and how to get its address or token. Put each password or token in the vault and the inventory as in step 6, and name its file in the destination's input that ends in_file. - No alerts yet. Keep
destinations: {}and skip steps 6, 7, and 11. MMWatchdog then runs the checks and restarts services, andstatussaysThis node sends alerts to nobody. - Step 4's login check fails. Fix the SSH login first: MMX logs in the
same way. A host name, a port, or a key file for the node can go in your
SSH settings, such as
~/.ssh/config, and the inventory'saddresscan name that host. - The run fails at
validate_config. The dryrun does not check your new files: it checks only the copy of the configuration that the last run left on the node. Onlymmx runchecks the new configuration. The output names the file, the line, and the fault, such as a mistake in a check group, a secret file that a destination names but the node does not have, or a module that neither the release nor the project holds. Fix it and run again. The node keeps its configuration, its modules, and its running MMWatchdog; only the secret files of step 6 are already written, and MMWatchdog already uses a changed Slack address. - The run fails at
write_secret_fileswith a line such as.secrets.mmwatchdog_secret_files[.secret_file]: value is unavailable at /secrets/mmwatchdog_secret_files/missing-file. The inventory lists a file, heremissing-file, that the vault'smmwatchdog_secret_filesdoes not hold. Each name in the inventory's list must be a key undermmwatchdog_secret_filesin the vault. - The status line of the destination says the alert is not delivered.
The line gives the reason, such as
the module slack reported: the file that webhook_url_file names must hold one web address that starts with https:// or http://, and nothing else. Fix the address withmmx secrets editand runmmx run hub_and_standalone; a changed secret file restarts MMWatchdog. - You build the archives yourself. Release MMWatchdog says how; the archives are the same as a downloaded release's.
- The node must not be installed by MMX. The
list of installed files says where each
file of an installation goes, the account
mmwatchdog, and the systemd unit, and Install with a tool of your own says how to check the new files before you install them.
What the install puts on the node
- The program
/usr/local/bin/mmwatchdog, and the systemd unit/etc/systemd/system/mmwatchdog.service, enabled and started. - The configuration in
/etc/mmwatchdog/:mmwatchdog_node_settings.yml,alert_destinations.yml,check_groups/, andsecrets/. - The modules that are not built in, in
/usr/local/lib/mmwatchdog/. - The system account
mmwatchdog, which runs the checks that need no root, with its home/var/lib/mmwatchdog-checks. MMWatchdog itself runs as root. - MMWatchdog's state, in
/var/lib/mmwatchdog/.
Related pages
This page is docs/how-to/install-on-one-node.md in the MMWatchdog repository.