Remove MMWatchdog from a node
When to use this page: MMWatchdog should no longer watch a node, and its program, files, and account should go from the node. To stop MMWatchdog for a while instead, follow Stop MMWatchdog for a while.
Who acts: you, or your coding agent. Steps 2 and 4 to 7 run on the
node, logged in with an account that can use sudo. Steps 1, 3, 8, and 9
run on your controller machine, the machine where you run mmx. Step 3 saves the
node's history into the MMX project, and step 5 deletes it from the node.
A coding agent always does step 3, unless the person said that the
history may go: a saved copy costs only disk space on the controller
machine, and a deleted history cannot come back.
What this page does: the MMX project that installed MMWatchdog has no
playbook that removes it, so you remove it with commands on the node. You
stop the service, save the node's history, remove every file that the
install made, and remove the account mmwatchdog. Then you prove that
nothing of MMWatchdog is left, and take the node out of the project so
that no later run installs MMWatchdog again.
Choose the route
| The node | Route |
|---|---|
| A standalone node: it sends its own alerts, with no fleet hub | Steps |
| A joined node: it reports to a fleet hub | Remove a node on the page Change a fleet. It sends you to steps 1 to 8 below, then takes the node out of the inventory and the hub |
| The fleet hub | First move the hub to another node, as Move the hub to another server says, then Steps |
Before you start
-
Find out which kind of node it is. On the node:
sudo grep -E '^(join_fleet_hub|fleet_hub_role):' /etc/mmwatchdog/mmwatchdog_node_settings.ymlThe command prints nothing on a standalone node,
join_fleet_hub:on a joined node, andfleet_hub_role:on the hub. -
Check whether the node uses other paths than the defaults. The steps remove the default paths. On the node:
sudo grep -E '^(state_dir|module_state_dir|socket):' /etc/mmwatchdog/mmwatchdog_node_settings.ymlThe command prints nothing when the node uses the defaults. Otherwise, remove the paths that it prints in place of the defaults in step 5.
-
Tell MMHotline, if the node sends alerts to it. A standalone node with an
mmhotlinedestination sends MMHotline a check-in every 30 seconds, and MMHotline alerts when the check-ins stop. Before step 1, take the node's alert source out ofalert_sourcesin MMHotline's settings file and runsudo mmhotline settings applyon the MMHotline server, as Connect a tool that sends alerts says. From the apply on, MMHotline no longer expects check-ins from the node. Done the other way round, the silence between the two steps raises a missed check-in warning, then an SOS.sudo mmhotline alert-sources listthen says the source is disabled. An incident that was already open stays open: close it as MMHotline's Handle an incident says.
Steps
-
Log in to the node:
ssh deploy@203.0.113.11Check: the prompt shows the node's host name. Steps 2 and 4 to 7 run in this login.
-
Stop MMWatchdog, and keep it from starting at boot:
sudo systemctl disable --now mmwatchdogRemoved '/etc/systemd/system/multi-user.target.wants/mmwatchdog.service'.From now on, MMWatchdog runs no check and sends no alert. It sends no message when it stops.
Check:
systemctl is-active mmwatchdogprintsinactive. -
Save the node's history.
/var/lib/mmwatchdog/holds the only copy of the node's events and measurement history, and step 5 deletes it. On the controller machine, in another terminal, go into the MMX project that installed MMWatchdog, and copy the directory into one archive there:ssh deploy@203.0.113.11 sudo tar -C / -czf - var/lib/mmwatchdog > mmwatchdog-state-app01.tar.gz tar -tzf mmwatchdog-state-app01.tar.gzvar/lib/mmwatchdog/ var/lib/mmwatchdog/measurement_history.db var/lib/mmwatchdog/daemon.lock var/lib/mmwatchdog/state.dbReason for stopping first: a stopped MMWatchdog has written everything into the two database files, so the copy is complete. To use the history again, put the directory back on a node before you install MMWatchdog there:
ssh deploy@203.0.113.11 sudo tar -C / -xzf - < mmwatchdog-state-app01.tar.gz. The install then keeps it, andsudo mmwatchdog eventslists the earlier events.Check: the list shows
state.dbandmeasurement_history.db. Skip this step only when the person said that the history may go. -
Remove the program and the systemd unit:
sudo rm -f -- /etc/systemd/system/mmwatchdog.service /usr/local/bin/mmwatchdog sudo systemctl daemon-reloadReason for
daemon-reload: systemd forgets the unit only when it reads its unit files again.Check:
systemctl status mmwatchdogprintsUnit mmwatchdog.service could not be found. -
Remove MMWatchdog's directories:
sudo rm -rf -- /usr/local/lib/mmwatchdog /etc/mmwatchdog sudo rm -rf -- /var/lib/mmwatchdog /var/lib/mmwatchdog-module-state /var/lib/mmwatchdog-checksDirectory What goes with it /usr/local/lib/mmwatchdog/The modules that are not built in, and the copies that each install checks before it installs them /etc/mmwatchdog/The configuration, and the secret files in secrets/, such as the Slack address/var/lib/mmwatchdog/The events, the measurement history, and open problems /var/lib/mmwatchdog-module-state/What checks of modules that are not built in keep from one run to the next. It exists only on a node with such a check /var/lib/mmwatchdog-checks/The home of the account mmwatchdogThe secret files are also in the vault of your MMX project, so you need no copy of them. A secret file that you put on the node by hand is gone after this step: copy it first if you need it. Files that an install puts on a node lists every file.
Check: none here; step 7 checks every path.
-
Remove the account and the group
mmwatchdog:sudo userdel mmwatchdog getent group mmwatchdog && sudo groupdel mmwatchdogOn many distributions, such as Debian,
userdelalso removes the group, andgetentthen prints nothing, sogroupdeldoes not run.Check:
getent passwd mmwatchdogandgetent group mmwatchdogprint nothing. -
Prove that nothing of MMWatchdog is left:
systemctl list-unit-files 'mmwatchdog*' ls -d /usr/local/bin/mmwatchdog /usr/local/lib/mmwatchdog /etc/mmwatchdog /var/lib/mmwatchdog /var/lib/mmwatchdog-module-state /var/lib/mmwatchdog-checks /run/mmwatchdog getent passwd mmwatchdog; getent group mmwatchdog pgrep -a mmwatchdogUNIT FILE STATE PRESET 0 unit files listed. ls: cannot access '/usr/local/bin/mmwatchdog': No such file or directory ls: cannot access '/usr/local/lib/mmwatchdog': No such file or directory ls: cannot access '/etc/mmwatchdog': No such file or directory ls: cannot access '/var/lib/mmwatchdog': No such file or directory ls: cannot access '/var/lib/mmwatchdog-module-state': No such file or directory ls: cannot access '/var/lib/mmwatchdog-checks': No such file or directory ls: cannot access '/run/mmwatchdog': No such file or directoryCheck: no unit file,
No such file or directoryfor each path, and nothing fromgetentorpgrep. systemd removed/run/mmwatchdog/itself in step 2. If a path is still there, remove it as in steps 4 to 6. Log out of the node. -
Remove MMX from the node too, if MMX should no longer manage it. MMX keeps its own program, service, and records on each node that it manages, and a record of the project's runs in
/var/lib/mmx/. If MMX still manages the node for other work, skip this step. Otherwise, in the MMX project that installed MMWatchdog, on the controller machine:mmx remove-mmx-from-node nodenode: stopped and disabled mmx-https-api.service node: removed /etc/systemd/system/mmx-https-api.service node: removed /opt/mmx node: removed /var/lib/mmx node: removed /run/mmx controller machine: removed .../.mmx/nodes/757251cdfcf6b7b5252ec4049deab203d849e183fbbdc9aa7f80596525d4983d.json MMX did not change the inventory or the playbooks. Take node out of the inventory, its groups, and the playbooks; until then, a run that selects node sets MMX up on it again.nodeis the node's name ininventories/hub_and_standalone.yml. A joined node is ininventories/hub_members.ymlinstead, so name that inventory, such asmmx remove-mmx-from-node web02 --inventory hub_members. MMX finds the node only while it is still in the inventory, so do this step before step 9. MMX's Remove MMX from a server says more.Check: the output lists each removed path, as above. For a joined node, go back now to step 2 of Remove a node, which takes the node out of the inventory and the hub in place of step 9.
-
Take the node out of the MMX project. Reason: the next
mmx run hub_and_standalonewould install MMWatchdog on every node of the inventory again.- The project serves other nodes too. Delete the node's entry under
nodes:ininventories/hub_and_standalone.yml. - The project served only this node. Keep the project as it is, as
a record of the configuration, or delete the project directory. Do
not delete only the node's entry: an inventory needs at least one
node, and
mmx validatethen fails withinventory requires at least one node.
Check: for a project that serves other nodes,
mmx nodes hub_and_standaloneno longer lists the node:Playbook: hub_and_standalone Inventory: hub_and_standalone (inventories/hub_and_standalone.yml, 1 node) Nodes web02 203.0.113.12 deploy ...Here the node
nodeis gone, and the nodeweb02stays. - The project serves other nodes too. Delete the node's entry under
What stays
- On the controller machine: your MMX project, with its
configuration, the vault with your secrets, and MMX's record of each run
in
.mmx/. They are yours to keep or delete. - On the node: the lines that MMWatchdog wrote to the system journal,
which
journalctl -u mmwatchdogshows until the journal deletes old lines by its own rules. MMX's own files, unless you did step 8. A secret file that another program uses, outside/etc/mmwatchdog/: step 5 removes only MMWatchdog's own directories. - In the MMX project, the archive of step 3,
mmwatchdog-state-app01.tar.gz, if you made it: the node's events and measurement history.
If something is different
- A
sudocommand asks for a password. The account can usesudowith a password: type it. Log in as root instead to run the commands withoutsudo. - You want to keep MMWatchdog installed but not running. Do step 2
only.
sudo systemctl enable --now mmwatchdogstarts it again. - A later
mmx runinstalled MMWatchdog again. The node was still in the inventory. Do step 9, then steps 1 to 7 again.
Related pages
- Files that an install puts on a node
- Install MMWatchdog on one node
- Pause alerts during planned work
- Update MMWatchdog
This page is docs/how-to/remove-mmwatchdog.md in the MMWatchdog repository.