Manage the controller machine itself
Who acts: you, or your coding agent, on a Linux controller machine: the
machine where you run mmx.
When to use this page: you run mmx on a Linux machine, such as an
administration server or a CI machine, and you want your playbooks to set
up that machine too. Or a run failed with MMX does not reboot the controller machine.
What this page does: you add the controller machine to your inventory
as a node at the address 127.0.0.1 and run a playbook on the controller
machine. Then you patch and reboot your servers with a playbook that leaves
the reboot of the controller machine to you.
Terms:
- The controller machine is the machine where you run
mmxcommands. A controller process is one runningmmxcommand. - A node is a Linux server that MMX manages. The inventory, a file
such as
inventories/example.yml, lists the nodes and how MMX logs in to each node. - A playbook is a file in
playbooks/that lists fleet stages. Each fleet stage selects nodes, and each selected node runs the fleet stage's node tasks, in order.
Choose your route
| Your situation | Route |
|---|---|
You want your playbooks to set up the Linux machine where you run mmx |
Add the controller machine as a node |
| A playbook patches and reboots every node, and the controller machine is one of the nodes | Patch every node and reboot the controller machine by hand |
You run mmx on a Mac |
A Mac cannot be a node, because mmx-runner, MMX's program for the node, is built for Linux only. Manage the Mac with another tool. |
You run mmx inside a container, such as a CI job |
The controller process runs in a container |
Before you start
mmxis installed on the controller machine, as in Get started, and you have a project made withmmx init.- The controller machine runs Linux with systemd, like any node. Limits on each distribution lists the tested distributions.
- Your account on the controller machine is
root, or an account that can run any command withsudo, because MMX installs software and changes system files.
The examples use the account barbara on a Debian 13 controller machine,
and the server web01 at 203.0.113.11. Put in your own account and
servers.
Add the controller machine as a node
MMX manages the controller machine as it manages web01. MMX has no
separate local mode. The controller process logs in to 127.0.0.1 over
SSH, installs mmx-runner on the controller machine, and sends the work
to mmx-runner. So the controller machine needs what every node needs: an
SSH server, a login for your account, and a host key that your account
trusts.
-
Install an SSH server. On Debian and Ubuntu:
sudo apt-get update sudo apt-get install -y openssh-serverOn other distributions, install the distribution's OpenSSH server package and start its service.
Check:
ss -ltn 'sport = :22'prints a line that starts withLISTEN, so an SSH server waits for logins on port 22. If the command prints only its header line, start the SSH server's service. -
Let your account log in to itself with an SSH key. The account that runs
mmxlogs in to the controller machine as the account in the inventory, here the same accountbarbara:ssh-keygen -t ed25519 # only if ~/.ssh/id_ed25519 does not exist yet cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keysssh-keygenasks for a file name and a passphrase. A coding agent, which cannot answer, runsssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519instead, which makes a key without a passphrase. A key with a passphrase works too, when the key is loaded in your SSH agent.Reason for
chmod 600: MMX refuses anauthorized_keysfile that the account's group can write, because another account could then add a key. A new file in the home directory is group-writable on some systems.Check:
ls -l ~/.ssh/authorized_keysstarts with-rw-------. -
Trust the controller machine's own SSH host key. MMX connects only to a machine whose host key your account already trusts, and the controller machine is no exception. Log in once and answer
yes:ssh barbara@127.0.0.1 trueBefore you answer, compare the fingerprint that
sshshows with the output ofssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub. That command reads the host key file of the machine you are on.A coding agent, which cannot answer the question, compares the output of these two commands. The two outputs must show the same
SHA256:fingerprint:ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub ssh-keyscan -t ed25519 127.0.0.1 | ssh-keygen -lf -Then the coding agent logs in with:
ssh -o StrictHostKeyChecking=accept-new barbara@127.0.0.1 trueCheck:
ssh barbara@127.0.0.1 trueruns again without a question and without a password. -
Add the controller machine to the inventory. Write
127.0.0.1as the address, and your account asssh_user:# inventories/example.yml nodes: controller_machine: # a name you choose for this machine address: 127.0.0.1 # the machine itself ssh_user: barbara # your own account, which can run any command with sudo web01: address: 203.0.113.11 ssh_user: rootCheck:
mmx nodeslists both nodes. The command contacts no machine:Inventory: example (inventories/example.yml, 2 nodes) Nodes controller_machine 127.0.0.1 barbara web01 203.0.113.11 root -
Write a playbook for every node. This playbook installs
unattended-upgrades, Debian's tool for automatic security updates, on every node in the inventory, the controller machine included. Createplaybooks/turn_on_security_updates.yml:# playbooks/turn_on_security_updates.yml fleet_stages: - id: turn_on_security_updates nodes: all # every node in the inventory, the controller machine too node_tasks: - id: install_unattended_upgrades apt_package: packages: - {name: unattended-upgrades, version: newest_when_first_installed}Check:
mmx validate turn_on_security_updatesprintsValid: mmx_example. -
Preview the playbook on the controller machine first.
mmx dryrun turn_on_security_updates --limit controller_machine echo "exit code $?"The first dryrun sets up MMX on the controller machine over SSH. When your account's
sudoasks for a password, MMX asks for the password once:127.0.0.1: setting up MMX through verified SSH sudo password for barbara@127.0.0.1:22: Progress: 1 running turn_on_security_updates / controller_machine: connecting ... turn_on_security_updates / controller_machine: succeeded turn_on_security_updates / controller_machine / install_unattended_upgrades: succeeded, would_changeA coding agent cannot type the password. The dryrun then fails with
sudo password for barbara@127.0.0.1:22 is required. Either you run this first dryrun once in a terminal, or the coding agent follows Passwords for automation.root, or an account whosesudoneeds no password, gets no question.Check:
install_unattended_upgradessayswould_change, and the exit code is 2, which for a dryrun means "something would change" (exit codes). -
Run the playbook on every node.
mmx run turn_on_security_updatesCheck: the run ends with
Run ...: succeededand exit code 0, and both nodes saysucceeded:turn_on_security_updates / controller_machine: succeeded turn_on_security_updates / web01: succeededThis run prints no
setting upline for127.0.0.1and asks for no password, because the run reaches the controller machine over HTTPS on port 7443.mmx results --jsonshows"connection_used": "https"forcontroller_machine.
What MMX keeps on the controller machine as a node
The controller machine now has two roles, and each role keeps its own files:
| What | Where | Owner |
|---|---|---|
| The controller process's saved runs and locks | .mmx/ in the project |
The account that runs mmx |
| The node's node jobs, results, and locks | /var/lib/mmx/ |
root |
mmx-runner, MMX's program for the node |
/opt/mmx/bin/ |
root |
mmx itself runs as your account and needs no root privileges. Only
mmx-runner runs as root. The systemd service mmx-https-api.service
runs mmx-runner and listens on TCP port 7443 on all of the controller
machine's addresses, as on every node. The service starts again after a
reboot. MMX never opens a firewall port itself.
If the mmx command stops during a run
The work on the controller machine runs in an mmx-runner process of its
own, separate from the mmx command. So when the mmx command ends in the
middle of a run, such as when you close the terminal, the work on the
controller machine goes on. Run mmx status later to collect the result,
or mmx resume to collect it and finish the rest of the run; Stop,
resume, and read a run explains both.
Patch every node and reboot the controller machine by hand
After a reboot node task, the controller process waits for the node to
come back, and then the run goes on. A reboot of the controller machine
would end the controller process, and nothing would continue the run. So
MMX does not reboot the controller machine. A fleet stage that ends with a
reboot node task fails on the controller machine, before any node task
of that fleet stage runs on the controller machine.
For example, this playbook upgrades the packages on every node and then reboots each node that an update asks to reboot:
# playbooks/patch.yml
fleet_stages:
- id: patch_and_reboot
nodes: all
node_tasks:
- id: upgrade_packages
apt_package:
upgrade_all_packages_within_their_version_rules: true
- id: reboot_when_an_update_asks
reboot:
run_only_if_this_path_exists: /var/run/reboot-required # a package hook creates this file when a reboot is due, such as after a kernel update
mmx run patch reports:
patch_and_reboot / controller_machine: failed
MMX does not reboot the controller machine, the machine where this mmx command runs. Run the playbook from another machine, or reboot this machine by hand after MMX finishes.
patch_and_reboot / web01: succeeded
patch_and_reboot / web01 / upgrade_packages: succeeded, unchanged
patch_and_reboot / web01 / reboot_when_an_update_asks: succeeded, changed
{"boot_id_after":"c7105a3d-9205-4248-a51e-168ec4f0eebc","boot_id_before":"1e536279-cd9a-482d-8360-0364ccfa8043"}
Run ed3d6667: failed
Read the result this way:
- The controller machine failed, although no update had asked for a
reboot on the controller machine. On the controller machine, MMX refuses
a fleet stage that ends with
reboot, whatever therebootnode task's conditions say. A playbook that reboots the controller machine only sometimes would otherwise work until the day the condition is true. - The packages of the controller machine were not upgraded either, because no node task of the fleet stage ran on the controller machine.
web01went on. The packages ofweb01were already up to date, soupgrade_packagessaysunchanged.web01rebooted, because the file/var/run/reboot-requiredexisted onweb01.- The command exits with code 1, because one node failed.
mmx dryrun patchandmmx resumereport the same failure. The failure counts as any failed node does, soon_node_failureapplies, as Choose what a failed server stops explains.
Split the work so that MMX patches the controller machine and you reboot the controller machine:
-
Leave the controller machine out of the fleet stage that reboots, and patch the controller machine in a fleet stage of its own. Replace
playbooks/patch.ymlwith:# playbooks/patch.yml fleet_stages: - id: patch_and_reboot nodes: all except_nodes: controller_machine # MMX does not reboot the machine where mmx runs node_tasks: - id: upgrade_packages apt_package: upgrade_all_packages_within_their_version_rules: true - id: reboot_when_an_update_asks reboot: run_only_if_this_path_exists: /var/run/reboot-required # a package hook creates this file when a reboot is due, such as after a kernel update - id: patch_controller_machine nodes: controller_machine node_tasks: - id: upgrade_packages apt_package: upgrade_all_packages_within_their_version_rules: trueexcept_nodesremoves nodes from whatnodesselects. Leave some servers out of a fleet stage explainsexcept_nodes.Check:
mmx nodes patchshows the controller machine in the second fleet stage only:Fleet stages patch_and_reboot all except controller_machine → web01 patch_controller_machine controller_machine → controller_machine -
Run the playbook.
mmx run patchCheck: the run ends with
Run ...: succeededand exit code 0:patch_and_reboot / web01: succeeded patch_controller_machine / controller_machine: succeeded -
Reboot the controller machine by hand when an update asks for a reboot. On Debian and Ubuntu:
test -f /var/run/reboot-required && sudo rebootThe command reboots only when the file
/var/run/reboot-requiredexists.Check: after the controller machine is back,
systemctl is-active mmx-https-apiprintsactive, andmmx run patchsucceeds again.
You can also run the playbook from another machine. That machine's controller process can reboot the controller machine like any node and wait for the reboot.
How MMX recognizes the controller machine
- MMX recognizes the machine, not the address. The refusal is the same
when the inventory reaches the controller machine as
127.0.0.1, by a host name, or by another of its addresses. The controller process sends the kernel's boot ID of its machine, from/proc/sys/kernel/random/boot_id, and a SHA-256 hash of the machine's systemd machine ID, from/etc/machine-id. The node compares the two values with the node's own values. Therebootnode task compares the values once more, just before the reboot. - Only the
rebootnode task is guarded. MMX does not read the commands in yourshellnode tasks or scripts. No setting turns the guard off: a playbook that must reboot the controller machine runs from another machine.
The controller process runs in a container
- A Docker container, such as a CI job. A process in a Docker container reads the boot ID of the server that runs the container. So MMX refuses to reboot that server, because the reboot would end the container and the controller process in the container.
- An LXC or Incus container. Such a container has a boot ID of its
own, new at each start. So MMX does not recognize the server that runs
the container, and reboots that server when a playbook says so. The
reboot ends the controller process. Leave that server out of the fleet
stage that reboots with
except_nodes. - A container that is a node on the controller machine can reboot. The reboot ends only that container, and the controller process goes on.
Module pages that rely on this section: reboot.
If something is different
| What you see | What it means and what to do |
|---|---|
MMX does not reboot the controller machine, the machine where this mmx command runs. Run the playbook from another machine, or reboot this machine by hand after MMX finishes. |
A fleet stage that ends with reboot selects the controller machine. See Patch every node and reboot the controller machine by hand. |
no SSH signing key on the controller machine matches the administrator account's SSH access (unsafe SSH authorization path /home/you/.ssh/authorized_keys: group or other accounts can write this file, so sshd and MMX do not trust it; run chmod go-w /home/you/.ssh/authorized_keys); ... |
The message names the file or directory that another account could change and says what to run: chmod go-w on that path, or chmod 600 ~/.ssh/authorized_keys for the file, as in step 2. When the message says the path belongs to another account, give the path to the administrator account or to root with chown. Each directory from / down to the file must belong to root or to the account. |
sudo password for barbara@127.0.0.1:22 is required; configure an environment variable for the credential or use an SSH key or ssh-agent (prompts are disabled) |
The first contact needs your sudo password, and MMX may not ask. Run the first dryrun from a terminal, or see Passwords for automation. |
SSH localhost: exit status 255: No ED25519 host key is known for localhost and you have requested strict checking. |
Your account does not trust the controller machine's host key under the address in the inventory. Do step 3 with that address. |
SSH localhost: exit status 255: ssh: connect to host localhost port 22: Connection refused |
No SSH server runs on the controller machine. Do step 1. |
this machine does not run Linux: it answered "Darwin" (macOS) to the command uname -s. MMX manages Linux nodes only, and a macOS controller machine cannot be a node. ... |
The inventory names your Mac as a node. Remove the Mac from the inventory, or leave the Mac out of the fleet stage with except_nodes. |
Where to go next
- Connect to your servers: passwords, a jump host, and the other ways to reach a node.
- Choose which servers a run touches:
--limit,except_nodes, and one inventory for each environment. - The
rebootmodule: every input of therebootnode task. - The
.mmx/directory and what MMX keeps on a node: what MMX keeps in.mmx/on the controller machine and in/var/lib/mmx/on each node.
This page is docs/how-to/manage-the-controller-machine-itself.md in the MMX repository.