When something fails

Who acts: you, or your coding agent, on your controller machine: the computer where you run mmx. Some fixes are commands on a server, as root.

When to use this page: mmx printed an error, or a run did something you did not expect, and you want the cause and the fix.

What this page does: for each message, the page says what the message means and what to do, and links the page that explains the topic. Search this page for a few words of your message, such as machine ID or uname -s.

Terms:

  • A node is a Linux server that MMX manages. The inventory, inventories/<name>.yml, lists your nodes and how to reach them.
  • A playbook lists fleet stages. Each fleet stage picks nodes, and each of those nodes runs the fleet stage's node tasks, in order.
  • A node job is the work of one fleet stage on one node.
  • mmx-runner is MMX's program on each node. The first run on a server logs in over SSH and installs mmx-runner. Later runs connect over HTTPS on port 7443, or over SSH when port 7443 is closed.

The examples use the project shop, the servers web01, web02, and so on at 203.0.113.11, 203.0.113.12, and so on, and the playbook example that mmx init creates.

Find the part of this page for the moment the message appeared:

The message appeared Go to
Before any server line, often with exit code 4 Before MMX connects to a server
On a server's first line, often after connecting or setting up MMX Connecting to a server and setting it up
On a node task's line, during a run During a run
From mmx dryrun with exit code 3, or from mmx status, mmx resume, or mmx results After a run
About the mmx program itself On the controller machine

Before MMX connects to a server

mmx checks the command, the project files, and the secrets on your computer before the command contacts a server. A failure here changes nothing on any server. Most of these failures end with exit code 4, and mmx validate ends with exit code 1.

What you see What it means and what to do
mmx: no mmx_project_settings.yml found in the current directory or its parents The current directory is not inside an MMX project. Change to the project directory, such as cd ~/shop. To start a project, see Get started.
mmx: there is no playbook set_up_web_servers: playbooks/set_up_web_servers.yml does not exist; the project's playbooks are: example No playbook file has that name. A playbook's name is its file name in playbooks/ without .yml. Use a name from the list.
mmx: dryrun requires a playbook name; available playbooks: example mmx dryrun and mmx run need a playbook name, such as mmx dryrun example.
mmx: unknown command "frobnicate" mmx help lists the commands. The commands reference explains each command.
mmx: playbooks/broken.yml: yaml: line 4: did not find expected ',' or ']' The file is not valid YAML, such as a list with no closing ]. Fix the line that the message names, then run mmx validate. The playbook file reference shows the fields.
mmx validate names a line, a column, and a field, such as (fleet_stages[0].node_tasks[0].shell.command): unknown field A field name is wrong or in the wrong place. See Write a playbook.
A message that starts with mmx: --limit, such as mmx: --limit: no node or inventory group named "web99" MMX refuses a limit before it contacts a server. See Choose which servers a run touches for each message.
mmx: vault passphrase required; use --vault-password-file, MMX_VAULT_PASSWORD_FILE, or MMX_VAULT_PASSWORD (prompts are disabled) The run needs a secret from the vault, and mmx may not ask for the passphrase. See Keep secrets.
secret database_password requires a nonempty environment variable SHOP_DATABASE_PASSWORD A secret comes from an environment variable that is not set. Set the variable. See Keep secrets.
reference .vars.http_port is not declared for node web01: declare it under vars in mmx_project_settings.yml or in inventories/example.yml (the inventory, a group, or the node), in command_line_vars of the playbook, or set it earlier in this fleet stage with set_vars A template reads a variable that no file declares for that node. Declare the variable in one of the places that the message lists. See Use variables and templates.

Connecting to a server and setting it up

The first contact with a server goes over SSH. MMX checks the server and the login account, and then installs mmx-runner. A failure here stops that server's work. The other servers go on, unless the fleet stage's on_node_failure stops them.

SSH cannot log in

What you see What it means and what to do
SSH replace-with-your-node: exit status 255: ssh: Could not resolve hostname replace-with-your-node: nodename nor servname provided, or not known The inventory still holds the address that mmx init wrote. Put your server's address in inventories/example.yml. On Linux, the message ends with Name or service not known.
ssh: connect to host 203.0.113.11 port 22: Connection timed out Your computer cannot reach the server's SSH port. See Connect to your servers.
Host key verification failed, or WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! Your computer does not trust the server's SSH host key, or the key changed. See Connect to your servers.
Too many authentication failures Your SSH agent offered too many other keys first. See Get started.
SSH password for deploy@203.0.113.12:22 is required; ... (prompts are disabled) The server needs a password, and mmx may not ask. See Passwords for automation.
account deploy does not have unrestricted root sudo access The login account must be root, or able to run any command with sudo. See Connect to your servers.

The first check of the server fails

MMX's first command over SSH asks the server for its system, its architecture, whether systemd runs it, and its systemd machine ID. The notice setting up MMX through verified SSH appears only after this first check passes, so a server that fails here shows no such notice. The machine ID is the 32 hexadecimal digits in the server's /etc/machine-id. MMX tells servers apart by the machine ID. The server fails at once when an answer is wrong, and the message says what MMX found.

The answer starts with other text. A login script of the SSH account, such as /root/.bashrc, prints text even when no person is logged in:

welcome / web03: failed
  the answer of this node to the command uname -s started with "Welcome to the shop server", and MMX expects the one word Linux there. If a login script of the SSH account prints this text, make it print nothing when the session has no terminal

Make the script print only when a terminal is attached. In /root/.bashrc on the server, change echo Welcome to the shop server to:

if [ -t 1 ]; then echo Welcome to the shop server; fi

Check: ssh root@203.0.113.13 uname -s prints only Linux. The text still prints once in the session that runs the fix, so check in a new session.

The answer is empty. The SSH account runs a fixed command instead of the commands that MMX sends:

welcome / web02: failed
  this node answered nothing to the command uname -s, and MMX expects the one word Linux there. The SSH account must run the commands that MMX sends

Look for a command="..." option before the key in the account's ~/.ssh/authorized_keys on the server, and for ForceCommand in /etc/ssh/sshd_config. Remove the fixed command for the key that MMX uses, or give MMX an account without one. On a server that MMX already set up, the same key fails over HTTPS with HTTPS 403: key is not authorized for unrestricted MMX access as root.

The account's login shell refuses commands.

welcome / web07: failed
  SSH 203.0.113.17: exit status 1: the first command of MMX printed nothing and exited with status 1. The likely cause is that the login shell of the SSH account refuses commands, as /usr/sbin/nologin and /bin/false do. Give the account a shell such as /bin/sh, or use another ssh_user

Check the account's shell on the server with getent passwd deploy. The last field must be a shell such as /bin/sh. Give the account that shell, or set another ssh_user in the inventory, such as root or another account that can run commands and sudo.

The server does not run Linux.

this machine does not run Linux: it answered "Darwin" (macOS) to the command uname -s. MMX manages Linux nodes only, and a macOS controller machine cannot be a node. Remove this node from the inventory, or leave it out of the fleet stage with except_nodes

mmx-runner runs on Linux only. Another system shows its own name in place of "Darwin" (macOS). Remove the server from the inventory, or leave it out with except_nodes. A Mac in its own inventory is covered in Manage the controller machine itself.

The server's processor is not supported. MMX ships mmx-runner for 64-bit x86 (x86_64) and 64-bit ARM (aarch64). A server with another processor, such as a 32-bit Raspberry Pi, fails with unsupported node architecture armv7l. MMX cannot manage that server. Limits on each distribution lists what MMX supports.

systemd does not run the server.

welcome / web04: failed
  this node does not run systemd: the directory /run/systemd/system, which exists on every machine that systemd started, is not there. MMX manages only Linux servers that run systemd. Remove this node from the inventory, or leave it out of the fleet stage with except_nodes

A server answers this way when another init system runs it, such as OpenRC on Alpine Linux. MMX installs mmx-runner as a systemd service and cannot manage the server. Remove the server from the inventory, or leave it out with except_nodes. Limits on each distribution lists the tested distributions.

The machine ID is missing or empty.

welcome / web05: failed
  this node has no systemd machine ID: its file /etc/machine-id is missing or empty. MMX identifies a node by the systemd machine ID. Run systemd-machine-id-setup as root on the node: it writes a new ID into the file. Then run MMX again

As root on the server:

systemd-machine-id-setup

Check: cat /etc/machine-id prints 32 hexadecimal digits. Then run mmx again.

The machine ID is not valid.

welcome / web05: failed
  this node's systemd machine ID is not valid: its file /etc/machine-id holds "uninitialized", not 32 hexadecimal digits. MMX identifies a node by the systemd machine ID. As root on the node, remove the file and run systemd-machine-id-setup: it creates the file with a new ID. Then run MMX again

systemd-machine-id-setup leaves a file that holds a word such as uninitialized as it is, so remove the file first. The command may also reuse /run/machine-id, so on a server cloned from an image, remove both files. As root on the server:

rm -f /etc/machine-id /run/machine-id
systemd-machine-id-setup

Check: cat /etc/machine-id prints 32 hexadecimal digits.

First contact: setup over SSH explains what the first check does and why it comes first.

Two inventory names reach one server

Two inventory names with different addresses must reach two servers. MMX compares the machine ID and the boot ID of each server in a run. The boot ID is a new random ID at each start of a server. When two names reach the same machine ID and the same boot ID, the second name fails before MMX sends any work:

welcome / web06: failed
  inventory nodes web01 and web06 reach the same machine: both answered with the same systemd machine ID and the same boot ID. MMX ran nothing for web06 on that machine, because the machine is web01's in this run. Check the addresses of web01 and web06 in the inventory: one of them probably names the wrong server; two names for one server need the same address. When an address is a Host alias in an SSH configuration file (~/.ssh/config, or the node's ssh_config), the wrong value may be that alias's HostName. Some containers share the boot ID of their host, so each container needs its own /etc/machine-id

Without this check, both names would run on one server, and the server that the second name means would never change. Which name fails depends on which server answers first.

  • An address is wrong: correct address in the inventory. When the address is a Host name from ~/.ssh/config, such as shop-web06, check the HostName line of that Host entry too.
  • Two names for one server on purpose, such as a load balancer that runs on an application server: give both names the same address and ssh_port. MMX then runs both names on the server.
  • Containers on one server: some containers share the server's boot ID. When such containers come from one image, they can share a machine ID too. Give each container its own machine ID, as The first check of the server fails shows for an invalid machine ID.

Virtual machines cloned from one image may share a machine ID, but each clone starts with its own boot ID. So MMX runs the clones as separate servers.

HTTPS and the setup of mmx-runner

What you see What it means and what to do
MMX could not connect to https://203.0.113.11:7443 (connection timed out). and a notice that MMX is using SSH Your computer could not reach port 7443 on the server. The run went on over SSH. The notice says what MMX found on the server, such as whether mmx-runner runs and whether the server's firewall has a rule for the port. See A server whose port 7443 is closed.
HTTPS 403: key is not authorized for unrestricted MMX access as root: its line in authorized_keys has options MMX does not accept (command, from); remove them, or use connection: ssh_only for other SSH authorization mechanisms The key's line in the account's authorized_keys carries options that MMX cannot check, such as command or from. Remove the options, or set connection: ssh_only on the node. See Login rules that only OpenSSH can check.
HTTPS 403: key is not authorized for unrestricted MMX access as root; check current SSH keys or use connection: ssh_only for other SSH authorization mechanisms mmx-runner did not find your key in the account's authorized_keys. See Connect to your servers.
automatic setup of the node runner executable from http://...: ... curl: (22) The requested URL returned error: 404 The server could not download mmx-runner from mmx_runner_download_url. See Connect to your servers.
HTTPS 409: node busy with job ..., or node busy with job ... while MMX sets up a server A node job of another run, maybe from another computer, works on the server. See Move or share a project.
cannot replace the node runner executable while owed restore unit ... exists; resolve the owed restore first The server still owes the restore of a network file. See Upgrade MMX.

During a run

When a node task fails on a server, that server runs none of its later node tasks or fleet stages, except a failed node task block's own on_failure and cleanup node tasks. The other servers go on, unless the fleet stage's on_node_failure stops them. Preview and run changes explains each result.

What you see What it means and what to do
this node refuses management because /etc/mmx/do_not_manage exists; the file says "...". A dryrun prints a run would be refused: ... Someone keeps MMX off the server on purpose, and the file says why. The node refuses at the start of each node job, before any node task runs. See Keep MMX off a server.
A node task's line ends with a long reason The reason comes from the module: what was missing, and what to do. mmx describe-module NAME lists the module's errors, such as mmx describe-module apt_package.
The run fails on one server and succeeds on the others The other servers are finished. Fix the cause, then run mmx resume or mmx run: Resume a run says which.
stopped with not started: on_node_failure stop_run after ... failed on web03, or blocked with requires same_node fleet stage ... A failure on another server stopped this work, as the playbook asks. See Roll out a change gradually.
A node task is blocked, and its reason names an earlier node task, a fleet stage output, or the limit The node task needs a value that no earlier work produced, such as an output of a server that failed or that --limit leaves out. See Use a result from one server on another and Choose which servers a run touches.
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. The server stopped inside dpkg during an earlier package change, such as at a power loss. See Packages after an interrupted upgrade.
another apt process still holds the lock on the package lists after 1m0s, and apt's line E: Could not get lock /var/lib/apt/lists/lock. It is held by process 1816 (python3) Another program used apt on the server for longer than MMX waits: one minute in each try, and a node task tries three times by default. Such a program can be the automatic security updates, which may run for minutes after a new server starts. Wait until that program ends, then run again. ps -p 1816 on the server shows the program that apt names. See apt_package.
node job interrupted before recording completion The node job worker process on the server stopped before the node job ended, such as at a reboot. See Change a configuration and reload its service.
details suppressed for a task using secrets; ... MMX hides the output of a failed node task that uses secrets. The message says where root can read it on the server. See Keep secrets.
After Ctrl-C or mmx stop, mmx status still says running for a server Each server finishes its current node task, or its whole node task block, before it stops. Run mmx status again later. See Stop a run.

Packages after an interrupted upgrade

A server that stops inside dpkg, such as at a power loss during an upgrade, refuses each later package change until someone finishes the work. A run with apt_package then fails at once, with no new tries, because another try changes nothing. The reason of the node task reads:

dpkg on the node stopped in the middle of a package change. Run dpkg --configure -a on the node first, then run MMX again: exit status 100

The line E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. stays in the standard error of the node task. mmx resume works after the fix.

  1. Finish the interrupted work on the server. As root on the server:

    dpkg --configure -a
    

    Check: apt-get check exits with code 0 and prints no line that starts with E:. Before the fix, apt-get check prints the same dpkg was interrupted line.

  2. Run the playbook again from your computer, such as mmx run upgrade_packages.

    Check: the run ends with Run ...: succeeded.

After a run

What you see What it means and what to do
A dryrun's exit code is 3, and lines say not_checked The dryrun could not check some work, often because an earlier node task would install something first. See What the results mean.
mmx status exits with code 3 and says unconfirmed; work may still be running MMX cannot reach the server to learn how its work ended. Start the server again, or, if the server is gone for good, see A server that is gone.
node was replaced since this run (machine identity changed); its recorded work belongs to the previous machine; use mmx run You resumed a run from before the server was rebuilt or replaced. Start a new mmx run. See A server that was replaced.
mmx: this playbook run was started by MMX 0.1.0, but this mmx program is version 0.1.1. ... after mmx resume You installed another release of MMX after the run started. See Finish a run that an earlier release started.
mmx: no matching saved run, Nothing to resume., mmx: cannot resume a dryrun; ..., or mmx: run ID 2a33c932 is ambiguous; use the full run ID The command found no run, or another run than you meant. See Stop, resume, and read a run.
mmx: history cleanup: MMX cannot read this saved run, which another version of MMX may have written: ... A saved run from another release blocks the removal of old runs. See Upgrade MMX.

On the controller machine

What you see What it means and what to do
mmx: command not found The shell does not find mmx. Add ~/.local/bin to your PATH, as Get started step 1 says.
mmx: the official extra modules in /home/you/.local/lib/mmx/official_extra_modules were shipped with MMX 0.1.0, but the mmx program is version 0.1.1; ... Only the mmx program was replaced, not the whole directory from the release archive. See Upgrade MMX.
the node runner executable /home/you/.local/lib/mmx/mmx-runner-linux-amd64 reports MMX version 0.1.0, but this mmx program is version 0.1.1. ... The mmx-runner files beside mmx come from another release. MMX checks the versions before a node runs any node task. See Upgrade MMX.
A message that names localhost or 127.0.0.1, or MMX does not reboot the controller machine, ... The inventory names your own computer as a node. See Manage the controller machine itself.

Where to go next

This page is docs/when-something-fails.md in the MMX repository.