How MMX works
This page follows one playbook from your keyboard to your servers and back. For each part of MMX, this page says what happens, on which machine, and why the step is safe. The documentation index lists the pages that show how to do each job, and the reference pages for every command and setting.
The controller machine and the nodes
You run mmx commands on your controller machine, usually your laptop
or a continuous integration (CI) machine. The Linux servers that MMX
manages are called nodes. A node must run systemd.
MMX has two programs, both written in Go:
mmxruns on the controller machine. Each runningmmxcommand is a controller process, which ends when the command finishes.mmx-runner, the node runner executable, runs on each node. The firstmmxcommand that connects to a node installsmmx-runneron that node.
CONTROLLER MACHINE (your laptop) NODES (your Linux servers)
you type: mmx run set_up_web_servers web01 web02
mmx reads the project files -- work --> each node does its own work
mmx shows and saves results <-- results -- and keeps its own results
The controller machine decides which nodes work, and when. Each node does
its work on itself and keeps its own results. So a node finishes its
current node job even if your laptop goes to sleep in the middle of a run.
The later fleet stages wait until you run mmx resume.
No MMX service runs on the controller machine between commands. On each
node that MMX reaches over HTTPS, which is every node without
connection: ssh_only, mmx-runner runs as the systemd service
mmx-https-api.service, so that mmx can reach the node over HTTPS.
A Linux controller machine can also be a node in its own inventory. MMX then never reboots the controller machine: Manage the controller machine itself says how to set that up.
Your project describes your servers
An MMX project is a directory of plain files on the controller machine.
Keep the project in Git, so that each change to your servers starts as a
commit that you can review and undo.
This page uses the project shop, which has two web servers:
shop/
mmx_project_settings.yml the project's settings, by which MMX finds the project
inventories/example.yml the inventory: your servers and how to reach them
playbooks/set_up_web_servers.yml a playbook: the work for your servers
roles/ reusable lists of work (optional)
external_modules/ modules of the project's own (optional)
secrets_vault.age the encrypted vault of secrets (optional)
.mmx/ what MMX saves on the controller machine, kept out of Git
You can run mmx from any directory inside the project. MMX looks for
mmx_project_settings.yml in the current directory and then in each
parent directory, as git looks for a repository.
The inventory lists the nodes, how to reach each node over SSH, and groups of nodes:
# inventories/example.yml
nodes:
web01: # a node name you choose
address: 203.0.113.11
ssh_user: root # root, or an account that can run any command with sudo
web02:
address: 203.0.113.12
ssh_user: root
groups:
web_servers: # a group name you choose
nodes: [web01, web02]
A playbook lists the work. The playbook set_up_web_servers installs
nginx, writes one site file, and reloads nginx only when the site file
changed:
# playbooks/set_up_web_servers.yml
fleet_stages:
- id: set_up_nginx # a fleet stage
nodes: web_servers # the nodes that do this fleet stage's work
node_tasks: # each node of the group runs these node tasks, in order
- id: install_nginx
apt_package: # a module
packages:
- {name: nginx, version: newest_when_first_installed}
- task_block: # a node task block
id: shop_site
changes: # first, make the change
- id: write_site_file
file:
path: /etc/nginx/conf.d/shop.conf
permissions_mode: "0644"
content: |
server {
listen 8080;
server_name {{ .node.name }}.example.com;
root /srv/shop/public;
}
activate: # then, put the change into use
- id: reload_nginx
systemd_service:
name: nginx
start_at_boot: true
state: running
refresh: reload
A playbook has these parts:
- A fleet stage is one step of a playbook. The fleet stage picks
nodes, such as the group
web_servers, and lists the work that each selected node does. Fleet stages run in the order you write them: for example, a fleet stage for the database servers, then one for the web servers. - A node task is one piece of work on one node, such as
install_nginx. Each node runs its own copy of the node tasks, in order. - A module is the code that does the work of a node task, such as
apt_package,file, orsystemd_service. A node task names one module, with the module's inputs below the module's name. - A node task block ties a change to what follows the change. In the
node task block
shop_site, nginx reloads only when the site file changed. A node task block can also check the change before nginx loads the change, and check the site afterwards. - A role is a list of node tasks in its own file, with inputs, that
several playbooks can call.
mmxreplaces each role call with the role's node tasks whenmmxreads the playbook, so the role's node tasks run on a node like any other node task.
{{ .node.name }} is a value that MMX fills in for each node: web01 on
web01, web02 on web02. Template files and values that only the node
knows, such as the node's memory size, are filled in on the node.
Write a playbook builds the
set_up_web_servers playbook step by step.
Reuse work with roles,
Use variables and templates, and
Roll out a change gradually cover
roles, values, and how many nodes work at once.
What happens when you run a playbook
Here is what mmx run set_up_web_servers does, in order.
On the controller machine:
- The controller process reads the project settings, the inventory, the playbook, and the roles. The controller process checks everything that it can check without a node, such as unknown fields and missing role inputs. A mistake stops the command before any node is contacted.
- The controller process gives the run an ID. The controller process
saves a copy of the playbook, the inventory, and every file, template,
and module to send, in
.mmx/runs/<run-id>/. The record of one run, with the run's results, is a playbook run.mmx resumelater uses the saved copy, even after you edit the project. - The controller process starts the first fleet stage. The controller
process starts as many nodes at once as the fleet stage's
max_nodes_at_onceallows.
For each node:
- The controller process connects to the node. The first connection sets up MMX on the node over SSH, as The two connections describes.
- The controller process records an ID for the node's work, then sends the node job: the work of one fleet stage on one node, with the files that the work needs. The files travel apart from the node job's instructions, so a large file needs no setting. A file that the node already keeps from an earlier run, such as a module file, is not sent again. The node job's instructions are JSON of at most 16 MiB: they hold each file's size and SHA-256, not the file's bytes, so the limit does not apply to the files. A file upload may take longer than 30 seconds as long as it makes progress. If no bytes move for 30 seconds, the transfer ends and the node job does not start. The node removes a file upload for a node job that never started after 72 hours.
- On the node,
mmx-runnerchecks the SHA-256 of each file, and saves the node job in/var/lib/mmx/jobs/<job-id>/. Thenmmx-runnerstarts a node job worker process for the node job. A node runs one node job at a time, and refuses a second node job as busy.
Once the node job worker process starts, the node job does not depend on the connection. If your laptop sleeps or the network drops, the node job worker process keeps going.
On the node:
- The node job worker process first looks for the file
/etc/mmx/do_not_manage. A person creates that file by hand to keep MMX off a server while the person works on the server. If the file exists, the node job fails before any node task runs: Keep MMX off a server explains the file. - The node job worker process measures the node's facts, such as its distribution and memory, once for the node job. Then the node job worker process runs the node tasks in order. The node job worker process applies each node task's conditions, loops, retries, and time limits.
- Each module checks the node's current state and changes only what differs. The node job worker process saves progress as the work goes, and then the result, on the node.
Back on the controller machine:
- The controller process collects the progress and the results, shows them, and saves them in the playbook run.
- When every node of the fleet stage has finished, the controller process starts the next fleet stage. A value that one fleet stage passes to a later fleet stage travels through the controller process, because nodes never contact each other: Use a result from one server on another shows how.
- After the last fleet stage, each node where every fleet stage
succeeded records the run in
/var/lib/mmx/last_successful_run/<project_name>/<playbook>. Monitoring on the node can then see when the playbook last succeeded on the node.
Every result keeps two facts apart: whether the node task succeeded,
and whether the node task changed anything. A node task can fail after
it changed something, or succeed without changing anything. Run the
playbook again with nothing new, and each node task reports unchanged.
A dryrun shows the changes without making them
mmx dryrun set_up_web_servers follows the same path as a run, with node
jobs and facts, but each module only checks:
- A core module inspects the node and reports what it would change.
- A
shellnode task runs its owndryruncommand, if it has one. Without one, MMX cannot know what the command would do, so the dryrun skips theshellnode task and reports its change asunknown. - A module of the project's own runs in a check mode, and its author makes sure that the check mode changes nothing: Modules says how.
MMX's own modules change nothing on a node during a dryrun, apart from
MMX's own setup. A shell node task's dryrun command, a program
module's check mode, and a script's run_command still run as root, and
their authors make sure that they change nothing. The first
dryrun on a new server installs mmx-runner on the server and starts
mmx-https-api.service. For a server that takes only a password, the
setup also adds MMX's own SSH key to the login account.
Here is the first dryrun of Get started:
Run fbe759f2
Project: mmx_example Playbook: example
Inventory: example (1 node) Selected: 1 node
Output: nodes (change with --output=nodes or --output=summary)
Progress: 1 not started
203.0.113.11: setting up MMX through verified SSH
Progress: 1 running
welcome / web01: connecting
welcome / web01: succeeded
welcome / web01 / task_1: succeeded, would_change
welcome / web01 / task_1: would create "/etc/mmx-example"
absent -> 16 bytes, permissions_mode 0644, uid 0, gid 0
Run fbe759f2: succeeded
1 finished
Saved: /home/you/shop/.mmx/runs/20261115T035229.429039873-fbe759f2091942e3
The line setting up MMX through verified SSH is the setup. The line
would create "/etc/mmx-example" is the change that a run would make.
Why a dryrun cannot check some work
A dryrun looks at the node as the node is now. The dryrun does not know
what an earlier node task will create when the run makes the earlier
change. On a server without nginx, the directory /etc/nginx/conf.d does
not exist yet, because the nginx package creates the directory. So the
dryrun cannot check the site file in /etc/nginx/conf.d, and reports the
node task write_site_file as not_checked, naming install_nginx. The
run then installs nginx first, and writes the site file.
A first dryrun on a new server can therefore report work as not_checked
that the run then does.
Preview and run changes
lists each case and each result.
Stopping, failures, and resume
A failed node task stops that node's work for the rest of the run.
Other nodes go on, or stop, as the fleet stage's on_node_failure setting
says. MMX does not undo a change that a node task already made.
A stop, with Ctrl-C or mmx stop, starts no new node job. A node that
is working finishes its current node task, or its whole current node task
block, and then stops. So a stop does not fall between writing a
configuration file and reloading its service.
A controller process that goes away, such as when your laptop closes,
leaves each started node job running on its node. No new node job starts,
because only a controller process starts node jobs. So a rollout does not
spread to more nodes without you. mmx status collects the results later.
If the controller machine and its .mmx/ directory are lost, the saved
runs are lost with them. Start a new run from a copy of the project:
Move or share a project
says how, and the nodes report unchanged for work that is done.
A lost answer is checked before anything is sent again. The controller
process records the node job's ID before the controller process sends the
node job. If the connection drops before the node answers, the node job's
outcome is unknown. mmx status and mmx resume then first ask the
node about that ID:
- A node job that still runs is followed to its end.
- A node job that finished keeps its result and does not run again.
- If the node has no record of the ID,
mmx resumesends a new node job. A missing record does not prove that the first node job never ran, so ashellcommand that must not run twice needs a guard. - If the node cannot be reached, the outcome stays
unknown.
A node job worker process that dies, such as when the node loses
power, leaves its node job failed, with the reason node job interrupted before recording completion. mmx resume or the next mmx run checks
the node again.
mmx resume repeats the failed and unfinished node jobs of a run,
with the files that the run saved. A repeated node job runs all of the
fleet stage's node tasks on that node from the first one. Work that is
already done reports unchanged, because each module checks the node
first. A shell node task runs its command again, unless a guard skips
the command.
A reload that a failure interrupted still happens on the next run.
Suppose the run writes the new site file, and then fails before nginx
reloads. On the next run, the site file is already correct, so
write_site_file reports unchanged. But the node recorded which version
of the site file nginx last loaded, so reload_nginx still reloads nginx.
The node keeps the record of what each service loaded in
/run/mmx/services/ until the node reboots. A changed file or checked-out
commit uses the node task block's refresh choice, reload or restart;
a changed package or unit file always restarts. After a reboot, the first
run records what each running service loaded as the starting point, so a
reboot alone causes no extra reload.
Reload or restart
has the table.
A run from another MMX release cannot be resumed. mmx resume refuses
a run that another release of mmx started, before the command contacts
any node. A run that two releases finish could leave a node without its
record of the last successful run.
Upgrade MMX
says how to finish such a run.
Stop, resume, and read a run has the commands and the messages.
The two connections: SSH and HTTPS
The controller machine talks to a node over SSH or over HTTPS, and over
nothing else. Both connections lead to the same code in mmx-runner, so a
node job works the same whichever connection carries it.
First contact: setup over SSH
The first command that connects to a node, even a dryrun, uses OpenSSH:
the same ssh you use, with your SSH configuration, keys, and agent. The
node's SSH host key must already be in your known_hosts file. MMX never
accepts an unknown host key itself, so OpenSSH's host key check decides
which server MMX trusts.
MMX's first SSH command asks the node what it runs:
uname -smust answerLinux, and systemd must run the node.- The node's machine ID, the 32 hexadecimal digits in
/etc/machine-id, identifies the node to MMX. - The kernel's boot ID, a new random ID at each start of the node, tells a machine apart from a clone of the same disk image.
A node that is not Linux, has no systemd, or has no valid machine ID fails at once, with what MMX found and what to do.
Then, over SSH, as root or through sudo, MMX:
- installs
mmx-runnerin/opt/mmx/bin/on the node, and checks thatmmx-runnercomes from the same MMX release asmmx - starts the node HTTPS API process,
mmx-runner serve, as the systemd servicemmx-https-api.service, which listens on TCP port 7443 - reads the node's HTTPS identity, called its pin: the SHA-256 of the node's HTTPS public key
- saves a SHA-256 hash of the machine ID, and the pin, on the controller
machine, in the project's
.mmx/nodes/
The pin arrives over SSH, which you already trust. So MMX can later trust the node's HTTPS connection without a public certificate authority.
Each project saves its own copy of a node's identity, in its own
.mmx/nodes/. A second project that manages the same node, or a
teammate's copy of the project, sets up MMX on the node again over SSH.
That setup finds MMX installed and keeps the node's HTTPS key, so the pin
that the first project saved stays valid.
A server that takes only a password gets an SSH key from MMX. MMX
asks for the password, creates its own SSH key in ~/.ssh/mmx/ on the
controller machine, and adds the key to the account's authorized_keys.
Later connections need no password. When you change ssh_port on such a
node, MMX copies MMX's own key for the new port only after a check.
Verified SSH at the new port must reach the same machine ID as before:
Connect to your servers
shows the message.
Normal work: HTTPS on port 7443
After setup, the controller process connects to the node over HTTPS on port 7443. The controller process and the node check each other:
- The controller process checks the node: the node must present the HTTPS key that matches the saved pin.
- The node checks you: the controller process signs a proof with your SSH key. The proof names the pin, the login account, and a value unique to the one HTTPS connection, so nobody can reuse the proof on another connection. Your SSH private key stays on the controller machine.
- The node checks your access against the login account's
authorized_keysfiles and sshd's allow and deny lists. The node does not run PAM, and refuses login rules that the node cannot check, such asForceCommand: Login rules that only OpenSSH can check lists them. The account must be root, or able to run any command withsudo. Over HTTPS the node asks for nosudopassword, so the SSH key alone gives root.
The node checks your access again for each new connection. Remove your key from the node, and your next connection is refused. A connection that is already open keeps its access until the connection closes, and node jobs that already started go on.
When HTTPS is not available
If the controller process cannot reach port 7443, the controller process does the same work over SSH. MMX then prints a notice that says what MMX checked on the node. MMX never opens a firewall port itself. If the node refuses your key over HTTPS, MMX does not try SSH, and reports the error.
Set connection: ssh_only on a node in the inventory to use SSH for all
work. You need ssh_only when only OpenSSH can check a login rule, such
as SSH certificates or a second login factor.
Connect to your servers
covers the notice and the choices.
One server, one inventory name
Two inventory names with different addresses must reach two machines. MMX compares the machine ID and the boot ID that each name's connection reports. When two names in one run reach the same machine, the second name's node job fails before anything is sent. The usual cause is a typo in an address. Without the check, both inventory names would run on one server, and the server that the second name means would never be touched.
Two names with the same address and port, such as a load balancer on an application server, are a choice that the inventory shows, and both run. Virtual machines cloned from one image may share a machine ID, but each clone has its own boot ID, so the clones run as separate nodes.
A replaced server
A rebuilt server has a new SSH host key. After you trust the new host key
in known_hosts, the next mmx run finds that the saved identity no
longer matches. The run sets up MMX on the new machine over SSH, saves the
new identity, and says so. mmx resume and mmx status refuse to apply
an older run's work to the new machine, because the older run's saved
progress describes the old machine.
A server that was replaced
shows the steps.
Both programs come from one release
mmx and mmx-runner must come from one MMX release. After MMX installs
mmx-runner and before the node runs any node task, MMX compares the
release that mmx-runner reports with the release of mmx, and fails the
node when the two differ. After you
upgrade mmx, the next command installs the new mmx-runner on each
node that the command connects to. Upgrade MMX
covers the upgrade.
Modules
A module is trusted code. A module runs as root on your nodes, so use only modules that you would trust with root access to your servers.
Every node task names a module. A module has one of three kinds, and a node task names each kind the same way:
- Core modules are built into
mmx-runner. Core modules cover everyday work, such as packages, files, accounts, services, and firewalls. - Official extra modules are scripts in
Starlark, a language that
reads like Python. The MMX project writes and tests the official extra
modules for less common tools, such as ZFS. The official extra modules
ship beside
mmx, from the same release asmmx. - Modules of the project's own live in the project's
external_modules/directory: a Starlark script, or any program for a special need.
MMX looks a module name up among the core modules first, then the
official extra modules, then external_modules/. A module that is not
core travels to the node with the node job, like any other file.
On the node, the node job worker process runs each module as root:
- A Starlark script reads and changes the node only through MMX's
primitives, functions such as
write_filethat the core modules use too. In a dryrun, the primitives change nothing and report what they would change. - A program runs as a child process and gets its inputs as JSON. In a dryrun, the program runs in a check mode and must change nothing. MMX cannot see what a program does, so the program's author makes sure of that.
MMX does the rest for every kind of module: which nodes run the module,
conditions, retries, time limits, stopping, secrets, and saved results.
A command that MMX runs as root gets a small fixed environment, with
HOME and LOGNAME set for root:
The environment of a command
lists the environment.
Find the module for a job and Write your own module cover choosing and writing modules.
How secrets travel
A secret, such as a database password, comes from the vault
secrets_vault.age in the project, or from an environment variable on the
controller machine. Suppose a node task writes a secret into a file:
- id: write_database_settings
file:
path: /etc/shop/database.env
content: "DATABASE_PASSWORD={{ .secrets.database_password }}\n"
permissions_mode: "0600"
The secret database_password then takes this path:
- On the controller machine, the controller process opens the vault. The vault's passphrase never leaves the controller machine.
- To the nodes, the controller process sends each node only the secrets that the node's node tasks use.
- On the node, the secret waits in
/run/mmx/private/, which is in memory. The node deletes the secret's file as the node job worker process reads the file. The node job records on the node hold no secret value. - In output and saved runs, anything made from the secret shows as
[redacted]. Statuses and exit codes stay visible. - In the file you asked for,
/etc/shop/database.env, the secret is written as plain text, with the permissions you set.
A secret can also start on a node, such as a join token that one node creates and other nodes need. The controller process passes such a secret from memory to the nodes that use the secret, and saves the secret nowhere on the controller machine.
Keep secrets shows how to create the vault and use a secret. The secrets reference lists which secrets each node receives and where a secret can appear.
What MMX protects
MMX is a tool for a person or a small team who already have root access to
their servers. MMX trusts the administrators, and the playbooks and
modules that the administrators run, as ssh and sudo do. MMX protects
five boundaries:
- Only a person whose SSH key the node accepts for an account with full
sudocan send the node work. - The controller machine talks only to the node whose identity MMX saved over trusted SSH.
- Each node receives only the secrets that its node tasks use, and its own connection passwords.
- Secrets stay out of output, logs, and saved runs.
- Text from a node, or from a web server that a node task called, cannot
control your terminal: MMX's text output shows terminal control
characters, such as the escape character that moves the cursor, erases
lines, sets the window title, or writes the clipboard, as visible escapes
like
\x1b.--jsoncarries the exact text.
MMX does not protect a server from the server's own administrators, and does not make an untrusted playbook or module safe to run. Two limits follow from how MMX works:
- Over HTTPS, a key alone gives root. On a node where MMX is set up,
a key whose account may use
sudogives root over HTTPS without thesudopassword, and no PAM runs. Over plainssh, the same account would need the password. Setconnection: ssh_onlyon a node that needs the extra login rules: Login rules that only OpenSSH can check lists them. - A faked result travels. Someone with root on one node can fake that node's results. A value that the node passes to a later fleet stage reaches the other nodes that use the value, so those nodes receive the faked value too.
Credentials and where they live
MMX has no accounts of its own. MMX uses your SSH access.
| Credential | Where the credential lives, and what the credential does | How to change or remove the credential |
|---|---|---|
| Your SSH private key or SSH agent | On the controller machine. Logs in over SSH, and signs the HTTPS proof. Never sent to a node. | Remove the public key from the node's authorized_keys. The next connection is refused. Open connections and started node jobs go on. |
| MMX's own SSH key | ~/.ssh/mmx/ on the controller machine, made only for a server that takes only a password. The key's public half is in the account's authorized_keys on the node. |
Remove the public key from the node, as for any key. Deleting only the file on the controller machine does not remove the key's access. |
SSH password, sudo password, key passphrase |
Typed at a prompt on the controller machine, or read from a secret. Used for SSH only, never for HTTPS. | Change the passwords and the passphrase the usual way. A new password does not remove a key that the node already accepts. |
| The node's HTTPS key and certificate | /var/lib/mmx/tls.key and tls.crt on the node, readable only by root. |
The certificate lasts 90 days and renews itself with the same key, so the pin stays the same. Only a setup over SSH can make MMX accept a new key. |
| The saved node identity | .mmx/nodes/ in the project on the controller machine. Says which machine and which HTTPS key to expect. Not a login credential. |
A new mmx run saves the identity again after verified SSH. |
| The vault passphrase | A prompt, a password file, or an environment variable on the controller machine. Opens secrets_vault.age. |
mmx secrets passwd encrypts the vault with a new passphrase. Old copies of the vault file keep the old passphrase. |
What a break-in would expose
| If someone takes control of | That person could |
|---|---|
The controller machine, its SSH keys, or a running mmx with an open vault |
Use the controller machine's SSH keys with all the access that the keys give, including root over HTTPS on a node whose account may use sudo, without the sudo password. Change the work sent to nodes, and read every secret that the controller machine can open. The vault's encryption protects a vault file that nobody opened, not values already open in memory. |
| A playbook or a module that you run | Do anything as root on the nodes where the playbook or the module runs. A dryrun, mmx validate, and file checksums do not make code trustworthy. |
| Root on one node | Read that node's secrets and HTTPS key, change that node's files, and fake that node's results. A node holds no key that works on other nodes, but values that the node passes to later fleet stages reach the other nodes that use them, so those nodes receive a faked value too. That person learns the node's sudo password and any other password sent to the node, and can use a shared password on every node that has the password. |
| A copy of the saved runs or of the vault file | Read server names, paths, and output that is not secret, from the saved runs. Try to guess the vault's passphrase. A new passphrase does not protect a copy that someone already took. |
MMX's security rests on each operating system protecting its accounts, processes, and files. HTTPS protects the connection between two trusted machines, but cannot make a machine that someone took over trustworthy.
A change that could cut a node off
A wrong network configuration file can cut a node off from the controller machine. MMX's network modules protect such a change:
- Before the change, the controller process checks that a new SSH or HTTPS connection reaches the node. If no new connection reaches the node, the node task fails and changes nothing.
- The node saves the old file and starts a systemd timer that will put the old file back. The node's promise to restore is an owed restore.
- The node makes the change. The controller process then opens new connections to the node again.
- If each kind of connection that reached the node before the change,
SSH or HTTPS, does not reach the node again in time, the node puts the
old file back by itself, even without the controller machine. An owed
restore has an ID of 16 lowercase hexadecimal characters, and its
service starts again after the node restarts.
mmx statusnever confirms a change; it only reads the outcome and refreshes a restore that an administrator started.
Change SSH or network settings safely says which modules protect a change this way, and which changes, such as sshd's settings, get no such protection.
What MMX keeps on a node
MMX keeps its state in ordinary files, with no central database. MMX
writes each node job's request and result to a temporary file first and
then moves the file into place, so a crash does not leave a half-written
request or result. The
.mmx/ directory holds what MMX keeps on the controller machine:
The .mmx directory
describes the .mmx/ directory. On each node:
| Path on the node | What the path holds | How long the node keeps it |
|---|---|---|
/opt/mmx/bin/<sha256>/mmx-runner |
mmx-runner, one directory for each build |
Until mmx remove-mmx-from-node |
/etc/systemd/system/mmx-https-api.service |
The service that runs the node HTTPS API process | Until mmx remove-mmx-from-node |
/var/lib/mmx/tls.key, tls.crt |
The node's HTTPS key and certificate. Only on a node that MMX reaches over HTTPS. | Until mmx remove-mmx-from-node. The certificate renews itself. |
/var/lib/mmx/jobs/<job-id>/ |
Each node job's request, progress, and result | At least 72 hours after the node job finishes, so that a controller process can still collect a result after a lost connection. The node removes the records at its next MMX request after that, and keeps them longer while a restore is owed. |
/var/lib/mmx/uploads/<job-id>/ |
The files sent for a node job | Until the node job finishes. Files for a node job that never started stay 72 hours. |
/var/lib/mmx/kept_files/<sha256> |
Module files, and files that download and unpack_archive fetched from a web address, so later runs do not send them again. Files from the project, which may hold a secret, are not kept. |
30 days after a run last used the file |
/var/lib/mmx/command_records/ |
One record for each shell node task that runs only when tracked inputs changed: the inputs of the last command run, and whether the command succeeded |
Until mmx remove-mmx-from-node |
/var/lib/mmx/podman_volumes_of_removed_quadlet_files |
A record of the Podman volumes that stay after MMX removed their Quadlet files | Until mmx remove-mmx-from-node |
/var/lib/mmx/active and admission.lock, execution.lock, owed_restores.lock in /var/lib/mmx/ |
The ID of the running node job, and lock files that let one node job run at a time | Until mmx remove-mmx-from-node |
/var/lib/mmx/owed_restores/<id>/ and units named mmx-owed-restore-<id> and mmx-activate-change-<id> in /etc/systemd/system/ |
One owed restore: the saved old file and the restore deadline | Until the change is confirmed or the old file is back. A restore that failed stays, so that a person can repair the node. |
/var/lib/mmx/last_successful_run/<project_name>/<playbook> |
One JSON line naming the last run of the playbook that succeeded on the node | Until the next such run replaces the line |
/run/mmx/services/ |
Which version of its configuration each service last loaded | Until the node reboots |
/run/mmx/private/ |
Secrets handed to a node job, secret values that a node job computed, and the hidden details of failed node tasks, which may quote a secret | Until read, or until the node reboots or the node job's records expire |
/run/mmx/private_task_directories/ |
One private directory for each running node task that hands files to a tool, such as a login for git or psql |
Until the node task ends. If the node job worker process is killed, until the node's next MMX request while no node job runs. |
/etc/mmx/systemd_scheduled_jobs/<name>.env |
The secret values of a scheduled job that uses secrets, readable only by root | Until the scheduled job is removed. mmx remove-mmx-from-node keeps the file. |
/etc/mmx/do_not_manage |
Created by a person, never by MMX: keeps runs off the node | Until the person removes the file |
The secrets reference
gives the owners and permissions of the directories in /run/mmx/.
No timer or job of MMX runs on a node between runs. Only
mmx-https-api.service stays, and a timer exists only while a restore is
owed. These processes of mmx-runner may appear on a node:
| Process | When the process runs |
|---|---|
Node HTTPS API process, mmx-runner serve |
On each node without connection: ssh_only, as mmx-https-api.service. Accepts HTTPS connections on port 7443. |
| Node SSH request process | For one request that comes over SSH. Ends when the request is done. |
| Node job worker process | For one node job. Keeps going if the connection drops. |
Node change activation process, in mmx-activate-change-<id>.service |
While a network module puts one changed file into use |
Node owed restore process, in mmx-owed-restore-<id>.service |
When an owed restore's timer fires, at boot, after a failed activation, or when an administrator starts it. Puts the old file back. |
Where to go next
- Get started: install
mmxand run a first playbook. - Write a playbook: the
shopplaybook, step by step. - Preview and run changes: what each result and exit code means.
- Connect to your servers: passwords, jump hosts, port 7443, and a replaced server.
- Commands reference: every command and flag.
This page is docs/how-it-works.md in the MMX repository.