Keep secrets
Who acts: you, or your coding agent, in your MMX project on your
controller machine: the computer where you run mmx. A coding agent cannot
type at a prompt, so a coding agent takes the vault passphrase from a file
or an environment variable, as step 1 of the first route shows.
When to use this page: a playbook needs a password, an API token, or a key, and you want the value out of your plain project files and out of MMX's output.
What this page does: you create the vault, an encrypted file of secrets in your project, add secrets without showing them, write two secrets into a file on your servers, and run without a terminal. The other routes change a secret, find where a secret is used, change the passphrase, take secrets from environment variables, and keep a separate vault for staging.
Terms:
- A node is a server that MMX manages. The inventory, a file such
as
inventories/example.yml, lists the nodes. A playbook inplaybooks/lists fleet stages; each fleet stage picks nodes, and each of those nodes runs the fleet stage's node tasks. - A secret is a value that MMX keeps out of its output and out of the records it saves of each run, such as a database password.
- A secret key is the name of one secret, such as
database_password. A node task reads the secret as{{ .secrets.database_password }}. - The vault is the file
secrets_vault.agein your project directory. The vault holds secret keys and their values, encrypted with a passphrase in the age format. MMX has age built in, so you install nothing else. You can commit the vault to Git. - The vault passphrase unlocks the vault. MMX reads the passphrase on your controller machine and never sends the passphrase to a node.
Choose your route
| You want to | Route |
|---|---|
| Start keeping secrets, and use one in a playbook | Create the vault and use a secret |
| Run from CI, a script, or a coding agent, with no one to type | Run without a terminal |
| Add a secret that another program gives you, or change a secret's value | Add or change a secret without showing it |
| Check which playbooks read a secret, and which secrets are missing or unused | Find where a secret is used |
| Change the vault passphrase | Change the passphrase |
| Keep secrets in environment variables, such as in CI, and no vault | Secrets from environment variables |
| Keep staging's secrets apart from production's | A vault for each inventory |
| Pass a value that one node makes, such as a join token, to other nodes | A secret that a node makes |
Before you start
- You have a project where
mmx runworked, as in Get started. The examples use the serversweb01at203.0.113.11andweb02at203.0.113.12, in the groupweb_serversofinventories/example.yml. - The project directory is a Git repository (
git init), becausemmx secrets diff-keyscompares the vault with the last commit. - Your team has a password manager, or another safe place for the vault passphrase. Nobody can open the vault without the passphrase, and MMX cannot recover a lost passphrase.
Create the vault and use a secret
-
Save a new passphrase in a file on your controller machine.
mkdir -p ~/.config/mmx (umask 077 && openssl rand -base64 24 > ~/.config/mmx/shop-vault-passphrase) export MMX_VAULT_PASSWORD_FILE="$HOME/.config/mmx/shop-vault-passphrase"openssl randmakes a random passphrase, andumask 077makes the file readable by you only. The file is outside the project, so the file never goes into Git. Add theexportline to your shell's startup file, such as~/.zshrcor~/.bashrc. Then eachmmxcommand that needs the vault reads the passphrase from the file and asks nothing. Copy the passphrase from the file into your password manager.You can type the passphrase instead: leave out the file and the
exportline. Then each command that opens the vault at a terminal asksVault passphrase:, and a command that creates the vault also asksConfirm vault passphrase:.mmx secrets setwith<or a pipe cannot ask, so it needs the file or an environment variable. A coding agent needs the file, or the environment variable of Run without a terminal.Check:
ls -l ~/.config/mmx/shop-vault-passphraseshows-rw-------, andecho "$MMX_VAULT_PASSWORD_FILE"prints the file's path. -
Create the vault with your first secret.
mmx secrets editMMX opens your editor: the editor that
VISUALnames, else the one thatEDITORnames, elsevi. A new vault starts with two comment lines:# Named secret values, one per line, such as: # database_password: "the password"Add one line with your database's password, then save and close the editor:
database_password: "the password of your database"MMX encrypts the text and writes the vault:
Saved encrypted vault: secrets_vault.agemmx secrets editsays how MMX handles the editor's file. A coding agent does not use an editor: the commands in step 3 and in Add or change a secret without showing it create the vault too.Check:
mmx secrets list-keysprintsdatabase_password, andhead -n 1 secrets_vault.ageprintsage-encryption.org/v1, the first line of an encrypted age file. -
Add a secret that MMX makes up. An application's cookie secret, for example, must be random and must then stay the same:
mmx secrets add-random session_cookie_secret --length 48Added the key session_cookie_secret to the vault secrets_vault.age, with a random value of 48 letters and digits. MMX does not show the value.Without
--length, the value has 32 letters and digits. The command fails when the vault already has the secret key, so a second run of the command never replaces a value by accident.Check:
mmx secrets list-keysprints both secret keys:database_password session_cookie_secret -
Use the secrets in a node task. Create
playbooks/write_shop_settings.yml:# playbooks/write_shop_settings.yml fleet_stages: - id: write_shop_settings nodes: web_servers node_tasks: - id: create_settings_directory directory: path: /etc/shop - id: write_database_settings file: path: /etc/shop/database.env permissions_mode: "0600" # only root can read the file content: | DATABASE_PASSWORD={{ .secrets.database_password }} SESSION_COOKIE_SECRET={{ .secrets.session_cookie_secret }}MMX unlocks the vault on your controller machine and sends each node only the secrets that the node's node tasks read. The file on the server holds the values in plain text, as an application needs them, so
permissions_mode: "0600"lets only root read the file. Files that hold secrets gives the rule for other modes.Check:
mmx secrets usagelists both secret keys with the line that reads them, and(none)afterReferenced but not defined::database_password vault secrets_vault.age playbooks/write_shop_settings.yml:14 file /etc/shop/database.env (content) session_cookie_secret vault secrets_vault.age playbooks/write_shop_settings.yml:14 file /etc/shop/database.env (content) Referenced but not defined: (none) Defined but never referenced: (none) -
Preview the change.
mmx dryrun write_shop_settings --diff echo "exit code $?"--diffshows each file's difference, except for a node task whose result MMX hides, such as one that writes a secret:write_shop_settings / web01 / write_database_settings: succeeded, would_change write_shop_settings / web01 / write_database_settings: would create "/etc/shop/database.env" absent -> 119 bytes, permissions_mode 0600, uid 0, gid 0 contents suppressed for a task using secretsOn a server where MMX has not run before, the output also has the line
203.0.113.11: setting up MMX through verified SSH, because the first dryrun installs MMX's node program there.Check:
write_database_settingssayswould_changeon each server, and the exit code is 2, which for a dryrun means "something would change". -
Make the change.
mmx run write_shop_settingsCheck:
write_database_settingssayscreated "/etc/shop/database.env"on each server, and the run ends withRun ...: succeeded. On the server, the file holds both values:$ ssh root@203.0.113.11 cat /etc/shop/database.env DATABASE_PASSWORD=the password of your database SESSION_COOKIE_SECRET=4UhcJwUvXy2bZuUT1ulOSdKJVI5zYuFO6kKYnKiURYt6Eh4w -
Commit the vault and the playbook.
git add secrets_vault.age playbooks/write_shop_settings.yml git commit -m "Keep the shop's secrets in the vault"Teammates get the vault with the project, and the passphrase from your password manager.
Check:
mmx secrets diff-keyscompares the vault with the last commit, and reports no difference:Vault: secrets_vault.age, compared with the last commit Added: (none) Removed: (none) Changed: (none)
Module pages that rely on this section: user.
Run without a terminal
MMX asks for the vault passphrase only at a terminal. A CI job, a script, or a coding agent gives the passphrase in one of these ways:
| Source | Use it for |
|---|---|
--vault-password-file PATH on the command |
One command, such as a command for another inventory's vault |
MMX_VAULT_PASSWORD_FILE, the path of a file |
Your own computer, or a coding agent on it |
MMX_VAULT_PASSWORD, the passphrase itself |
CI, which keeps the passphrase in the CI system's secret storage |
MMX uses the first source in the table that is set. Unlocking the vault says what happens when that source fails.
-
Give the passphrase to the command. In CI, store the passphrase as a secret of the CI system, under the environment variable name
MMX_VAULT_PASSWORD. On your own computer, theMMX_VAULT_PASSWORD_FILEof the first route works too.Check:
mmx secrets list-keys --non-interactiveprints the secret keys.--non-interactivemakes MMX fail instead of asking a question. -
Run with
--non-interactive.mmx run write_shop_settings --non-interactive --output=summary--output=summaryputs the servers with the same result on one line, which keeps a CI log short:write_shop_settings / 2 nodes [web01, web02]: succeeded write_shop_settings / 2 nodes [web01, web02] / create_settings_directory: succeeded, unchanged write_shop_settings / 2 nodes [web01, web02] / create_settings_directory: directory "/etc/shop" checked 1 directories and 0 files; 0 metadata updates write_shop_settings / 2 nodes [web01, web02] / write_database_settings: succeeded, unchanged Run 86a155d0: succeededCheck: the run ends with
Run ...: succeededand exit code 0. Without a passphrase source, the run stops before any server changes, with exit code 4:mmx: vault passphrase required; use --vault-password-file, MMX_VAULT_PASSWORD_FILE, or MMX_VAULT_PASSWORD (prompts are disabled)
MMX also asks nothing, and fails in the same way, when the command has no
terminal or when the command has --json. Unlocking the vault
lists which commands open the vault.
Add or change a secret without showing it
mmx secrets set and mmx secrets add-random write one secret into the
vault. Neither command prints the value, so a coding agent can store a
value that the agent never sees. Both commands create the vault when the
project has none.
-
Add a value that you have in a file, or that a program prints. The value comes from standard input:
mmx secrets set mail_api_token < ~/Downloads/mail-api-token.txtA program's output works through a pipe, such as
some_program | mmx secrets set mail_api_token. At a terminal without<or a pipe, the command asksValue of mail_api_token:and hides what you type.mmx secrets setlists the rules for the value.Check: the command prints nothing and exits with code 0, and
mmx secrets list-keysprintsmail_api_token. -
Change the value of a secret key that the vault has. Add
--replace:mmx secrets set mail_api_token --replace < ~/Downloads/mail-api-token.txt mmx secrets add-random session_cookie_secret --replaceReplaced the value of the key session_cookie_secret in the vault secrets_vault.age with a random value of 32 letters and digits. MMX does not show the value.Without
--replace, both commands refuse a secret key that the vault has, and change nothing. To change a value by hand, usemmx secrets edit.Check:
mmx secrets diff-keyslists the secret keys that you added or changed since the last commit:Vault: secrets_vault.age, compared with the last commit Added: mail_api_token Removed: (none) Changed: session_cookie_secret -
Deliver the new value.
mmx run write_shop_settingsA new value in the vault changes nothing on a server until a run gives the value to the node tasks that read the value.
Check:
write_database_settingssaysupdated "/etc/shop/database.env"on each server.A password that another program also keeps, such as a database's own record of a user's password, must change in that program too. Let the same run change both, such as with a
postgresql_rolenode task that reads the same secret. -
Commit the vault.
git add secrets_vault.age git commit -m "Add the mail API token, replace the cookie secret"Check:
mmx secrets diff-keysshows(none)on all three lines.
Find where a secret is used
mmx secrets usage reads every playbook, the roles that the playbooks
call, and the inventory, and lists where each secret key is read. The
command contacts no server and prints no value.
-
List every secret key.
mmx secrets usage echo "exit code $?"database_password vault secrets_vault.age playbooks/write_shop_settings.yml:14 file /etc/shop/database.env (content) mail_api_token vault secrets_vault.age (never referenced) session_cookie_secret vault secrets_vault.age playbooks/write_shop_settings.yml:14 file /etc/shop/database.env (content) Referenced but not defined: (none) Defined but never referenced: mail_api_token exit code 0mmx secrets usageexplains each line of the output.Check:
Referenced but not defined:says(none), and the exit code is 0. A secret key in that list is read by a playbook and is in neither the vault norsecrets_from_env, so a run of that playbook would fail. The command then exits with code 1, so CI can runmmx secrets usageas a check. -
Look up one secret key.
mmx secrets usage database_passwordCheck: the output has only the lines of
database_password.
A secret key under Defined but never referenced: is in the vault and
nothing reads the key. Remove the key with mmx secrets edit when no
playbook needs the key any more.
Change the passphrase
mmx secrets passwd encrypts the vault again with a new passphrase. The
secrets in the vault stay the same.
-
Save the new passphrase in a second file.
(umask 077 && openssl rand -base64 24 > ~/.config/mmx/shop-vault-passphrase.new)Check:
ls -l ~/.config/mmx/shows both files with-rw-------. -
Change the passphrase.
mmx secrets passwd --new-vault-password-file ~/.config/mmx/shop-vault-passphrase.newSaved encrypted vault: secrets_vault.ageMMX opens the vault with the old passphrase, from
MMX_VAULT_PASSWORD_FILE, and encrypts the vault with the new one. At a terminal, you can leave out--new-vault-password-file: MMX then asksNew vault passphrase:andConfirm new vault passphrase:.Check: the command prints
Saved encrypted vault:and exits with code 0. -
Put the new passphrase where the old one was.
mv ~/.config/mmx/shop-vault-passphrase.new ~/.config/mmx/shop-vault-passphrasePut the new passphrase in your password manager and in each CI system that has the old one too.
Check:
mmx secrets list-keysprints the secret keys. -
Commit the vault.
git add secrets_vault.age git commit -m "Change the vault passphrase"Check:
mmx secrets diff-keysshows(none)on all three lines. Before this commit,mmx secrets diff-keysfails, because the committed vault needs the old passphrase.
The old passphrase still opens the older copies of the vault that it encrypted, such as the copies in your Git history. If the old passphrase got out, also give each secret a new value, as in Add or change a secret without showing it.
Secrets from environment variables
A secret key can come from an environment variable on the controller machine instead of the vault. A project whose secret keys all come from environment variables needs no vault and no passphrase. CI systems often keep secrets as environment variables.
-
Name an environment variable for each secret key. Add
secrets_from_envtommx_project_settings.yml:# mmx_project_settings.yml project_name: mmx_example default_inventory: example secrets_from_env: database_password: SHOP_DATABASE_PASSWORD # secret key: environment variable session_cookie_secret: SHOP_SESSION_COOKIE_SECRETThe playbook stays the same:
{{ .secrets.database_password }}now readsSHOP_DATABASE_PASSWORD. Other secret keys still come from the vault. Where a secret's value comes from says which source wins.Check:
mmx secrets usagenames the environment variable on each secret key's first line:database_password environment variable SHOP_DATABASE_PASSWORD playbooks/write_shop_settings.yml:14 file /etc/shop/database.env (content) -
Run with the environment variables set.
SHOP_DATABASE_PASSWORD='...' SHOP_SESSION_COOKIE_SECRET='...' mmx run write_shop_settings --non-interactiveIn CI, the CI system sets the environment variables from its secret storage, so the command needs no values in front of
mmx.Check: the run ends with
Run ...: succeeded. With an environment variable that is not set or is empty, the run stops before any server changes:mmx: secret database_password requires a nonempty environment variable SHOP_DATABASE_PASSWORD
Passwords that MMX logs in with, ssh_password and sudo_password, come
from secrets too.
Connect to your servers
shows how to set these passwords from secrets.
A vault for each inventory
An inventory can name a vault of its own with vault_file. Staging then
has its own secrets and its own passphrase, and a person who works on
staging needs no production secret.
-
Name the vault in the staging inventory.
# inventories/staging.yml vault_file: staging_secrets_vault.age # a path from this inventory's directory nodes: staging-web01: {address: 203.0.113.41, ssh_user: root} groups: web_servers: nodes: [staging-web01]The path starts at the inventory's directory, so the vault is
inventories/staging_secrets_vault.age. Which vault file lists the other places that choose the vault.Check:
mmx validate --inventory staging write_shop_settingsprintsValid: mmx_example. -
Save staging's passphrase in a file of its own.
(umask 077 && openssl rand -base64 24 > ~/.config/mmx/shop-staging-vault-passphrase)Check:
ls -l ~/.config/mmx/shop-staging-vault-passphraseshows-rw-------. -
Add staging's secrets.
--inventory stagingselects staging's vault, and--vault-password-filegives staging's passphrase. The first command creates the vault:mmx secrets add-random database_password --inventory staging --vault-password-file ~/.config/mmx/shop-staging-vault-passphrase mmx secrets add-random session_cookie_secret --inventory staging --vault-password-file ~/.config/mmx/shop-staging-vault-passphraseAdded the key database_password to the vault inventories/staging_secrets_vault.age, with a random value of 32 letters and digits. MMX does not show the value.Check:
mmx secrets usage --inventory staging --vault-password-file ~/.config/mmx/shop-staging-vault-passphraseshowsvault inventories/staging_secrets_vault.ageon each secret key, and(none)afterReferenced but not defined:. -
Run on staging.
mmx run write_shop_settings --inventory staging --vault-password-file ~/.config/mmx/shop-staging-vault-passphrase--vault-password-filewins over the production file inMMX_VAULT_PASSWORD_FILE.Check: the run ends with
Run ...: succeeded. Without--vault-password-file, MMX tries the production passphrase and fails withcannot decrypt vault: wrong passphrase or invalid age file.
A secret that a node makes
A secret can start on a node, such as a join token that the first server
of a cluster writes and the other servers need. content_is_secret: true
on a read_file node task marks the file's
content as a secret, and an output made from the content is a secret too.
In this example, a program on web01 keeps a token in
/etc/shop/join_token, and web02 needs the token:
# playbooks/share_join_token.yml
fleet_stages:
- id: read_join_token
nodes: web01
node_tasks:
- id: join_token
read_file:
path: /etc/shop/join_token
content_is_secret: true
outputs:
join_token: "{{ .node_tasks.join_token.content }}"
- id: write_join_token
nodes: web02
node_tasks:
- id: write_join_token
file:
path: /etc/shop/join_token
permissions_mode: "0600"
content: "{{ .outputs.read_join_token.web01.join_token }}"
Use a result from one server on another explains outputs in full.
The token goes from web01 through the memory of mmx on your controller
machine to web02, and to no other server. mmx results shows the output
as {"join_token":"[redacted]"}. A shell node task with
output_is_secret: true marks a command's output as a secret in the same
way.
After an interruption, mmx resume reads the file on web01 again, but
does not run a completed shell command again. So keep a value that later
fleet stages need after an interruption in a file on the node, and read
the file with read_file, as this example does. Pass the content on whole,
as "{{ .node_tasks.join_token.content }}" does: an output with text around
the content blocks its users on mmx resume, so put the extra text in the
node task that uses the output instead. The
secrets reference
gives the rules of travel, saved records, and mmx resume.
If something is different
| What you see | What it means and what to do |
|---|---|
vault passphrase required; use --vault-password-file, MMX_VAULT_PASSWORD_FILE, or MMX_VAULT_PASSWORD (prompts are disabled) |
MMX could not ask for the passphrase: no terminal, --non-interactive, or --json. Give a passphrase source, as in Run without a terminal. |
cannot decrypt vault: wrong passphrase or invalid age file |
The passphrase does not open this vault. Check which source MMX used: --vault-password-file first, then MMX_VAULT_PASSWORD_FILE, then MMX_VAULT_PASSWORD. For another inventory's vault, give that vault's passphrase. |
read vault password file /home/you/.config/mmx/shop-vault-passphrase: open /home/you/.config/mmx/shop-vault-passphrase: no such file or directory |
MMX_VAULT_PASSWORD_FILE or --vault-password-file names a file that is not there. Fix the path. MMX does not then try MMX_VAULT_PASSWORD or ask. |
vault is missing required secret mail_token |
A playbook reads a secret key that the vault does not have, often a misspelled name. The run stopped before any server changed, unless the missing key is a node's ssh_password or sudo_password, which MMX reads when it connects to that node. mmx secrets usage lists the key under Referenced but not defined: with its file and line. Fix the name, or add the secret. |
secret database_password requires a nonempty environment variable SHOP_DATABASE_PASSWORD |
secrets_from_env binds the secret key to an environment variable that is not set or is empty. Set the environment variable. |
the vault already has the key session_cookie_secret; to give it a new random value, add --replace |
add-random or set refuses to replace a value by accident. Add --replace if you want a new value. |
secret key database_password is bound to environment variable SHOP_DATABASE_PASSWORD by secrets_from_env in mmx_project_settings.yml. ... |
A run reads this secret key from the environment variable, so a value in the vault would never be read. Remove the secrets_from_env entry, or choose another secret key. |
vault must contain one YAML mapping with unique secret names after mmx secrets edit |
The text you saved is not a list of key: value lines, or has a secret key twice. MMX kept the vault as it was. At a terminal, MMX asks Edit again? [Y/n]; answer Enter to fix the text in your editor. Without a terminal, your changes are lost, so run mmx secrets edit again. |
role input database_password receives a secret; declare secret: true |
A role call passes a secret to a role's input. Add secret: true to that input in the role's file, below type and required. |
details suppressed for a task using secrets; root can read them on the node in /run/mmx/private/<job-id>-hidden-details-of-failed-node-tasks.txt for 72 hours, or until the node reboots |
A node task with a hidden result failed, such as one that reads a secret, and MMX hid the reason because the reason may hold the secret. Read the file on that server with sudo cat. The secrets reference explains which results MMX hides. |
mmx secrets diff-keys says the last committed version cannot be opened with the supplied passphrase |
You changed the passphrase, and the committed vault still needs the old one. Commit the vault. |
mmx secrets diff-keys says the repository has no commit yet |
Commit the vault once. Later, mmx secrets diff-keys compares with that commit. |
new vault passphrase required; use --new-vault-password-file (prompts are disabled) |
mmx secrets passwd could not ask for the new passphrase. Give the new passphrase in a file with --new-vault-password-file. |
Where to go next
- Secrets reference: which secrets a run needs
and each server receives, which results MMX hides, where secrets stay on
a server, and every rule of the vault and the
mmx secretscommands. - Connect to your servers:
SSH and
sudopasswords from secrets. shell: give a command a secret throughenv, and show a command's output when the command prints no secret.- Choose which servers a run touches: one inventory for each environment.
examples/secrets/: a whole project that passes a secret from the vault to a role.
This page is docs/how-to/keep-secrets.md in the MMX repository.