The mmhotline command
This page lists every mmhotline command: its flags, its exit codes, and
the fields of its --json result. Most commands run on the MMHotline
server, as root, and talk to the running service through its command
socket, a private Unix socket that only root can reach. A few run
anywhere, such as settings check on a laptop. Each entry links to the
how-to page that uses the command step by step;
Change the settings is the one most
people start with.
Common flags, results, and exit codes
Every command of these groups takes the common flags: people,
sending-services, triage-agent, review-agent, agent-playbook-rules,
contact-methods, alert-sources, schedules, status-page,
down-alarm, incidents, alerts, triage-sessions,
approval-requests, and commit-reviews. settings apply and
settings show take them too. The exceptions are named in their entries:
status-page set-up and triage-agent check-preparation take only
--json and --help, and the commands
that systemd, SSH, or the triage agent run take none.
| Flag | Meaning |
|---|---|
--command-socket PATH |
The command socket. Default /run/mmhotline/command.sock |
--json |
Print one JSON object instead of sentences |
--help |
Show the command's flags. It needs no running service |
mmhotline --helplists every command group.mmhotline GROUP --helplists the actions of one group, such asmmhotline schedules --help.- Flags come after the action:
mmhotline schedules show --json. - Never put a secret in an argument: other users on the server can read a program's arguments. The flags that take a secret read it from a file or from standard input instead.
- Use
sudofor every command that uses the command socket.
Who the command acts as. A command run through sudo acts for the
whole team, also on another person's personal contact methods. Its
actions are recorded as "the mmhotline command, run by barbara" when
SUDO_USER is barbara, or as root without sudo. They carry no team
member ID. Changes from the settings file are recorded as "the settings
file /etc/mmhotline/mmhotline_settings.yml". On the pages, only a personal
contact method's owner edits that method.
Results. Every successful command returns message, one sentence
about the result. Every group except settings, alert-sources,
down-alarm, and status-page also returns web_url, the settings
file's web_url. Optional fields and empty lists may be absent unless an
entry says "present also when empty". Times are RFC 3339 UTC strings, such
as 2026-11-15T08:51:06Z. Successful JSON goes to standard output.
Text output replaces every control character except the line feed and
the tab with the replacement character � (U+FFFD). Reason: alert
titles, alert messages, and the triage agent's text come from other people
and programs, and a terminal acts on escape sequences in them, which can
rewrite earlier lines, set the window title, or write to the clipboard.
--json keeps the original characters, escaped, such as \u001b.
Exit codes, unless an entry says otherwise:
| Exit code | Meaning |
|---|---|
0 |
Success, or --help |
1 |
An installation, service, or setup failure |
4 |
MMHotline refused the command, the arguments are wrong, or the command socket does not answer |
130 |
Interrupted, such as with Ctrl-C |
A refused command prints MMHotline's reason on standard error, also with
--json. A missing command socket says mmhotline serve is not running at command socket /run/mmhotline/command.sock.
mmhotline install
Installs MMHotline on the Linux server where it runs, from the settings
file /etc/mmhotline/mmhotline_settings.yml.
sudo mmhotline install
Install MMHotline runs it step by step and shows its output; Upgrade MMHotline runs it again with a new program.
- Flags: only
--help, which needs no root and checks and changes nothing. - Needs: Linux, root, running systemd, and the settings file.
- Refuses, and changes nothing, until the settings file exists and
passes the owner and mode check and the same checks as
settings check. The refusal names the settings file's path and lists each problem with its key path. - What it does: creates the
mmhotlineUnix user, installs the program, and creates the database with the file'steam_namewhen the database is missing. It sets the settings file's owner toroot, groupmmhotline, and mode0640, so MMHotline can read the file and only root can change it. It starts the service, the triage agent's run socket, and the fleet view socket. Installed files lists every file it writes. - What it leaves alone: the settings file's contents, which the team
writes; an existing database and its team name (a changed
team_nameapplies withsettings apply); andcloudflared, Cloudflare Access, and the private network. - The backup unit: install creates
mmhotline-backup.serviceonly when/etc/mmhotline/backup.envand thelitestreamprogram exist. See Back up and restore the database. - Restarts the service when the program or a generated unit changed, the database is new, the settings file's owner or mode was corrected, or the service was stopped. An unchanged rerun changes nothing.
- Output: one line for each step, then
MMHotline:and the web address. Whenweb_urlis anhttps://address and the settings file has nocloudflare_access, it prints the next step: Put MMHotline behind Cloudflare Access. - To upgrade: run
sudo ./mmhotline installfrom the folder that holds the new program.
mmhotline version
Prints the program's version, the Git revision it was built from, and its system.
mmhotline version
mmhotline 0.1.0 (COMMIT, linux/amd64, modified=false)
COMMITstands for the full hash of the commit the program was built from, andmodified=truemeans the checkout had uncommitted changes.- Flags: only
--json. Any other argument prints the usage and exits 4. - Needs no
sudoand no running service, so another program, such as a check that compares installed versions with the latest releases, can run it. - JSON fields:
version,revision,modified,go,os,arch.
Used in Install MMHotline, Upgrade MMHotline, and Post to the status page.
mmhotline backup check-uploads
Checks whether MMHotline's continuous backup keeps up: whether Litestream captured every change of the database, and whether the backup in object storage holds every change Litestream captured.
sudo mmhotline backup check-uploads
MMHotline's backup holds every change Litestream captured, up to transaction 0000000000000002.
Back up and restore the database explains when to run it and what to do with each answer.
- Flags:
--jsonand--helponly. - Runs as root on the MMHotline server. It never connects to
mmhotline serveand changes nothing. - Reads
/etc/mmhotline/backup.env, the database's write-ahead log/var/lib/mmhotline/mmhotline.db-wal, and the changes Litestream keeps in/var/lib/mmhotline-backup/mmhotline.db-litestream/ltx/0/until they are uploaded. It lists the backup withlitestream ltx, found onPATHand started with only the variables ofbackup.envandPATH. - Waits up to 30 seconds for Litestream to capture a transaction the log holds.
- Prints one line, or one JSON object with
--json, whatever the answer.
Exit codes (this command is a check, so its codes differ):
| Exit code | Line |
|---|---|
0 |
MMHotline's backup holds every change Litestream captured, up to transaction 00000000000004d2. |
0 |
MMHotline's backup is 30 seconds behind, within 5 minutes: the newest in the backup is transaction 00000000000004d1; Litestream has captured up to 00000000000004d2. |
1 |
MMHotline's backup has stopped: the database has changes that Litestream has not captured after waiting 30 seconds; Litestream has captured up to transaction 0000000000000004. Read why with: sudo journalctl -u mmhotline-backup |
1 |
MMHotline's backup is missing changes captured 12 minutes ago: the newest in the backup is transaction 00000000000004d1; Litestream has captured up to 00000000000004d2. Read why with: sudo journalctl -u mmhotline-backup |
3 |
MMHotline's backup is not set up: /etc/mmhotline/backup.env is missing. To set it up, follow and the guide's address |
3 |
Litestream has captured no change of the database, so the backup has not started; and how to check and start it |
3 |
Cannot list MMHotline's backup: and Litestream's last error line, with each backup.env value of 8 characters or more replaced by its variable's name, such as $LITESTREAM_SECRET_ACCESS_KEY |
4 |
The arguments are wrong |
It also exits 3 when Litestream is missing, when backup.env cannot be
read (run the check with sudo) or sets no
MMHOTLINE_BACKUP_REPLICA_URL, when the newest captured change has no LTX
header, and when the list has no max_txid column.
JSON fields. A transaction is Litestream's transaction ID, as Litestream writes it: 16 hexadecimal digits.
| Field | Meaning |
|---|---|
message |
The line from the table above |
uploads_keep_up |
true for exit 0, false for exit 1, null when the check cannot tell |
newest_captured_transaction |
The newest transaction Litestream captured. Absent before the first, and when the check cannot tell |
newest_uploaded_transaction |
The newest transaction in the backup. Absent when the backup holds none, when Litestream did not capture the database's changes (the check then lists nothing), and when the check cannot tell. It can be newer than newest_captured_transaction, because Litestream may capture and upload between the check's local read and its listing of the backup |
oldest_missing_change_captured_at |
When Litestream captured the oldest change the backup lacks. Absent when the backup lacks none |
mmhotline setup
Creates MMHotline's database, mmhotline.db, with the settings file's
team_name and no people. sudo mmhotline install runs it for you; run it
by hand only on a development machine.
sudo mmhotline setup --state-dir /var/lib/mmhotline \
--settings-file /etc/mmhotline/mmhotline_settings.yml
Created MMHotline state in /var/lib/mmhotline.
| Flag | Meaning |
|---|---|
--state-dir DIR |
Required. The folder for the database |
--settings-file FILE |
Required. The settings file. Setup checks it, its owner and mode included |
--help |
Show the flags |
- Refuses when the folder already holds MMHotline's state, and when the settings file has problems, which it lists.
- Prints sentences, never JSON.
- Exit codes: 4 for a missing flag; 1 for a refused setup.
- Local development runs setup and serve on a development machine.
mmhotline serve
Runs MMHotline: the pages, the alert intake, and the command socket.
systemd runs it as mmhotline.service; a person runs it only on a
development machine.
sudo mmhotline serve --state-dir /var/lib/mmhotline \
--settings-file /etc/mmhotline/mmhotline_settings.yml \
--command-socket /run/mmhotline/command.sock
| Flag | Meaning |
|---|---|
--state-dir DIR |
Required. The folder that holds the database |
--settings-file FILE |
Required. The installed service passes /etc/mmhotline/mmhotline_settings.yml |
--command-socket PATH |
Required. The command socket, which checks that the caller is root |
--triage-agent-run-socket PATH, --review-agent-run-socket PATH, --triage-agent-proxy-socket PATH |
Give all three or none. Without them, triage agent and review agent runs are unavailable. The installed service gives all three |
--fleet-view-socket PATH |
The socket where systemd runs the settings file's fleet_view_command for the dashboard's servers. The installed service gives /run/mmhotline-fleet-view.sock. Without it, the dashboard shows no servers |
--help |
Show the flags |
- Listens on
web_listen_addressfor the pages, over plain HTTP, such as127.0.0.1:8480, and onalert_intake_listen_addressfor signed alerts and check-ins, such as10.20.0.1:8484. - Prints its sign-in methods on standard error at start:
Sign-in: Cloudflare Access for example.cloudflareaccess.com, and sign-in links, orSign-in: sign-in links only; the settings file has no cloudflare_access. Withcloudflare_access, serve checks the token Cloudflare Access adds to each page request. - Reads the settings file at start and checks its owner, mode, and
values.
web_url,web_listen_address,alert_intake_listen_address, andcloudflare_accesschange only at a start. Parts without an outside test apply before serve tells systemd it is ready; parts with one apply in the background, assettings applywould. - A settings file with problems: serve starts with the settings it
last applied and sends the administrators a warn alert that names each
problem by key path. On the very first start, with nothing applied yet,
serve refuses to start, exits 1, and prints
the settings file FILE has problems, and MMHotline has never applied a settings file, so it cannot start:and each problem. - A listen address serve cannot open, such as one the server does not have: serve falls back to the addresses it last applied, and warns. The settings file design gives the reasons.
- Exit codes: 4 for a missing
--state-dir,--settings-file, or--command-socket, a socket flag that needs the other two, or an unexpected argument. - Development switch: the environment variable
MMHOTLINE_FAKE_SENDING_SERVICES_URLpoints chat, push, and phone sending services at the fake sending services.
mmhotline settings check
Checks a settings file's shape, keys, and values, without a network or a running MMHotline. It runs anywhere, as any user, such as on a laptop in a deploy tool's dry run.
mmhotline settings check --settings-file mmhotline_settings.yml
The settings file /etc/mmhotline/mmhotline_settings.yml has 1 problem:
sending_services.pushover.application_token: must be one line of exactly 30 letters and digits, as Pushover shows the application's API token
Change the settings uses it before each apply.
| Flag | Meaning |
|---|---|
--settings-file FILE |
Required. The file to check |
--json |
Print one JSON object |
--help |
Show the flags |
-
Prints each problem with its key path and what is wrong, never a secret's value. Other problem texts look like
on_call_schedule.escalate_if_unclaimed_after_minutes: must be a whole number of minutes from 1 to 60andalert_sources[0].signing_secrets[0]: must be 64 hexadecimal characters, such as the output of openssl rand -hex 32. A problem of the whole file has no key path, such asthe settings file is not valid YAML: line 1: did not find expected node content. -
A file without problems prints
The settings file mmhotline_settings.yml has no problems. -
Notes follow the problems: something the file leaves out, which MMHotline fills with its safest behavior, but which a person should see. A note never fails the check, so a dry run shows it and continues. For example, a kind of action that an inventory's
action_kindsleaves out:The settings file mmhotline_settings.yml has no problems. Notes, which do not stop MMHotline from applying the settings file: rules_for_running_agent_playbooks.production: check_only has no rule; it needs a person until you add one -
What it checks: every rule of
settings applythat needs no saved state: keys, value types, and each value; at least one administrator inpeople; and every sign-in email address inon_call_schedulelisted inpeople. -
What it skips: the file's owner and mode, because a dry run writes the file on a laptop without root (
settings apply, serve's start, andinstallcheck them); custom schedule code; and every outside service.settings applyruns those.
| Exit code | Meaning |
|---|---|
0 |
No problems, with or without notes |
1 |
The file has problems |
4 |
--settings-file is missing, or another argument is wrong |
JSON fields:
| Field | Meaning |
|---|---|
settings_file |
The path given with --settings-file |
problems |
Each problem: key_path and problem. Present also when empty |
notes |
Each note: key_path and note. Present also when empty |
mmhotline settings apply
Tells the running MMHotline to read its settings file, check it, and apply each part that changed.
sudo mmhotline settings apply
people: did not apply: Dana Example owns 1 claimed incident(s). Resolve the claimed incidents or have someone take over the claimed incidents before removing the person. The earlier setting, applied at 2026-11-15T06:17:42Z, stays.
on_call_schedule: applied.
1 part did not apply and keeps its earlier setting: people.
Change the settings walks through an apply and each thing that can go differently.
| Flag | Meaning |
|---|---|
--command-socket PATH |
The command socket. Default /run/mmhotline/command.sock |
--json |
Print one JSON object; the fields are under Settings results |
--help |
Show the flags |
-
Which file: the file at the path
servewas given, normally/etc/mmhotline/mmhotline_settings.yml. MMHotline checks its owner and mode, then its values. -
A file with problems applies nothing. The command prints each problem, as
settings checkdoes, thenThe settings file has problems, so nothing applied and the earlier settings stay. Fix the settings file problems, then apply the settings file again. -
A file someone other than root could change, or every user could read, is refused with the command that fixes it:
/etc/mmhotline/mmhotline_settings.yml can be read by every user on the server (mode 0644), and it holds the team's secrets. Fix the settings file owner and mode with: sudo chown root:mmhotline /etc/mmhotline/mmhotline_settings.yml; sudo chmod 0640 /etc/mmhotline/mmhotline_settings.yml -
Parts. MMHotline checks each part against what it saved and writes each part on its own. A part that fails keeps its earlier setting while the other parts apply. A part whose value is already in effect is
unchanged, and is not written again, unless an attempt to apply it failed after its last success. -
One apply at a time. A second apply waits for the first. The command waits without a time limit, because an outside test can take minutes. Ctrl-C stops the wait, not the apply.
The output starts with Settings file: and the path, and
Settings file first comment line: when the file starts with a comment.
Then comes one line for each part, in the order they apply, people
before the schedule that names them:
web_url, web_listen_address, alert_intake_listen_address, cloudflare_accessteam_namefleet_view_commandpeopleon_call_schedulerules_for_running_agent_playbooksservices_whose_changes_a_person_must_approvealert_sourcestriage_agent.commit_signing_public_key, triage_agent.trusted_commit_signing_public_keystriage_agent.commits_without_a_trusted_signature_are_approved_by, triage_agent.paths_only_a_person_may_approvetriage_agent.coding_agent, triage_agent.credential_type, triage_agent.credential, triage_agent.credential_expires_on, triage_agent.model, triage_agent.reasoning_efforttriage_agent.internet_accesstriage_agent.listed_hoststriage_agent.sos_waits_for_the_agent_minutestriage_agent.warn_waits_for_the_agent_minutestriage_agent.sos_waits_for_a_server_replacement_minutestriage_agent.software_review_interval_hoursmmhotline_down_alarm.wake_the_person_on_call_after_minutes_without_a_check_in, mmhotline_down_alarm.wake_the_backup_after_minutes_without_a_check_inreview_agentsending_services.pushoversending_services.twiliosending_services.telegramsending_services.slacksending_services.mail_servermmhotline_down_alarmstatus_page
Each line ends in one of these:
applied.unchanged.not set up, because the settings file leaves it out.did not apply:, the reason, then when the earlier setting applied:The earlier setting, applied at 2026-11-15T03:54:04Z, stays., orIt stays not set up, as before., orNo value has ever applied for the settings file part, so the part is not set.
The last line is Every part of the settings file is applied., or
1 part did not apply and keeps its earlier setting: and the parts.
Notes come on the lines under a part, indented by two spaces. A note never makes the command fail. They have three sources:
-
The static check's notes, under the part their key path belongs to, also when the part is unchanged. MMHotline saves them from the last apply. Example:
rules_for_running_agent_playbooks.production: check_only has no rule; it needs a person until you add one. -
The next step after a sending service applies for the first time; the note starts with
Next step::sending_services.mail_server: applied. sending_services.mail_server: Next step: Add your own email address under Contact methods. Saving the contact method sends a test message. -
Hosts listed automatically, under
triage_agent.listed_hosts, only insettings show.
Other notes the parts print:
- Under
triage_agent.software_review_interval_hourswhile the review is off:Scheduled software review: off. The triage agent wakes only for alerts; sudo mmhotline triage-agent run-software-review still runs one review, such as to learn what one costs. - Under
triage_agent.commits_without_a_trusted_signature_are_approved_bywitha_person_or_the_triage_agent:The triage agent reviews each commit no trusted key signed and approves an ordinary change; a person reviews the triage agent's own commits outside its folders, and every commit that changes a path of triage_agent.paths_only_a_person_may_approve.
The parts with an outside test are saved only when the test passes:
| Part | Outside test |
|---|---|
triage_agent.coding_agent, ... (part 11) |
One test prompt, run with the file's coding agent, credential, model, and reasoning effort. It must answer MMHotline triage agent test passed. The server ends the test after 5 minutes |
review_agent |
The same, with the review agent's setup, in a run like a review's. It must answer MMHotline review agent test passed |
sending_services.pushover |
Pushover's limits endpoint |
sending_services.twilio |
Twilio's check of the calling number in the account |
sending_services.telegram |
Telegram's getMe. MMHotline then starts polling: it keeps asking Telegram for updates |
sending_services.slack |
Slack's token checks. MMHotline then starts Socket Mode, a connection that receives Slack messages and button presses |
sending_services.mail_server |
A connection to the mail server that checks TLS and the sign-in, and sends nothing |
mmhotline_down_alarm |
Uploads the Worker, sets its check-in token, turns on its workers.dev address, saves the setup, and checks in |
status_page |
Posts the current page with the file's R2 key |
- A setup test that cannot start because another triage agent run is
active fails the part with
a triage agent run was active, so the new agent setup was not tested. Apply the settings file again after the triage agent run ends. A failed setup test names the reason and the coding agent's answer, when there is one. - A section left out removes that setup. A sending service that contact
methods still use is refused and keeps its setup:
sending_services.pushover: did not apply: cannot remove the Pushover setup: 2 Pushover notification contact methods use the sending service setup. Remove those contact methods on the Contact methods page, or with mmhotline contact-methods remove --contact-method-id ID, first. The earlier setting, applied at 2026-11-15T05:08:44Z, stays.Leaving outmmhotline_down_alarmmakessettings applycheck in with nothing held, then remove the setup. An MMHotline-down alarm whose Worker cannot be told to hold nothing is refused the same way (settings file design).
The rules part is the only part that writes the rest of its value when
it fails. When an inventory in MMHotline's copy of the systems
repository has no entry in rules_for_running_agent_playbooks, the line
is:
rules_for_running_agent_playbooks: did not apply: inventory staging has no rules for running the agent's playbooks. Every fix in the inventory needs a person until you add its rules. The rules for running the agent's playbooks applied for the inventories listed in the settings file. Any inventory without rules requires a person to approve every fix until you add its rules.
Restart. web_url, web_listen_address,
alert_intake_listen_address, and cloudflare_access change only when
MMHotline restarts. When one of them changed, MMHotline applies nothing
and names the changed keys. The command then:
- prints
Restarting MMHotline to apply the settings that require a restart: web_listen_address.; - runs
systemctl restart mmhotline.service; - waits up to one minute for the command socket to answer;
- applies again, and the result says
Restarted MMHotline first, because these keys change only when it restarts: web_listen_address.
When the socket does not answer within the minute, or MMHotline still
runs with the earlier values, such as when it could not open a new listen
address, the command exits 1 and says to read
sudo journalctl -u mmhotline -n 50. A restart drops no alert: senders
send an alert again until MMHotline answers it
(settings file design).
| Exit code | Meaning |
|---|---|
0 |
Every part applied or was unchanged |
1 |
The file has problems, a part did not apply, or the restart failed |
4 |
The command socket does not answer, or the arguments are wrong |
mmhotline settings show
Shows the settings file's path, its first comment line, when it last applied, and each part as applied or not, with why.
sudo mmhotline settings show
on_call_schedule: applied at 2026-11-15T03:58:24Z.
That is one line of the output, as
Change the settings shows it. The
whole output starts with these lines, then has one line for each part, in
the order of settings apply:
Settings file: /etc/mmhotline/mmhotline_settings.yml
Settings file first comment line: Managed by MMX from mmhotline_settings.yml in the systems repository
Last applied: 2026-11-15T03:58:24Z
- Flags: the common flags.
- Reads MMHotline's record, not the settings file, so it shows what MMHotline uses even when the file has changed since.
Last appliedis when any part last applied. A part that never applied saysnot applied yet.- Notes: each part's notes saved by the last apply, under the part.
Under
triage_agent.listed_hosts,settings showadds the vendor pages' hosts that the latest triage agent run listed fromsoftware_outside_the_distribution.yml, such astriage_agent.listed_hosts: Also listed automatically from software_outside_the_distribution.yml, as of the latest run: github.com port 443. Reason: a shortlisted_hostsin the file must not hide what the agent reaches. That note comes from MMHotline's run record, and never changes the part (settings file design). - Commit signing keys:
settings showlists each applied commit signing key with its type, comment, and fingerprint. With the triage agent set up and no keys, it saysNo commit signing keys are applied, so MMHotline refuses every MMX run and commit check. - The dashboard shows administrators the parts that failed.
- Exit code:
0whatever the parts' states.
Settings results
The --json result of settings apply and settings show. It has
message and settings_file, and no web_url.
| Field | Meaning |
|---|---|
settings_file.settings_file |
The settings file's path, as serve was given it |
settings_file.first_comment_line |
The file's first line without its #, when that line is a comment; empty otherwise |
settings_file.last_applied_at |
show only: the latest time any part applied. Absent before any part applied |
settings_file.problems |
apply only: the file's problems, each with key_path and problem. With any, nothing applied |
settings_file.restart_needed_for |
The restart keys that changed. The command restarts MMHotline, so its output has this field only when the restart did not take the new values |
settings_file.restarted_for |
The restart keys that changed, when the command restarted MMHotline before it applied the rest |
settings_file.parts |
Each part, in the order they apply |
settings_file.parts[].part |
The part's key paths, such as people or triage_agent.listed_hosts |
settings_file.parts[].state |
applied, unchanged (apply only), failed, or never_applied (show only) |
settings_file.parts[].applied_at |
When the setting in effect applied. Absent when none ever did |
settings_file.parts[].failure_reason, settings_file.parts[].failed_at |
Why and when the latest attempt failed; only with failed |
settings_file.parts[].left_out_of_the_settings_file |
true when the setting in effect is the section left out of the file, so the part is not set up; with failed, that is the setting that stays. The text then says not set up, because the settings file leaves it out. or It stays not set up, as before. in place of a time |
settings_file.parts[].notes |
The notes under the part, each with key_path and note. Absent when there are none |
mmhotline people list
Lists every active person: the settings file's people, as it last
applied.
sudo mmhotline people list
Barbara Example (barbara@example.com) is administrator.
Carlos Example (carlos@example.com) is member.
Listed 2 people. Open MMHotline at https://ops.example.com.
Set up people and the on-call schedule adds people with the settings file and checks them with this command.
- Flags: the common flags.
- Roles are
administratorandmember.
JSON fields:
| Field | Meaning |
|---|---|
people |
The active people |
people[].id |
The person's stable internal member ID. Commands take the sign-in email address instead |
people[].sign_in_email_address |
The sign-in email address, in lower case |
people[].name |
The person's display name |
people[].role |
administrator or member |
mmhotline people create-sign-in-link
Creates a one-time sign-in link, so a person can open MMHotline without Cloudflare Access: during a Cloudflare outage, or for a team that runs without Cloudflare.
sudo mmhotline people create-sign-in-link --sign-in-email-address barbara@example.com
Sign-in link: http://localhost:8480/sign-in/TOKEN
Sign-in link valid until (UTC): 2026-11-15T07:33:04Z
SSH forward command: ssh -L 8480:127.0.0.1:8480 YOUR_SSH_HOST
This sign-in link works once.
TOKEN stands for the random token in the real link.
Sign in during a Cloudflare outage
walks through the whole procedure.
| Flag | Meaning |
|---|---|
--sign-in-email-address ADDRESS |
Required. An active team member's sign-in email address |
| Common flags | As above |
- How the person uses it: on their own computer, they run the SSH
forward command with
YOUR_SSH_HOSTreplaced by the host name they use for SSH to the server, then open the link there. The port and host in the command come fromweb_listen_address. - The link works once, for 10 minutes, because it is printed in a terminal where someone could copy it. Opening it sets a sign-in cookie for 12 hours, long enough for a long incident, and opens the dashboard. A used or expired link shows how to create a new one.
- Refused for an address with no active member, such as
no member has sign-in email address nobody@example.com, exit 4. - Safety: MMHotline stores only a SHA-256 hash of each token, and the
service log names the person but never the token. Removing the person
from
peopleends their links and cookies at once, when the file applies. Withcloudflare_accessin the settings file, a page request that carries a Cloudflare Access token is judged by that token alone and ignores any sign-in cookie. - Sign-in email addresses: MMHotline removes whitespace at both ends
and stores the address in lower case. The stored address has 3 to 254
characters, exactly one
@with text on each side, and no spaces or control characters. With Cloudflare Access, it must be the address Cloudflare Access returns for the person.
JSON fields: the fields of people list,
and:
| Field | Meaning |
|---|---|
sign_in_link_url |
The one-time link, http://localhost:PORT/sign-in/TOKEN |
sign_in_link_valid_until |
The link's expiry, 10 minutes after it was made |
ssh_forward_command |
The SSH forward to run on the person's computer |
mmhotline sending-services list
Lists every sending service, set up or not, with its saved settings and never a secret. A sending service is the mail server or outside service MMHotline sends notifications through.
sudo mmhotline sending-services list
Pushover: set up by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T05:08:50Z. Checked with Pushover: monthly message limit 10000.
That is one line of the output.
Connect a sending service sets
each one up in the settings file's sending_services, which
settings apply tests and saves.
- Flags: the common flags.
- To test a sending service, add a contact method that uses it, or run
contact-methods send-testfor one that exists. - The Sending services page shows administrators the same setups, without edits.
- The saved tokens, the Twilio API key secret, and the mail server password are never returned.
JSON fields:
| Field | Meaning |
|---|---|
sending_service_setups |
One record for each sending service; no secret |
sending_service_setups[].sending_service |
mail_server, telegram, slack, twilio, or pushover |
sending_service_setups[].sending_service_title |
The name the pages show: Mail server, Telegram, Slack, Twilio, or Pushover |
sending_service_setups[].is_set_up |
Whether the sending service is set up |
sending_service_setups[].settings |
The settings that are not secret; empty before setup |
sending_service_setups[].settings.smtp_server |
The mail server's host and port |
sending_service_setups[].settings.smtp_connection_security |
starttls, implicit_tls, or unencrypted |
sending_service_setups[].settings.smtp_user_name |
The mail server user name; empty without sign-in |
sending_service_setups[].settings.email_from_address |
The sender address, without a display name |
sending_service_setups[].settings.telegram_bot_username |
The bot's username from Telegram, without @; present after setup |
sending_service_setups[].settings.telegram_bot_id |
The bot's numeric ID from Telegram; present after setup |
sending_service_setups[].settings.slack_workspace_id, .slack_workspace_name, .slack_bot_user_id, .slack_app_id |
The workspace, bot user, and app, as Slack confirmed them |
sending_service_setups[].settings.twilio_account_sid |
The account that owns the calling number |
sending_service_setups[].settings.twilio_api_key_sid |
The API key used for calls and polling |
sending_service_setups[].settings.twilio_phone_number_to_call_from |
The calling number, in E.164 form: +, country code, and number |
sending_service_setups[].settings.extra_phone_numbers_on_the_contact_card |
Comma-separated E.164 numbers, or an empty string |
sending_service_setups[].settings.pushover_monthly_message_limit |
The monthly message limit from Pushover; present after setup |
sending_service_setups[].set_up_at |
When it was set up; absent before setup |
sending_service_setups[].set_up_by |
the settings file and its path; absent before setup |
mmhotline triage-agent show
Shows the triage agent's setup, whether its sign-in works, its internet access and listed hosts, its latest setup test, its software reviews, and its recent connections. The triage agent is the team's Claude Code or Codex, run as a separate Unix user in a systemd sandbox on the MMHotline server.
sudo mmhotline triage-agent show
Triage agent sandbox installed: true.
Triage agent: Claude Code with a subscription token, model opus, reasoning effort high (coding_agent claude_code, credential_type subscription_token, model opus, reasoning_effort high), set up by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:51:06Z.
Sign-in to Claude Code: works; last checked at 2026-11-15T08:51:06Z.
Subscription token expires at: 2027-11-15T08:51:06Z
Internet access: listed_hosts_only
Listed host: api.anthropic.com:443; listed because: Claude Code's model API.
Listed host: api.osv.dev:443; listed because: OSV.dev's API.
Listed host: raw.githubusercontent.com:443; listed because: CISA's list of known exploited vulnerabilities, from CISA's copy on GitHub.
Setup test bbd54461c72208a751219d09b0050026: passed; run state: finished; purpose: setup_test; coding agent: claude_code; credential type: subscription_token.
...
Scheduled software review: off; set by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:51:06Z.
Set up the triage agent sets it up and reads this output; Keep the agents signed in reads its sign-in lines.
- Flags: the common flags.
- Only administrators use the Triage agent page; the command calls the same administrator-only operations with root's command access.
- The setup comes from the settings file's
triage_agentsection (settings file); no command changes it.
Credentials. The setup's coding_agent and credential_type:
coding_agent |
credential_type |
The credential |
|---|---|---|
claude_code |
api_key |
An Anthropic API key |
claude_code |
subscription_token |
A token from claude setup-token. show prints Subscription token expires at:, one year after the credential applied |
codex |
api_key |
An OpenAI API key |
codex |
access_token |
A Codex access token from a ChatGPT Business or Enterprise workspace. show prints Codex access token expires on: with the settings file's credential_expires_on |
codex |
chatgpt_sign_in |
The whole content of auth.json after codex login, made just for MMHotline |
Administrators get a reminder 30 days and 7 days before a subscription token or Codex access token expires (design). No credential is ever returned.
Listed hosts. MMHotline lists these hosts itself:
- the coding agent's service hosts, OSV.dev's API, and
raw.githubusercontent.com, for CISA's list of known exploited vulnerabilities (design); - at the start of each run in the systems repository, each server in the inventories of MMHotline's copy, by name at port 22, so the agent can read the servers over SSH (design);
- at the same time, the host of each vendor page in
software_outside_the_distribution.yml, from each program'sreleases_pageandsecurity_advisories_page, at port 443 or the port the address names (design).
Only runs in the systems repository reach the inventory servers and the
vendor pages' hosts; setup tests, sign-in checks, and review agent runs do
not. The settings file's triage_agent.listed_hosts adds other hosts,
each a host and a port from 1 to 65535, and neither adds nor removes the
automatic ones. With internet_access: listed_hosts_only, the agent
reaches only listed hosts; any_site also allows unlisted hosts at public
internet addresses. Listed hosts may use private addresses with either
value. Loopback, link-local, unspecified, and multicast addresses are
always refused.
show prints one line for each listed host, with why, and when the
inventory servers and vendor pages' hosts were listed:
Listed host: app001:22; listed because: A server in the inventory “production”.
Listed host: github.com:443; listed because: The releases page of caddy and of cloudflared, and the security advisories page of caddy, from software_outside_the_distribution.yml.
Inventory servers and vendor pages' hosts listed at the start of the latest run: 2026-10-28 09:00:00 UTC. A server or a page removed since stays listed until the next run.
When the latest run left a vendor page out, a line says why, such as
At the start of the latest run, MMHotline could not list the host of every vendor page in software_outside_the_distribution.yml: the releases_page of caddy, http://github.com/caddyserver/caddy/releases, is not an https:// address with a host.
When MMHotline cannot read the file, it lists no vendor page hosts, and
that line names the failure. The run continues either way.
The setup test. A setup test passes when its run finishes without a
failure reason and the answer contains
MMHotline triage agent test passed. JSON has no passed field; the text
output and the page work it out. Only one triage agent run is active at a
time, and a setup test or sign-in check of the triage agent or the review
agent takes that one place.
JSON fields. The result has message, web_url, and triage_agent;
the fields below are inside triage_agent.
| Field | Meaning |
|---|---|
sandbox_installed |
Whether the run socket and logging proxy socket are configured. It reports configuration, not the sandbox's health |
setup |
The saved setup. Absent before setup or after removal. Never holds a credential |
setup.coding_agent, setup.credential_type |
From the table above |
setup.model, setup.reasoning_effort |
Passed to the coding agent on every run |
setup.set_up_at, setup.set_up_by |
When MMHotline saved the credential, and the settings file and its path. Changing only the model or reasoning effort keeps set_up_at |
setup.credential_expires_at |
When a subscription_token stops working, 365 days after set_up_at; absent for other types |
setup.sign_in_last_checked_at |
The end of the agent's latest successful run or latest failed sign-in check, whichever is later. A passed setup test counts |
setup.sign_in_works |
false when that latest run is a failed sign-in check |
setup.sign_in_failure_reason |
Why that sign-in check failed; empty when sign-in works |
commit_signing_public_key |
The applied triage_agent.commit_signing_public_key, a public key; absent while none is applied. check-preparation compares it with the agent's git settings |
internet_access |
listed_hosts_only or any_site |
listed_hosts |
The automatic and added hosts |
listed_hosts[].host, listed_hosts[].port |
The host and port |
listed_hosts[].listed_because |
Why it is listed, such as A server in the inventory “production” |
listed_hosts[].added_by |
For a host from the settings file, the settings file and its path; absent for automatic hosts |
hosts_from_the_systems_repository_listed_at |
When the latest run in the systems repository started and listed the inventory servers and vendor pages' hosts. Absent before any such run |
inventory_servers_not_listed_because |
Why that run could not list every inventory server, such as a failed mmx nodes; absent when it listed them all |
vendor_page_hosts_not_listed_because |
Why that run left a vendor page out; absent when it left none out |
latest_setup_test |
The latest setup test; absent when none ran |
latest_setup_test.id, .purpose |
The run ID, and setup_test |
latest_setup_test.coding_agent, .credential_type |
What the test tested, which can differ from the saved setup |
latest_setup_test.coding_agent_session_id |
The coding agent's session ID; empty until reported |
latest_setup_test.state |
running, finished, stopped, or failed. finished alone does not mean it passed |
latest_setup_test.started_at, .started_by, .ended_at |
Start, author, and end; ended_at absent until recorded |
latest_setup_test.answer, .failure_reason |
The coding agent's answer, and why the test did not pass; empty when not available |
latest_setup_test.cost_in_us_dollars |
The reported cost; absent when not reported |
latest_setup_test.input_tokens, .output_tokens |
Reported tokens; zero when not reported |
latest_setup_test.cached_input_tokens |
The part of input_tokens read from the model service's cache, which costs less; zero when not reported |
latest_setup_test.steps |
Each step: sequence, occurred_at, kind (message, command, or tool), and text, cut to at most 2,000 characters |
latest_sign_in_check |
The latest sign-in check, with the fields of latest_setup_test and the purpose sign_in_check. started_by is MMHotline for the daily check. A check stopped for a triage session has the state stopped. Absent when none ran |
software_review_interval_hours |
"off", or the hours between scheduled software reviews, 1 to 8760. Also "off" before the settings file set the key |
software_review_interval_changed_at, software_review_interval_changed_by |
When the setting applied, and the settings file and its path; absent while not set |
next_software_review_due_at |
One interval after the latest counted review started. Absent while the review is off, and before any counted review, since one is then due at once. A due review still waits while a triage agent run is active |
latest_software_review |
The latest software review, without its steps; absent when none ran |
latest_software_review.run_id, .state |
The run ID; running, finished, stopped, or failed |
latest_software_review.started_at, .started_by, .ended_at |
Start; MMHotline or the person who ran run-software-review; end, absent until recorded |
latest_software_review.failure_reason |
Why it failed or stopped, such as a stop for a triage session; empty when it finished |
latest_software_review.cost_in_us_dollars, .input_tokens, .output_tokens, .cached_input_tokens |
As for latest_setup_test |
latest_software_review.report_counts_by_alert_severity.sos, .warn, .info |
How many reports the review sent at each alert severity. A refused report does not count |
latest_commit_review |
The latest commit review run, which reviews commits no trusted key signed, with the fields of latest_software_review and started_by MMHotline. Absent when none ran |
recent_connections |
The latest 50 connection attempts, newest first |
recent_connections[].id, .run_id |
The connection and its run |
recent_connections[].started_at, .ended_at |
Start and end; end absent until recorded |
recent_connections[].host, .port, .address |
The requested host and port, and the checked IP address, empty if refused before the name was looked up |
recent_connections[].decision, .refusal_reason |
allowed or refused, and why |
recent_connections[].bytes_sent, .bytes_received |
Bytes each way, updated when the connection ends |
recent_connections[].can_add_listed_host |
Whether the page offers to add the refused destination to triage_agent.listed_hosts; never for a review agent run, which a listed host does not let through |
mmhotline triage-agent check-sign-in
Runs the triage agent's sign-in check now and waits for it.
sudo mmhotline triage-agent check-sign-in
The output is that of triage-agent show,
with Sign-in to Claude Code: works; last checked at TIME. or fails and
the reason. Keep the agents signed in
uses it.
- Flags: the common flags.
- The check: an API key is checked with the vendor's models list; any other credential with the setup test's prompt, as the daily sign-in check does. It counts as the daily check: the next is due 24 hours later.
- Waits at most 5 minutes, checking every 2 seconds.
- Refused while another triage agent run is active: "A triage agent run is already active." A triage session that needs the run stops the check after the agent's current step, and the command fails with that reason.
- A failed check sends the administrators a warn alert; the next successful run sends an info alert.
- JSON: the fields of
triage-agent show, andtriage_agent_run_id, the check's run ID.
| Exit code | Meaning |
|---|---|
0 |
The sign-in works |
4 |
The check failed, did not finish within 5 minutes, or was refused |
mmhotline triage-agent run-software-review
Starts one software review now, also while the scheduled review is off, such as to learn what one review costs.
sudo mmhotline triage-agent run-software-review
Turn on the scheduled software review uses it.
- Flags: the common flags.
- Returns once the run has started, without waiting for the review, which takes minutes.
- Counts as a review: the next scheduled one is due one interval after it started.
- Refused while another triage agent run is active: "A triage agent run is already active." A triage session that needs the run stops the review after the agent's current step.
- The review uses the team's task in
scheduled_software_review_task.mdin the systems repository. A task file that is missing, empty, or over 64 KiB ends the review at once as failed;showprints the reason asSoftware review failure reason:. - What one review costs: wait until
showprintsSoftware review ended at:, then readlatest_software_reviewinshow --json:cost_in_us_dollarswhen the coding agent reports it, and the token counts. - The review's reports reach people as alerts;
showprints how many it sent at each alert severity. - JSON: the fields of
triage-agent show, andtriage_agent_run_id, the review's run ID.
mmhotline triage-agent check-preparation
Checks the eight things a team prepares by hand before the triage agent
works, and prints one line for each: Present: or Missing:.
sudo mmhotline triage-agent check-preparation
Present: 1. Server names resolve: web1 -> 10.0.0.11.
Present: 2. The triage user accepts the agent's key: triage@ on web1 accepts /var/lib/mmhotline-triage-agent/.ssh/id_ed25519.
...
Present: 8. mmx and its node runners are installed: /usr/local/bin/mmx, /usr/local/bin/mmx-runner-linux-amd64.
All 8 checks are present.
Set up the triage agent runs it after the preparation steps.
| Flag | Meaning |
|---|---|
--json |
Print one JSON object |
--help |
Show the flags and exit codes |
- Runs as root, without the command socket for its checks. Reason: it
reads the triage agent's home folder,
/var/lib/mmhotline-triage-agent(mode0700), and logs in with the agent's private key and with MMHotline's own, which the sandboxedmmhotline servecannot do. It asks the running MMHotline only for the applied setup and commit signing key. It takes no--command-socket. - Changes nothing, and never prints a private key, a token, or a credential from the settings.
- The checks, each with the preparation step that fixes it:
- Each server name in the inventories resolves to an address that is not loopback (step 1).
- The
triageuser on each server accepts the agent's key,id_ed25519in its home (step 2). - The agent's
.gitconfigsigns commits with SSH, using the appliedtriage_agent.commit_signing_public_key(step 3). This line stays missing until the agent is set up. triage_agent_rules.mdexists in MMHotline's copy of the systems repository (step 4).- The copy's
ssh/known_hostsholds a host key for each server (step 4). - MMHotline's own key logs in to each server, as the inventory's
ssh_userorroot(step 5). - The copy's
originis an SSH address, andgit ls-remote --heads originworks as themmhotlineuser (step 6). mmxis installed, withmmx-runner-linux-amd64ormmx-runner-linux-arm64beside it for each server's processor (step 7).
- A missing line ends with
Fix:and the how-to page's address with the step. A check whose input is missing, such as no copy or no servers, is missing with the reason. The last line isAll 8 checks are present.or, such as,7 of 8 checks are present; 1 is missing.
| Exit code | Meaning |
|---|---|
0 |
All eight are present |
1 |
At least one is missing |
3 |
It cannot run, such as without root |
4 |
The arguments are wrong |
JSON fields: checks, one for each check:
| Field | Meaning |
|---|---|
checks[].number, checks[].name |
The check's number and name, such as 1 and Server names resolve |
checks[].present |
Whether the check passed |
checks[].detail |
What the check found |
checks[].fix |
The how-to page's address and step that fixes a missing check |
mmhotline triage-agent run-inside-sandbox
systemd starts this command for each triage agent run; people do not run it.
mmhotline triage-agent run-inside-sandbox --proxy-socket PATH
--proxy-socket PATHis required: the logging proxy socket mounted in the sandbox. Its installed path is/run/mmhotline-triage-agent-proxy.sock.- It takes
--helpand none of the common flags. It exchanges run messages over standard input and output, not the command socket.
mmhotline triage-agent ssh-proxy
SSH runs this command inside a triage agent run, as the ProxyCommand of
the agent's ~/.ssh/config; people do not run it.
mmhotline triage-agent ssh-proxy HOST PORT
- Carries one SSH connection through MMHotline's logging proxy.
- Refuses an IP address and says to connect by the server's name in the inventory. Refuses outside a run.
- When the proxy refuses, it exits 255 and prints the proxy's reason on standard error.
- It takes
--helpand none of the common flags.
mmhotline triage-agent ssh
The agent's ssh runs this command: each triage agent run puts it first
on the agent's search path. People do not run it.
mmhotline triage-agent ssh SSH_ARGUMENTS
- Drops each
-Foption and the file it names, with a notice on standard error, then replaces itself with the realsshand the remaining arguments, so the agent's~/.ssh/configapplies (the agent's ssh). - Exits 255 outside a run. It takes
ssh's arguments and has no--helpof its own.
mmhotline triage-agent request-playbook-run
The triage agent runs this command inside a run to ask MMHotline to run an MMX playbook; people do not run it (requests from the agent).
mmhotline triage-agent request-playbook-run --playbook PATH --inventory NAME --diagnosis-file FILE [--json]
FILEholds the agent's diagnosis in Markdown, at most 64 KiB. The review agent and the people who approve read it.- The request carries the agent's commits, which MMHotline pushes when
they change nothing outside
one_time_playbooks/andtriage_agent_notes/.
| Exit code | Meaning |
|---|---|
0 |
The playbook ran and succeeded |
1 |
The playbook ran and failed |
3 |
The request waits for a person |
4 |
Refused or invalid, or the run is ending. Also outside a run |
mmhotline triage-agent report
The triage agent runs this command inside a run to send a report, which MMHotline records with the run and sends as an alert; people do not run it.
mmhotline triage-agent report --alert-severity info|warn|sos --report-file FILE [--json]
FILEholds the report in Markdown, at most 64 KiB.- Exit codes:
0recorded;4refused or invalid, or the run is ending, or outside a run.
mmhotline triage-agent review-commit
The triage agent runs this command in a commit review run to give its decision on a commit no trusted key signed; people do not run it.
mmhotline triage-agent review-commit --commit-id COMMIT_ID --decision approve|needs_a_person --report-file FILE [--alert-severity info|warn|sos] [--json]
approvemarks the commit reviewed and sends one info alert with the report.needs_a_personleaves the commit waiting and sends the report at--alert-severity, whichneeds_a_personrequires andapproverefuses.FILEholds the report in Markdown, at most 64 KiB.- Exit codes:
0recorded;4refused, such asapproveof a commit that changes a path only a person may approve, or invalid, or outside a run (the agent's decision).
Review a commit says what happens to the decision.
mmhotline run-fleet-view-command
systemd starts this command as root, from mmhotline-fleet-view@.service,
for each connection to the fleet view socket; people do not run it.
mmhotline run-fleet-view-command --settings-file /etc/mmhotline/mmhotline_settings.yml
- Reads
fleet_view_commandfrom the settings file on each connection and runs its words without a shell. The command gets onlyPATHin its environment and the null device as input; its output goes to the connection, and its errors to the journal. - Exits 0 whatever the fleet view command's exit status, because a fleet view exits nonzero while a server fails. Exits 1 when the command cannot start at all.
- Exits 4 when
--settings-fileis missing, whenfleet_view_commandis missing or invalid, or when the settings file fails its owner and mode check: root runs the command, so only root may change the file. Problems in other settings do not stop it.
Connect a tool that sends alerts
sets fleet_view_command.
mmhotline review-agent show
Shows the review agent's setup, whether its sign-in works, its latest setup test, and its latest sign-in check. The review agent is an optional second coding agent that must approve each playbook the triage agent asks to run, before the rules for running the agent's playbooks apply.
sudo mmhotline review-agent show
Triage agent sandbox installed: true.
Review agent: Codex with an API key, model gpt-6-astra, reasoning effort high (coding_agent codex, credential_type api_key, model gpt-6-astra, reasoning_effort high), set up by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:55:26Z.
Sign-in to Codex: works; last checked at 2026-11-15T08:55:26Z.
Setup test 17916abc6746d58900e9d6953b6202d4: passed; run state: finished; purpose: review_agent_setup_test; coding agent: codex; credential type: api_key.
...
Set up the triage agent adds it.
- Flags: the common flags.
- Setup: the settings file's
review_agentsection, with the triage agent's five setup keys and their rules. Its setup test runs insettings apply. Leaving the section out removes it, and playbooks are no longer reviewed. - Reviews: when the review agent refuses, its reason goes back to the triage agent, which may revise the playbook and ask again. After the third refusal in a row on a problem, the request waits for a person. Any coding agent may review, also the triage agent's own; one from another vendor is recommended, because it is less likely to share the triage agent's mistakes (design).
- JSON:
message,web_url, andreview_agent, withsandbox_installed,setup,latest_setup_test, andlatest_sign_in_check, in the fields oftriage-agent show.latest_setup_test.purposeisreview_agent_setup_test, andlatest_sign_in_check.purposeisreview_agent_sign_in_check.
mmhotline review-agent check-sign-in
Runs the review agent's sign-in check now, as a review runs, and waits for
it, as triage-agent check-sign-in
does, with the same exit codes.
sudo mmhotline review-agent check-sign-in
- Flags: the common flags.
- JSON: the fields of
review-agent show, andreview_agent_run_id, the check's run ID.
Used in Keep the agents signed in.
mmhotline agent-playbook-rules show
Shows the rules for running the agent's playbooks: for each inventory and kind of action, whether MMHotline runs a playbook the triage agent asked for without a person. It also shows the services whose changes a person must approve.
sudo mmhotline agent-playbook-rules show --inventory production
Services whose changes a person must approve, on every inventory: lab-database.service; changed by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:39:15Z.
Rules for running the agent's playbooks on inventory production:
Start or restart a service (start_or_restart_a_service): Act, then report (act_then_report); changed by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:39:15Z.
...
Reboot a server (reboot_a_server): Get permission, then act (1 approval) (get_permission_then_act); changed by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:39:15Z.
...
The rules for running the agent's playbooks and the services whose changes a person must approve are set in /etc/mmhotline/mmhotline_settings.yml, in rules_for_running_agent_playbooks and services_whose_changes_a_person_must_approve.
Decide what the triage agent may do sets the rules. The design owns the kinds of action, the presets, and how a playbook's rule is chosen.
| Flag | Meaning |
|---|---|
--inventory NAME |
Optional. Show only this inventory: letters, digits, _, and -, at most 100 characters |
| Common flags | As above |
- Without
--inventory, it lists every inventory in MMHotline's copy of the systems repository and every inventory with saved rules. - An inventory without an entry in
rules_for_running_agent_playbooksfails that part ofsettings apply, and every kind of action on it uses Get permission, then act with 1 approval until the entry is added. - A kind of action an entry's
action_kindsleaves out has no saved rule and uses Get permission, then act with 1 approval. The part still applies, with a note.showand the page addcheck_only has no rule; it needs a person until you add oneto that kind's line. - Members read the same rules on the page of rules for running the agent's playbooks.
JSON fields: message, web_url, and:
| Field | Meaning |
|---|---|
services_whose_changes_a_person_must_approve.service_names |
The unit names, such as pgautofailover.service, in the order given; an empty list when none |
services_whose_changes_a_person_must_approve.changed_at, .changed_by |
When the list was saved, and the settings file and its path; absent while not set |
agent_playbook_rules |
One entry for each inventory |
agent_playbook_rules[].inventory |
The inventory's name |
agent_playbook_rules[].inventory_file_exists |
Whether MMHotline's copy has inventories/<inventory>.yml |
agent_playbook_rules[].action_kinds |
One entry for each kind of action: start_or_restart_a_service, free_disk_space, pull_a_container_image_again, reboot_a_server, block_a_network_range, change_dns_records, check_only, and unrecognized_task |
action_kinds[].action_kind, action_kinds[].action_kind_title |
The kind's ID and name, such as reboot_a_server and "Reboot a server" |
action_kinds[].rule, action_kinds[].rule_title |
get_permission_then_act, warn_wait_then_act, or act_then_report, and its name with its wait or approvals, such as "Warn, wait 10 minutes, then act" |
action_kinds[].warning_wait_minutes |
Minutes to wait after the warning; only with warn_wait_then_act |
action_kinds[].approvals_needed |
People who must approve, 1 or 2; only with get_permission_then_act |
action_kinds[].changed_at, action_kinds[].changed_by |
When the rule was saved, and the settings file and its path; absent while the kind has no saved rule |
agent_playbook_rules[].listed_playbooks |
The playbooks listed by name under the inventory's playbooks in the settings file, in name order; an empty list when none |
listed_playbooks[].playbook_name |
NAME of playbooks/NAME.yml |
listed_playbooks[].rule, .rule_title, .warning_wait_minutes, .approvals_needed, .changed_at, .changed_by |
As in action_kinds |
listed_playbooks[].its_approval_also_approves_in_the_same_session |
The listed playbooks its approval also approves once in the same triage session; absent when none. The text adds "its approval also approves in the same session: every_server, destroy_replaced_servers" after the rule |
listed_playbooks[].starts_a_server_replacement |
true when a run of the playbook starts a server replacement, during which an SOS of the session waits longer; absent otherwise |
mmhotline agent-playbook-rules check-whether-a-playbook-needs-a-person
Shows whether a playbook needs a person when the triage agent asks to run it on an inventory, and why. It changes nothing.
sudo mmhotline agent-playbook-rules check-whether-a-playbook-needs-a-person \
--inventory production --playbook playbooks/stop_lab_database.yml
Needs a person: Get permission, then act (1 approval).
Why: A node task acts on a service on the list of services whose changes a person must approve, so the playbook waits for a person whatever the rules for its kinds of action, with the number of approvals required by the strictest rule, and at least 1 approval: Get permission, then act (1 approval).
Checked playbooks/stop_lab_database.yml in the inventory “production” at commit a4fce954720b97d948bdb9f80640cbc5ee5ede5e of MMHotline's copy of the systems repository.
Node tasks, as mmx explain listed them, with their kinds of action:
stop/task_1 on web1: systemd_service; Unrecognized task
Rules for the playbook's kinds of action on the inventory “production”:
Unrecognized task (unrecognized_task): Get permission, then act (1 approval) (get_permission_then_act); changed by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T08:39:15Z.
Tasks on services whose changes a person must approve:
stop/task_1 (systemd_service) stops lab-database.service on web1, which is on the list of services whose changes a person must approve
An administrator changes the list in services_whose_changes_a_person_must_approve in MMHotline's settings file.
Tasks MMHotline did not recognize, and why:
stop/task_1 (systemd_service): systemd_service.state must be running
MMHotline checked the playbook as it checks a triage agent's request to run it, and changed nothing: it ran only mmx explain, and recorded no request, asked for no approval, sent no alert, and ran no playbook.
Decide what the triage agent may do uses it.
| Flag | Meaning |
|---|---|
--playbook PATH |
Required. The playbook's path in the systems repository, as the triage agent gives it: playbooks/NAME.yml or one_time_playbooks/YYYY-MM-DD_what_it_does.yml |
--inventory NAME |
Required. NAME of inventories/NAME.yml |
| Common flags | As above |
-
How it checks: exactly as MMHotline checks the agent's request (design): it runs
mmx explainin MMHotline's copy of the systems repository, at the copy's current commit. It fetches nothing, so the playbook must be in a commit the copy already holds. -
Changes nothing: no request, no approval, no alert, no playbook run.
-
The first line is one of three:
First line What MMHotline does with the agent's request Needs a person: Get permission, then act (1 approval).Runs the playbook only after people approve it, 1 or 2 as the rule says Runs without a person unless someone presses Stop: Warn, wait 10 minutes, then act.Warns the team, waits, then runs it unless someone presses Stop Runs without a person: Act, then report.Runs it at once -
With a review agent set up, a playbook that would run without a person gets this line after
Why::A review agent would also review the playbook before the playbook runs. The check command does not ask the review agent.A playbook that needs a person is not reviewed and gets no such line. -
Refused, exit 4, where the agent's request would be refused, with the same message: a path outside
playbooks/andone_time_playbooks/, an invalid inventory name, or a playbook MMX cannot read. For example:MMHotline refused the request: the playbook path "one_time_playbooks/../deploy.yml" must not have an empty, ".", or ".." part. -
Waits while an MMX run holds the copy's lock, and times out after 30 seconds; it works again once the MMX run ends.
JSON fields: message, web_url, and
whether_a_playbook_needs_a_person:
| Field | Meaning |
|---|---|
playbook, inventory |
What was checked |
needs_a_person |
true only for get_permission_then_act; false for act_then_report, and for warn_wait_then_act, which runs unless someone presses Stop |
rule, rule_title |
The rule the playbook takes, with approvals_needed for get_permission_then_act and warning_wait_minutes for warn_wait_then_act |
reason |
Why, in plain words |
listed_playbook |
The rule a person listed for the playbook by name, in the fields of show's listed_playbooks; absent when not listed |
review_agent_would_review_the_playbook |
true when a review agent is set up and the playbook would run without a person |
commit_id |
The commit of MMHotline's copy that mmx explain read |
tasks |
Each node task: path, node (the server it runs on), module, action_kind, and action_kind_title |
action_kinds |
The playbook's kinds of action with their rules, in the fields of show's action_kinds |
tasks_on_services_whose_changes_a_person_must_approve |
path, node, module, service_name, and reason, as in approval-requests show. Present also when empty |
unrecognized_tasks |
path, module, and reason, as in approval-requests show. Present also when empty |
mmhotline contact-methods list
Lists contact methods: all of them, one person's, or the team's shared ones.
sudo mmhotline contact-methods list --sign-in-email-address barbara@example.com
Contact method fe9d9b2757973d600bf0142391667e35 belongs to barbara@example.com.
Contact method type: email. Sends to: barbara@example.com. Alert severities: info,sos,warn.
Send while unavailable or muted: true. Last test message: accepted (queued at 2026-11-15T05:41:53Z).
Listed 1 contact method.
Add contact methods uses every
contact-methods command.
| Flag | Meaning |
|---|---|
--sign-in-email-address ADDRESS |
Optional. Only this person's contact methods |
--shared-with-team |
Optional. Only the team's shared contact methods |
| Common flags | As above |
JSON fields of every contact-methods command: message, web_url,
and:
| Field | Meaning |
|---|---|
contact_methods |
The listed methods, or the one a save or test affected. remove returns none |
contact_methods[].contact_method_id |
The ID that change, remove, and send-test take |
contact_methods[].sign_in_email_address |
The owner; empty for a shared method |
contact_methods[].shared_with_team |
Whether the method belongs to the team |
contact_methods[].contact_method_type |
email, outgoing_webhook, automated_voice_call, pushover_notification, telegram_message, or slack_message |
contact_methods[].sends_to |
The destination the pages show. A webhook's path and query are hidden, because they may hold tokens |
contact_methods[].alert_severities |
The alert severities the method gets |
contact_methods[].send_while_unavailable_or_muted |
Whether it sends while the person is unavailable or the team is muted, within the Pushover and phone rules |
contact_methods[].last_test_message |
The latest test message, or null |
contact_methods[].last_test_message.delivery_state |
Such as queued, accepted, failed, or uncertain |
contact_methods[].last_test_message.delivery_state_explanation |
Why, such as an error, a wait, or the service's answer (HTTP 200) |
contact_methods[].last_test_message.queued_at |
When the test was queued |
outgoing_webhook_signing_secret_id |
add of a webhook only: the new signing secret's ID |
outgoing_webhook_signing_secret_shown_once |
add of a webhook only, without an output file: the new signing secret in hex |
outgoing_webhook_signing_secret_file |
add of a webhook only, with an output file: its path, in place of the secret |
mmhotline contact-methods add
Adds a contact method for a person or for the team, and queues a test message.
sudo mmhotline contact-methods add --sign-in-email-address barbara@example.com \
--contact-method-type email --email-address barbara@example.com \
--alert-severities sos,warn,info --send-while-unavailable-or-muted
Contact method fe9d9b2757973d600bf0142391667e35 belongs to barbara@example.com.
Contact method type: email. Sends to: barbara@example.com. Alert severities: info,sos,warn.
Send while unavailable or muted: true. Last test message: queued (queued at 2026-11-15T05:41:53Z).
Contact method saved. A test message for the contact method is queued. Inspect the last test result to check delivery.
| Flag | Meaning |
|---|---|
--sign-in-email-address ADDRESS or --shared-with-team |
Required: the owner, one person or the team |
--contact-method-type TYPE |
Required: email, outgoing_webhook, automated_voice_call, or pushover_notification |
--alert-severities VALUES |
Comma-separated sos, warn, info, in any order, without repeats. An empty string gets no alerts. Required, except for Pushover, which defaults to sos,warn,info. Pass --alert-severities sos for SOS only. An automated voice call takes only sos |
--send-while-unavailable-or-muted |
Optional. Without it, the method follows the person's unavailability and the team's mute. A team mute always silences Pushover and phone calls, even with this flag |
--email-address ADDRESS |
With email: the address |
--outgoing-webhook-url URL |
With outgoing_webhook: the receiver's https:// URL |
--outgoing-webhook-bearer-token-file FILE |
With outgoing_webhook, optional: a file holding only the bearer token; surrounding whitespace is trimmed |
--outgoing-webhook-signing-secret-file FILE |
With outgoing_webhook, optional: where to write the new signing secret, instead of printing it |
--phone-number-to-call NUMBER |
With automated_voice_call: E.164 form, such as +12025550123. Personal methods only |
--pushover-user-key-file FILE|- |
With pushover_notification: a file only its owner can read, or - for standard input, holding the person's own user key, exactly 30 letters and digits, not a delivery-group key. Personal methods only; needs the Pushover sending service |
| Common flags | As above |
- The test message is queued in the same save, and the method gets
alerts at once. A queued test is not proof of delivery; read the last
test result with
list. An automated voice call's test counts against the daily test limits; at the limit, the method is still added, no test is queued, andmessagesaysTest not sent:and why. - Telegram and Slack are connected by the person on the Contact methods page. Asking for one returns that instruction with the page's full address.
- Phone numbers keep the page's limits on allowed countries, duplicates, and numbers per person.
- The Pushover user key never goes on the command line, where it
would stay in the shell history and the list of running processes. The
command request and the stored settings call it
pushover_user_key; results show it insends_to. Saving queues an emergency test alert. - The webhook signing secret is made once and shown once. The output
file gets mode
0600and holds only the secret, without a newline. The command writes a private temporary file in the same folder first, then renames it over the path, which replaces an existing file. The folder must exist. - Refused, exit 4, for a flag of another contact method type, a key
that is not 30 letters and digits, or a duplicate, such as
you already added this Pushover notification contact method.
mmhotline contact-methods change
Changes a contact method's alert severities, or whether it sends while the person is unavailable or the team is muted.
sudo mmhotline contact-methods change --contact-method-id fe9d9b2757973d600bf0142391667e35 --alert-severities sos
| Flag | Meaning |
|---|---|
--contact-method-id ID |
Required |
--alert-severities VALUES |
As for add |
--send-while-unavailable-or-muted true|false |
Give one or both of these two flags |
| Common flags | As above |
A change queues no new test message.
mmhotline contact-methods remove
Removes a contact method.
sudo mmhotline contact-methods remove --contact-method-id fe9d9b2757973d600bf0142391667e35
- Flags:
--contact-method-id ID, required, and the common flags. - Removing the contact methods that use a sending service lets
settings applyremove that sending service.
mmhotline contact-methods send-test
Sends a contact method a test message now.
sudo mmhotline contact-methods send-test --contact-method-id 2c35b3af0c71bc6da2ea6e1c9cd3ce59
Contact method 2c35b3af0c71bc6da2ea6e1c9cd3ce59 belongs to barbara@example.com.
Contact method type: pushover_notification. Sends to: FakePushoverPersonalUserKey001. Alert severities: info,sos,warn.
Send while unavailable or muted: false. Last test message: queued (queued at 2026-11-15T05:08:44Z).
Test message queued; inspect the last test result for delivery.
- Flags:
--contact-method-id ID, required, and the common flags. - It keeps the page's test rate and phone-call cost limits.
- Set up the MMHotline-down alarm and Connect Pushover use it to check a contact method.
mmhotline alert-sources list
Lists the alert sources, each with its addresses and check-in state, and never a signing secret's value. An alert source is one sender of signed alerts and check-ins, such as a monitoring tool.
sudo mmhotline alert-sources list
Alert source Monitoring (monitoring) is active.
Intake URL: http://10.20.0.1:8484/alert-intake/monitoring
Check-in URL: http://10.20.0.1:8484/check-in/monitoring
No check-in has arrived yet. MMHotline starts watching for missing check-ins when the first check-in arrives.
...
Alert intake address: http://10.20.0.1:8484
Connect a tool that sends alerts
adds sources with the settings file's alert_sources
(settings file) and reads this list;
Declare planned downtime reads
its downtime. Alert requests shows a sender that uses
these addresses.
- Flags: the common flags only.
- A source left out of the settings file is turned off: its signing secrets stop working at once, and its alerts and check-ins stay. Listing it again turns it back on with the listed secrets.
- Senders send to the alert intake address, never to the web URL: the web URL sits behind Cloudflare Access, which refuses a sender without a person's sign-in. A sender on the MMHotline server uses the same address as senders elsewhere.
- The Alert sources page shows the same addresses.
JSON fields: message, alert_intake_address, and alert_sources;
no web_url.
| Field | Meaning |
|---|---|
alert_intake_address |
Once, for every source: http:// and alert_intake_listen_address, such as http://10.20.0.1:8484 |
alert_sources[].alert_source_id |
The source's ID, such as monitoring |
alert_sources[].name |
Its display name |
alert_sources[].active |
Whether signed requests are accepted; false for a source the settings file no longer lists |
alert_sources[].signing_secrets |
Each secret's signing_secret_id and created_at, when it was first saved; never the value |
alert_sources[].intake_url |
The intake address, /alert-intake/, and the source ID |
alert_sources[].check_in_url |
The intake address, /check-in/, and the source ID |
alert_sources[].check_in_state |
The check-in state, below |
check_in_state matches the source's page:
| Field | Meaning |
|---|---|
expected |
At least one check-in has arrived |
last_check_in_at |
The latest check-in's receipt time |
last_heard_at |
The latest receipt of an alert or check-in |
sent_by_node_name |
The server the last check-in named |
warn_after_no_check_in_for_minutes |
Minutes without a check-in before a Warning; 0 means off |
sos_after_no_check_in_for_minutes |
Minutes without a check-in before an SOS |
nodes_in_whole_node_maintenance |
The maintenance windows the sender listed, each with node_name, maintenance_until, and optional reason and started_by |
planned_downtime |
The downtime in effect, or null |
planned_downtime.planned_downtime_id |
The ID of downtime a team member set; empty for downtime the sender listed |
planned_downtime.started_by_name |
Who started the downtime |
planned_downtime.sender_maintenance_node_name |
The server whose sender-listed window gives the downtime; empty for downtime set by hand |
planned_downtime.until, planned_downtime.reason |
When the downtime ends, and why |
mmhotline schedules show
Shows the saved on-call schedule, and who an SOS wakes first now and who next, with unavailability and taking on-call counted, as the dashboard's On call panel shows.
sudo mmhotline schedules show
Active schedule type: weekly_rotation. Schedule revision: 2.
Time to claim before MMHotline escalates: 5 minutes. Each escalation step waits this long for a claim before the next escalation step starts.
Saved weekly_rotation schedule in America/New_York. Handoff: 09:00; weekday: 1 (Sunday is 0); first start date: 2026-11-02. Backup option: everyone_available; fixed backup: no one scheduled.
Rotation position 1: barbara@example.com.
Rotation position 2: carlos@example.com.
On call now: Carlos Example (carlos@example.com). Backup now: everyone else available: Barbara Example (barbara@example.com).
Listed the saved schedule, set in /etc/mmhotline/mmhotline_settings.yml, in on_call_schedule.
Set up people and the on-call schedule
sets the schedule; Be away or take over on call
and Mute the team read the On call now
line.
- Flags: the common flags.
- The schedule comes from the settings file's
on_call_schedule(settings file).settings applysaves and activates it and removes saved schedules of other types. The backup is where an SOS escalates; nothing else sets escalation. - The
On call nowline saysOn call now: nobody.when nobody is scheduled or available, andeveryone available:oreveryone else available:when the schedule calls for everyone. It adds the reason when coverage differs from the plain schedule, such asThe person who took on-call is contacted first., and while the team is muted, that an SOS wakes nobody until the mute ends. - People are named by sign-in email address. MMHotline saves member IDs and turns them back into addresses when it reads them.
The time to claim is
on_call_schedule.escalate_if_unclaimed_after_minutes, 1 to 60: how long
each escalation step, such as the person on call and then the backup,
waits for a claim before MMHotline contacts the next step
(incident design).
Each step stores its end time when it starts. A new time to claim applies
to every step that starts later, also the next step of an open incident;
the current step keeps its end time, which the incident page shows. For
example, an SOS opened at 09:00 with five minutes still reaches the
backup at 09:05 when two minutes applies at 09:01; the backup then has
until 09:07.
JSON fields: message, web_url, and schedule_state:
| Field | Meaning |
|---|---|
schedule_state.active_schedule_type |
The active type; empty before the schedule first applied |
schedule_state.revision |
The schedule's revision; it grows with each save |
schedule_state.escalate_if_unclaimed_after_minutes |
The time to claim |
schedule_state.schedule_configurations |
The saved schedule, as a list of one object, below; empty before the schedule first applied |
schedule_state.on_call_now |
Who an SOS wakes now, below |
schedule_configurations[] uses the names of on_call_schedule's keys:
| Field | Meaning |
|---|---|
schedule_type |
day_of_week, hour_of_day, weekly_rotation, or custom_schedule_code |
time_zone |
An IANA time zone name, such as America/New_York |
handoff_hour, handoff_minute |
The local handoff time, 0–23 and 0–59; daily and weekly schedules |
assignment_sign_in_email_addresses |
Seven daily slots, Sunday first, or 24 hourly slots, midnight first. An empty address means nobody is scheduled first for an SOS then. Empty for rotation and custom |
rotation_sign_in_email_addresses |
The rotation, in order; empty for other types |
handoff_weekday |
Sunday 0 to Saturday 6; weekly rotation |
first_start_date |
The first person's local start date, YYYY-MM-DD, on the handoff weekday |
backup_option |
everyone_available, fixed_person, or, for a weekly rotation only, next_in_rotation |
backup_sign_in_email_address |
The fixed backup with fixed_person; empty otherwise |
custom_schedule_code |
The Starlark code; custom schedules only. Its results primary_sign_in_email_address and backup_sign_in_email_address are sign-in email addresses (Write a custom schedule) |
custom_schedule_test_cases |
Each with local_time, expected_primary_sign_in_email_address, and expected_backup_sign_in_email_address |
on_call_now:
| Field | Meaning |
|---|---|
at |
The time the answer is for |
on_call |
Who an SOS wakes first, each with name and sign_in_email_address; present also when empty |
on_call_is_everyone_available |
true when that is everyone available |
backup |
Who it wakes next, each with name and sign_in_email_address; present also when empty |
backup_is_everyone_available |
true when that is everyone else available |
reason |
Why coverage differs from the plain schedule, such as when nobody is scheduled; absent otherwise |
team_muted_until |
While the team is muted, when the mute ends: until then, no Pushover notification or automated voice call goes out, so an SOS wakes nobody |
mmhotline schedules preview
Works out a settings file's schedule before it applies, and saves nothing.
sudo mmhotline schedules preview --settings-file /etc/mmhotline/mmhotline_settings.yml
At 2026-11-15T06:18:01Z, primary: carlos@example.com; backup option: fixed_person; backup: barbara@example.com.
At 2026-11-16T14:00:00Z, primary: barbara@example.com; backup option: fixed_person; backup: barbara@example.com.
At 2026-11-21T14:00:00Z, primary: carlos@example.com; backup option: fixed_person; backup: barbara@example.com.
Schedule preview complete. Nothing was saved.
Set up people and the on-call schedule and Write a custom schedule use it.
| Flag | Meaning |
|---|---|
--settings-file FILE |
Required. The file whose on_call_schedule to preview |
| Common flags | As above |
- How it works: it reads the file's
on_call_scheduleand asks the running MMHotline to work out the assignments. - Refused, exit 4, for a file with problems, which it lists as
settings checkdoes; and for a sign-in email address thepeoplepart has not applied yet:unknown sign-in email address dana@example.com; the schedule names only people already on the team, so apply the settings file with those people first. Custom schedule code runs in its sandbox with its test cases, and a failed run or test case is refused with the reason, assettings applyrefuses it. - The period: the next 24 hours for hourly assignments, eight future handoffs for daily assignments and weekly rotations, and 14 days for custom code; the On call page covers the same.
- It shows the schedule's own answer. Personal unavailability and
taking on call count only when an incident needs someone; see
schedules showfor the live answer.
JSON fields: message, web_url, and preview, the current
assignment and each later change:
| Field | Meaning |
|---|---|
preview[].at |
When the change happens |
preview[].primary_sign_in_email_address |
Who is scheduled first; empty means nobody |
preview[].backup_option |
How the backup is chosen |
preview[].backup_sign_in_email_address |
The chosen or fixed backup; empty when the backup is everyone available |
mmhotline down-alarm show
Shows the MMHotline-down alarm: its Worker, the last accepted check-in or the failing check-ins, who it would wake, and what it last sent. The MMHotline-down alarm is a Cloudflare Worker in the team's own Cloudflare account that wakes the person on call when MMHotline stops checking in with it.
sudo mmhotline down-alarm show
MMHotline-down alarm Worker mmhotline-down-alarm-ops-example-com-7db8a22e at https://mmhotline-down-alarm-ops-example-com-7db8a22e.example.workers.dev.
Set up by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T07:46:30Z.
Last check-in accepted at 2026-11-15T07:46:30Z.
If MMHotline stops checking in: Pushover notification to Carlos Example (u...) after 10 minutes.
If MMHotline stops checking in: Automated voice call to Carlos Example (+12025550124) after 10 minutes.
If MMHotline stops checking in: Pushover notification to Barbara Example (u...) after 20 minutes.
Set up the MMHotline-down alarm
sets it up with the settings file's mmhotline_down_alarm and uses both
down-alarm commands. The
design
explains how it works.
- Flags: the common flags.
- Any member may show the alarm and send a test.
- Without the alarm, it prints
The MMHotline-down alarm is not set up.
JSON fields: message and mmhotline_down_alarm; no web_url. Results
never include the Cloudflare API token, the check-in token, or a held
message's request.
| Field | Meaning |
|---|---|
mmhotline_down_alarm.is_set_up |
Whether the alarm is set up |
mmhotline_down_alarm.cloudflare_account_id, .worker_name, .worker_url |
The account, the Worker's name, and its workers.dev address |
mmhotline_down_alarm.set_up_at, .set_up_by |
When setup last ran, and the settings file and its path |
mmhotline_down_alarm.last_check_in_accepted_at |
The last accepted check-in since MMHotline started |
mmhotline_down_alarm.check_ins_failing_since, .last_check_in_error |
Present while check-ins fail: since when, and why |
mmhotline_down_alarm.held_messages |
Who would be woken: person_name, contact_method_type, sent_to, and delay_minutes |
mmhotline_down_alarm.recent_sends |
The sends the alarm reported in its answer to the last accepted check-in: send_id, person_name, contact_method_type, due_at, send_outcome, and http_status_code |
mmhotline_down_alarm.test_sends |
send-test only: each test's person_name, contact_method_type, send_outcome, and http_status_code |
send_outcome is delivered, refused_by_service,
not_delivered_within_10_minutes, stopped_by_check_in, or, for a test,
no_answer or service_unavailable.
mmhotline down-alarm send-test
Checks in, then has the MMHotline-down alarm send a test of each held message now, such as an emergency Pushover notification titled "Test of the MMHotline-down alarm".
sudo mmhotline down-alarm send-test
Test: Pushover notification to Carlos Example: refused by the service (HTTP 400).
Test: Automated voice call to Carlos Example: refused by the service (HTTP 401).
Test: Pushover notification to Barbara Example: refused by the service (HTTP 400).
The MMHotline-down alarm sent a test alert for each held message; 3 of 3 were not accepted, so those people would not be woken. Check the contact methods with sudo mmhotline contact-methods send-test.
The output starts with the lines of
down-alarm show. Its last line is one of:
-
The MMHotline-down alarm sent a test alert for each held message, and each service accepted it. -
... 1 of 3 was not accepted, so that person would not be woken. Check the contact method with sudo mmhotline contact-methods send-test., or the line above for several. -
The MMHotline-down alarm holds no message, so it would wake nobody. -
Flags: the common flags.
-
JSON: the fields of
down-alarm show, withtest_sends.
| Exit code | Meaning |
|---|---|
0 |
Every test was accepted (send_outcome delivered) |
1 |
A test was not accepted, or the alarm holds no message |
4 |
Refused, such as when the alarm is not set up, or wrong arguments |
mmhotline status-page set-up
Creates the status page in the team's Cloudflare account, from a laptop, and prints what the settings file needs next. The status page is a public page where the team tells its customers about a problem.
mmhotline status-page set-up --cloudflare-account-id 0123456789abcdef0123456789abcdef \
--host-name status.example.com --cloudflare-api-token-file ~/cloudflare-api-token
The status page is set up in Cloudflare account 0123456789abcdef0123456789abcdef: the R2 bucket mmhotline-status-status-example-com-18a8ea11, the host name status.example.com connected to it, and the rule that serves index.html at https://status.example.com/.
r2_bucket_url: https://0123456789abcdef0123456789abcdef.r2.cloudflarestorage.com/mmhotline-status-status-example-com-18a8ea11
Next step: on Cloudflare's site, open R2, then Manage API tokens, and create an API token with the permission Object Read & Write for the bucket mmhotline-status-status-example-com-18a8ea11 only. Put r2_bucket_url above and the token's Access Key ID and Secret Access Key in the settings file's status_page section, as r2_bucket_url, r2_access_key_id, and r2_secret_access_key (with a deploy tool, in its vault), then apply the file, with your deploy tool or with sudo mmhotline settings apply on the MMHotline server. The first apply publishes the page.
Publish a status page runs it step by step. The design explains how the page works.
| Flag | Meaning |
|---|---|
--cloudflare-account-id ID |
Required. The team's Cloudflare account ID, 32 hexadecimal characters |
--host-name HOST |
Required. The page's public host name, in one of the account's zones |
--cloudflare-api-token-file FILE|- |
Required. A file only its owner can read, or - for standard input, holding the person's own Cloudflare API token. A file other users can read is refused with the chmod 600 command that fixes it |
--json |
Print one JSON object |
--help |
Show the flags |
- Runs anywhere with the
mmhotlineprogram; needs no MMHotline service and no command socket, and talks only to Cloudflare. It takes no--command-socket. - The token needs Account › Workers R2 Storage › Edit, Zone › Zone › Read, and Zone › Transform Rules › Edit, for the host name's zone. The command uses it only during setup and never stores it.
- What it creates, each only when missing: the R2 bucket, the host name's connection to it, and the URL rewrite rule. A rerun completes only what is missing. When Cloudflare refuses a step, it stops with Cloudflare's reason and exits 4; what it already created stays.
- It uploads no first page, because writing to the bucket needs the R2
key the person makes next. The page's title is the settings file's
page_title. - JSON fields:
cloudflare_account_id,host_name,bucket_name,r2_bucket_url,page_url, andnext_step.
mmhotline status-page show
Shows the status page's setup, its address, and its open and recently resolved notices.
sudo mmhotline status-page show
Status page: https://status.example.com/
R2 bucket: mmhotline-status-status-example-com-18a8ea11 (https://0123456789abcdef0123456789abcdef.r2.cloudflarestorage.com/mmhotline-status-status-example-com-18a8ea11)
Set up by the settings file /etc/mmhotline/mmhotline_settings.yml at 2026-11-15T07:51:11Z.
Example Inc. status, published at 2026-11-15T07:51:10Z.
No open notices: the page says that everything is working.
Post to the status page uses
show, post, and resolve, on the server and from a laptop.
- Flags: the common flags.
- Any member may post.
- Without the status page, it prints
The status page is not set up.
From a laptop. With these three environment variables set, show,
post, and resolve go straight to the R2 bucket, with no MMHotline
service and no command socket. Their output then starts with
Status page in R2 bucket and the bucket's URL.
| Environment variable | Holds |
|---|---|
MMHOTLINE_STATUS_PAGE_R2_BUCKET_URL |
The bucket's URL for its API that works like Amazon S3's; r2_bucket_url in status-page show |
MMHOTLINE_STATUS_PAGE_R2_ACCESS_KEY_ID |
The R2 API token's Access Key ID |
MMHOTLINE_STATUS_PAGE_R2_SECRET_ACCESS_KEY |
The R2 API token's Secret Access Key |
The bucket URL alone is refused: the two keys are required with it.
JSON fields: message and status_page; no web_url. post and
resolve also return notice_id, the notice they changed. Results never
include the R2 Secret Access Key or an API token.
| Field | Meaning |
|---|---|
status_page.is_set_up |
Whether the settings file's status_page section applied on this MMHotline; false from a laptop |
status_page.page_url |
The page's address, such as https://status.example.com/ |
status_page.host_name, .cloudflare_account_id |
The settings file's host_name, and the account that r2_bucket_url names |
status_page.bucket_name, .r2_bucket_url |
The R2 bucket and its S3 API address |
status_page.r2_access_key_id |
The saved R2 Access Key ID; absent before the section applied |
status_page.set_up_at, .set_up_by |
When the section last applied, and the settings file and its path |
status_page.published_status_json |
The published status.json: page_title, published_at, and notices (its format) |
status_page.read_error |
Why status.json could not be read, or why index.html may not show a saved post until the next post |
mmhotline status-page post
Opens a notice on the status page, or adds an update to an open notice.
sudo mmhotline status-page post --message "We are looking into a problem."
sudo mmhotline status-page post --notice-id 1 --message "Sign-in works again for most customers."
The output is that of status-page show,
then Posted to notice 1., or Posted to notice 1, directly in the R2 bucket. from a laptop.
| Flag | Meaning |
|---|---|
--message TEXT |
Required. 1 to 5,000 characters |
--notice-id N |
Optional. Add the message as an update to open notice N; without it, a new notice opens |
| Common flags | As above |
- Each post uploads
status.jsonandindex.htmlto the bucket. - From a laptop, a post is refused while the bucket has no
status.json, which the firstsettings applywith thestatus_pagesection creates.
mmhotline status-page resolve
Resolves a notice, with an optional last update.
sudo mmhotline status-page resolve --notice-id 1 --closing-message "Everything works again."
The output is that of status-page show,
then Notice 1 is resolved., or Notice 1 is resolved, directly in the R2 bucket. from a laptop.
| Flag | Meaning |
|---|---|
--notice-id N |
Required. The notice to resolve |
--closing-message TEXT |
Optional. Added as the notice's last update |
| Common flags | As above |
Commands that read what happened
The incidents, alerts, triage-sessions, and approval-requests
commands let a person, or the team's coding agent, read what happened over
SSH on the MMHotline server. The pages sit behind Cloudflare Access, which
an SSH session cannot pass. Each command shows the facts of a page and
calls the same operation that page calls: the dashboard and incident page,
Alerts, the incident page's triage agent panel, and Approvals.
- They change nothing. To claim, resolve, approve, refuse, or stop, use the pages, because each of those actions records a person.
- Each reads in at most a few transactions the size of a page's, so it holds the database no longer than opening the page does.
- They need root's command access; any member sees the same facts on the pages.
TIMEis an RFC 3339 time, such as2026-10-28T09:00:00Zor2026-10-28T11:00:00+02:00. A span includes its start and excludes its end.
mmhotline incidents list
Lists SOS incidents: by default every one not resolved, newest first, as the dashboard does.
sudo mmhotline incidents list
SOS incident acc896d49ce5b2ef40cd2c944da4f08f: Test SOS from the first run
Status: Not claimed yet. Opened at 2026-11-15T04:40:40Z.
Being contacted now: Barbara Example.
Page: https://ops.example.com/incidents/acc896d49ce5b2ef40cd2c944da4f08f
Listed 1 SOS incident not resolved, newest first.
Handle an incident and Back up and restore the database use it.
| Flag | Meaning |
|---|---|
--opened-since TIME |
Optional. Every incident opened at or after TIME, resolved or not |
--opened-before TIME |
Optional. Every incident opened before TIME, resolved or not |
| Common flags | As above |
With one or both time flags, it lists the incidents opened in that span,
at most the newest 1,000, and ends with a line such as
Listed 2 SOS incidents opened since 2026-10-28T00:00:00Z, resolved or not, newest first.
JSON fields: message, web_url, and incidents, present also when
empty. Each incident:
| Field | Meaning |
|---|---|
incident_id, title, message |
The incident, and its opening alert's title and message |
status |
open (shown as "Not claimed yet"), claimed, or resolved (states) |
opened_at, claimed_at, resolved_at |
When it opened, was last claimed, and was resolved; absent until then |
claimed_by |
The name of the person who claimed it; absent while nobody has |
escalation_step |
primary, backup, everyone_available, only_member, chosen_person, or triage_agent, the step that waits for the triage agent |
held_for_triage_agent |
true while the incident waits for the triage agent and nobody is contacted yet |
being_contacted_now |
While open: who is contacted, such as Barbara Example or No one yet: the triage agent is working on it |
next_person_contacted_at |
While open: when MMHotline contacts the next person unless someone claims it, or, while held, when it wakes the person on call; absent when there is no next step |
opening_alert_id, alert_source_id |
The alert that opened it, and its alert source |
sender_incident_key |
The sender's incident key; absent when the sender gave none |
page_url |
The incident page |
mmhotline incidents show
Shows one SOS incident with its message, its triage session, its action history, and its newest 100 alerts.
sudo mmhotline incidents show --incident-id acc896d49ce5b2ef40cd2c944da4f08f
The output starts with the lines of incidents list, then adds one line
each for the message and the triage session, then:
Action history:
2026-10-28T01:01:03Z Opened: SOS incident opened: Nightly backup of db001 failed. By: Backups.
2026-10-28T01:01:03Z Contacting: Contacting Barbara Example (only team member). By: MMHotline.
Related alerts, newest first:
2026-10-28T01:01:03Z SOS 02692794cbcd552987b12e1bbe245467: Nightly backup of db001 failed (SOS incident cd1eec890f8a52f59c04f3f6982767a2)
A claim or resolve with a note ends with the note, such as
Resolved: Resolved by Barbara Example. Note from Barbara Example: “This was a test.” By: Barbara Example.
Handle an incident uses it.
| Flag | Meaning |
|---|---|
--incident-id ID |
Required |
| Common flags | As above |
JSON fields: message, web_url, and incident, with the fields of
incidents list and:
| Field | Meaning |
|---|---|
triage_session_id |
The triage session the incident's newest alert joined, as the incident page shows; absent when none did |
alerts |
The incident's newest 100 alerts, newest first, in the fields of alerts list. The message names the command for older ones |
history |
The incident's history, oldest first, which the incident page's What happened says in words |
history[].kind, history[].label |
Such as claimed or triage_agent_report, and a short name, such as Triage agent report |
history[].message, history[].occurred_at |
What happened, and when |
history[].by |
The person, alert source, triage agent, or MMHotline |
history[].related_alert_id |
The alert that caused the entry, when one did |
history[].note |
What the person wrote with a claim or resolve; absent when they wrote none |
mmhotline alerts list
Lists the newest 100 alerts that match, newest first. Warn and info alerts open no incident, so this is where to find them.
sudo mmhotline alerts list --severity warn
2026-11-15T08:51:45Z Warning 048bd52b16047999b0cee4512075e678: The AI triage agent reports: Disk check playbook (PR 7) looks harmless, but a person must review it: my rules file is missing
That is the first line of one alert. Each alert has a line with its
received time, severity, ID, and title, then a line with its alert source
and page, then its message. The list ends with a line such as
Listed 1 alert, newest first.
Back up and restore the database
uses it.
| Flag | Meaning |
|---|---|
--severity SEVERITY |
info, warn, sos, resolved-info, resolved-warn, or resolved-sos (severities) |
--alert-source-id ID |
An alert source's ID from alert-sources list; system for MMHotline's own alerts, such as the triage agent's reports; manual for SOS incidents a person created. Another ID is refused, because an empty list would look like a source that went quiet |
--incident-id ID |
Only the alerts of this SOS incident |
--received-since TIME, --received-before TIME |
Only alerts received in that span |
--older-than-alert-id ID |
Continue after a list of 100: give the oldest listed alert's ID, which the list's message names. Reason: a burst arrives within one second, and a time would skip or repeat alerts in it |
| Common flags | As above |
JSON fields: message, web_url, and alerts, present also when
empty. Each alert:
| Field | Meaning |
|---|---|
alert_id, severity, title, message |
The alert as received |
notes |
MMHotline's note about the alert, such as a default severity; absent when none |
received_at |
When MMHotline received it |
happened_at |
When it happened on the sender's clock; only when the sender gave the time |
alert_source_id |
The alert source's ID, or system or manual |
incident_id |
The SOS incident it opened or joined; absent when none |
sender_alert_id, sender_incident_key |
The sender's IDs, when it gave them |
page_url |
The alert's page |
mmhotline triage-sessions list
Lists the newest 50 triage sessions, newest first. A triage session is the triage agent's work on a stretch of trouble: the alerts, and the runs in which the agent looked into them (design).
sudo mmhotline triage-sessions list
Triage session 89fda90d8160696485d427a1d654db1c: Waiting: the triage agent is not running. MMHotline resumes the triage agent when another alert or the result of a playbook run arrives.
Started at 2026-11-15T08:56:16Z. Alerts: 2. Runs: 2. SOS incidents: none.
Listed 1 triage session, newest first.
Set up the triage agent reads sessions with both commands.
- Flags: the common flags.
- An older incident's session:
incidents showgives itstriage_session_id.
JSON fields: message, web_url, and triage_sessions, present also
when empty. Each session:
| Field | Meaning |
|---|---|
triage_session_id, started_at |
The session, and when its first alert started it |
ended_at, ended_because |
When and why it ended; absent while open |
phase |
working, waiting, stopping, handing_over, or ended, as the incident page names them |
phase_text |
The session page's line for the phase |
taken_over |
Who pressed Stop and when, such as Barbara pressed Stop at 2026-11-03 03:10:00 UTC; absent when nobody did |
hand_over_run_id |
The run in which the agent handed over after a Stop; absent before |
alert_count, run_count |
How many alerts and runs |
incident_ids |
The session's SOS incidents, by the last alert from each to join the session, newest first |
mmhotline triage-sessions show
Shows one triage session with its alerts, its SOS incidents, and its runs, each run with its latest 50 steps.
sudo mmhotline triage-sessions show --triage-session-id 89fda90d8160696485d427a1d654db1c
Triage session 89fda90d8160696485d427a1d654db1c: Waiting: the triage agent is not running. MMHotline resumes the triage agent when another alert or the result of a playbook run arrives.
Started at 2026-11-15T08:56:16Z. Alerts: 2. Runs: 2. SOS incidents: none.
Alerts, oldest first:
2026-11-15T08:56:15Z Warning ecbc50fea74cc1b695d8ec8028355a67: lab-web.service on web1 is not running; sent in run b52fc544df25dfe3d85c77343caee088
2026-11-15T08:58:28Z Warning dc0af42deeb7f2f68967254e5bf19676: lab-web.service on web1 is still not running; sent in run 8a3b1687b4e4087ff040ad6987ae1dc2
SOS incidents:
Runs, oldest first:
Run b52fc544df25dfe3d85c77343caee088: finished. Started at 2026-11-15T08:56:16Z, ended 2026-11-15T08:57:10Z. Cost in US dollars: 0.233324
2026-11-15T08:56:19Z command: cat triage_agent_rules.md; echo ----; cat AGENTS.md; ls -la; ...
| Flag | Meaning |
|---|---|
--triage-session-id ID |
Required |
| Common flags | As above |
Each run shows its latest 50 steps, as the incident page does; the coding agent's session files keep the whole transcript.
JSON fields: message, web_url, and triage_session, with the
fields of triage-sessions list and:
| Field | Meaning |
|---|---|
alerts |
The session's alerts, oldest first, in the fields of alerts list, with joined_at and sent_in_run_id, the run that carried the alert to the agent, absent while the alert waits for the next run |
incidents |
The session's SOS incidents, in the fields of incidents list |
runs |
The session's runs, oldest first, in the fields of triage-agent show's latest_setup_test: id, purpose (triage_session), state, started_at, ended_at, failure_reason, the run's own cost_in_us_dollars or tokens, steps with the latest 50, and earlier_steps_not_shown |
mmhotline approval-requests list
Lists the playbook runs that wait for a person, newest first, as the Approvals page does: open approval requests, and warned runs that have not started. An approval request is a playbook run the triage agent asked for that waits for a person, because its rule is "Get permission, then act". A warned run, under "Warn, wait, then act", is listed so that a person can stop it before it runs.
sudo mmhotline approval-requests list
Playbook run e63b3c6fe869617623da0ebcb22d1fba: one_time_playbooks/2026-11-15_install_security_fixes_brotli_openssl_expat_apt_sasl_on_web1.yml in the inventory “production”. Waiting for approval. Asked at 2026-11-15T09:16:02Z.
Listed 1 playbook run waiting for a person, newest first. Team members approve, refuse, or stop playbook runs at https://ops.example.com/approvals.
Decide what the triage agent may do approves and refuses on the page.
- Flags: the common flags.
- JSON:
message,web_url, andapproval_requests, present also when empty, each in the fields ofapproval-requests show.
mmhotline approval-requests show
Shows one approval request or warned run, waiting or decided, with what it would run.
sudo mmhotline approval-requests show --approval-request-id e63b3c6fe869617623da0ebcb22d1fba
Playbook run e63b3c6fe869617623da0ebcb22d1fba: one_time_playbooks/2026-11-15_install_security_fixes_brotli_openssl_expat_apt_sasl_on_web1.yml in the inventory “production”. Refused. Asked at 2026-11-15T09:16:02Z.
Rule: Get permission, then act (1 approval). Commit: cc72003b1e209a56dd79d94ca48972db9d58ba69.
SOS incident: f9a143aca619c1e2a0d8a555e9f04548
Kinds of action in the playbook: Check only, Unrecognized task.
What the playbook can change on each server, as listed by mmx explain:
install_security_fixes_on_web1/task_1 on web1: assert (core), ready
...
After those lines come the triage session, "Why the triage agent asks" with the agent's diagnosis, the playbook's text at the checked commit, and the Approvals page's address.
| Flag | Meaning |
|---|---|
--approval-request-id ID |
Required |
| Common flags | As above |
JSON fields: message, web_url, and approval_request:
| Field | Meaning |
|---|---|
approval_request_id, playbook, inventory, commit_id |
The request, and the playbook and inventory at the commit MMHotline judged |
rule |
get_permission_then_act, with approvals_needed, 1 or 2; or warn_wait_then_act, with warning_wait_minutes and runs_at |
listed_playbook_rule |
When the rule is one a person listed for the playbook by name, says so, such as "The playbook reboot_servers_after_updates is listed in rules_for_running_agent_playbooks for the inventory “servers” with Warn, wait 10 minutes, then act."; absent otherwise. The text adds it after Rule: |
its_approval_also_approves_in_the_same_session |
The listed playbooks this approval also approves once later in its triage session (one approval for a known sequence); absent when none. The text adds "Its approval also approves, once each later in the same triage session: …" |
covered_by_approval_request_id, covered_by |
For a run an earlier approval covered: that approval request, and who approved it; absent otherwise. The text adds "Approved without a new request: …" |
state |
open, approved, refused, stopped, running, ran_and_succeeded, ran_and_failed, could_not_run, or stale (states) |
state_label |
The state in words, such as Waiting for approval or Runs in 7 minutes unless someone presses Stop |
asked_at, state_changed_at |
When MMHotline asked, and when the state last changed |
incident_id, triage_session_id |
The incident of the request's alert, and the triage session that asked; absent when none |
kinds_of_action |
What the playbook does, such as reboot_a_server |
tasks |
What the playbook would run, as mmx explain lists it: path, node, module, module_kind, status, and arguments with field, value, and status |
unrecognized_tasks |
What MMHotline did not recognize: path, module (or save_on_controller_machine for a file a fleet stage saves on MMHotline's server), and reason, such as systemd_service.start_at_boot is not an allowed argument for Start or restart a service (…) |
tasks_on_services_whose_changes_a_person_must_approve |
The tasks on services whose changes a person must approve, with the current list: path, node, module, service_name (absent when MMX knows the unit's name only during the run), and reason. Present also when empty |
playbook_text |
The playbook's text at the commit |
review_agent_answer |
The review agent's answer, when one ran |
diagnosis |
The triage agent's Markdown explanation: findings, cause, evidence, and why the playbook fixes the problem |
earlier_versions_refused_by_the_review_agent |
When the review agent's refusals made the request wait, the versions it refused in a row before this one, oldest first (refusal rounds): playbook, inventory, commit_id, diagnosis, review_agent_reason, and playbook_text. Present also when empty. The text lists them under "Earlier versions the review agent refused, oldest first" |
decisions |
Each person's member_name, decision (approve or refuse), reason, and decided_at |
stopped_by, stop_reason |
For a warned run a person stopped |
result_message |
What happened, once decided or run |
result_for_the_agent |
none, waits_for_the_next_run, sent, or could_not_reach_the_agent |
mmhotline commit-reviews check
Checks every new commit of the systems repository through a commit, as MMHotline does before each MMX run and each triage agent run in the repository (the check).
sudo mmhotline commit-reviews check --commit-id 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1
Checked commit 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1.
Trusted through commit a4fce954720b97d948bdb9f80640cbc5ee5ede5e: every commit through it passed the check or a person reviewed it.
Commit 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1, which no key the team trusts signed.
Author: Git host <noreply@githost.example>. Commit date: 2026-11-15T08:51:26Z. Found by MMHotline: 2026-11-15T08:51:26Z.
Message: Merge pull request 7: add a disk check playbook
Waits for review at https://ops.example.com/approvals/commits/5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1.
1 commit waits for a person's review at https://ops.example.com/approvals.
Review a commit uses every
commit-reviews command.
| Flag | Meaning |
|---|---|
--commit-id COMMIT_ID |
Required. A full commit ID, such as the commit util01 is about to apply |
| Common flags | As above |
- The rule: a commit signed by a key in
triage_agent.trusted_commit_signing_public_keyspasses. A commit the triage agent's key signed passes when it changes only the agent's two folders. Every other commit waits for a review on the Approvals page. Withtriage_agent.commits_without_a_trusted_signature_are_approved_by: a_person_or_the_triage_agent, the triage agent reviews a commit no trusted key signed that changes no path oftriage_agent.paths_only_a_person_may_approve, and approves it or decides that a person must review it (design). - What it does: fetches origin into MMHotline's copy under the copy's
lock, as runs do, then checks every commit from the last checked commit
through
COMMIT_ID. Each commit that newly waits gets one alert: MMHotline's warning at once, or, for a commit the triage agent reviews, the agent's approval or report. - Waits without a time limit while an MMX run or a person's helper
holds the copy's lock,
/var/lib/mmhotline/systems_repository.lock. A caller that holds the lock itself must release it first. - Refused for a
COMMIT_IDthe copy lacks after the fetch. - A scheduled job on the server, such as
util01's job that applies the systems repository, runscheckbefore applying a commit, and applies only throughtrusted_through_commit_id.
| Exit code | Meaning |
|---|---|
0 |
Every commit through COMMIT_ID passed or was reviewed |
1 |
A commit waits for review; the answer is printed first |
4 |
Refused, or wrong arguments |
JSON fields: message, web_url, and commit_check:
| Field | Meaning |
|---|---|
commit_check.checked_commit_id |
The commit the check went through |
commit_check.trusted_through_commit_id |
The newest commit on the checked commit's first-parent line through which every commit passed or was reviewed: the checked commit when nothing waits, the commit before a merge that brings in a waiting commit, and empty when no commit is trusted yet |
commit_check.commits_waiting_for_review |
The commits through the checked commit that wait, newest first, in the fields of commit-reviews show without diff. Present also when empty |
mmhotline commit-reviews trust-through
Trusts a commit of origin's branch and every commit before it, such as after a person's first run on a new or rebuilt server.
sudo mmhotline commit-reviews trust-through --commit-id a4fce954720b97d948bdb9f80640cbc5ee5ede5e
Review a commit uses it.
| Flag | Meaning |
|---|---|
--commit-id COMMIT_ID |
Required. A full commit ID of origin's branch |
| Common flags | As above |
- Fetches origin without checking, then makes
COMMIT_IDthe checked-through commit. Each commit before it that waited is recorded as reviewed by the Unix user through the trust-through, and one info alert tells the team, such asroot on util01 marked every commit through abc1234 trusted. The checkout then moves to the trusted commit, and later commits go through the check. - A
COMMIT_IDat or before the checked-through commit changes nothing, and the message says so. - Refused for a
COMMIT_IDthe copy lacks after the fetch, or one not on origin's branch. - Waits for the copy's lock, answers, and exits as
checkdoes, and returnscommit_check, withmessagesaying what changed.
mmhotline commit-reviews mark-reviewed
Marks a waiting commit reviewed, as the page's Mark commit reviewed does. It is recorded as "the mmhotline command, run by barbara".
sudo mmhotline commit-reviews mark-reviewed --commit-id 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1
- Flags:
--commit-id COMMIT_ID, required, and the common flags. - Marking a reviewed commit changes nothing.
- JSON:
message,web_url, andcommit_review, in the fields ofcommit-reviews show.
mmhotline commit-reviews list
Lists the commits waiting for review, oldest found first, then the latest 50 commits the triage agent approved, newest first, as the Approvals page lists them. It changes nothing.
sudo mmhotline commit-reviews list
- Flags: the common flags.
- Output: each commit as
commit-reviews showprints it, without the diff. - JSON:
message,web_url,commit_reviews, andcommits_the_triage_agent_approved, newest first, each present also when empty, in the fields ofcommit-reviews show.
mmhotline commit-reviews show
Shows one commit that waits or waited, with its diff, who reviewed it, and the triage agent's decision and report. It changes nothing.
sudo mmhotline commit-reviews show --commit-id 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1
Commit 5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1, which no key the team trusts signed.
Author: Git host <noreply@githost.example>. Commit date: 2026-11-15T08:51:26Z. Found by MMHotline: 2026-11-15T08:51:26Z.
Message: Merge pull request 7: add a disk check playbook
Waits for review at https://ops.example.com/approvals/commits/5c281a5cc65bb8cb9a6ecbe9d2e4f8bd524453c1.
The triage agent decided that a person must review the commit, with its report at alert severity warn at 2026-11-15T08:51:45Z, in commit review run a9653b004da1f20f02a6cc2473c5da73. Its report:
...
Diff:
diff --git a/playbooks/check_disk.yml b/playbooks/check_disk.yml
...
| Flag | Meaning |
|---|---|
--commit-id COMMIT_ID |
Required |
| Common flags | As above |
JSON fields: message, web_url, and commit_review:
| Field | Meaning |
|---|---|
commit_id |
The commit's full ID |
reason_kind |
not_signed_by_a_trusted_key: unsigned, signed by a key the settings file does not list, or a signature Git cannot verify. triage_agent_change_needs_review: signed by the triage agent's key, and changes paths outside its folders or adds a file type it may not add |
reason |
The reason in words, such as which no key the team trusts signed |
author, committed_at, message |
The commit's author, date, and message |
paths_that_need_review |
Each path and reason: for triage_agent_change_needs_review, the paths outside the agent's folders; for a commit no trusted key signed, the changed paths of triage_agent.paths_only_a_person_may_approve, as applied now; otherwise empty |
changed_paths |
Every path the commit changes, against each parent of a merge |
found_at |
When MMHotline first found the commit |
reviewed_by, reviewed_at |
Who reviewed it and when, such as Barbara Example, the mmhotline command, run by barbara, the mmhotline command, run by root, marking every commit through abc1234 trusted, or the triage agent; absent while it waits |
reviewer_kind |
member, unix_user, trust_through, or triage_agent; absent while it waits |
triage_agent_review |
The triage agent's latest decision; absent when it gave none |
triage_agent_review.run_id, .decided_at |
The commit review run, and when MMHotline recorded the decision |
triage_agent_review.decision |
approve, or needs_a_person, which leaves the commit waiting |
triage_agent_review.alert_severity |
With needs_a_person, the severity the report went out at; absent with approve, which sends one info alert |
triage_agent_review.report |
The agent's whole report |
review_url |
The commit's page under Approvals |
diff, diff_truncated |
show and mark-reviewed only: the diff against each parent, cut after 128 KiB with diff_truncated true |
This page is docs/reference/mmhotline-command.md in the MMHotline repository.