The settings file

This page lists every key of the settings file, /etc/mmhotline/mmhotline_settings.yml: its type, its allowed values, and what mmhotline settings check says when a value is wrong. A section is a key that holds other keys, such as people or triage_agent. A part is a group of keys that settings apply applies together and reports on one line, such as people: applied.. To edit, check, and apply the file step by step, see Change the settings. Each key below links to the how-to page that sets it up and shows its example.

A minimal settings file

This file has every required key and no optional section. It sets up one administrator, a weekly rotation, and one alert source:

# Written by hand with sudoedit. Apply a change with sudo mmhotline settings apply.
team_name: Operations
web_url: https://ops.example.com
web_listen_address: 127.0.0.1:8480
alert_intake_listen_address: 10.20.0.1:8484

rules_for_running_agent_playbooks: {}
services_whose_changes_a_person_must_approve: []
sending_services: {}

people:
  - name: Barbara Example
    sign_in_email_address: barbara@example.com
    role: administrator

on_call_schedule:
  escalate_if_unclaimed_after_minutes: 5
  schedule_type: weekly_rotation
  time_zone: America/New_York
  handoff_hour: 9
  handoff_minute: 0
  handoff_weekday: 1
  first_start_date: "2026-11-02"
  rotation_sign_in_email_addresses: [barbara@example.com]
  backup_option: everyone_available

alert_sources:
  - alert_source_id: monitoring
    name: Monitoring
    signing_secrets: ["YOUR_64_HEXADECIMAL_CHARACTER_SIGNING_SECRET"]
    warn_after_no_check_in_for_minutes: 2
    sos_after_no_check_in_for_minutes: 5

With a real signing secret in place, sudo mmhotline settings check --settings-file /etc/mmhotline/mmhotline_settings.yml prints:

The settings file /etc/mmhotline/mmhotline_settings.yml has no problems.

Get started writes this file in step 3.

Rules for the whole file

Rule Detail
Format YAML, one section of keys at the top.
Size At most 1 MiB. Reason: the largest value, custom schedule code, is at most 64 KiB.
Owner and mode MMHotline refuses a file whose owner is not an allowed owner (root), that group or other users can write (any of mode bits 0022), or that other users can read (mode bit 0004). The install's choice is owner root, group mmhotline, mode 0640, so the mmhotline user can read it. Reason: the file decides what MMHotline does, fleet_view_command runs as root, and the file holds the team's secrets. settings check skips this rule, so a deploy tool can check a copy on a laptop; settings apply, install, and MMHotline's start apply it. Installed files says who sets the owner.
Unknown keys Refused, so a typo never passes silently.
Required keys Every top-level key without "optional" below is required. Inside a section that is present, every key is required unless its table says optional.
Defaults None, except triage_agent.warn_waits_for_the_agent_minutes, which is 0 when left out, and an action kind that action_kinds leaves out, which needs a person (rules_for_running_agent_playbooks). Reason: the person or agent who writes the file states every setting.
Optional sections Leaving out an optional section, such as status_page, turns its feature off. settings apply then reports the part as not set up, because the settings file leaves it out.
First line When the first line is a comment, the pages and settings show print it next to the file's path. Use it to say how the file is changed, such as # Managed by MMX from mmhotline_settings.yml in the systems repository.
Secrets Values marked secret below never appear in a problem, a page, a log, or command output. A YAML syntax error names only its line.

These problems apply to every key. settings check prints each as KEY_PATH: PROBLEM and exits 1:

Problem text Cause
required A required key is missing.
unknown key The key is not on this page.
unknown key; it was renamed to rules_for_running_agent_playbooks An old key name. The same text names the new key for trust_settings, trust_setting (now rule), response_window_minutes (now escalate_if_unclaimed_after_minutes), services_that_always_need_a_person (now services_whose_changes_a_person_must_approve), and lifeline (now mmhotline_down_alarm).
has no value The key is present with nothing after the colon.
appears twice The key is repeated in one section.
must be a section of keys, each on its own line as key: value A section has a single value or a list.
must be a list, such as [a, b], or [] for none A list key has a single value.
must be a single value, not a list or a section A text key has a list or a section.
must be a whole number A number key has text or a fraction.
must be true or false A switch has another value.

Problems with the whole file have no key path: the settings file is not valid YAML: ... (with the line), the settings file is empty, the settings file exceeds MMHotline's limit of 1 MiB, the settings file must be a section of keys, each on its own line as key: value, such as team_name: Example Operations (the file holds a list or a single value), and MMHotline cannot read the settings file: ... (the system's reason follows, such as permission denied). The owner and mode refusals are ... is owned by user ID N, not root. Only root may change it ..., ... can be changed by users other than root (mode 0664). Only root may change it ..., and ... can be read by every user on the server (mode 0644), and it holds the team's secrets. Fix the settings file owner and mode with: sudo chown root:mmhotline /etc/mmhotline/mmhotline_settings.yml; sudo chmod 0640 /etc/mmhotline/mmhotline_settings.yml.

An example with several problems, from a real check:

team_nam: unknown key
lifeline: unknown key; it was renamed to mmhotline_down_alarm
team_name: required
on_call_schedule.escalate_if_unclaimed_after_minutes: must be a whole number of minutes from 1 to 60
on_call_schedule: the first start date must be a date such as 2026-11-02 on the handoff weekday (0 is Sunday)
alert_sources[0].sos_after_no_check_in_for_minutes: the time before waking the person on call must be longer than the time before a warning

A key path shows each list item by its position from 0, such as people[1].role. A key gets one problem at a time; fix it and check again.

The top-level keys

Key Required Section below
team_name Yes team_name
web_url Yes web_url
web_listen_address Yes web_listen_address
alert_intake_listen_address Yes alert_intake_listen_address
cloudflare_access Optional cloudflare_access
fleet_view_command Optional fleet_view_command
people Yes people
on_call_schedule Yes on_call_schedule
alert_sources Yes, may be [] alert_sources
triage_agent Optional triage_agent
review_agent Optional review_agent
rules_for_running_agent_playbooks Yes, may be {} rules_for_running_agent_playbooks
services_whose_changes_a_person_must_approve Yes, may be [] services_whose_changes_a_person_must_approve
sending_services Yes, may be {} sending_services
mmhotline_down_alarm Optional mmhotline_down_alarm
status_page Optional status_page

settings apply applies the parts in this order, one line each: the four server and sign-in keys together, team_name, fleet_view_command, people, on_call_schedule, rules_for_running_agent_playbooks, services_whose_changes_a_person_must_approve, alert_sources, the triage agent's parts, the MMHotline-down alarm's wake times, review_agent, each sending service, mmhotline_down_alarm, and status_page. People apply before the schedule that names them. A part that has not changed is not applied again. A part that MMHotline refuses keeps its earlier setting, and the other parts still apply. The command exits 0 only when every part applied.

MMHotline's pages show the settings file's values read-only. Each settings page folds the file's path, first comment line, and editing instructions under a heading that names the settings. Personal actions stay on the pages: contact methods, unavailability, taking on-call, team mute, claims, approvals, status page posts, and incidents.

Server and sign-in

web_url, web_listen_address, alert_intake_listen_address, and cloudflare_access apply as one part and change only when MMHotline restarts. When one of them changes, settings apply restarts MMHotline with systemctl restart mmhotline.service first, then applies the rest (why).

team_name

The team's name, shown on every page and in alerts.

team_name: Operations
Fact Value
Required Yes
Type Text, 1 to 100 characters; spaces around it are dropped
Default None
Wrong value team_name: team name must contain 1–100 characters
Set up in Get started, step 3, and Install MMHotline

mmhotline install names the new database after team_name. A later change applies as its own part.

web_url

The address people open. Links in pages, notifications, and command output start with it.

web_url: https://ops.example.com
Fact Value
Required Yes
Type An origin: scheme, host name, and optional port, with no path, query, or user name. A trailing / is dropped.
Allowed values Any https:// origin, such as https://ops.example.com. For a team without Cloudflare that opens MMHotline through an SSH forward: http://localhost:PORT or http://127.0.0.1:PORT, where the port is a whole number from 1 to 65535 and must be written.
Default None
Wrong value web_url: must be an HTTPS origin, such as https://mmhotline.example.com, or http://localhost:PORT or http://127.0.0.1:PORT
Takes effect At a restart, which settings apply does
Set up in Get started, step 3, and Put MMHotline behind Cloudflare Access

web_listen_address

The host and port where MMHotline serves its pages over plain HTTP on the server.

web_listen_address: 127.0.0.1:8480
Fact Value
Required Yes
Type host:port, with a port from 0 to 65535
Allowed values Normally 127.0.0.1:8480. Reason: only programs on the server, such as cloudflared and SSH forwards, then reach the pages.
Default None
Wrong value web_listen_address: must be a host:port address, such as 127.0.0.1:8480
Takes effect At a restart, which settings apply does
Set up in Get started, step 3

alert_intake_listen_address

The host and port where MMHotline accepts signed alerts and check-ins. Only those two routes answer there; every other path returns 404.

alert_intake_listen_address: 10.20.0.1:8484
Fact Value
Required Yes
Type host:port, with a port from 0 to 65535
Allowed values An address of the MMHotline server on its private network, such as a WireGuard address. 127.0.0.1 works when every tool that sends alerts runs on the same server.
Default None
Wrong value alert_intake_listen_address: must be a host:port address, such as 127.0.0.1:8480
Takes effect At a restart, which settings apply does
Set up in Get started, step 3, and Connect a tool that sends alerts

settings check cannot know the server's addresses. An address that is not the server's own passes the check, and MMHotline then fails to start with MMHotline cannot listen on alert_intake_listen_address 10.20.0.9:8484: listen tcp 10.20.0.9:8484: bind: cannot assign requested address. Install MMHotline says what to do.

cloudflare_access

Sign-in through Cloudflare Access. MMHotline checks the token that Cloudflare Access signs for each request.

cloudflare_access:
  team_domain: example.cloudflareaccess.com
  audience_tag: YOUR_APPLICATION_AUDIENCE_TAG
Key Required Type and allowed values Wrong value
cloudflare_access Optional Section. Without it, people sign in only with sign-in links.
cloudflare_access.team_domain Yes The Cloudflare Access team domain, a host name without https://, path, port, or spaces the Cloudflare Access team domain must be a host name without a URL scheme, path, port, or spaces
cloudflare_access.audience_tag Yes The Access application's Application Audience (AUD) Tag: ASCII letters and digits the Cloudflare Access audience tag must contain only ASCII letters and digits (or ... must not be empty and must contain only ASCII letters and digits)

Takes effect at a restart, which settings apply does. At start, MMHotline logs which sign-in methods it accepts: Sign-in: Cloudflare Access for example.cloudflareaccess.com, and sign-in links, or Sign-in: sign-in links only; the settings file has no cloudflare_access.

Set up in Put MMHotline behind Cloudflare Access. Without the section, see Sign in during a Cloudflare outage.

fleet_view_command

The command that prints your monitoring tool's fleet view as JSON. The dashboard shows servers that need attention, with their failing checks, and healthy servers by name, from the command's output.

fleet_view_command: [/usr/local/bin/MONITORING_TOOL, fleet, --json]
Fact Value
Required Optional. Without it, the dashboard says in one line that there is no fleet view.
Type A list of words: the program, then each argument as its own item
Allowed values The first item is the program's absolute path. No item is empty or holds a control character.
Default None
Wrong value fleet_view_command: must list the program and its arguments, such as [/usr/local/bin/PROGRAM, fleet, --json]; fleet_view_command[0]: must be the program's absolute path, such as /usr/local/bin/PROGRAM; fleet_view_command[N]: must be one word without control characters
Set up in Connect a tool that sends alerts
  • The command runs with no shell, so quotes, variables, and pipes in its words mean nothing.
  • The command runs as root, through systemd, in the unit mmhotline-fleet-view@.service. Reason: a monitoring tool's fleet view usually admits only root, and MMHotline itself runs as mmhotline and cannot gain privileges. MMHotline can only request a run of the configured command; only root can change the command, because only root can change the settings file.
  • Each run is one connection to /run/mmhotline-fleet-view.sock, which only the mmhotline user may open.
  • Each run has a read-only file system, no network except local Unix sockets, only PATH in its environment, and 20 seconds before systemd ends it. MMHotline's database, the backup's files, backup.env, and the triage agent's home are hidden from it.
  • The root unit reads the command from the settings file at every run. Run sudo mmhotline settings apply after adding the command, so the dashboard knows there is one. A problem elsewhere in the settings file does not stop the fleet view.
  • Errors go to the journal: sudo journalctl -u 'mmhotline-fleet-view@*'.
  • The design lists the JSON fields MMHotline reads.

People and the schedule

people

Everyone who uses MMHotline. A person's sign-in email address identifies the person at sign-in.

people:
  - name: Barbara Example
    sign_in_email_address: barbara@example.com
    role: administrator
  - name: Carlos Example
    sign_in_email_address: carlos@example.com
    role: member
Key Required Type and allowed values Wrong value
people Yes A list with at least one administrator people: needs at least one person with role administrator, who opens the administration pages
people[].name Yes Text, 1 to 100 characters person name must contain 1–100 characters
people[].sign_in_email_address Yes The address the person's Cloudflare Access sign-in gives: 3 to 254 characters, one @ with text on both sides, no spaces. Saved in lowercase. Unique. sign-in email address must have 3–254 characters, exactly one @ with text on both sides, and no spaces or control characters; carlos@example.com is also the sign-in email address of people[0]; each person must have a unique sign-in email address
people[].role Yes administrator or member. Administrators also open the administration pages, such as the Triage agent page, and see other people's contact methods. person role must be administrator or member
  • A person removed from the list can no longer sign in, and their contact methods are removed; their history stays.
  • MMHotline refuses to remove a person who owns a claimed incident or has taken on-call, and the part keeps its earlier list, such as: people: did not apply: Bruno Costa owns 1 claimed incident(s). Resolve the claimed incidents or have someone take over the claimed incidents before removing the person. The earlier setting, applied at 2026-11-02T09:00:00Z, stays.

Set up in Set up people and the on-call schedule.

on_call_schedule

Who is on call, who is the backup, and how long each escalation step waits for a claim. Each schedule type uses its own keys, and a key another type uses is refused.

Key Required Type and allowed values
on_call_schedule.escalate_if_unclaimed_after_minutes Yes 1 to 60: how long each escalation step, such as the person on call and then the backup, waits for someone to claim an SOS before MMHotline contacts the next step. The wait starts again at each step.
on_call_schedule.schedule_type Yes weekly_rotation, day_of_week, hour_of_day, or custom_schedule_code
on_call_schedule.time_zone Yes A named time zone, such as America/New_York
on_call_schedule.backup_option Yes everyone_available, fixed_person, or, for weekly_rotation only, next_in_rotation
on_call_schedule.backup_sign_in_email_address With fixed_person only The fixed backup's sign-in email address
on_call_schedule.handoff_hour, on_call_schedule.handoff_minute weekly_rotation, day_of_week The local handoff time, 0 to 23 and 0 to 59
on_call_schedule.handoff_weekday weekly_rotation 0 (Sunday) to 6
on_call_schedule.first_start_date weekly_rotation The date the first person starts, in quotes, such as "2026-11-02", on the handoff weekday
on_call_schedule.rotation_sign_in_email_addresses weekly_rotation The rotation's people in order, each once
on_call_schedule.assignment_sign_in_email_addresses day_of_week, hour_of_day 7 sign-in email addresses, Sunday first, or 24, hour 0 first; "" for no assigned person
on_call_schedule.custom_schedule_code custom_schedule_code Starlark code with a function on_call(at), at most 64 KiB
on_call_schedule.custom_schedule_test_cases custom_schedule_code At most 50 test cases, or []. Each has local_time, such as "2026-11-02T09:30", expected_primary_sign_in_email_address ("" for no one), and optionally expected_backup_sign_in_email_address.

Every sign-in email address in on_call_schedule must name a person in people.

What settings check says when a value is wrong:

Problem Text
Escalation wait out of range on_call_schedule.escalate_if_unclaimed_after_minutes: must be a whole number of minutes from 1 to 60
Unknown type on_call_schedule.schedule_type: must be day_of_week, hour_of_day, weekly_rotation, or custom_schedule_code
A key the type needs on_call_schedule.first_start_date: required for a weekly_rotation schedule
A key the type does not use on_call_schedule.handoff_weekday: a day_of_week schedule does not use the setting. Remove the unused setting
Fixed backup without a person on_call_schedule.backup_sign_in_email_address: required with backup_option fixed_person
A backup person without fixed_person on_call_schedule.backup_sign_in_email_address: used only with backup_option fixed_person. Remove the backup_sign_in_email_address setting
A person not in people dana@example.com is not the sign-in email address of any person listed in people
Time zone on_call_schedule: the schedule needs a named time zone, such as America/New_York, or the time zone X is unknown; use a name such as America/New_York
Start date on_call_schedule: the first start date must be a date such as 2026-11-02 on the handoff weekday (0 is Sunday)
Rotation rotation needs at least one member; select a person for each rotation slot and do not list a person more than once
Assignments a day_of_week schedule has 7 assignment slots, one for each day of the week, Sunday first; an hour_of_day schedule has 24 assignment slots, one for each hour of the day, 0 first
Handoff time the handoff hour must be 0 to 23, and the handoff minute 0 to 59
Backup option the backup option must be everyone_available, fixed_person, or next_in_rotation (for a weekly rotation), else the backup option must be everyone_available or fixed_person
Custom code write your code, with a function on_call(at); code must be text of at most 64 KiB; line N has a control or text-direction character; remove it; at most 50 test cases; test case time must look like 2026-10-05T09:30

Applying the part saves and activates the schedule. To see whose turn the file's schedule gives before you apply it, run sudo mmhotline schedules preview --settings-file /etc/mmhotline/mmhotline_settings.yml. Taking on-call and unavailability stay on the On call page.

Set up in Set up people and the on-call schedule, and for custom_schedule_code, in Write a custom schedule.

Alert sources

alert_sources

The programs that send alerts and check-ins, such as a monitoring tool. Each source signs its requests with its signing secret.

alert_sources:
  - alert_source_id: monitoring
    name: Monitoring
    signing_secrets: ["YOUR_64_HEXADECIMAL_CHARACTER_SIGNING_SECRET"]
    warn_after_no_check_in_for_minutes: 2
    sos_after_no_check_in_for_minutes: 5
Key Required Type and allowed values Wrong value
alert_sources Yes A list of at most 100 sources; may be [] alert_sources: lists 101 alert sources; an installation has at most 100
alert_sources[].alert_source_id Yes Your own ID, 1 to 64 letters, digits, _ and -, such as monitoring; unique; not system or manual. The sender's URLs end with it: /alert-intake/monitoring and /check-in/monitoring. must be 1 to 64 letters, digits, _ and -, such as monitoring; system names MMHotline's own alerts; choose another ID; monitoring is also the ID of alert_sources[0]; each alert source must have a unique ID
alert_sources[].name Yes Text, 1 to 100 characters, shown on pages and in alerts alert source name must contain 1–100 characters
alert_sources[].signing_secrets Yes, secret One signing secret of 64 hexadecimal characters, which openssl rand -hex 32 prints, or two different ones while the sender moves to a new one must list one signing secret, or two while the sender moves to a new one; signing_secrets[0]: must be 64 hexadecimal characters, such as the output of openssl rand -hex 32; signing_secrets[1]: is the same secret as signing_secrets[0]
alert_sources[].warn_after_no_check_in_for_minutes Yes 1 to 60, or off: a warn alert after this many minutes without a check-in must be a whole number of minutes from 1 to 60, or off
alert_sources[].sos_after_no_check_in_for_minutes Yes 1 to 60, and longer than the warning: an SOS after this many minutes without a check-in the time before waking the person on call must be 1 to 60 minutes; the time before waking the person on call must be longer than the time before a warning
  • A source removed from the list is disabled: its signing secrets stop working, and its alerts and check-ins stay.
  • To move a sender to a new signing secret, list both secrets, apply, give the sender the new one, then remove the old one and apply again.
  • sudo mmhotline alert-sources list prints each source's intake and check-in URLs.

Set up in Connect a tool that sends alerts; Change a source later covers the secret change.

The agents

triage_agent

The triage agent: the team's coding agent, which investigates alerts and asks MMHotline to run playbooks. Leaving the section out removes the triage agent's setup.

Key Required Type and allowed values
triage_agent.commit_signing_public_key Yes The public key line the triage agent signs its commits with, from /var/lib/mmhotline-triage-agent/.ssh/id_ed25519.pub. The agent's own commits pass only within one_time_playbooks/ and triage_agent_notes/.
triage_agent.trusted_commit_signing_public_keys Yes The public key lines of the team's people and machines, such as each person's desk and util01. A commit no listed key signed waits for a review before any MMX run goes past it. Never the triage agent's own key. May be [].
triage_agent.commits_without_a_trusted_signature_are_approved_by Yes a_person, or a_person_or_the_triage_agent: the triage agent then reviews each such commit and approves an ordinary change
triage_agent.paths_only_a_person_may_approve Yes Paths in the systems repository, as Git names them, that only a person may approve; an entry ending in / is a directory. List the files that set the triage agent's own powers and the trusted keys. May be [].
triage_agent.coding_agent Yes claude_code or codex
triage_agent.credential_type Yes For claude_code: api_key or subscription_token. For codex: api_key, access_token (a Codex access token from a ChatGPT Business or Enterprise workspace), or chatgpt_sign_in (a ChatGPT sign-in made just for MMHotline). api_key is the most dependable.
triage_agent.credential Yes, secret An API key, the token from claude setup-token, or a Codex access token: one line, at most 4,096 bytes. For chatgpt_sign_in: the whole content of Codex's auth.json after codex login, at most 64 KiB.
triage_agent.credential_expires_on With access_token only The Codex access token's end date, such as 2027-02-06, as ChatGPT showed it. MMHotline reminds administrators 30 and 7 days before.
triage_agent.model Yes A model name of at most 200 characters without spaces, such as opus. MMHotline checks only the shape; the agent's setup test proves the model works.
triage_agent.reasoning_effort Yes One lowercase word, such as high. Claude Code takes low, medium, high, xhigh, or max; codex debug models lists Codex's levels. MMHotline checks only the shape.
triage_agent.internet_access Yes listed_hosts_only, or any_site, which also allows unlisted hosts at public addresses, through MMHotline's logging proxy
triage_agent.listed_hosts Yes Extra hosts the agent may reach, each {host: NAME, port: N}, such as [{host: packages.example.com, port: 443}]; usually []. Do not list the hosts MMHotline adds by itself: it lists the coding agent's service, OSV.dev's API, CISA's list of known exploited vulnerabilities on raw.githubusercontent.com, each inventory server at port 22, and the vendor pages' hosts in software_outside_the_distribution.yml.
triage_agent.software_review_interval_hours Yes off, or 1 to 8760 hours between scheduled software reviews
triage_agent.sos_waits_for_the_agent_minutes Yes 0 to 15 minutes an SOS from an alert source waits for the agent before MMHotline wakes the person on call. 0 wakes the person at once while the agent still looks. 5 is a good start: the wait gives the agent a chance to fix the problem before anyone is woken. An SOS that joins a triage session which already woke the person on call does not wait or wake them again.
triage_agent.warn_waits_for_the_agent_minutes Optional; left out means 0 0 to 15 minutes MMHotline holds a warning that starts a triage session, or joins one while its SOS waits for the agent. 0 holds nothing. 3 makes a routine fix reach people as one message, the agent's report. The rules for a held warning are below the table.
triage_agent.sos_waits_for_a_server_replacement_minutes Yes 0 to 60 minutes a waiting SOS keeps waiting after the agent starts replacing a server, counted from the replacement's start. 0 turns the longer wait off. MMHotline sends one warn alert as the replacement starts.

How warn_waits_for_the_agent_minutes works:

  • MMHotline delivers a held warning at the end of the wait.
  • If the agent asked for a playbook run or sent a report in that session meanwhile, the agent's request or report takes the warning over, and MMHotline sends nothing for the warning.
  • MMHotline delivers the warning at once when the agent cannot take it: the agent is not set up, its sandbox is not installed, MMHotline's copy of the systems repository is missing, or its run fails or ends without a request or report.
  • A warning that joins a session while an SOS waits is held together with that SOS. An SOS keeps its own wait.

Limits and their reasons:

  • sos_waits_for_the_agent_minutes and warn_waits_for_the_agent_minutes stop at 15. Reason: a stuck or wrong agent must not hold an SOS or a warning longer.
  • sos_waits_for_a_server_replacement_minutes stops at 60. Reason: one MMX run ends after one hour, so a replacement cannot take longer. 10 is a good start: a measured replacement by the agent took about 5 to 6 minutes.

What settings check says when a value is wrong, each after its key path:

Key Text
commit_signing_public_key, trusted_commit_signing_public_keys[N] must be one OpenSSH public key line: the key type, such as ssh-ed25519, then the key, then an optional comment; starts with "X", which is not an SSH key type: ...; has a key that is not valid base64: ...; must be one OpenSSH public key line, without a line break or a NUL character
trusted_commit_signing_public_keys[N] is the triage agent's key, triage_agent.commit_signing_public_key; list only the keys of the team's people and machines, because the triage agent's commits may change only one_time_playbooks/ and triage_agent_notes/
commits_without_a_trusted_signature_are_approved_by must be a_person or a_person_or_the_triage_agent
paths_only_a_person_may_approve[N] must be a path in the systems repository, such as AGENTS.md, or a directory ending in /, such as roles/every_server/files/; "/etc/sudoers" must be relative to the systems repository, ...; ... must not have an empty, ".", or ".." part, ...; AGENTS.md is listed twice
coding_agent must be claude_code or codex
credential_type must be api_key or subscription_token for claude_code, or must be api_key or access_token or chatgpt_sign_in for codex
credential CREDENTIAL_TYPE requires a credential (when credential is empty, with the credential type you wrote in place of CREDENTIAL_TYPE, such as api_key requires a credential); credential must be at most 4,096 bytes; credential must be one line without control characters; for chatgpt_sign_in, credential must be the whole content of Codex's auth.json after codex login, but it ... with the reason
credential_expires_on is required with credential_type access_token: ...; must be a date such as 2027-02-06; is only for credential_type access_token; ...
model must be a model name of at most 200 characters without spaces, such as opus; the coding agent's setup test checks whether the service recognizes the model name
reasoning_effort must be a lowercase word, such as high; the coding agent's setup test checks whether the model supports the reasoning effort
internet_access must be listed_hosts_only or any_site
listed_hosts[N] port must be 1 to 65535; host must be a DNS name or an IP address; Loopback addresses are never allowed. (also link-local, unspecified, and multicast addresses); packages.example.com:443 is listed twice
software_review_interval_hours must be off, or a whole number of hours from 1 to 8760
sos_waits_for_the_agent_minutes must be a whole number of minutes from 0 to 15: 0 wakes the person on call at once, and a stuck or wrong agent must not hold an SOS longer than 15 minutes
warn_waits_for_the_agent_minutes must be a whole number of minutes from 0 to 15: 0 holds no warning, and a stuck or wrong agent must not hold a warning longer than 15 minutes
sos_waits_for_a_server_replacement_minutes must be a whole number of minutes from 0 to 60: 0 turns the longer wait off, and an MMX run longer than its one-hour limit cannot finish a replacement

Notes that settings check and settings apply print, which never fail the check:

  • Under triage_agent.commits_without_a_trusted_signature_are_approved_by, what the setting means, such as A person reviews every commit no trusted key signed before MMX runs past it. With a_person_or_the_triage_agent and an empty paths_only_a_person_may_approve, the note adds The list is empty, so the triage agent may approve a commit that changes its own rules or the trusted keys: list the files that set them.
  • Under triage_agent.software_review_interval_hours: Scheduled software review: off. The triage agent wakes only for alerts; sudo mmhotline triage-agent run-software-review still runs one review, such as to learn what one costs., or Scheduled software review: every 24 hours, with the team's task in ....

The keys apply as these parts: the two signing key settings; the two commit approval settings; the agent setup (coding_agent, credential_type, credential, credential_expires_on, model, reasoning_effort); and each other key on its own. The agent setup runs one test prompt first and is saved only when the test passes. While another triage agent run is active, the agent setup does not apply; apply again after the run ends.

Start with software_review_interval_hours: off and test what one review costs first. A playbook listed with starts_a_server_replacement: true starts the replacement; MMHotline wakes the person on call at once when a run of it fails. How one approval can cover a known sequence of playbooks: Decide what the triage agent may do.

Set up in Set up the triage agent. The signing keys and approvals: Review a commit. The credential: Keep the agents signed in. software_review_interval_hours: Turn on the scheduled software review.

review_agent

A second coding agent, which must approve each playbook the triage agent asks to run before the team's rules apply. It has its own Unix user and runs with no systems repository.

review_agent:
  coding_agent: codex
  credential_type: api_key
  credential: "YOUR_OPENAI_API_KEY"
  model: YOUR_MODEL
  reasoning_effort: high
Fact Value
Required Optional. Leaving it out removes the review agent's setup.
Keys coding_agent, credential_type, credential, credential_expires_on, model, and reasoning_effort, with the same values and messages as in triage_agent
Extra rule Two chatgpt_sign_in credentials must be different sign-ins: review_agent.credential: The review agent needs its own ChatGPT sign-in. Sign in again in a new empty folder and use that auth.json. Reason: two copies of one sign-in end each other when either refreshes.
Applies As one part, after one test prompt passes
Set up in Set up the triage agent

rules_for_running_agent_playbooks

For each inventory, which playbook runs the triage agent may start alone and which need a person.

rules_for_running_agent_playbooks:
  production:
    preset: routine_fixes_without_permission
Key Required Type and allowed values
rules_for_running_agent_playbooks Yes; may be {} One entry per inventory, named as in inventories/NAME.yml: letters, digits, _, and -, at most 100 characters
<inventory>.preset preset or action_kinds, not both permission_for_every_fix, routine_fixes_without_permission, or every_fix_without_permission
<inventory>.action_kinds preset or action_kinds, not both A rule for each kind of action: start_or_restart_a_service, free_disk_space, pull_a_container_image_again, reboot_a_server, block_a_network_range, change_dns_records, check_only, unrecognized_task
<inventory>.action_kinds.<kind>.rule Yes get_permission_then_act, warn_wait_then_act, or act_then_report
...rule with approvals_needed With get_permission_then_act only 1 or 2
...rule with warning_wait_minutes With warn_wait_then_act only 1 to 1440
<inventory>.playbooks Optional Ordinary playbooks listed by name, NAME of playbooks/NAME.yml, each with a rule as for a kind. A listed playbook takes its own rule instead of the strictest rule of its kinds, even when a task acts on a service whose changes a person must approve. A one-time playbook cannot be listed.
<inventory>.playbooks.<name>.its_approval_also_approves_in_the_same_session Optional, with get_permission_then_act only Other playbooks listed on the same inventory that a person's approval of this one also approves, once each, later in the same triage session
<inventory>.playbooks.<name>.starts_a_server_replacement Optional true marks the playbook whose run starts a server replacement, so a waiting SOS waits triage_agent.sos_waits_for_a_server_replacement_minutes longer

Every inventory in MMHotline's copy of the systems repository needs an entry. An inventory without one fails the part, which still applies the inventories it lists, and every fix on the missing inventory needs a person until you add the entry: The rules for running the agent's playbooks applied for the inventories listed in the settings file. Any inventory without rules requires a person to approve every fix until you add its rules.

An action kind that action_kinds leaves out needs a person, with 1 approval. Reason: a newer MMHotline may add a kind, and an older file must still apply. settings check and settings apply print a note, which does not fail the check:

rules_for_running_agent_playbooks.production: check_only has no rule; it needs a person until you add one

What settings check says when a value is wrong, each after its key path:

Problem Text
Inventory name the inventory name "..." may have only letters, digits, "_", and "-", and at most 100 characters
Both or neither has preset and action_kinds. Specify only preset or only action_kinds; needs preset or action_kinds
Preset must be permission_for_every_fix, routine_fixes_without_permission, or every_fix_without_permission
Kind unknown kind of action; the kinds are start_or_restart_a_service, ...
Rule rule must be get_permission_then_act, warn_wait_then_act, or act_then_report; warning_wait_minutes must be 1 to 1440 with warn_wait_then_act; warning_wait_minutes goes only with warn_wait_then_act; approvals_needed must be 1 or 2; approvals_needed goes only with get_permission_then_act
Playbook name must be NAME of a playbook file playbooks/NAME.yml, starting with a letter or "_", then letters, digits, "_", or "-"; a one-time playbook in one_time_playbooks/ cannot be listed
Also-approves list goes only with rule get_permission_then_act: ...; names replace_failed_servers itself; an approval approves its own playbook already; every_server is named twice; each named playbook is approved once; every_server is not listed under rules_for_running_agent_playbooks.servers.playbooks; list it there with its own rule, so the approval covers only a playbook a person chose by name

Set up in Decide what the triage agent may do; listed playbooks in Give one playbook its own rule.

services_whose_changes_a_person_must_approve

The systemd units that no playbook may start, stop, restart, or change without a person, on every inventory.

services_whose_changes_a_person_must_approve: [pgautofailover.service]
Fact Value
Required Yes; may be []
Type A list of systemd unit names. A name without a unit type means a .service, so pgautofailover is pgautofailover.service.
Default None
Wrong value "..." is not a systemd unit name: a unit name has only letters, digits, and the characters _ . @ : \ -, and starts with a letter, a digit, or _; systemd allows at most N characters in a unit name, ...; pgautofailover.service is listed twice; a name without a unit type means a .service, so pgautofailover is pgautofailover.service
Set up in Decide what the triage agent may do

Sending services

sending_services

The accounts MMHotline sends notifications through. Each service is an optional section inside sending_services, and each applies as its own part after the service's own check passes; a failed check keeps the earlier setup.

sending_services: {}
Fact Value
Required Yes; may be {}
Sections pushover, twilio, telegram, slack, mail_server, each optional
Set up in Connect a sending service
  • A service left out while contact methods use it is refused, and the setup stays, such as: cannot remove the Pushover setup: 1 Pushover notification contact method uses the sending service setup. Remove those contact methods on the Contact methods page, or with mmhotline contact-methods remove --contact-method-id ID, first. A different Telegram bot or Slack workspace is also refused while contact methods use the existing one, and a new Pushover application token is refused while a Pushover emergency message is open.
  • When a service is newly set up, settings apply prints its next step under its part, such as sending_services.pushover: Next step: Add your own Pushover user key under Contact methods. Saving the Pushover contact method sends an emergency test alert.

sending_services.pushover

Key Required Type and allowed values Wrong value
application_token Yes, secret The API token of the team's Pushover application: 30 letters and digits must be one line of exactly 30 letters and digits, as Pushover shows the application's API token

Set up in Connect Pushover.

sending_services.twilio

Key Required Type and allowed values Wrong value
account_sid Yes AC and 32 hexadecimal digits must be AC followed by 32 hexadecimal digits
api_key_sid Yes A dedicated API key: SK and 32 hexadecimal digits must be SK followed by 32 hexadecimal digits
api_key_secret Yes, secret The API key's secret: one line, at most 4,096 bytes required; must be at most 4,096 bytes; must be one line
phone_number_to_call_from Yes The Twilio voice number, in E.164 form: +, the country code, and the number, without spaces must be an E.164 phone number, such as +12025550102
extra_phone_numbers_on_the_contact_card Yes; may be [] More E.164 numbers for the contact card that people save must be an E.164 phone number, such as +12025550102

Set up in Connect Twilio.

sending_services.telegram

Key Required Type and allowed values Wrong value
bot_token Yes, secret The token from BotFather, Telegram's bot-management chat: digits, a colon, then letters, digits, _ and - must be the token from BotFather: digits, a colon, and letters, digits, _ and -

Set up in Connect Telegram.

sending_services.slack

Key Required Type and allowed values Wrong value
bot_token Yes, secret The Bot User OAuth Token, which starts with xoxb- must be the Bot User OAuth Token, which starts with xoxb-
app_level_token Yes, secret The app-level token with connections:write, which starts with xapp- must be the app-level token, which starts with xapp-

Set up in Connect Slack.

sending_services.mail_server

Key Required Type and allowed values
smtp_server Yes host:port, such as smtp.example.com:587
smtp_connection_security Yes starttls, implicit_tls, or unencrypted
email_from_address Yes One plain ASCII email address, such as alerts@example.com
smtp_user_name Optional One line; set together with smtp_password, or neither
smtp_password Optional, secret One line; set together with smtp_user_name, or neither

settings check reports a wrong value under sending_services.mail_server: smtp_server must be a host name and port, such as smtp.example.com:587; smtp_connection_security must be starttls, implicit_tls, or unencrypted; email_from_address must be one plain ASCII email address, such as alerts@example.com; smtp_user_name and smtp_password are required together, or both must be empty; SMTP credentials must be one line of UTF-8 text without control characters.

Set up in Connect a mail server.

Cloudflare features

mmhotline_down_alarm

The MMHotline-down alarm: a Cloudflare Worker in the team's own Cloudflare account that wakes the person on call when MMHotline stops checking in.

Key Required Type and allowed values Wrong value
mmhotline_down_alarm Optional Section. Leaving it out tells the Worker to hold no notifications, then removes the setup.
cloudflare_account_id Yes 32 lowercase hexadecimal characters the Cloudflare account ID must be 32 lowercase hexadecimal characters, as the dashboard shows it
cloudflare_api_token Yes, secret An account API token with one permission, Account › Workers Scripts › Edit: 1 to 1,024 bytes, one line, no white space. MMHotline keeps it to upload the new Worker after an upgrade. the Cloudflare API token must be one line of text
wake_the_person_on_call_after_minutes_without_a_check_in Yes 3 to 60; 5 is a good start must be a whole number of minutes from 3 to 60: MMHotline checks in every minute and at once when it starts, so a shorter wait would wake someone for a server reboot, and a longer one would leave a stopped MMHotline unnoticed for over an hour
wake_the_backup_after_minutes_without_a_check_in Yes At least 5 more than the person on call's wait; 10 is a good start must be a whole number of minutes, at least 5 more than wake_the_person_on_call_after_minutes_without_a_check_in (10), so the person on call has time to answer before the backup is woken

A server reboot leaves at most about 2.5 minutes without a check-in, so a wait under 3 minutes would wake someone for a reboot.

The two wake times apply as their own part, which takes effect within seconds. The account and token apply as the mmhotline_down_alarm part, after MMHotline uploads the Worker and checks in.

Set up in Set up the MMHotline-down alarm.

status_page

The public status page, which MMHotline publishes to a Cloudflare R2 bucket.

Key Required Type and allowed values Wrong value
status_page Optional Section. Leaving it out removes the setup from MMHotline; the bucket and the page stay in Cloudflare.
page_title Yes The page's title and heading, shown exactly as written: 1 to 100 characters on one line, such as Example Inc. status the page title must have 1 to 100 characters on one line
host_name Yes The page's host name, such as status.example.com the host name must be a DNS name such as status.example.com
r2_bucket_url Yes The bucket's S3 API address, as the laptop's status-page set-up printed it the R2 bucket URL must be the bucket's S3 API address, such as https://ACCOUNT_ID.r2.cloudflarestorage.com/BUCKET_NAME (or ... must end with the bucket's name, ...)
r2_access_key_id Yes, secret The Access Key ID of an R2 API token with Object Read & Write on that bucket only: letters and digits must be letters and digits, as Cloudflare shows the R2 Access Key ID
r2_secret_access_key Yes, secret That token's Secret Access Key: one line, at most 4,096 bytes required; must be at most 4,096 bytes; must be one line

The part applies after MMHotline publishes the current page with the file's R2 key.

Set up in Publish a status page.

This page is docs/reference/settings-file.md in the MMHotline repository.