MMHotline documentation
Each page answers one question. Start with Get started if you have not run MMHotline yet.
Start here
- Get started: install MMHotline on one server, put it behind Cloudflare Access, and send and claim your first SOS.
- How MMHotline works: the parts, how an alert becomes an incident that wakes someone, and why each part is built the way it is.
- When something fails: a message or a surprise, what it means, and what to do.
Install and look after the server
- Install MMHotline: the release, the settings file,
and
mmhotline install. - Put MMHotline behind Cloudflare Access: the team's sign-in in front of every page.
- Change the settings: edit the settings file, check it, and apply it.
- Back up and restore the database: a copy of every change off the server, a check that it keeps up, and a restore on the same or a new server.
- Upgrade MMHotline: install a newer release.
- Sign in during a Cloudflare outage: reach the pages when Cloudflare Access is down.
- Remove MMHotline: take MMHotline off a server.
People, contact methods, and the schedule
- Set up people and the on-call schedule: who is on the team, who is on call, and who is the backup.
- Write a custom schedule: a schedule rule the built-in rotations cannot express.
- Connect a sending service: choose among Pushover, Twilio, a mail server, Slack, and Telegram.
- Add contact methods: how each person is reached, and for which severities.
- Be away or take over on call: mark yourself unavailable, or take on-call from someone.
Alerts and incidents
- Connect a tool that sends alerts: an alert source, its signing secret, and its check-ins.
- Handle an incident: claim it, escalate it, and resolve it.
- Declare planned downtime: silence an alert source while you work on its servers.
- Mute the team: stop every notification for a while, and what that risks.
When MMHotline itself is down, and your customers
- Set up the MMHotline-down alarm: someone is still woken when MMHotline's server dies.
- Publish a status page: a public page for your customers, in your own Cloudflare account.
- Post to the status page: open, update, and resolve a notice, also from a laptop.
The triage agent
- Set up the triage agent: prepare the servers and the systems repository, and give MMHotline your team's Claude Code or Codex.
- Decide what the triage agent may do: which fixes run without a person, and approving or refusing the others.
- Review a commit: the commits that wait before MMX runs past them.
- Turn on the scheduled software review: learn what one review costs, then set the interval.
- Keep the agents signed in: replace a credential before it ends.
Reference
- The mmhotline command: every command, flag, exit code, and JSON field.
- The settings file: every key of
/etc/mmhotline/mmhotline_settings.yml. - Alert requests: the format a tool uses to send alerts and check-ins, and how to sign them.
- Severities and incident states: sos, warn, and info, and how an incident moves from unclaimed to resolved.
- Outgoing webhooks: the request MMHotline sends to a webhook contact method, and how to check its signature.
- Installed files: every file, user, and
unit that
mmhotline installcreates.
Changing MMHotline itself
- Changing MMHotline: the design, testing, validation record, and acceptance contracts, for people who change MMHotline's own code.
This page is docs/README.md in the MMHotline repository.