Installed files
This page lists every file, directory, Unix user, systemd unit, and socket
that sudo mmhotline install creates or uses on the MMHotline server, with
its owner and mode, what it holds, and who reads it. It also lists the
files the continuous backup adds and the files a person adds by hand to
prepare the triage agent. A unit is a systemd service or socket file
under /etc/systemd/system. A socket here is a Unix socket file
through which one process asks another for something. To install step by
step, see Install MMHotline.
What install prints
mmhotline install checks each item and prints one line for it: what it
changed, or that the item was already correct. An unchanged rerun writes
no file and leaves running services alone. A changed program or unit, a
new database, or a corrected settings file restarts mmhotline.service,
and so does a stopped mmhotline.service. Install then waits up to 30
seconds for the command socket to answer. When MMHotline does not start,
install prints the last 30 lines of the service's journal.
On a new server, from a real run, the first lines are:
Checks: Linux, root, and systemd confirmed
Unix user: created mmhotline with no login shell
Unix user: created mmhotline-triage-agent with no login shell
Unix user: created mmhotline-review-agent with no login shell
Triage agent home: installed or corrected /var/lib/mmhotline-triage-agent
Service: mmhotline-triage-agent-run.socket unit updated; enabled; restarted
Service: mmhotline-review-agent-run.socket unit updated; enabled; restarted
Service: mmhotline-fleet-view.socket unit updated; enabled; restarted
Binary: installed or corrected /usr/local/bin/mmhotline
Data directory: installed or corrected /var/lib/mmhotline
Database: created mmhotline.db for Operations
Settings file: set owner root, group mmhotline, and mode 0640, so MMHotline can read the settings file and only root can change the settings file: /etc/mmhotline/mmhotline_settings.yml
Then come a line for mmhotline.service, such as
Service: mmhotline unit updated; enabled; restarted, the Backup: line,
the MMHotline: line with web_url, and, without cloudflare_access in
the settings file, a Next: line.
An unchanged rerun, from a real run:
Checks: Linux, root, and systemd confirmed
Unix user: mmhotline already exists
Unix user: mmhotline-triage-agent already exists
Unix user: mmhotline-review-agent already exists
Triage agent home: already correct (/var/lib/mmhotline-triage-agent)
Service: mmhotline-triage-agent-run.socket unit already correct; already enabled; already running
Service: mmhotline-review-agent-run.socket unit already correct; already enabled; already running
Service: mmhotline-fleet-view.socket unit already correct; already enabled; already running
Binary: already correct (/usr/local/bin/mmhotline)
Data directory: already correct (/var/lib/mmhotline)
Database: mmhotline.db already exists; kept
Settings file: already has owner root, group mmhotline, and mode 0640: /etc/mmhotline/mmhotline_settings.yml
Service: mmhotline unit already correct; already enabled; already running
Backup: not set up; see https://github.com/monkeymaxxing/mmhotline/blob/main/docs/how-to/back-up-and-restore-the-database.md#set-up
MMHotline: https://ops.example.com
Next: set up cloudflared and a Cloudflare Access application for https://ops.example.com, add cloudflare_access to /etc/mmhotline/mmhotline_settings.yml, and apply the file, with your deploy tool or with sudo mmhotline settings apply (https://github.com/monkeymaxxing/mmhotline/blob/main/docs/how-to/put-mmhotline-behind-cloudflare-access.md)
With the backup set up, the Backup: line is
Backup: Litestream service configured; check that uploads keep up with sudo mmhotline backup check-uploads,
after a Service: mmhotline-backup ... line.
Before it changes anything, install refuses to run without Linux, root,
systemd at /run/systemd/system, or a settings file that passes the
check. Install also refuses a symbolic link, or a file of another kind, at
any path below where it expects a regular file or a directory, and says
to move it aside.
Unix users
All four users are system users with the shell /usr/sbin/nologin.
Install creates each one that is missing and keeps an existing one.
mmhotline
| Fact | Value |
|---|---|
| Created with | groupadd --system --force mmhotline, then useradd --system --gid mmhotline --home-dir /var/lib/mmhotline --no-create-home --shell /usr/sbin/nologin mmhotline |
| Group | mmhotline. An existing group stays, such as one a deploy tool made so it could write the settings file with that group before the first install. |
| Home | /var/lib/mmhotline |
| Runs | mmhotline.service, and the MMX runs MMHotline starts for the triage agent |
mmhotline-triage-agent
| Fact | Value |
|---|---|
| Created with | useradd --system --user-group --home-dir /var/lib/mmhotline-triage-agent --no-create-home --shell /usr/sbin/nologin mmhotline-triage-agent |
| Group | Its own, mmhotline-triage-agent. It is not in group mmhotline. |
| Home | /var/lib/mmhotline-triage-agent |
| Runs | Each triage agent run, in mmhotline-triage-agent-run@.service |
mmhotline-review-agent
| Fact | Value |
|---|---|
| Created with | useradd --system --user-group --home-dir /nonexistent --no-create-home --shell /usr/sbin/nologin mmhotline-review-agent |
| Group | Its own, mmhotline-review-agent |
| Home | None. Each review gets a new empty home in its run's private /tmp, which systemd discards when the run ends. |
| Runs | Each review agent run, in mmhotline-review-agent-run@.service |
Reason for a separate user: a triage run, which may run code planted in the triage agent, can then neither see a review run's processes nor write anything the review agent reads.
mmhotline-backup
| Fact | Value |
|---|---|
| Created | Only when /etc/mmhotline/backup.env exists and litestream is on the search path |
| Created with | useradd --system --gid mmhotline --home-dir /nonexistent --no-create-home --shell /usr/sbin/nologin mmhotline-backup |
| Group | mmhotline, so Litestream may write the database and SQLite's working files |
| Home | None |
| Runs | mmhotline-backup.service (Litestream) |
Reason for a separate user: MMHotline's own process then cannot read the bucket's keys in Litestream's process.
Programs
/usr/local/bin/mmhotline
| Fact | Value |
|---|---|
| Owner and mode | root:root, 0755 |
| Holds | The MMHotline program |
| Written by | Install, which copies the program it runs from, through a new file renamed into place. When install runs from /usr/local/bin/mmhotline itself, it only corrects the owner and mode. |
| Read by | mmhotline.service, the agent run units, and the fleet view unit, which all run this path; people, for sudo mmhotline ... commands |
Programs install and MMHotline look for
Install does not put these in place; the how-to pages do.
| Program | Looked for | Used by | Set up in |
|---|---|---|---|
litestream |
On the search path of sudo mmhotline install, such as /usr/local/bin/litestream. Install writes the path it finds into mmhotline-backup.service. |
The backup | Back up and restore the database |
mmx and mmx-runner-linux-amd64 or mmx-runner-linux-arm64 |
/usr/local/bin, then /usr/bin |
MMHotline, to read and run the triage agent's playbooks | Set up the triage agent, step 7 |
claude, or codex with codex-code-mode-host and rg |
/usr/local/bin |
The agent runs | Set up the triage agent, step 1 |
Settings
/etc/mmhotline/
| Fact | Value |
|---|---|
| Owner and mode | root, 0755 when install creates it. Install does not change an existing directory. |
| Holds | The settings file, and with the backup, backup.env and litestream.yml |
/etc/mmhotline/mmhotline_settings.yml
| Fact | Value |
|---|---|
| Owner and mode | root:mmhotline, 0640. Install sets them; before the first install, the group may not exist yet, so the file starts as root:root. |
| Holds | Every team setting that is not one person's own, including secrets. The settings file lists its keys. |
| Written by | A person with sudoedit, or the team's deploy tool. Install never changes the contents and refuses to run when the file is missing or has problems. |
| Read by | mmhotline.service at start and on settings apply, through the group; install; the fleet view unit at each run, as root |
MMHotline refuses the file when it is not a regular file, not owned by root, writable by others than root, or readable by every user (why).
/etc/mmhotline/backup.env
| Fact | Value |
|---|---|
| Owner and mode | root:root, 0600 |
| Holds | The backup bucket's address, MMHOTLINE_BACKUP_REPLICA_URL, and its key, such as LITESTREAM_ACCESS_KEY_ID and LITESTREAM_SECRET_ACCESS_KEY, as KEY=value lines |
| Written by | A person, as Back up and restore the database says. Install never writes it; its presence turns the backup on. |
| Read by | systemd, which passes it to mmhotline-backup.service only; install, to list the bucket before it starts the backup of a new database. mmhotline.service never gets it, and the fleet view unit hides it. |
/etc/mmhotline/litestream.yml
| Fact | Value |
|---|---|
| Owner and mode | root:root, 0644 |
| Written by | Install, with the backup unit, when the backup is set up. It replaces different contents. |
| Holds | Litestream's configuration: the database, where Litestream keeps its working files, and the replica URL, which Litestream reads from backup.env when it starts. No credential. |
| Read by | Litestream, in mmhotline-backup.service |
The whole file:
# Managed by mmhotline install.
dbs:
- path: /var/lib/mmhotline/mmhotline.db
meta-path: /var/lib/mmhotline-backup/mmhotline.db-litestream
replica:
url: "${MMHOTLINE_BACKUP_REPLICA_URL}"
State
/var/lib/mmhotline/
| Fact | Value |
|---|---|
| Owner and mode | mmhotline:mmhotline, 0770 |
| Holds | The database, and the files the triage agent preparation adds: MMHotline's SSH key, its copy of the systems repository, and the MMX vault passphrase (below) |
| Read by | mmhotline.service, which may write only here and in its private /tmp; Litestream, through the group. The agent runs and the fleet view cannot see it. |
The group may write so that the backup's own user can write the database.
/var/lib/mmhotline/mmhotline.db
| Fact | Value |
|---|---|
| Owner and mode | mmhotline:mmhotline, 0660, and the same for SQLite's working files beside it, mmhotline.db-wal and mmhotline.db-shm |
| Holds | All of the team's state: people, schedules, alerts, incidents, contact methods, and the applied settings, including the sending services' tokens and webhook bearer tokens |
| Written by | Install creates it, named after team_name, only when it is missing. An existing database stays, with every row. |
| Read by | mmhotline.service; Litestream; sudo mmhotline backup check-uploads |
/var/lib/mmhotline/new-database-waits-for-the-backup-bucket-check
| Fact | Value |
|---|---|
| Owner and mode | root:root, 0600 |
| Holds | One line: This database was created by mmhotline install. Install checks that the backup bucket holds no earlier backup before it starts the backup, then removes this file. |
| Written by | Install, when it creates a new database |
| Read by | Install, on this run and every later run |
| Removed by | Install, once backup.env exists and the bucket folder holds no backup |
While the marker exists and the bucket already holds a backup, install
does not start the backup and says:
Backup: not started: this server's database is new, and the bucket in MMHOTLINE_BACKUP_REPLICA_URL already holds a backup, up to transaction ... Starting the backup now would save the empty database as the newest backup. ....
Reason: on a new server, Litestream would save the empty database as the
bucket's newest state, and a restore would then bring back nothing. A
restore replaces the state directory and so removes the marker; a
database a deploy tool restored before install never had one.
Restore on a new server
uses this.
/var/lib/mmhotline-triage-agent/
| Fact | Value |
|---|---|
| Owner and mode | mmhotline-triage-agent:mmhotline-triage-agent, 0700 |
| Holds | The triage agent's home: its SSH key and Git settings from the preparation, its copy of the systems repository, the coding agent's own files, and the files MMHotline writes at the start of each run (below) |
| Written by | Install creates it and corrects its owner and mode. The triage agent runs write in it. |
| Read by | The triage agent runs; sudo mmhotline triage-agent check-preparation. The review agent runs and the fleet view cannot see it. |
/var/lib/mmhotline-backup/
| Fact | Value |
|---|---|
| Owner and mode | mmhotline-backup, 0700. systemd creates it for the backup unit. |
| Holds | mmhotline.db-litestream/: Litestream's working files, copies of the database changes it captured, kept until they are uploaded |
| Read by | Litestream. The mmhotline user cannot write it; the fleet view cannot see it. |
Units
Install writes each unit as root:root, mode 0644, and replaces
different contents. After a change it runs systemctl daemon-reload.
Each unit starts with the line # Managed by mmhotline install.. Install
enables each unit and starts or restarts it when needed.
mmhotline.service
| Fact | Value |
|---|---|
| Runs as | mmhotline |
| Runs | mmhotline serve, below |
| Restarts | Always, 2 seconds after a failure, with no limit on repeated restarts |
| First line | # Managed by mmhotline install. Binary SHA256: ..., so a rerun of install from the installed path still sees a new program |
/usr/local/bin/mmhotline serve --state-dir /var/lib/mmhotline --settings-file /etc/mmhotline/mmhotline_settings.yml --command-socket /run/mmhotline/command.sock --triage-agent-run-socket /run/mmhotline-triage-agent/run.sock --review-agent-run-socket /run/mmhotline-review-agent/run.sock --triage-agent-proxy-socket /run/mmhotline/triage-agent-proxy.sock --fleet-view-socket /run/mmhotline-fleet-view.sock
The file system is read-only for the service except
/var/lib/mmhotline and its private /tmp. The service cannot see home
directories, cannot gain privileges, and has no capabilities. Its
system call filter allows the seccomp call and resource limits, which
custom schedules use. systemd creates /run/mmhotline/ for it.
MMHotline serves its pages over plain HTTP at web_listen_address and
takes signed alerts at alert_intake_listen_address; it listens on no
public address unless the settings file names one.
The private network needs no TLS, but signatures do not encrypt alert contents.
mmhotline-triage-agent-run.socket and mmhotline-triage-agent-run@.service
| Fact | Value |
|---|---|
| Socket | Listens at /run/mmhotline-triage-agent/run.sock and starts one service instance per connection, at most 4 at once |
| Service runs as | mmhotline-triage-agent |
| Service runs | /usr/local/bin/mmhotline triage-agent run-inside-sandbox --proxy-socket /run/mmhotline-triage-agent-proxy.sock, with HOME and the working directory /var/lib/mmhotline-triage-agent |
| Limits | No network of its own; its only way out is MMHotline's logging proxy socket, mounted at /run/mmhotline-triage-agent-proxy.sock. Writes only its home and private /tmp. Cannot see /var/lib/mmhotline, /etc/mmhotline, or /run/mmhotline. Ended after one hour. |
Reason: MMHotline cannot switch users, so systemd starts each run as the agent's user, in its own control group.
mmhotline-review-agent-run.socket and mmhotline-review-agent-run@.service
| Fact | Value |
|---|---|
| Socket | Listens at /run/mmhotline-review-agent/run.sock, one service instance per connection, at most 4 at once |
| Service runs as | mmhotline-review-agent |
| Service runs | The same command as a triage agent run |
| Limits | Those of a triage agent run, and it also cannot see /var/lib/mmhotline-triage-agent. Writes only its private /tmp, where its new home is. |
mmhotline-fleet-view.socket and mmhotline-fleet-view@.service
| Fact | Value |
|---|---|
| Socket | Listens at /run/mmhotline-fleet-view.sock, one service instance per connection, at most 2 at once |
| Service runs as | root |
| Service runs | /usr/local/bin/mmhotline run-fleet-view-command --settings-file /etc/mmhotline/mmhotline_settings.yml, which runs the settings file's fleet_view_command with no shell |
| Limits | Read-only file system, Unix sockets only, ended after 20 seconds. Cannot see /var/lib/mmhotline, /var/lib/mmhotline-backup, /var/lib/mmhotline-triage-agent, /etc/mmhotline/backup.env, or /run/mmhotline. |
| Written | On every install, whether or not the settings file has fleet_view_command, so adding the command needs only settings apply |
fleet_view_command says more.
mmhotline-backup.service
| Fact | Value |
|---|---|
| Written | Only when /etc/mmhotline/backup.env exists and litestream is found |
| Runs as | mmhotline-backup, group mmhotline |
| Runs | "LITESTREAM_PATH" replicate -config /etc/mmhotline/litestream.yml, with the path install found |
| Reads | backup.env, as its environment, and litestream.yml |
| Restarts | Always, 2 seconds after a failure; install also restarts it when litestream.yml changed |
| Limits | MMHotline's own restrictions: read-only file system except /var/lib/mmhotline and its state directory /var/lib/mmhotline-backup |
When backup.env exists but litestream is missing, install stops with
Backup: /etc/mmhotline/backup.env exists but litestream is missing; install Litestream 0.5.18 as ... says, and rerun sudo mmhotline install.
After you change backup.env, run sudo systemctl restart mmhotline-backup.
Sockets
| Socket | Owner and mode | Made by | Who connects |
|---|---|---|---|
/run/mmhotline/command.sock |
mmhotline, 0600, in /run/mmhotline/ (mmhotline, 0700) |
mmhotline serve |
sudo mmhotline ... commands. MMHotline checks that the connecting process is root. |
/run/mmhotline/triage-agent-proxy.sock |
mmhotline, 0666, in the 0700 directory, which blocks other users |
mmhotline serve |
Each agent run, through systemd's mount at /run/mmhotline-triage-agent-proxy.sock. Every site the agent reaches goes through this logging proxy. |
/run/mmhotline-triage-agent/run.sock |
mmhotline:mmhotline, 0600 |
systemd, from the triage agent socket unit | MMHotline, to start a triage agent run |
/run/mmhotline-review-agent/run.sock |
mmhotline:mmhotline, 0600 |
systemd, from the review agent socket unit | MMHotline, to start a review agent run |
/run/mmhotline-fleet-view.sock |
mmhotline:mmhotline, 0600 |
systemd, from the fleet view socket unit | MMHotline, to get the fleet view for the dashboard |
The files under /run disappear at a restart of the server; systemd and
MMHotline make them again.
Files the triage agent preparation adds
A person or the team's deploy tool adds these once, as
Prepare the servers and the systems repository
says. Install neither creates nor removes them.
sudo mmhotline triage-agent check-preparation checks eight of them,
prints Present or Missing for each, and names the step that fixes a
missing one.
On the MMHotline server:
| Path | Owner and mode | Holds | Read by | Step |
|---|---|---|---|---|
/etc/hosts lines |
root |
Each inventory server's name with its private address, such as 10.0.0.11 web1 |
The logging proxy, which looks up the names the agent connects to | 1 |
/var/lib/mmhotline-triage-agent/.ssh/ |
mmhotline-triage-agent, 0700 |
The agent's SSH key, id_ed25519 and id_ed25519.pub |
The agent's ssh, to log in to the servers as triage; the agent's Git, to sign commits |
2 |
/var/lib/mmhotline-triage-agent/.gitconfig |
mmhotline-triage-agent |
The agent's Git name and email, and SSH signing with id_ed25519.pub for every commit |
The agent's Git | 3 |
/var/lib/mmhotline/.ssh/ |
mmhotline, 0700 |
MMHotline's own SSH key, id_ed25519, and known_hosts with the Git host's key |
MMHotline's Git, to fetch and push the systems repository; MMX, to reach the servers as the inventory's ssh_user |
5, 6 |
/var/lib/mmhotline/systems_repository/ |
mmhotline |
MMHotline's clone of the team's systems repository | MMHotline and MMX | 6 |
/var/lib/mmhotline/mmx_vault_passphrase |
mmhotline, 0600 |
The MMX vault's passphrase, when the team's playbooks use a vault | MMX, only when a task needs the vault | 6 |
/usr/local/bin/mmx, /usr/local/bin/mmx-runner-linux-amd64 |
root, 0755 |
MMX and its node runner | MMHotline; MMX copies the runner to each server | 7 |
On each server of each inventory:
| Path | Holds | Step |
|---|---|---|
The Unix user triage, in group systemd-journal, with no sudo or only read-only commands |
The account the agent investigates as | 2 |
/home/triage/.ssh/authorized_keys |
The agent's public key, after from="10.0.0.5" , the MMHotline server's private address |
2 |
The inventory ssh_user's authorized_keys, such as /root/.ssh/authorized_keys |
MMHotline's public key, after the same from= |
5 |
In the systems repository: triage_agent_rules.md at the top, which the
agent reads at the start of every run, and ssh/known_hosts, with each
server's host key under its inventory address (step 4).
Files MMHotline writes in the triage agent's home
MMHotline writes these at the start of each triage agent run and replaces them each time:
Path in /var/lib/mmhotline-triage-agent |
Mode | Holds |
|---|---|---|
.ssh/config |
0600 |
Every host through the logging proxy, as the user triage, with the agent's key, the systems repository's ssh/known_hosts, and no prompts |
.mmhotline/bin/ssh |
0700 |
The agent's ssh, first on its search path, which runs mmhotline triage-agent ssh |
.mmhotline/bash_env |
0600 |
Puts that ssh first again in a bash login shell |
systems_repository/ |
The agent's own copy of the systems repository, at the commit MMHotline hands the run; the agent's new commits go back to MMHotline from here | |
.codex/ |
Codex's own files, with Codex as the coding agent |
Each run also makes /tmp/mmhotline-triage-agent-requests-*/request.sock,
mode 0600, in its private /tmp, where the agent sends its requests,
such as a playbook run. systemd discards it with the run.
This page is docs/reference/installed-files.md in the MMHotline repository.