email
email sends each alert as one email. MMWatchdog hands the email over SMTP to a
mail server: the program that accepts email and passes it on to the mailboxes
of the recipients, also called an SMTP server. MMWatchdog delivers to no mailbox
itself, so the destination needs a mail server that takes email from the
node, such as the mail service of the team or a mail server on the node
itself.
Choose email for people who read their alerts in a mailbox, and for an
address that turns each email into an SMS. For a chat, use an official extra
module such as slack; for a
program of the team's own, use webhook. A
destination names its module and states the settings that every destination
takes, as the alert destinations file
describes.
Set up
The destination states how its connection to the mail server is protected,
in smtp_connection_security. The setting is required and has no default,
so a reader sees on the destination how the connection is protected:
smtp_connection_security |
The connection to the mail server |
|---|---|
starttls |
Starts in plain text and then switches to encryption. A mail server that does not offer the switch fails the attempt |
implicit_tls |
Is encrypted from the first byte, as mail servers offer on port 465 |
unencrypted |
Is not encrypted |
starttls, implicit_tls, and unencrypted are also the values that the
smtp, imap, and pop3
checks take as connection_security.
- With
starttlsandimplicit_tls, MMWatchdog checks the certificate and host name with TLS 1.2 or later. The default uses the system's certificate authorities.trusted_certificate_authority_filereplaces those authorities with the PEM certificates in that file.dangerously_skip_server_certificate_check: truedisables certificate and identity checks; encryption remains. - A login needs
login_userandlogin_password_filetogether. MMWatchdog refuseslogin_userwithoutlogin_password_file, and refuseslogin_password_filewithoutlogin_user. The password file holds one line with the password: UTF-8 text of at most 1,024 bytes, without a tab or another control character. A space at the start or at the end of that line belongs to the password; only the line break at the end does not. MMWatchdog reads the file for every delivery, so a new password in the same file needs no reload.sudo mmwatchdog validatereads the file too, and names it when it is missing or holds anything other than one line. - MMWatchdog logs in with the method that mail servers call PLAIN, or with the method LOGIN when the mail server offers only that. These are the two methods that mail servers offer for a name and a password.
unencryptedaccepts a login to any mail server. The password crosses the connection in plain text. Use a login overunencryptedonly when the network itself protects the connection, such as a private WireGuard network.- Secret files follow the credential permission rule.
Place permission settings beside
email, wherealert_severitiesstands. - Each address is written plain and in ASCII, such as
mmwatchdog@example.com, with at most 254 characters. A display name is refused:from: "MMWatchdog <mmwatchdog@example.com>"is a validation error.tolists at least one address and no address twice.
Inputs
dangerously_skip_server_certificate_check(boolean; optional): With true, encrypt the connection without checking the server's certificate or identity. By default, verify the server's certificate.trusted_certificate_authority_file(string; optional): The absolute path of PEM certificates that replace the system certificate authorities for this mail server.smtp_server(string; required): The mail server's host name or address and its port, such as smtp.example.com:587.smtp_connection_security(starttls,implicit_tls, orunencrypted; required): How the connection to the mail server is protected.from(string; required): The address that the email comes from, such as mmwatchdog@example.com.to(list_of_strings; required): The addresses that get the email. All of them get one message.login_user(string; optional): The name that MMWatchdog logs in to the mail server with. It needs login_password_file.login_password_file(string; optional): The absolute path of a file that holds the password of login_user on one line. MMWatchdog reads it for each delivery.
What it sends
The subject is the alert's title with the alert's severity in front, so a person sees in the list of emails how urgent each one is. The text is the alert's body, then an empty line, then three lines that identify the alert.
The check http belongs to the check group myapp, which has
alert_severity: sos. The check fails on the node web-3 at 14:02 UTC, and the alert goes
out at 14:07. Each recipient of the destination on_call_mailbox gets
the following email:
Subject: SOS: web-3: myapp/http failed
expected HTTP 200, received 503
failing since 14:02 UTC
Time of the alert: 2026-09-29 14:07:05 UTC
Incident ID: 9f2c4e6a8b0d1f3a5c7e9b1d3f5a7c9e
Event ID: 4b1d7f3a9c5e2b8d6f0a4c8e2b6d0f4a
Time of the alertis when the alert happened, in UTC. The date of the email itself is the time of the delivery attempt, which is later when MMWatchdog had to try more than once.Incident IDis the same in the failure, each raise, each reminder, and the recovery of one incident. An alert without an incident, such as a notice or a reboot notice, has no such line.Event IDidentifies the individual alert event. The two IDs are theincident_idand theidthatmmwatchdog events --jsonshows.- The event ID is also the email's
Message-ID, in the example<4b1d7f3a9c5e2b8d6f0a4c8e2b6d0f4a@example.com>, with the domain offrom. Delivery is at least once, so an email can arrive twice after a retry, and a mail program can recognize a duplicate email by the event ID in theMessage-IDheader. - The email goes out once, to all addresses in
to, and each recipient sees the others in theToheader. - The email says that a program wrote it (
Auto-Submitted: auto-generated), so a mailbox with an out-of-office reply does not answer it. - The text is UTF-8. A subject with a character outside ASCII is encoded so that no line is longer than a mail server takes.
With send_only_the_alert_title: true,
the email's subject is still the alert's title with the severity in front,
but the email has no text at all, not even the three closing lines. The
send_only_the_alert_title setting is for an address that turns each email
into an SMS.
When a delivery fails
When an attempt counts as delivered. The mail server accepting the email acknowledges delivery. MMWatchdog knows that the mail server took the email; what the mail server does with it afterwards, MMWatchdog cannot see.
time_limit_per_delivery_attempt limits the whole conversation with the
mail server, from connecting to the accepted email. For an email destination
time_limit_per_delivery_attempt is 30s unless the destination states
another time; other destinations have 5s. A mail server may take seconds
to accept an email, for example while it scans the email for spam. If
MMWatchdog gave up while the mail
server was still accepting the email, the mail server would still deliver
the email, MMWatchdog would send it again, and the recipients would get the
same alert once for each attempt.
When an attempt fails. Every failure is a failed attempt, and MMWatchdog tries again as for every destination: after one second, then with the delay doubling up to 256 seconds (delivery). A failure is a connection that does not open, a certificate that MMWatchdog does not trust, a refused login, a refused sender, recipient, or message, the time limit, or a password file that MMWatchdog cannot read.
A refused sender stops the attempt. A refused recipient does not block the accepted recipients: MMWatchdog sends their email and reports every refused address as the attempt's error. A retry sends to all recipients again, so an accepted recipient can get a duplicate with the same event ID. Sending to accepted recipients despite a refused address avoids losing an alert for healthy mailboxes because one address is wrong.
When the mail server answered with a refusal, the error says what the mail
server refused and ends with the mail server's own answer. An error without
a refusal from the mail server, such as a connection failure, names the SMTP
step where the attempt failed. The error is one line, cut after 300 bytes.
mmwatchdog send-notice prints the error, and mmwatchdog status shows the
error in the line of the destination
(a destination that is failing).
The errors below list each one.
Examples
A mail service with a login
# /etc/mmwatchdog/alert_destinations.yml
destinations:
on_call_mailbox:
email:
smtp_server: smtp.example.com:587 # the mail server: its host name and its port
smtp_connection_security: starttls # how the connection to the mail server is protected
from: mmwatchdog@example.com # the sender of every email
to: [oncall@example.com, ops@example.com] # each address gets every alert of on_call_mailbox
login_user: mmwatchdog@example.com # the name that MMWatchdog logs in to the mail server with
login_password_file: /etc/mmwatchdog/secrets/smtp # the file that holds the password of mmwatchdog@example.com
remind_every: 4h
alert_severities: [warn, sos]
For each warn and sos alert, MMWatchdog on the standalone node:
- reads the password from
/etc/mmwatchdog/secrets/smtp; - connects to
smtp.example.comon port 587, switches to encryption, and checks that the certificate is valid forsmtp.example.com; - logs in as
mmwatchdog@example.com, names the sender and the recipientsoncall@example.comandops@example.com, and sends one email.
While an incident is open, oncall@example.com and ops@example.com get a
reminder every four hours.
The mail server on the same machine
# /etc/mmwatchdog/alert_destinations.yml
destinations:
operations_mailbox:
remind_every: never
email:
smtp_server: localhost:25 # the mail server on the MMWatchdog node
smtp_connection_security: unencrypted
from: mmwatchdog@app-1.example.com
to: [operations@example.com]
alert_severities: [warn, sos]
MMWatchdog hands each email to the mail server on the node itself, without
encryption and without a login, because nothing crosses the network. The
mail server on the node passes the email on to operations@example.com.
An address that turns each email into an SMS
# /etc/mmwatchdog/alert_destinations.yml
destinations:
on_call_sms:
remind_every: never
email:
smtp_server: smtp.example.com:587
smtp_connection_security: starttls
from: mmwatchdog@example.com
to: [15551234567@sms-gateway.example.com] # a gateway that turns each email into an SMS
login_user: mmwatchdog@example.com
login_password_file: /etc/mmwatchdog/secrets/smtp
send_only_the_alert_title: true # on_call_sms gets the title and no text
alert_severities: [sos]
For each sos alert, the gateway gets an email with a subject such as
SOS: web-3: myapp/http failed and no text, and sends that line as an SMS.
Test the destination
Quiet verification connects, applies the certificate policy, logs in when
configured, names the sender and every recipient, and sends QUIT before
DATA. It sends no email. A successful verification does not prove final
mailbox delivery; send a notice for that test.
Send a notice to the new destination
alone, on the standalone node or on the MMWatchdog fleet hub that holds the
destination. In the following test, one address in to is wrong:
$ sudo mmwatchdog send-notice --alert-severity info --title "Test of the on-call mailbox. Nothing is wrong." --destination on_call_mailbox
Notice sent to the destination on_call_mailbox:
on_call_mailbox failed: the mail server smtp.example.com:587 refused the recipient ops@example.com: 550 5.1.1 User unknown
mmwatchdog: the notice was not delivered at the first attempt to 1 of 1 destinations. MMWatchdog keeps the notice queued and tries again
Correct the address and run sudo mmwatchdog reload. A change of the recipients
makes the destination a new one, so the reload cancels the waiting notice
(delivery); send the
notice again. When the line says delivered, the mail server has taken the
email. Look in each mailbox for it.
How it works
- One conversation for each delivery attempt. MMWatchdog connects to
smtp_server, starts the encrypted connection at once forimplicit_tls, waits for the mail server's greeting, and greets it with the node's host name, which some mail servers require in place oflocalhost. Withstarttls, MMWatchdog switches to encryption. With a login, MMWatchdog logs in. Then MMWatchdog names the sender and each recipient, sends the email, and says goodbye. The email is delivered when the mail server has accepted it, whatever happens after that. - MMWatchdog names the sender without asking for anything more. Every line of the email is ASCII or encoded, so it needs no feature that only some mail servers have, and a mail server that passes it on to another one can always deliver it.
- MMWatchdog reads at most 1 MiB from a mail server in one delivery attempt. No mail server needs more to take one email.
- What makes the destination a new one is its mail server, its list of
recipients, and the path of its password file. The order of the
recipients does not matter, and a change of
from,login_user, orsmtp_connection_securitykeeps the alerts that wait. A reload that changes the identity cancels the alerts that wait for the destination (delivery). - In a fleet, the email destination is in the hub's
alert_destinations.yml, and the hub sends the emails. A joined node sends an email itself only when it delivers an alert without the hub (alerts in a fleet). The node then connects to the mail server insmtp_serverfrom its own place in the network, with the password from its copy of the fleet settings:- The mail server must be reachable from every node, and accept the login from every node.
smtp_server: localhost:25means the mail server on the node that sends the email: the hub's own for an alert that goes through the hub, and each joined node's own for an alert that the node delivers without the hub. A joined node without a mail server of its own fails the attempt to deliver throughlocalhost:25and tries again. Withunencryptedand a login, a joined node sends the password to the mail server on the joined node's machine.- The hub reads the password file when it starts and reloads, and gives the password to the joined nodes with its settings. After a change of the password, reload the hub.
Errors and what they mean
trusted_certificate_authority_file must be absolute. Certificate settings
have no use with smtp_connection_security: unencrypted: remove them or
choose starttls or implicit_tls.
The validation errors, from mmwatchdog validate, mmwatchdog reload, and a deploy. MMWatchdog
writes the file and the line of the setting in front of each one:
| Error | What to do |
|---|---|
destinations.on_call_mailbox.email.smtp_server: expected the mail server's host name and port, such as smtp.example.com:587 |
Write the port too: it has no default |
destinations.on_call_mailbox.email.smtp_connection_security: required: starttls, implicit_tls, or unencrypted |
State how the connection is protected |
destinations.on_call_mailbox.email.from: expected one email address in ASCII, such as mmwatchdog@example.com |
Write the address alone, without a display name such as MMWatchdog <...>. The error expected one email address in ASCII, such as mmwatchdog@example.com at to.0, to.1, and so on names an address in to |
destinations.on_call_mailbox.email.to: expected a list of at least one email address, such as [oncall@example.com] |
List at least one address |
destinations.on_call_mailbox.email.to.1: ops@example.com is listed twice |
Remove the second ops@example.com entry from to |
destinations.on_call_mailbox.email.login_password_file: required with login_user: the file that holds the password of mmwatchdog@example.com |
Set login_user and login_password_file together, or omit login_user and login_password_file |
destinations.on_call_mailbox.email.login_user: required with login_password_file: the name that MMWatchdog logs in to the mail server with |
Set login_user and login_password_file together, or omit login_user and login_password_file |
destinations.on_call_mailbox.email.login_user: expected one line of text of at most 256 bytes |
Shorten the login name, or remove control characters from the login name |
destinations.on_call_mailbox.email.login_password_file: must be absolute |
Write the whole path, starting with / |
The errors of a delivery attempt, from mmwatchdog send-notice and mmwatchdog status:
| Error | What happened |
|---|---|
the conversation with the mail server smtp.example.com:587 failed while MMWatchdog was connecting: ... |
MMWatchdog could not connect: nothing listens at smtp.example.com:587, the name does not resolve, or a firewall is in the way |
the mail server smtp.example.com:587 does not offer STARTTLS, which smtp_connection_security: starttls requires |
The mail server has no switch to encryption on port 587 at smtp.example.com. MMWatchdog does not go on without encryption |
the conversation with the mail server smtp.example.com:587 failed while MMWatchdog was switching to encryption: tls: failed to verify certificate: ... |
MMWatchdog does not trust the mail server's certificate, or the certificate names another host. With implicit_tls the step reads starting the encrypted connection |
the mail server smtp.example.com:587 offers no login on this connection, and the destination states login_user and login_password_file |
The mail server offers no login on the SMTP connection MMWatchdog opened to smtp.example.com:587. Some mail servers offer a login only on an encrypted connection |
the mail server smtp.example.com:587 offers no login that MMWatchdog can use: MMWatchdog uses PLAIN or LOGIN, and the mail server offers CRAM-MD5, GSSAPI |
The mail server offers only other login methods |
the mail server smtp.example.com:587 refused the login of mmwatchdog@example.com: 535 5.7.8 Authentication failed |
The name or the password is wrong |
the mail server smtp.example.com:587 refused the sender mmwatchdog@example.com: ... |
The mail server does not accept the address in from as a sender |
the mail server smtp.example.com:587 refused the recipient ops@example.com: ... |
The mail server does not take email for ops@example.com |
the mail server smtp.example.com:587 refused the message: ... |
The mail server took the addresses and then refused the email itself |
the mail server smtp.example.com:587 did not finish within 30s (time_limit_per_delivery_attempt); MMWatchdog was logging in |
The time limit passed. The last words say which step MMWatchdog was at: connecting, starting the encrypted connection, waiting for the mail server's greeting, greeting the mail server, switching to encryption, logging in, naming the sender and the recipients, or sending the message |
/etc/mmwatchdog/secrets/smtp: expected one line with the password: UTF-8 text of at most 1,024 bytes, without a tab or another control character |
The password file is empty, has more than one line, is too long, or holds a character that a password cannot have. mmwatchdog validate also reports the displayed password-format error |
No error holds the password.
Tested on
- Go tests, on macOS and natively on
every target distribution,
against a mail server inside the test that speaks SMTP and records what MMWatchdog
sends (
internal/daemon/email_test.go):TestEmailSenderContract(one message to every recipient overstarttls,implicit_tls, andunencrypted; a PLAIN login, a LOGIN login, and no login; the subject, the text, theMessage-ID, andAuto-Submitted),TestEmailMessageText(no text withsend_only_the_alert_title, a subject outside ASCII, no line over 78 bytes, a notice without an incident line),TestEmailSenderFailures(each error above, a certificate that MMWatchdog does not trust with both kinds of encryption, a cancelled delivery; nobody gets a message, and the password is in no error),TestReadSMTPPassword,TestEmailSettingsAreValidated,TestValidateRuntimeReadsTheEmailPasswordFile,TestEmailDestinationIdentity,TestTheDaemonDeliversAlertsByEmail,TestAJoinedNodeSendsEmailWithoutTheHub, andTestFleetSettingsCarryEmailPasswords. - VM suites, on every target distribution,
against three loopback-only Mailpit servers in each guest: STARTTLS with
a required login, TLS from the first byte with a required login, and
unencrypted SMTP without a login. The guest trusts their localhost
certificate through its system trust store. The scenarios verify a check's
failure and recovery
over STARTTLS after a login, as one message for both recipients; an email
without text for
send_only_the_alert_title: true;unencryptedtolocalhostwithout a login;mmwatchdog send-noticeoverimplicit_tlsand with a title outside ASCII; a wrong password that fails the first attempt with the mail server's answer, after which the queued notice arrives once the file is right, without a reload; andmmwatchdog validatenaming a missing password file. The fleet suite has a joined node that sends the email itself while the hub is down. - Not verified: internet delivery and spam filtering. These cases are covered by the Go tests only: a refused sender, recipient, or message, a mail server without STARTTLS or without a login that MMWatchdog uses, a certificate that MMWatchdog does not trust, the time limit, reminders, and a reload that changes the destination's identity (validation record).
Why it works this way
-
Permission refusals belong to validation and reload. Startup warns once per file and continues. Deliveries and check-ins keep reading secrets after a mode change, because refusing then would drop alerts. The credential permission rule owns the settings and the full behavior.
-
Every email destination states
smtp_connection_security. The reader sees on the destination how the connection is protected. A mail server on the MMWatchdog node needs no encryption, sounencryptedstays possible. -
unencryptedwith a password is allowed to any host. The explicit connection setting lets operators use a protected network without adding TLS. -
Certificate checks stay enabled by default. A private authority can replace the system store; skipping identity checks requires
dangerously_skip_server_certificate_check: true, whose name makes the danger explicit, so the trust decision remains readable. -
Quiet verification stops before
DATA. MMWatchdog logs in, names the sender and every recipient, and sendsQUIT. The server checks the addresses without receiving a message to deliver. -
The time limit is
30s, not5s. An attempt that gives up while the mail server still accepts the email sends a second copy at every retry. -
Accepted recipients receive the message even when another address fails. A duplicate on retry is preferable to an alert lost for every recipient.
-
MMWatchdog logs in with PLAIN or LOGIN, written in MMWatchdog itself. These are the two methods that mail servers offer for a name and a password, and Go's own PLAIN method refuses a connection that is encrypted from the first byte.
-
MMWatchdog names the sender without parameters. Go's mail client asks for SMTPUTF8, the feature for addresses outside ASCII, whenever a mail server offers it. A mail server that must then pass the email on to one without SMTPUTF8 accepts the email and cannot deliver the email. MMWatchdog's addresses are ASCII, so MMWatchdog needs no SMTPUTF8.
-
Plain addresses in ASCII, without a display name. MMWatchdog writes each address as it is into the SMTP commands and into the headers.
-
The event ID is the
Message-ID, and the IDs are in the text. Delivery is at least once, and a person or a mail program can recognize a repeated email. -
The text ends with the time of the alert. The date of the email is the time of the attempt, which a retry makes later.
-
No text at all with
send_only_the_alert_title. An SMS gateway sends the subject alone, and the closing lines would only cost it characters. -
The password lives only in a file, read for every delivery. It stays out of the configuration, and a new password needs no reload.
This page is docs/reference/modules/email.md in the MMWatchdog repository.