system_log
system_log writes one line for each alert to the delivering machine's
systemd journal: the alert's title with its severity in front.
An example title is SOS: web-3: myapp/http failed. MMWatchdog writes the
line itself, through the journal's own socket, and needs no other program.
Choose system_log when the machine's log already goes somewhere.
A product that collects the system log of the product's machines carries
the alerts further through the product's log collection and needs no other destination. To reach people directly, use email,
an official extra module such
as slack, or
mmhotline. A destination names its module and
states the settings that every destination takes, as the alert destinations file describes.
The system_log module has no inputs, so the module's value is {}.
Inputs
The module takes no inputs.
What it sends
The system_log destination writes one journal entry for each alert, with
these fields:
- The message is the alert's
title with the alert's severity in front,
such as
SOS: web-3: myapp/http failedorResolved SOS: web-3: myapp/http is ok again. - The identifier is
mmwatchdog-alert, sojournalctl --identifier mmwatchdog-alertshows the alerts and nothing else. The daemon's own messages are in the journal of its unit,mmwatchdog.service. - The priority follows the alert's severity: critical for
sos, warning forwarn, and informational forinfo. An alert that closes an incident, a recovery, a check that MMWatchdog can check again, or a retirement, is informational whatever the severity of the incident was. So a log collector that acts on the priority does not take the end of an outage for a new outage.
The alert's body is not written. mmwatchdog events on the node has the whole
event. Because system_log writes only the title,
send_only_the_alert_title is refused on a system_log destination, as a
validation error.
When a delivery fails
A line that the journal does not accept is a failed attempt, such as when
the journal's socket /run/systemd/journal/socket does not exist or does
not take the line within time_limit_per_delivery_attempt, 5s unless the
destination states another time. The error reads
MMWatchdog could not write the alert to the system log: ..., with the reason. MMWatchdog
tries again as for every destination: after one second, then with the delay
doubling up to 256 seconds
(delivery), and
mmwatchdog status shows the destination as
a destination that is failing.
Examples
Every alert in the system log
# /etc/mmwatchdog/alert_destinations.yml
destinations:
machine_log:
remind_every: never
system_log: {} # one line in the system log for each alert
alert_severities: [info, warn, sos]
The check http in the check group myapp, with alert_severity: sos,
fails on the standalone node web-3. MMWatchdog on web-3 writes the line to the
node's journal, at the priority critical. The journalctl output below shows
the alert entry:
$ journalctl --identifier mmwatchdog-alert
Sep 29 14:07:05 web-3 mmwatchdog-alert[812]: SOS: web-3: myapp/http failed
When the check recovers, MMWatchdog writes
Resolved SOS: web-3: myapp/http is ok again at the priority informational.
journalctl --identifier mmwatchdog-alert --priority warning shows only the lines
of warn and sos alerts.
The system log beside a mailbox
# /etc/mmwatchdog/alert_destinations.yml
destinations:
machine_log:
remind_every: never
system_log: {}
alert_severities: [info, warn, sos]
operations_mailbox:
remind_every: never
email:
smtp_server: localhost:25 # the mail server on this same machine
smtp_connection_security: unencrypted
from: mmwatchdog@app-1.example.com
to: [operations@example.com]
alert_severities: [warn, sos]
Every alert goes to the journal, and the warn and sos alerts also go by
email. Each destination has its own queue, so a mail server that does not
answer never holds up the lines in the journal.
Test the destination
Reload and daily quiet verification open the journal socket through MMWatchdog's shared connection primitive without writing an entry. To test the complete path, send a notice explicitly as shown below.
Send a notice to the destination alone, then look for the notice's line:
$ sudo mmwatchdog send-notice --alert-severity info --title "Test of the system log. Nothing is wrong." --destination machine_log
Notice sent to the destination machine_log:
machine_log delivered
$ journalctl --identifier mmwatchdog-alert --lines 1
Sep 29 14:10:12 web-3 mmwatchdog-alert[812]: Info: web-3: Test of the system log. Nothing is wrong.
How it works
- The line is written on the machine that delivers the alert. A
standalone node writes the line to the standalone node's journal. In a
fleet, the destination is in the hub's
alert_destinations.yml, and the hub writes the alerts of all its joined nodes to the hub's own journal. A joined node writes a line to its own journal only for an alert that it delivers without the hub (alerts in a fleet).
Errors and what they mean
| Error | What to do |
|---|---|
destinations.machine_log.send_only_the_alert_title: a system_log destination takes no send_only_the_alert_title: it writes only the title with the alert's severity in front |
Remove the setting: the line holds only the title already |
destinations.machine_log.system_log: the module system_log takes no inputs and is written system_log: {} |
Write system_log: {}, not system_log: with nothing after it |
destinations.machine_log.system_log.facility: unknown field "facility": the module system_log takes no inputs |
The module has no inputs. Write system_log: {} |
failed: MMWatchdog could not write the alert to the system log: ... |
The journal did not take the line. Check that systemd-journald runs on the machine that delivers the alert |
MMWatchdog prefixes a validation error with the setting's configuration file
and line, such as
/etc/mmwatchdog/alert_destinations.yml:4:5:.
Tested on
- Go tests, on macOS and natively on
every target distribution:
TestSystemLogDestinationininternal/checksandalerts/node_checks_test.go, against a stand-in for the journal's socket: entries forsosandwarnfailures, a reminder, a maintenance alert, and two alerts that close an incident; a journal that takes no entry as a failed attempt; a message of several lines; the settings of the destination and the refusal ofsend_only_the_alert_title; and the destination read back from JSON. - VM suites, on every target distribution: the node scenarios
(
tests/integration/node_scenarios.py) find a failure and its recovery in the guest's real journal, under the identifiermmwatchdog-alert, with the priorities critical and informational.
Why it works this way
-
Quiet verification opens the socket without writing. Reload checks whether the journal can be reached without producing a test alert.
-
Destination health waits ten minutes before notifying other destinations. Temporary failures and delivery retries have time to clear. The shared delivery rules own the fixed delay and notices.
-
The system log is a core destination. Every Linux server has one, and a product that ships MMWatchdog inside the product carries the alerts through its own log collection.
-
One line, the title with the severity in front. A log line is read in a list of many, and the title is complete in one line. The whole event stays in
mmwatchdog events. -
The identifier
mmwatchdog-alertis not the daemon's. A person or a log collector selects the alerts without the daemon's own messages. -
An alert that closes an incident is informational. A log collector that alerts on the priority must not take the end of an outage for a new one.
-
MMWatchdog writes to the journal's socket itself. A node needs no program such as
loggerto deliver an alert.
This page is docs/reference/modules/system_log.md in the MMWatchdog repository.