webhook
webhook posts each alert to a web address that you name, as MMWatchdog's own alert
event in JSON over HTTP or HTTPS. A receiver at the configured web address
reads the event and decides how to act. The receiver can be a small
program of the team's own or a relay to another service.
Choose webhook for a receiver that reads MMWatchdog's event as it is. A service
with a format of its own, such as Slack or Telegram, takes a
destination module that speaks that
format, such as the official extra module
slack. A destination names
its module and states the settings that every destination takes, as the alert destinations file describes.
Set up
HTTPS verifies the server certificate and name by default.
trusted_certificate_authority_file replaces the system's authorities with
PEM certificates from trusted_certificate_authority_file. dangerously_skip_server_certificate_check: true encrypts without checking the certificate or identity.
The token file holds one token of 1 to 4,096 bytes on one line. MMWatchdog sends the token to another service, which decides what token is valid; MMWatchdog imposes no minimum beyond one byte. Secret files follow the credential permission rule.
Place permission settings beside webhook, where alert_severities stands.
Inputs
dangerously_skip_server_certificate_check(boolean; optional): With true, encrypt the connection without checking the server's certificate or identity. By default, verify the server's certificate.trusted_certificate_authority_file(string; optional): The absolute path of PEM certificates that replace the system certificate authorities for this webhook.url(string; required): The address that takes the alerts, with http:// or https://, without a user name or a password in it.bearer_token_file(string; optional): The absolute path of a file that holds a bearer token. MMWatchdog reads it for each delivery and sends it in the Authorization header.
What it sends
For each alert, MMWatchdog sends one POST request to url with the alert event as
its JSON body:
Content-Type: application/json.Idempotency-Key, which holds the event ID, matching the body'sid. Each retry preserves the original POST's event ID in the header and body, so the receiver can recognize an alert already received.Authorization: Bearer TOKENwhen the destination statesbearer_token_file. Withoutbearer_token_file, the request has noAuthorizationheader.
The request body contains the whole alert event. The table lists the event's fields. The alert events section of the design explains the event fields:
| Field | Field contents |
|---|---|
id, incident_id |
The event ID, and the incident that the event belongs to. A notice and a reboot notice have no incident |
fleet_incident_key |
On every alert about an incident: 32 lowercase hexadecimal digits, the same for one check on every node of a fleet and on a standalone node, so a receiver can take the alerts of one check on many nodes for one incident (one incident per check across the fleet). A notice, a reboot notice, and a maintenance alert have none |
node, check_group, check |
The node's name, and the check group and the check that the alert is about |
finding |
On an alert about one failed unit of a every_systemd_unit check, that unit, such as backup.service |
type |
The change: failed, recovered, cannot_check, can_check_again, failure_alert_severity_raised, cannot_check_alert_severity_raised, closed_without_recovery, maintenance_on, maintenance_off, rebooted, reminder, or notice |
alert_severity |
info, warn, or sos. A recovery or a retirement carries the highest severity that the incident reached |
alert_severities_sent_for_incident |
On a recovery or a retirement, every severity that the incident's alerts went out with |
at |
When the change happened |
title, title_was_shortened, title_with_alert_severity, body, message |
The alert's title and body, and the readable explanation of the observation |
evidence |
On an incident's alerts, the observation that opened the incident (evidence) |
no_alerts_or_recovery_actions_for_these_checks_while_this_check_fails, number_of_checks_with_no_alerts_or_recovery_actions_while_this_check_fails |
On a failure, its raise, and its reminders: at most 20 checks that MMWatchdog keeps quiet while the alert's check fails, and the total number of quiet checks (one cause, one alert) |
check_groups, maintenance |
On a maintenance alert: the check groups of the maintenance command, or every_check_group for the whole node, and the window's until, reason, and user. When the command names one check group, check_group holds the check group's name too. A maintenance_off alert says whether the window was turned off, expired, or ended because the window's check group was removed |
nodes, node_count |
On a message in which the MMWatchdog fleet hub grouped the alerts of several nodes: up to 20 nodes, and the total number of nodes (alerts in a fleet) |
destination |
On a reminder, the destination that the reminder is for |
send_alerts_only_to_destinations |
The destinations that the alert's check names (which destination gets an alert), or the one destination that mmwatchdog send-notice --destination names |
A field without a value is left out, except check, which is an empty text
on an alert that is about no check. A destination with
send_only_the_alert_title: true
gets an event without body, message, and evidence; title,
title_with_alert_severity, and the fields that identify the alert stay.
A receiver can get a recovery or a retirement whose failure it never got,
after a gap in delivery. Match the alerts of one incident by incident_id.
When a delivery fails
An answer with a status code from 200 to 299 acknowledges the delivery.
Everything else is a failed attempt:
- any status code outside
200to299, such as401or503, which MMWatchdog reports aswebhook returned HTTP 503; - a redirect, such as
301or307: MMWatchdog does not follow it; - no answer within
time_limit_per_delivery_attempt, which is5sunless the destination states another time; - a connection that does not open, or a certificate that MMWatchdog does not trust;
- a token file that MMWatchdog cannot read, or whose content is not one token.
MMWatchdog tries a failed alert again as it does for every destination: after one
second, then with the delay doubling up to 256 seconds, with an unchanged event
ID on each retry. The alerts of one destination go out in order, so the later
alerts wait behind the one that fails, and other destinations do not wait
for the failed alert. The delivery
section of How MMWatchdog works has the rules for every destination, and
mmwatchdog status shows the destination as
a destination that is failing.
Examples
Every alert to a receiver of the team's own
# /etc/mmwatchdog/alert_destinations.yml
destinations:
operations:
webhook:
url: https://alerts.example.com/mmwatchdog
bearer_token_file: /etc/mmwatchdog/secrets/webhook # holds one token; MMWatchdog reads it for each delivery
remind_every: 4h
alert_severities: [info, warn, sos]
The check http in the check group myapp fails on the standalone node
web-3. When the alert is due, MMWatchdog on web-3:
- reads the token from
/etc/mmwatchdog/secrets/webhook; - posts the event to
https://alerts.example.com/mmwatchdog, with the token in theAuthorizationheader and the event ID inIdempotency-Key; - counts the alert as delivered when the receiver answers
204, or tries again after one second when it answers503.
While the incident is open, the receiver gets a reminder every four hours.
When the check recovers, the receiver gets a recovered event whose
incident_id matches the opening failed event's incident_id.
A second receiver for sos alerts only
# /etc/mmwatchdog/alert_destinations.yml
destinations:
operations:
remind_every: never
webhook:
url: https://alerts.example.com/mmwatchdog
bearer_token_file: /etc/mmwatchdog/secrets/webhook
time_limit_per_delivery_attempt: 5s
alert_severities: [info, warn, sos]
pager:
remind_every: never
webhook:
url: https://pager.example.com/mmwatchdog # takes no token
alert_severities: [sos]
Every alert goes to operations. Only sos alerts go to pager, and the
recovery of an incident goes to pager when pager got the incident's
failure alert or an alert reporting the incident's severity increase to
sos. Each destination has its own queue: when pager does not answer,
operations still gets its alerts at once.
A receiver that gets the title only
# /etc/mmwatchdog/alert_destinations.yml
destinations:
on_call_sms:
remind_every: never
webhook:
url: https://sms-relay.example.com/mmwatchdog
send_only_the_alert_title: true # this destination gets the title and no body
alert_severities: [sos]
The relay turns each alert into an SMS. The relay gets each sos alert
without body, message, and evidence, and sends
title_with_alert_severity, such as SOS: web-3: myapp/http failed.
Test the destination
Quiet verification checks only the connection and server certificate policy. MMWatchdog sends no HTTP request, so verification tests neither the URL path nor the bearer token. Send a notice to test whether the receiver accepts a real alert.
Send a notice to the destination alone, on the standalone node or on the MMWatchdog fleet hub that holds the destination.
$ sudo mmwatchdog send-notice --alert-severity info --title "Test of the pager. Nothing is wrong." --destination pager
Notice sent to the destination pager:
pager delivered
The receiver gets an event of the type notice with the title
web-3: Test of the pager. Nothing is wrong. A failed attempt prints the
reason, such as pager failed: webhook returned HTTP 401, and the command
exits 1. The notice stays queued, and MMWatchdog tries again.
How it works
- The token file is read for each delivery. A new token in the configured token file
needs no reload. The file holds one token of 1 to 4,096 bytes on one
line; white space at the token's start and end does not belong to the
token.
sudo mmwatchdog validatereads the token file and names the file when the file is missing or holds anything else. urltakeshttp://orhttps://, and never a user name or a password. Withhttps://, MMWatchdog checks the receiver's certificate against the system trust store of the node. Withhttp://, the event and the token cross the network unencrypted. Usehttp://only for a receiver on the node that sends the POST, or in a network that you trust.- In a fleet, the destination is in the hub's
alert_destinations.yml, and the hub posts the alerts of every node. A joined node posts an alert itself only when the hub did not accept it in time, with the token from its copy of the fleet settings and from its own place in the network (alerts in a fleet). After a new token, reload the hub, so the joined nodes get the new token. - A change of
urlor of the token contents inbearer_token_filemakes the destination a new one: a reload cancels the alerts queued for the destination (delivery).
Errors and what they mean
trusted_certificate_authority_file must be absolute. Certificate settings
have no use with an http:// URL: remove them or choose https://.
| Error | What to do |
|---|---|
destinations.operations.webhook.url: expected HTTP(S) URL without credentials and with a host, such as https://example.com/health |
Write the address with http:// or https://, and put a token in bearer_token_file instead of a user name and a password in the address |
destinations.operations.webhook.bearer_token_file: must be absolute |
Write the whole path, starting with / |
destinations.operations.webhook.bearer_token_file: /etc/mmwatchdog/secrets/webhook: expected a single token of 1..4096 bytes |
From mmwatchdog validate or a delivery attempt: put one token of 1 to 4,096 bytes on one line into the file |
failed: webhook returned HTTP 401 |
The receiver refused the alert. Check the token in the file named by bearer_token_file, and the token expected by the receiver |
failed: MMWatchdog could not post the alert to the webhook: ... |
No answer: the name does not resolve, nothing listens at the address, the certificate is not trusted, or time_limit_per_delivery_attempt passed |
MMWatchdog writes the file and the line of the setting in front of a validation
error, such as /etc/mmwatchdog/alert_destinations.yml:5:7:.
Tested on
- Go tests on macOS and natively on
every target distribution:
TestWebhookSenderContractinnotify_test.go(thePOST, theContent-Type,Idempotency-Key, andAuthorizationheaders,2xxas delivery,503and a redirect as failures, no redirect followed, a cancelled delivery),TestWebhookSlowDestinationDoesNotBlockAnother, and the tests of delivery, identity, and token files instore_test.go,notify_test.go, anddestination_test.go. - VM suites on every target distribution: the acceptance, diagnostics, and fleet suites deliver failures, recoveries, reminders, and notices to a webhook receiver in the guest, and the fleet suite delivers through the hub and directly from a joined node with the token from its copy of the fleet settings.
Why it works this way
-
Permission refusals belong to validation and reload. Startup warns once per file and continues. Deliveries and check-ins keep reading secrets after a mode change, because refusing then would drop alerts. The credential permission rule owns the settings and the full behavior.
-
Quiet verification sends no HTTP request. A generic webhook has no standard request that tests its path or token without triggering work.
-
Certificate checks stay enabled by default. A private authority can replace the system store; skipping identity checks requires an explicit dangerous setting.
-
Sent tokens can be one byte. The receiving service owns the token's format and strength; MMWatchdog only bounds the file size and keeps one line.
-
A generic JSON webhook with a stable event ID and an
Idempotency-Keyheader. Any receiver or relay can read the JSON alert event and use the event ID to recognize a retry. -
Redirects are not followed. The destination is the address that the operator wrote, and a redirect could send the alert and its token elsewhere.
-
The token lives only in a file. The token stays out of the configuration, version control, the address, and every output of MMWatchdog.
-
webhookis a core module.webhookspeaks a general protocol, so the module never needs a newmmwatchdogprogram when a company changes its service.
This page is docs/reference/modules/webhook.md in the MMWatchdog repository.