Write your own check or destination
When to use this page: no module of MMWatchdog checks what you need, such as a number that only your application knows, or no module delivers alerts to a service that your team uses, such as an alarm system of your own. Use this page also when you want alerts in your own words: MMWatchdog has no message templates, and a destination module gets the whole alert and writes what it sends.
Who acts: you, or your coding agent. Each step runs on your controller
machine, the machine where you run mmx, except where the step runs a
command on the node over ssh. A coding agent asks the person to run
mmx secrets edit in step 12 and to type the token themselves, so the token
never passes through the chat.
What this page does: you write queued_jobs, a check module that reads
how many jobs wait in your application's queue. You try the module on the
node, install it with MMX, and watch a check that uses it fail and recover
in Slack. Then you write alarm_system, a destination module that posts
each alert to your team's alarm system with a token, install it, and prove
that a test alert arrives.
Terms:
- A module is the code that does the work of one check or of one alert
destination, such as
httporslack. A check or a destination names exactly one module, and writes the module's inputs below the name. - A module of the project's own is a directory in the project's
mmwatchdog_external_modules/, named after the module. Each deploy installs the directory on the node. - A Starlark script is the usual form of such a module:
module.star, written in Starlark, a small language that reads like Python. - A primitive is a function that MMWatchdog gives the script to read the
node, such as
read_fileorhttp_request. A script reaches the node only through primitives. - The module description is the file
module_description.yml: what the module does, its inputs, its measurements, and examples.
| What you need | Where |
|---|---|
| A check that measures or tests something of your own | Steps, steps 1 to 9 |
| Alerts delivered to a service that no module serves | Steps, steps 10 to 15 |
| A module in Bash, Go, or another language | Writing modules, the part about an executable |
| State kept between runs, one incident for each failed thing, or a destination checked without a message | Writing modules |
| Every primitive, field, and limit | Writing modules |
Before you start
-
MMWatchdog runs on the node, installed with MMX from your project, and alerts reach your Slack channel. If not, follow Install MMWatchdog on one node first. The examples use the node
app01. -
Open a terminal in the project: the directory that holds
mmx_project_settings.yml. -
No module fits the job.
ssh deploy@app01 mmwatchdog describe-modulelists every module with one line each. A test that one shell command answers, such astest -f /var/lib/myapp/ready, needs no module: write ashellcheck, as Add a check says. A module is worth writing when the check takes inputs, measures a number, or serves several checks. -
The example application
myappwrites the number of its waiting jobs into the file/var/lib/myapp/pending. To follow the example without such an application, create the file on the node:ssh deploy@app01 'sudo mkdir -p /var/lib/myapp && echo 12 | sudo tee /var/lib/myapp/pending'
Steps
-
Create the check module's directory:
mkdir -p mmwatchdog_external_modules/queued_jobsReason: the directory's name is the module's name, which a check writes. A name uses lower-case letters, digits, and underscores, starts with a letter, and is not the name of a module that MMWatchdog already has, such as
httporslack.Check:
ls mmwatchdog_external_moduleslistsqueued_jobsand theREADME.mdthat came with the release. -
Write the script. Create
mmwatchdog_external_modules/queued_jobs/module.star:# mmwatchdog_external_modules/queued_jobs/module.star # Reads the number of waiting jobs from a file that the application writes. # The module only measures: the check states the limit. def run_check(inputs, file_contents): queue_file = read_file(inputs["queue_file_path"]) if queue_file.error: return {"status": "unknown", "message": "MMWatchdog cannot read the queue file: " + queue_file.error} waiting = queue_file.content.strip() if not waiting.isdigit(): return {"status": "unknown", "message": "the queue file does not hold a whole number"} return { "status": "ok", "message": waiting + " queued jobs", "measurements": {"queued_jobs": int(waiting)}, }What the script does, at each run of a check that uses it:
- MMWatchdog calls
run_checkwith the check's inputs, such as{"queue_file_path": "/var/lib/myapp/pending"}.file_contentsis empty here; Writing modules says when it is not. read_fileis a primitive. Its result has the fieldcontent, and the fielderror, which is empty when the call worked. A failed call does not stop the script, so the script decides what the failure means.- The script returns a status, a message, and the measurement
queued_jobs.okmeans that the module could measure, andunknownmeans that it could not. The script does not compare the number with a limit: the check states the limit in step 5, and MMWatchdog applies it.
Check: none yet, because MMWatchdog reads a module only with its description. Step 4 runs both.
- MMWatchdog calls
-
Describe the module. Create
mmwatchdog_external_modules/queued_jobs/module_description.yml:# mmwatchdog_external_modules/queued_jobs/module_description.yml summary: Counts the jobs that wait in a queue file. used_as: check inputs: queue_file_path: type: string required: true description: The absolute path of the file that holds the number of waiting jobs. measurements: queued_jobs: unit: count description: The number of jobs that wait. examples: - description: Fail when more than 500 jobs wait. config: | queued_jobs: queue_file_path: /var/lib/myapp/pending fail_when_measurement_above: queued_jobs: 500 alert_severity: warnReason: MMWatchdog checks every check that names the module against this description before anything runs.
used_as: checksays that checks use the module. Each input states itstype, whether it isrequired, and adescription, which MMWatchdog prints when a check leaves out a required input. Each measurement states itsunit.Check: step 4 reads the description.
-
Try the module on the node, before a deploy. The
mmwatchdogprogram in the project is the Linux program for the node, so try the module there. Copy the modules directory to the node, print the module's description, and run the script once:scp -r mmwatchdog_external_modules deploy@app01:/tmp/modules-to-try ssh deploy@app01 mmwatchdog describe-module queued_jobs --external-module-dir /tmp/modules-to-try echo '{"protocol":1,"inputs":{"queue_file_path":"/var/lib/myapp/pending"}}' | ssh deploy@app01 mmwatchdog run-starlark-module /tmp/modules-to-try/queued_jobsqueued_jobs: a module of the project's own (Starlark), used as a check Counts the jobs that wait in a queue file. Files: /tmp/modules-to-try/queued_jobs/ Inputs queue_file_path string required The absolute path of the file that holds the number of waiting jobs. Measurements queued_jobs count The number of jobs that wait. Example: Fail when more than 500 jobs wait. queued_jobs: queue_file_path: /var/lib/myapp/pending fail_when_measurement_above: queued_jobs: 500 alert_severity: warn{"measurements":{"queued_jobs":12},"message":"12 queued jobs","protocol":1,"status":"ok"}run-starlark-modulereads one request on its standard input, with the inputs that a check would give, and prints the script's result. It runs the script asdeploy, the account of yoursshcommand, where a scheduled run uses the accountmmwatchdog.Check:
describe-moduleprints the description, and the result says"status":"ok"with the number in the file. An error names the file and the line to fix; see If something is different. When the module works, remove the copy:ssh deploy@app01 rm -r /tmp/modules-to-try. -
Write the check that uses the module. Create
files/hub_and_standalone/check_groups/myapp_jobs.yml:# files/hub_and_standalone/check_groups/myapp_jobs.yml check_groups: myapp_jobs: check_every: 30s time_limit_per_run: 10s alert_after_failing_for: 1m recovered_after_healthy_for: 30s checks: job_queue_is_short: queued_jobs: # the module, in mmwatchdog_external_modules/queued_jobs/ queue_file_path: /var/lib/myapp/pending # an input of the module fail_when_measurement_above: queued_jobs: 500 # MMWatchdog applies this limit to the measurement alert_severity: warnA check names a module of the project's own exactly as it names a core module: the module's name, with its inputs below. A check with such a module always states an
alert_severity, because the module can answerfailedby itself. Add a check explains the four timings.Check: step 6 checks the check group against the module's description on the node, before it changes anything.
-
Install the module and the check:
mmx run hub_and_standaloneinstall / node / install_monitoring/changes/install_modules: succeeded, changed install / node / install_monitoring/activate/start_maintenance: succeeded, changed — ran because tracked inputs /etc/mmwatchdog/check_groups/myapp_jobs.yml and 2 more changed ... Run c739e4ba: succeededThe run copies
mmwatchdog_external_modules/to the node and checks the whole configuration against it. Then it installs the modules in/usr/local/lib/mmwatchdog/external_modules/and the check group in/etc/mmwatchdog/check_groups/, and restarts MMWatchdog inside a maintenance window. The node then has exactly the modules of the project.Check: the run ends with
succeeded, andinstall_modulessayschanged. If the run fails atvalidate_config, the node keeps its earlier configuration and modules; see If something is different. -
Confirm that the check runs:
ssh deploy@app01 sudo mmwatchdog status myapp_jobsmyapp_jobs/job_queue_is_short ok queued_jobs is 12 (fails above 500); the module reported: 12 queued jobs module queued_jobs: the project's own, Starlark Config: /etc/mmwatchdog/check_groups/myapp_jobs.yml:9:7 Rule: check every 30s, time limit per run 10s, alert after failing for 1m, recovered after healthy for 30s Alert severity: warn Observed: 2026-11-15T10:31:03.405753404Z queued_jobs: 12 count Alert deliveries: 0 waiting, 0 droppedThe first line gives MMWatchdog's comparison with the limit, then the module's own message.
module queued_jobssays where the module comes from, because the check group does not say so.Check: the check shows
okwith the number in the file. A check that showsunknownwithwaiting for a new runhas not run yet: wait 30 seconds and run the command again. -
Make the check fail, and see the alert. Write a number above the limit into the file, and ask for a fresh run:
ssh deploy@app01 'echo 780 | sudo tee /var/lib/myapp/pending' ssh deploy@app01 sudo mmwatchdog check-now myapp_jobsmyapp_jobs/job_queue_is_short failed queued_jobs is 780 (fails above 500); the module reported: 780 queued jobs Observed: 2026-11-15T10:31:42.246729786Z queued_jobs: 780 count mmwatchdog: one or more checks need attentioncheck-nowshows a fresh run and sends no alert. The scheduled runs fail too, and after 1 minute of failing (alert_after_failing_for), Slack shows:Warning: app01: myapp_jobs/job_queue_is_short failed queued_jobs is 780 (fails above 500); the module reported: 780 queued jobs failing since 10:32 UTCCheck: the message arrived, about a minute after the first failed run. If not, run
ssh deploy@app01 sudo mmwatchdog status myapp_jobs: its last line names a destination that fails, and See what is happening says how to read it. -
Recover:
ssh deploy@app01 'echo 12 | sudo tee /var/lib/myapp/pending'After 30 seconds of healthy runs (
recovered_after_healthy_for), Slack shows:Resolved warning: app01: myapp_jobs/job_queue_is_short is ok again queued_jobs is 12 (fails above 500); the module reported: 12 queued jobsCheck: the message arrived. The check module is done. Steps 10 to 15 write a destination module.
-
Write the destination module's script. The example alarm system takes each alert as JSON at
https://alarm.example.com/alerts, and wants a token in theAuthorizationheader. Create the directorymmwatchdog_external_modules/alarm_system/and, in it,module.star:mkdir -p mmwatchdog_external_modules/alarm_system# mmwatchdog_external_modules/alarm_system/module.star # Posts each alert as JSON to the team's alarm system, with a token. def deliver_alert(alert, inputs, file_contents): alert_for_the_alarm_system = { "title": alert["title"], "alert_severity": alert["alert_severity"], "type": alert["type"], } # A notice has no incident, and a notice sent without a body has no body. for name in ["incident_id", "body"]: if name in alert: alert_for_the_alarm_system[name] = alert[name] response = http_request( method = "POST", url = inputs["alarm_system_url"], headers = { "Content-Type": "application/json", "Authorization": "Bearer " + file_contents["api_token_file"], }, body = json.encode(alert_for_the_alarm_system), ) if response.error: return {"status": "failed", "message": "MMWatchdog could not reach the alarm system: " + response.error} if response.http_status_code // 100 != 2: return {"status": "failed", "message": "the alarm system answered HTTP %d" % response.http_status_code} return {"status": "delivered", "message": "the alarm system accepted the alert (HTTP %d)" % response.http_status_code}What the script does, once for each alert that the destination gets:
- MMWatchdog calls
deliver_alertwith the alert, the destination's inputs, andfile_contents. Alert format lists the alert's fields. file_contents["api_token_file"]holds the token. An input whose name ends in_filenames a file, and MMWatchdog reads the file as root and gives the module its text. So the token file can stay readable by root alone, and the module never opens it.- The script returns
deliveredorfailed, with a message. MMWatchdog tries a failed delivery again later, so the module does not retry. response.errornever holds the address of the request, so the message is safe to show. Never put the token into a message.
Check: none yet. Step 11 checks the script with its description.
- MMWatchdog calls
-
Describe the destination module. Create
mmwatchdog_external_modules/alarm_system/module_description.yml:# mmwatchdog_external_modules/alarm_system/module_description.yml summary: Posts each alert as JSON to the team's own alarm system, with a token. used_as: alert_destination inputs: alarm_system_url: type: string required: true description: The address of the alarm system that takes the alerts, such as https://alarm.example.com/alerts. api_token_file: type: string required: true description: The absolute path of the file that holds the alarm system's token. MMWatchdog reads the file for every alert. examples: - description: Send every warning and SOS alert to the alarm system. config: | alarm_system: alarm_system_url: https://alarm.example.com/alerts api_token_file: /etc/mmwatchdog/secrets/alarm-system-token alert_severities: [warn, sos] remind_every: neverused_as: alert_destinationsays that destinations use the module. A destination module has no measurements.Check: copy the modules to the node and print the description, as in step 4:
scp -r mmwatchdog_external_modules deploy@app01:/tmp/modules-to-try ssh deploy@app01 mmwatchdog describe-module alarm_system --external-module-dir /tmp/modules-to-try ssh deploy@app01 rm -r /tmp/modules-to-tryThe first line of the description says
alarm_system: a module of the project's own (Starlark), used as an alert destination. -
Put the token in MMX's vault. The token is a secret, so MMX keeps it in the encrypted file
secrets_vault.ageand writes it to the node during the install. Open the vault:mmx secrets editAdd one line below
mmwatchdog_secret_files:, beside the Slack address, with the alarm system's token, then save and close the editor:mmwatchdog_secret_files: slack-webhook-team: "https://hooks.slack.com/services/YOUR/WEBHOOK/ADDRESS" alarm-system-token: "the token of your alarm system"Then list the new file in
inventories/hub_and_standalone.yml, undervars:vars: mmwatchdog_config_source_dir: files/hub_and_standalone mmwatchdog_secret_files: [slack-webhook-team, alarm-system-token]Each install writes each listed file to
/etc/mmwatchdog/secrets/on the node, readable only by root.Check:
mmx secrets editends withSaved encrypted vault: secrets_vault.age, andmmx validateprintsValid: mmwatchdog. -
Add the destination. In
files/hub_and_standalone/alert_destinations.yml, addteam_alarm_systembelow the Slack destination:# files/hub_and_standalone/alert_destinations.yml destinations: team_slack_channel: slack: webhook_url_file: /etc/mmwatchdog/secrets/slack-webhook-team alert_severities: [info, warn, sos] remind_every: 4h team_alarm_system: alarm_system: # the module, in mmwatchdog_external_modules/alarm_system/ alarm_system_url: https://alarm.example.com/alerts api_token_file: /etc/mmwatchdog/secrets/alarm-system-token # MMWatchdog reads this file and gives the module its content alert_severities: [warn, sos] remind_every: neverteam_alarm_systemgets the warning and SOS alerts of every check, and no reminders. Send alerts where your team sees them explainsalert_severitiesandremind_every.Check:
mmx validateprintsValid: mmwatchdog. Step 14 checks the destination against the module's description. -
Install the destination:
mmx run hub_and_standalonewrite_secret_files / node / secret_files[1]: created "/etc/mmwatchdog/secrets/alarm-system-token" ... install / node / install_monitoring/changes/install_modules: succeeded, changed ... Run 26eced38: succeededCheck: the run ends with
succeeded, and it created/etc/mmwatchdog/secrets/alarm-system-token. -
Prove that an alert reaches the alarm system. Send a test notice to the new destination alone:
ssh deploy@app01 sudo mmwatchdog send-notice --alert-severity info \ --title "Test of the alarm system. Nothing is wrong." --destination team_alarm_systemNotice sent to the destination team_alarm_system: team_alarm_system deliveredThe notice goes through the real delivery path: MMWatchdog reads the token file, runs the module, and prints what the module answered.
--destinationsends the notice to the named destination, whatever itsalert_severitiessay. The alarm system receives this JSON:{"alert_severity":"info","title":"app01: Test of the alarm system. Nothing is wrong.","type":"notice"}Check: the command says
delivered, and the alarm system shows the test notice. From now on, the warning and SOS alerts of every check, such as the one in step 8, reach the alarm system too.
If something is different
| What you see | What it means and what to do |
|---|---|
The run fails at validate_config with mmwatchdog: /usr/local/lib/mmwatchdog/candidate_config/check_groups/myapp_jobs.yml:11:11: check_groups.myapp_jobs.checks.job_queue_is_short.queued_jobs.queue_file: unknown field "queue_file": the inputs of the module queued_jobs are queue_file_path |
The check states an input that the description does not list. The line and column are those of your file in files/hub_and_standalone/check_groups/. Fix the input's name, or add the input to the description, and run mmx run hub_and_standalone again. Nothing changed on the node. |
mmwatchdog: /tmp/modules-to-try/queued_jobs/module_description.yml:7: inputs.queue_file_path: unknown field "requird"; the fields are type, required, allowed_values, description |
The description has a mistake at the line after the file name. Fix the line. |
mmwatchdog: /tmp/modules-to-try/queued_jobs/module.star:6: key "queue_file_path" not in dict from run-starlark-module, with exit code 4 |
The script stopped at line 6: here, the request lacks the input that line 6 reads. In a scheduled run, the same error makes the check unknown, with the message the script of the module queued_jobs ended with an error: and the error. |
the module slack in /tmp/modules-to-try has the name of the official extra module slack, which every deploy installs. A name means one module on every node: give your module another name |
Another module has the name. Rename the directory and every check or destination that names it. |
/tmp/modules-to-try/queue-length is no module: the name of a module uses lower-case letters, digits, and underscores, starts with a letter, and has at most 64 characters, such as queued_jobs |
A directory in mmwatchdog_external_modules/ has a name that no module can have, such as one with a hyphen. Rename the directory. |
the module queued_jobs has no code: ... must hold the script module.star or the executable module_executable |
The module's directory has no script. Check the file name. |
The check shows unknown with MMWatchdog cannot read the queue file: open /var/lib/myapp/pending: permission denied |
The module runs as the account mmwatchdog, which cannot read the file. Make the file readable by that account, or run the module as the application's account: add run_as_user_account: myapp beside queue_file_path. |
The check shows unknown with the module queued_jobs reported the measurement jobs, which its description does not list |
The script reports a measurement under another name than the description. Use the same name in both files. |
team_alarm_system failed: the module alarm_system reported: the alarm system answered HTTP 401 |
The module reached the alarm system, and the alarm system refused the token. Check the token in mmx secrets edit, run mmx run hub_and_standalone, and send the notice again. MMWatchdog keeps a failed alert and tries again. |
| The node is in a fleet | Destinations live on the hub, but a joined node that cannot reach the hub delivers its alerts itself, with its own copy of the module. Put the module in the project once: the hub's and the joined nodes' playbooks install the same modules on every node. |
| You change a node that MMX did not install | Copy the module's directory into /usr/local/lib/mmwatchdog/external_modules/ on the node, as root, then run sudo mmwatchdog validate and sudo mmwatchdog reload. reload reads the modules again, and uses the new configuration only if all of it is valid. |
Related pages
- Writing modules: every primitive, the description's fields, findings, state kept between runs, a destination checked without a message, and a module as an executable with its JSON protocol.
- Add a check
- Send alerts where your team sees them
- Alert format: the fields of an alert that a destination module gets.
- The module pages, one for each module that comes with MMWatchdog.
This page is docs/how-to/write-your-own-check-or-destination.md in the MMWatchdog repository.