Writing modules reference
This page lists what a module of the project's own may use and must do, for
looking up while you write one: the module's name and directory, the fields
of module_description.yml, what a check module returns, the Starlark
language as MMWatchdog runs it, every primitive and what it returns, state
kept between runs, findings, destination modules, the account and the
environment of a module, and the JSON protocol of an executable module with
its Bash helper. To write a first check module and a first destination
module step by step, see
Write your own check or destination.
The modules that come with MMWatchdog have their own pages:
module pages.
A module is the code that does the work of one check or of one alert
destination, such as http or slack. A check or a destination names
exactly one module, and writes the module's inputs below the module's
name. A module has one of three kinds, which say where the module comes
from:
| Kind | What the module is | Where a node has the module |
|---|---|---|
| Core module | Go code built into the mmwatchdog program, such as http |
In mmwatchdog itself |
| Official extra module | A Starlark script that the MMWatchdog project writes, such as slack |
/usr/local/lib/mmwatchdog/official_extra_modules/NAME/, from official_extra_modules/ of the release |
| Module of the project's own | A Starlark script or an executable that you write | /usr/local/lib/mmwatchdog/external_modules/NAME/, from mmwatchdog_external_modules/ of your project |
A check calls a module of every kind in the same way, so the configuration never states the kind. Write a Starlark script unless the module needs something that a script cannot do: see An executable in place of a script.
The module's name
| Rule | Detail |
|---|---|
| Characters | Lower-case letters, digits, and underscores. The first character is a letter. At most 64 characters, such as queued_jobs. A hyphen is not allowed. |
| Taken names | The name of a core module, of an official extra module, or of a setting that stands beside a module, such as alert_severity. A name means one module on every node. |
| In a check or a destination | The module's name as the key, with the inputs below it, as for a core module. |
A directory in mmwatchdog_external_modules/ with a taken name, or with a
name that no module can have, stops mmwatchdog validate and every deploy,
also when nothing names the module:
the module slack in /usr/local/lib/mmwatchdog/external_modules has the name of the official extra module slack, which every deploy installs. A name means one module on every node: give your module another name
/usr/local/lib/mmwatchdog/external_modules/queue-length is no module: the name of a module uses lower-case letters, digits, and underscores, starts with a letter, and has at most 64 characters, such as queued_jobs
Do not edit an official extra module, and do not copy one into
mmwatchdog_external_modules/: the next release brings the module again.
For a module that works differently, write a module of your own under
another name.
The module's directory
| File | Required | What it holds |
|---|---|---|
module_description.yml |
Yes | The module's summary, inputs, measurements, and examples, at most 256 KiB: The module description |
module.star |
One of the code files | A Starlark script of at most 1 MiB: A Starlark script |
module_executable |
One of the code files | Any program that runs on the node, such as a Bash script or a Go program: An executable in place of a script |
README.md |
No | The module's page. mmwatchdog describe-module NAME prints the page after the description. |
- A directory holds
module.starormodule_executable, not both. A directory with neither stops validation:the module queued_jobs has no code: ... must hold the script module.star or the executable module_executable. - A file in
mmwatchdog_external_modules/itself, such as theREADME.mdthat comes with the release, is no module. - Each deploy makes each
module_executablerunnable, with mode 0755, whatever its mode on the controller machine. - Keep
mmwatchdog_external_modules/in the project even without modules of your own. MMX refuses a missing directory before it connects to any node, so a missing directory cannot erase the modules on the nodes.
The module description
Every module has a module description. A module of the project's own keeps
it in module_description.yml, beside the script or the executable:
# mmwatchdog_external_modules/queued_jobs/module_description.yml
summary: Counts the jobs that wait in a queue file.
used_as: check
inputs:
queue_file_path:
type: string
required: true
description: The absolute path of the file that holds the number of waiting jobs.
measurements:
queued_jobs:
unit: count
description: The number of jobs that wait.
examples:
- description: Fail when more than 500 jobs wait.
config: |
queued_jobs:
queue_file_path: /var/lib/myapp/pending
fail_when_measurement_above:
queued_jobs: 500
alert_severity: warn
| Field | Required | Meaning |
|---|---|---|
summary |
Yes | One sentence that says what the module does. mmwatchdog describe-module shows it in its list. |
used_as |
Yes | check, or alert_destination for a destination module. |
inputs |
No | The module's inputs by name, each with the fields below. |
measurements |
No | Only for a check: the measurements that the module reports, by name. Each measurement has a unit of at most 64 bytes, such as count or percent, and a description. A measurement's name follows the rule for a module's name. |
one_incident_for_each_finding |
No | Only for a check: true for a module that reports findings. |
service_keeps_incidents |
No | Only for a destination: true for a module whose service keeps incidents of its own, such as a pager: A service that keeps incidents. |
examples |
Yes, at least one | Each example has a description and a config: the YAML of one check or one destination that uses the module. |
| Input field | Required | Meaning |
|---|---|---|
type |
Yes | One of the types below. |
required |
Yes | true when every check or destination must state the input, false when it may leave the input out. |
description |
Yes | What the input holds, with an example value. mmwatchdog validate ends its error about a missing required input with this text, so write it for the person who reads that error. |
allowed_values |
No | A list of the values the input may have. Only for the types string and integer. |
| Type | A value of the type |
|---|---|
string |
Text, such as /var/lib/myapp/pending. |
integer |
A whole number. |
number |
Any number. |
boolean |
true or false. |
duration |
A time written as MMWatchdog's own settings write one, such as 30s, 5m, 2h, or 7d. MMWatchdog checks the form, and the module gets the text as written. |
list_of_strings |
A list of texts. |
- An input has no default value. A script reads an input that may be left
out with
inputs.get("NAME", DEFAULT). - The name of an input follows the rule for a module's name. No input can
be named
run_as_user_account,run_from_directory, orenv, because these run settings stand beside the inputs. - An unknown field, or a key written twice, is an error at its line, such
as
module_description.yml:7: inputs.queue_file_path: unknown field "requird"; the fields are type, required, allowed_values, description. A description with an error makes the module unusable. - MMWatchdog checks each example against the description when it reads the description.
- MMWatchdog reads the description of each module that the configuration names, when MMWatchdog starts and at each reload. A module that nothing names is never read and never run.
mmwatchdog validate checks every check and every destination against
the description of its module, before anything runs. Each error starts
with the file and the line of the check:
| The check | mmwatchdog validate answers |
|---|---|
| States an input that the module does not have | check_groups.myapp_jobs.checks.job_queue_is_short.queued_jobs.queue_file: unknown field "queue_file": the inputs of the module queued_jobs are queue_file_path |
| Leaves out a required input | check_groups.myapp_jobs.checks.job_queue_is_short.queued_jobs.queue_file_path: is required: The absolute path of the file that holds the number of waiting jobs. |
| States a value of the wrong type | ...queued_jobs.queue_file_path: expected a string |
States a value that allowed_values does not list |
...: expected one of and the allowed values |
| States a limit on a measurement that the module does not report | check_groups.myapp_jobs.checks.job_queue_is_short.fail_when_measurement_above.jobs: this check has no such measurement; it measures queued_jobs |
| Names a module that is used as an alert destination | check_groups.myapp_jobs.checks.job_queue_is_short.slack: the module slack is used as an alert destination, not as a check: its description says used_as: alert_destination |
| Names a module that no directory holds | unknown field "queued_jobs": the field name matches neither a setting of a check nor an installed module. and where MMWatchdog looked |
On the node, mmwatchdog validate also reads each script that the
configuration names, without running it. A script that is not valid
Starlark, that uses a name that nothing defines, that loads another file,
or that does not define the function that MMWatchdog calls with its
arguments is an error with the file and the line. So such a script stops
a deploy. mmwatchdog validate --config-only leaves out the checks that
need the node, such as this one.
A check module's result
A check module's script defines run_check(inputs, file_contents), and
MMWatchdog calls it once for each run of a check that uses the module.
inputs is a dictionary of the check's inputs. file_contents holds the
text of each file that an input names: see
Inputs that name a file. The function returns a
dictionary:
| Key | Meaning |
|---|---|
status |
Required. ok, failed, or unknown. |
message |
Required. One text that says what the module found, such as 12 queued jobs. |
measurements |
Optional. A dictionary from a measurement's name to its number, such as {"queued_jobs": 780}. At most 128. |
findings |
Only for a module that reports findings. |
The module measures, and MMWatchdog does the rest. MMWatchdog owns the check's schedule, limits, waiting times, incidents, recovery actions, and alerts.
statusisokwhen the module could measure,failedwhen the thing that the module checks does not work at all, andunknownwhen the module could not measure.- The limit belongs to the check. A check states a limit with
fail_when_measurement_aboveorfail_when_measurement_below. MMWatchdog applies the limits to anokresult: a reading strictly above itsabovenumber, or strictly below itsbelownumber, fails the observation, with a message such asqueued_jobs is 780 (fails above 500); the module reported: 780 queued jobs. - A
failedorunknownresult keeps its status. MMWatchdog applies no limit to it. - A limit needs its measurement. An
okresult without a measurement that the check has a limit on makes the observationunknown, with the messagethe result has no measurement named queued_jobs, which this check has a limit on. A module that cannot measure at one run answersunknown. - A measurement is a plain, finite number. The description states its
unit. A measurement that the description does not list makes the
observation
unknown:the module queued_jobs reported the measurement jobs, which its description does not list. - Keep the name and the unit of a measurement. MMWatchdog stores the readings of every scheduled run in the node's measurement history, under the name that the module reported. A measurement with a new name is another measurement in the history.
- Every check that uses your module states an
alert_severity, also a check without a limit, because the module can answerfailedby itself. - A result that breaks a rule makes the observation
unknown, with a message that names the module, such asthe module queued_jobs wrote a result that is not valid: a message is required.
A Starlark script
MMWatchdog runs a script in a process of its own, once for each run of a
check or each delivery attempt: mmwatchdog run-starlark-module with the
module's directory, as the module's account. That process reads the
request of the JSON protocol and calls the script's
function. So a script and an executable run in the same way.
The Starlark language
Starlark reads like Python: def, if, for, lists, dictionaries, and
strings with methods such as strip() and split(). The
language specification
lists everything. These are the differences from Python that a module
writer meets:
- A script has no
whileloop and no function that calls itself, so every script comes to an end. - A script is one file. It cannot load another file and has no
import. - An
ifor aforcannot stand outside a function. Outside a function, a script defines its functions and gives names to values, such asMOST_CHARACTERS_OF_A_MESSAGE = 4000. - Starlark has no classes, no exceptions, and no f-strings. Format a text
with
%, as in"%d queued jobs" % queued_jobs. - A script reaches the node only through the primitives. It cannot open a file, start a program, or use the network by itself.
- A script also has
json.encode,json.decode, andjson.indent: json. - A value that the script returns, or stores with
write_state, isNone,True,False, a number, a text, a list, a tuple, or a dictionary with text keys. A whole number must fit in 64 bits. print(...)writes a line to the standard error of the script's process.sudo mmwatchdog check-now CHECK_GROUPshows the printed lines below the check, soprintis how you look at a value while you write a script. MMWatchdog passes on the first 8 KiB that one run prints.
The limits of a run
| Limit | What happens |
|---|---|
The check's time_limit_per_run, or a destination's time_limit_per_delivery_attempt |
Each call of a primitive that waits ends shortly before the time limit, with an error, so the script still decides the result: see Primitives. A run that takes longer all the same is ended, and the observation is unknown: the module queued_jobs did not finish within 10s (time_limit_per_run), so MMWatchdog stopped the module. |
| 10 million steps | The run ends. A check needs far fewer, and the limit keeps a script that counts without end from keeping a processor busy until the time limit. |
module.star at most 1 MiB |
MMWatchdog does not read a larger script. |
A script that ends with an error makes the observation unknown. The
message names the file and the line, such as
the script of the module queued_jobs ended with an error: /usr/local/lib/mmwatchdog/external_modules/queued_jobs/module.star:6: key "queue_file_path" not in dict
for a script that reads an input that the check does not state.
Check input values before a run
A script may define validate_inputs(inputs), for rules that the
description cannot state. After MMWatchdog has checked the inputs against
the description, it calls validate_inputs during mmwatchdog validate,
mmwatchdog validate --config-only, and reload. The function returns
None for valid inputs, or a text that says what to change:
def validate_inputs(inputs):
if not inputs["queue_file_path"].startswith("/"):
return "queue_file_path: use an absolute path, such as /var/lib/myapp/pending"
return None
- The text becomes a validation error at the configuration file and line. A text that starts with the name of an input and a colon points at that input. Any other text points at the module's mapping.
validate_inputschecks values only. A call of a primitive inside it ends with the errorread_file cannot be called during input validation; validate_inputs checks input values only. So validation also works on a machine that is not the node.jsonworks.- Only a script has
validate_inputs. An executable checks its inputs at each run.
Primitives
The primitives are the functions through which a script reads the node. The core modules do their work through the same functions.
| Primitive | What it does |
|---|---|
read_file(path) |
Reads a text file. |
run_command(arguments, standard_input = "") |
Runs a program with its arguments, without a shell. |
http_request(method, url, headers = {}, body = "", trusted_certificate_authority_file = "") |
Sends one HTTP or HTTPS request. |
verify_connection(url) |
Opens and closes a connection to an HTTP or HTTPS address, and checks an HTTPS certificate, without a request. |
tcp_exchange(address, text_to_send = "", read_until_text = "") |
Opens a TCP connection, and can send a text and read the answer. |
udp_exchange(address, text_to_send) |
Sends a text in one UDP packet and reads the packet that comes back. |
read_state() and write_state(value) |
Keep one value from one run of a check to the next. |
- Each primitive returns a result with named fields, such as
queue_file.content.read_stateandwrite_stateare the exception. - The field
errorsays whether the call worked. It is empty text when the call worked, and otherwise says what went wrong. A failed call does not stop the script, so the script decides what the failure means: a missing file can make one checkfailedand anotherunknown. - A call written wrong ends the script, such as a call with a number where the primitive takes a text.
- A call that waits ends shortly before the time limit. A call of
http_request,verify_connection,tcp_exchange, orudp_exchangewithout an answer by then ends with theerrorMMWatchdog ended the call shortly before the time limit of the run (time_limit_per_run). A program ofrun_commandthat has not ended by then is ended, with theerrorMMWatchdog ended the program shortly before the time limit of the run (time_limit_per_run). "Shortly before" is a tenth of the time limit, and at most one second: after 9 seconds of a 10-second limit, and after 1.8 seconds of a 2-second limit. After that moment, every further call that waits ends at once with the sameerror, so a script that tries several servers in a loop still returns its result in time. In a destination module, the error namesthe time limit of the delivery attempt (time_limit_per_delivery_attempt). The script decides whether the timeout is a failure, so a check can fail for a server that hangs, and its recovery action can restart the server. - Every primitive acts as the module's account, with that account's rights and no more.
read_file
read_file(path) reads the file at path, which holds at most 1 MiB.
| Field | Meaning |
|---|---|
content |
The content of the file. |
error |
Empty, or why MMWatchdog could not read the file, such as open /var/lib/myapp/pending: no such file or directory. |
queue_file = read_file("/var/lib/myapp/pending")
if queue_file.error:
return {"status": "unknown", "message": "MMWatchdog cannot read the queue file: " + queue_file.error}
waiting = queue_file.content.strip()
run_command
run_command(arguments, standard_input = "") runs a program and waits for
its end. arguments is a list: the program and each of its arguments.
MMWatchdog runs no shell, so a |, a >, or a * in an argument has no
special meaning. With standard_input, the program gets that text on its
standard input.
| Field | Meaning |
|---|---|
exit_code |
The program's exit code, or -1 for a program that MMWatchdog could not start. |
standard_output |
What the program wrote to its standard output, at most 1 MiB. |
standard_error |
What the program wrote to its standard error, at most 16 KiB. |
error |
Empty for a program that ran to its end, whatever its exit code. Otherwise why the program did not finish, such as exec: "myapp-cli": executable file not found in $PATH. |
ran = run_command(["myapp-cli", "queue-length"])
if ran.error:
return {"status": "unknown", "message": "MMWatchdog could not run myapp-cli: " + ran.error}
if ran.exit_code != 0:
return {"status": "unknown", "message": "myapp-cli ended with exit code %d: %s" % (ran.exit_code, ran.standard_error.strip())}
waiting = ran.standard_output.strip()
- An exit code other than 0 is no error of the call: the script reads
exit_codeand decides. - These are errors: a program that writes more than 1 MiB to its standard output, that a signal ends, or that has not ended shortly before the time limit.
- The program runs as the script's account, in its directory, and with its environment.
- When MMWatchdog ends the script at the time limit, it ends every program that the script started too, so no program of a run stays behind.
http_request
http_request(method, url, headers = {}, body = "", trusted_certificate_authority_file = "")
sends one HTTP or HTTPS request and reads the answer.
| Field | Meaning |
|---|---|
http_status_code |
The answer's status code, such as 200, or 0 when no answer came. |
headers |
The answer's headers, as a dictionary, such as {"Content-Type": "application/json"}. |
body |
The answer's body, at most 1 MiB. |
error |
Empty when an answer came, whatever its status code. Otherwise why none came, such as dial tcp 127.0.0.1:8080: connect: connection refused. |
response = http_request(method = "GET", url = "http://127.0.0.1:8080/queue")
if response.error:
return {"status": "failed", "message": "myapp did not answer: " + response.error}
if response.http_status_code != 200:
return {"status": "failed", "message": "myapp answered HTTP %d" % response.http_status_code}
queue = json.decode(response.body, default = None)
- A status code is no error. An answer with the status code 503 has an
empty
error. The script readshttp_status_codeand decides. - A body larger than 1 MiB is an error. The result then has the status
code and the headers, and an empty
body. - Only the time limit of the run limits the request. There is no separate time limit to connect or for the TLS handshake.
- A redirect is not followed. The script gets the answer with the status
code, such as 302, and its
Locationheader. - An HTTPS certificate is always verified, with the certificate
authorities that the node trusts.
trusted_certificate_authority_filenames a file of PEM certificates, of at most 1 MiB, that MMWatchdog trusts for this call in place of the node's. The module's account reads that file. - A
Hostheader selects the virtual host, and for HTTPS also the name that MMWatchdog verifies in the certificate. - The
errornever holds the address of the request, because an address can hold a secret, such as the token in a chat webhook's address. Theerrorcan name the host, never the path behind it.
verify_connection
verify_connection(url) opens and closes a TCP connection to an http
address, or a TLS connection to an https address. For HTTPS, it checks
the certificate against the node's trusted certificate authorities and the
address's host name. It sends no HTTP request, and ignores the address's
path and query. Its result has only error: empty when the connection and
the certificate passed, and otherwise
the connection or server certificate check failed, without the address
or a credential.
Use verify_connection in verify_destination
when the service has no request that checks a credential without sending
a message. A connection that opens does not prove that the credential is
right or that the destination may post, so say that in the verification
message.
tcp_exchange
tcp_exchange(address, text_to_send = "", read_until_text = "") opens a
TCP connection to address, written as host, colon, port, such as
127.0.0.1:9000.
- With neither
text_to_sendnorread_until_text, the call only establishes that the connection opens. - With
text_to_send, MMWatchdog sends that text. - With
read_until_text, MMWatchdog then reads until the answer containsread_until_text, the other side closes the connection, or MMWatchdog has read 64 KiB.
| Field | Meaning |
|---|---|
received |
What MMWatchdog read from the connection. |
error |
Empty, or why the exchange failed, such as dial tcp 127.0.0.1:9000: connect: connection refused. |
answer = tcp_exchange(address = "127.0.0.1:9000", text_to_send = "status\n", read_until_text = "\n")
if answer.error:
return {"status": "failed", "message": "myapp does not answer on port 9000: " + answer.error}
if answer.received.strip() != "ready":
return {"status": "failed", "message": "myapp answered " + answer.received.strip()}
udp_exchange
udp_exchange(address, text_to_send) sends text_to_send in one UDP packet
to address, written as host, colon, port, such as 127.0.0.1:9001, and
waits for one packet in return.
- UDP has no connection: only an answer says that a program listens. So a
call without an answer ends with an
errorshortly before the time limit. - A machine that reports that no program listens on the port ends the call
with an
errorat once. - A host name with several addresses gets a try on each address, each with an equal share of the time that is left.
- MMWatchdog reads at most 64 KiB of the packet that comes back.
| Field | Meaning |
|---|---|
received |
The content of the packet that came back. |
error |
Empty, or why the exchange failed. |
answer = udp_exchange(address = "127.0.0.1:9001", text_to_send = "status\n")
if answer.error:
return {"status": "failed", "message": "myapp does not answer on UDP port 9001: " + answer.error}
json
json.encode(value) writes a value as JSON text, json.decode(text) reads
JSON text into a value, and json.indent(text) writes JSON text with
indentation. An object becomes a dictionary, and a list becomes a list.
queue = json.decode('{"queued_jobs": 780}') # {"queued_jobs": 780}
text = json.encode({"title": "the queue is long"}) # '{"title":"the queue is long"}'
json.decode ends the script when the text is not JSON.
json.decode(text, default = None) returns None then, and the script
decides.
State kept between runs
Some checks must remember something from one run to the next, such as a count at the last run. MMWatchdog keeps one state directory for each check whose module is not core. A script keeps one value there:
write_state(value)stores a value for the next run of the calling check.read_state()returns the value that an earlier run of the calling check stored, orNonewhen no run has stored one.
The value is a number, a text, True, False, None, a list, or a
dictionary with text keys, of at most 1 MiB as JSON.
The example module finished_jobs
reports how many jobs the application finished since the last run. The
application writes into /var/lib/myapp/finished how many jobs it has
finished since it started:
# mmwatchdog_external_modules/finished_jobs/module.star
def run_check(inputs, file_contents):
counter_file = read_file(inputs["finished_jobs_counter_file_path"])
if counter_file.error:
return {"status": "unknown", "message": "MMWatchdog cannot read the counter file: " + counter_file.error}
counter = counter_file.content.strip()
if not counter.isdigit():
return {"status": "unknown", "message": "the counter file does not hold a whole number"}
finished_jobs_now = int(counter)
finished_jobs_at_last_run = read_state()
write_state(finished_jobs_now)
if finished_jobs_at_last_run == None:
return {"status": "unknown", "message": "this is the first run, so there is no earlier count to compare with"}
# The application starts its count at 0 again when it restarts.
finished = finished_jobs_now - finished_jobs_at_last_run
if finished < 0:
finished = finished_jobs_now
return {
"status": "ok",
"message": "%d jobs finished since the last run" % finished,
"measurements": {"jobs_finished_since_last_run": finished},
}
A check with fail_when_measurement_below: {jobs_finished_since_last_run: 1}
and check_every: 10m then fails when no job finished in 10 minutes. With
4,100 finished jobs at one run and 4,130 at the next, the module reports
30. The first run has nothing to compare with and answers unknown.
- The state directory is
/var/lib/mmwatchdog-module-state/CHECK_GROUP/CHECKon the node, such as/var/lib/mmwatchdog-module-state/myapp_jobs/jobs_keep_finishing. It belongs to the module's account, and no other account can read it. The node settingmodule_state_dirnames another directory in place of/var/lib/mmwatchdog-module-state: Configuration files. - Each check has its own state. Two checks that use the same module do not see each other's value.
mmwatchdog check-nowworks on a copy. Its run reads the value of the last scheduled run, and MMWatchdog discards what the run stores. So looking at a check never changes what the check's next scheduled run sees. For an executable, the copy holds the directories and regular files of the state directory that belong to the module's account. A symbolic link is not copied. MMWatchdog copies at most 64 MiB and at most 10,000 files and directories, and with more the run ofcheck-nowisunknownand says so.- MMWatchdog removes the state with the check: when you remove the check, or change it so that MMWatchdog treats it as a new check.
- A destination module keeps no state.
read_stateandwrite_stateend a destination module's script with the errorwrite_state: only a check keeps state between runs.
Findings
Some checks watch many things at once. One run of such a check reports each
thing that is wrong as a finding, and each finding is its own incident,
with its own alert and its own recovery. The core module
every_systemd_unit works this way: each
failed systemd unit is one finding.
A module reports findings when its description states
one_incident_for_each_finding: true. The result then lists the findings
under findings, each with an id and a message. The example module
failed_jobs reads a file
in which the application writes the name of each failed job, one on a
line:
# mmwatchdog_external_modules/failed_jobs/module.star
def run_check(inputs, file_contents):
failed_jobs_file = read_file(inputs["failed_jobs_file_path"])
if failed_jobs_file.error:
return {"status": "unknown", "message": "MMWatchdog cannot read the file of failed jobs: " + failed_jobs_file.error}
findings = []
for line in failed_jobs_file.content.splitlines():
job = line.strip()
if job:
findings.append({"id": job, "message": "the job " + job + " failed"})
if not findings:
return {"status": "ok", "message": "no job failed"}
return {
"status": "failed",
"message": "%d jobs failed" % len(findings),
"findings": findings,
}
With the two lines nightly_report and export_invoices in the file,
MMWatchdog opens one incident for each job. When the application takes
nightly_report out of the file, the nightly_report incident recovers,
and the export_invoices incident stays open.
sudo mmwatchdog check-now CHECK_GROUP shows each finding on a line below
the check, such as Finding nightly_report: the job nightly_report failed,
so you see the id of each incident.
| Field of a finding | Meaning |
|---|---|
id |
Required. The finding's identity: the same thing has the same id at every run. Text on one line, of at most 255 bytes. No two findings of one result have the same id. For a thing whose name can be longer, use a stable checksum as the id and name the thing in the message. |
message |
Required. What is wrong, such as the job nightly_report failed. |
recovery_message |
Optional. The text to use when the finding becomes healthy. MMWatchdog keeps it with the finding's incident, also across restarts, so a finding with a checksum id still names its thing in the recovery. |
- A
failedresult holds at least one finding, and anokorunknownresult holds none. A result holds at most 500 findings. - Each finding follows the check's timings by itself: its incident opens
after
alert_after_failing_for, and recovers when no run has reported it forrecovered_after_healthy_for. - Every finding takes the check's
alert_severity. Two levels need two checks. - An
unknownresult is the check's, not a finding's. An open incident of a finding stays open while the module cannot check. - A result that breaks a rule makes the observation
unknown, with a message such asthe module failed_jobs answered failed without a finding, and its description states one_incident_for_each_finding.
Destination modules
A destination module delivers alerts to one service. An alert
destination names the module, and MMWatchdog runs the module once for each
delivery attempt. The description states used_as: alert_destination,
and the script defines deliver_alert(alert, inputs, file_contents).
Write your own check or destination
builds the example module alarm_system.
The three arguments:
alertis the whole alert event, as awebhookdestination gets it: Alert format lists the fields.titleandtitle_with_alert_severityare always there, and some fields can be missing:body,message, andevidencefor a destination that statessend_only_the_alert_title: true,bodyfor a notice sent without one, andincident_idandfleet_incident_keyfor an alert without an incident, such as a notice. Read a field that can be missing withalert.get("body", ""), or ask with"body" in alert.inputsare the destination's inputs as written.file_contentsholds the text of each file that an input names, under the input's name: Inputs that name a file.
title_with_alert_severity is the title with the severity in front, such
as Warning: app01: myapp_jobs/job_queue_is_short failed. Send it where a
person reads the alert, such as the first line of a chat message. A service
that takes the severity as a field of its own gets title and
alert_severity.
deliver_alert returns a dictionary with a status and a message:
statusisdeliveredorfailed, andmessageis required. Forfailed,messagesays why, and MMWatchdog shows it as the error of the delivery attempt:the module alarm_system reported: the alarm system answered HTTP 503.- Anything else is a failed attempt too: a script that ends with an error,
a run past the destination's
time_limit_per_delivery_attempt, or another result.
Rules for a destination module:
- A module delivers once and reports. MMWatchdog tries a failed attempt again and runs the module anew each time. A module does not retry, wait, or keep a queue, and it keeps no state.
- A credential stays out of the message. Never write the content of a
file into the
message.http_requestkeeps the address out of itserror, soresponse.erroris safe to pass on. MMWatchdog also replaces a whole file content that a message holds with[content of NAME]. - In a fleet, every node needs the module. A joined node that delivers an alert without the hub runs the module itself, with the hub's destination and the node's copy of the credential files. The release's MMX workflows install the same modules on every node.
- Test a destination with
sudo mmwatchdog send-notice --alert-severity info --title "Test of the alarm system. Nothing is wrong." --destination team_alarm_system, which sends a notice through the real delivery path and prints what the module answered: Commands.
When a delivery attempt fails
The error of a failed attempt names the module and is one line. The text
that comes from the module, such as its message, is cut after 300 bytes.
sudo mmwatchdog status and mmwatchdog send-notice show the error.
MMWatchdog replaces the content of each file that an input names, even a
short one, with [content of INPUT], such as
[content of api_token_file].
| Error | What happened |
|---|---|
the module alarm_system reported: TEXT |
The module answered failed. TEXT is its message |
the script of the module alarm_system ended with an error: TEXT |
The script ended with an error, such as a call of fail() |
the module alarm_system ended with exit status 3: TEXT |
An executable ended with another exit status. TEXT is the first line of its standard error |
the module alarm_system did not finish within 5s (time_limit_per_delivery_attempt), so MMWatchdog stopped the module |
The time limit passed. MMWatchdog ended the module and the programs that it started, unless a program left the module's process group |
the module alarm_system is not installed on this node |
This node has no module of that name. In a fleet, every node needs the module |
the module alarm_system on this node is used as a check, not as an alert destination |
This node's module of that name is a check module |
MMWatchdog cannot read the file of the input api_token_file of the module alarm_system: ... |
A file that an input names cannot be read, so MMWatchdog did not start the module |
MMWatchdog cannot use the files of the module alarm_system: ... |
The files that the inputs name are larger than 64 KiB together |
the module alarm_system could not run: ... |
MMWatchdog could not start the module, such as for an account that does not exist on the node |
the module alarm_system was ended by a signal |
Something else ended the module, such as the kernel when memory ran out |
the module alarm_system ended without an answer on its standard output |
An executable ended with exit status 0 and wrote no answer |
the module alarm_system did not answer with one JSON object of protocol, status, and message: ... |
The answer is not one JSON object |
the module alarm_system returned protocol 2; MMWatchdog requires protocol: 1 |
The answer has another protocol |
the module alarm_system answered with the status "ok", and a destination module answers delivered or failed |
The module answered as a check module does |
the module alarm_system answered delivered without a message, and a message is required |
The answer has no message |
the module alarm_system wrote more than 65536 bytes to its standard output |
The answer is too large |
A module that delivered the alert but did not answer delivered, such as
one that ran past the time limit, gets the same alert again at the next
attempt.
Check a destination without a message
A script may define verify_destination(inputs, file_contents), which
checks the address and the credential without sending a message. It
returns a pair: True or False for whether the service accepted the
address and the credential, and a one-line text that says what was
checked, such as (True, "verified address and credential"). A refused
credential returns False with the service's reason. Never send a message
from verify_destination.
- MMWatchdog reads the credential files as for a delivery, so
file_contentsholds them, and runs the function as the destination's account, withintime_limit_per_delivery_attempt. - MMWatchdog calls it for a new or changed destination at reload, and every 24 hours. A failed check is tried again every minute.
mmwatchdog reloadprints the outcome for each new or changed destination, such asDestination team_alarm_system: not tested: MMWatchdog can test the destination only by sending a messagefor a module withoutverify_destination.- Only a script has
verify_destination. MMWatchdog checks an executable destination only by sending a message.
A service that keeps incidents
Some services keep incidents of their own: a pager opens an incident for a
failure and resolves it for the recovery. In a fleet, one check can fail on
many nodes at once, and the service should have one incident for the
check, which ends when the last node has recovered. A module for such a
service states service_keeps_incidents: true in its description.
MMWatchdog then sends a destination with the module:
- every failure, raise, and reminder, as for every destination;
- nothing when one node recovers while another node still has an incident of the same check open;
- one closing alert when the last node's incident closes: the node's own
recovery when only that node failed, and otherwise an alert about the
check across the fleet, without an
incident_id, such asmyapp/http is ok on every node again; - rarely, a failure a second time after the closing alert, when a node's report shows that the node still failed while its reports did not reach the hub. The module opens the incident again, as for any failure.
Every failure, raise, reminder, and closing alert carries
fleet_incident_key: 32 hexadecimal digits, the same for one check across
the fleet, or on a standalone node. So the module opens and closes the
service's incident by fleet_incident_key, and keeps no state:
# mmwatchdog_external_modules/team_pager/module.star
def deliver_alert(alert, inputs, file_contents):
closes = alert["type"] in ["recovered", "can_check_again", "closed_without_recovery"]
# A notice has no key; it gets an incident of its own, by its ID.
key = alert.get("fleet_incident_key", alert["id"])
response = http_request(
method = "POST",
url = inputs["pager_url"],
headers = {"Content-Type": "application/json"},
body = json.encode({"incident": key, "resolve": closes, "summary": alert["title_with_alert_severity"]}),
)
if response.error:
return {"status": "failed", "message": "MMWatchdog could not reach the pager: " + response.error}
if response.http_status_code // 100 != 2:
return {"status": "failed", "message": "the pager answered HTTP %d" % response.http_status_code}
return {"status": "delivered", "message": "the pager took the alert"}
The closing alert's alert_severities_sent_for_incident lists the
severities that the destination got for the key, and its alert_severity
is the highest of them. A module that closes only the incidents that it
opened at one severity reads alert_severities_sent_for_incident.
The account and the environment of a module
MMWatchdog runs a module that is not core in a process of its own, once for each run of a check or each delivery attempt. Three run settings, which a check or a destination writes beside the module's inputs, say how:
| Run setting | Meaning | When left out |
|---|---|---|
run_as_user_account |
The account that the module runs as. | mmwatchdog, the unprivileged account of the checks |
run_from_directory |
The absolute directory that the module starts in. | / |
env |
More environment variables, as a mapping of texts. | None |
# files/hub_and_standalone/check_groups/myapp_jobs.yml
check_groups:
myapp_jobs:
check_every: 30s
time_limit_per_run: 10s
alert_after_failing_for: 1m
recovered_after_healthy_for: 30s
checks:
job_queue_is_short:
queued_jobs:
queue_file_path: /var/lib/myapp/pending # an input
run_as_user_account: myapp # run settings
run_from_directory: /var/lib/myapp
env:
MYAPP_MODE: production
fail_when_measurement_above:
queued_jobs: 500
alert_severity: warn
- The module has the rights of its account and no more. It cannot read MMWatchdog's own state, the alert queue, or a credential file that only root reads.
- The module gets a small, fixed environment: a standard
PATH, the account'sHOMEandUSER,LANG=C.UTF-8, and the variables ofenv. It gets nothing from MMWatchdog's own environment. Configuration files lists the names thatenvrefuses. - An account or a directory that does not exist on the node is a validation error on that node.
Inputs that name a file
An input whose name ends in _file names a file by its absolute path, such
as api_token_file: /etc/mmwatchdog/secrets/alarm-system-token. MMWatchdog
reads the file as root before each run of a check and for each delivery,
and gives the module the text in file_contents, under the input's name.
inputs keeps the path.
- Use
_filefor a credential that the module's account must not open, such as a root-owned file of mode 0600 in/etc/mmwatchdog/secrets/. MMWatchdog refuses a secret file that other accounts can read, unless the check or the destination allows that access: Configuration files. An input whose name ends incertificate_authority_fileholds a public trust bundle, so this rule does not apply to it, but MMWatchdog still reads the file and passes it on. - MMWatchdog removes one final line break from the text.
- Each file holds UTF-8 text of at most 16 KiB, and the files of one check or one destination hold at most 64 KiB together. The limits keep the module's request within 1 MiB.
- A file that MMWatchdog cannot read makes a check's run
unknown, and fails a delivery attempt. - Never repeat a file's content in a result or in diagnostics. MMWatchdog
also replaces each whole file content in the module's output with
[content of NAME]before it stores the observation. - A file input whose name does not end in
_file, such asqueue_file_path, is plain text: the module opens the path itself, withread_file, as the module's account. - The inputs of one destination are at most 256 KiB as JSON.
An executable in place of a script
Write an executable for a need that Starlark cannot meet, such as a library
that exists only in another language. The module's directory then holds the
file module_executable in place of module.star: a Bash script, a Go or
Rust program, or anything else that runs on the node.
mmwatchdog_external_modules/
file_age/
module_executable the program that MMWatchdog starts
module_description.yml the module's inputs, measurements, and examples
Choose a script when you can:
- A script needs nothing on the node but MMWatchdog. An executable needs whatever it uses on every node, such as an interpreter or a program that it calls.
- MMWatchdog sees what a script does, because a script reaches the node
only through the primitives. MMWatchdog does not see what an executable
does, and
mmwatchdog describe-moduleandmmwatchdog module-usagesay so beside each executable:MMWatchdog does not see what an executable does on this node. The local node must provide everything the executable needs. - A compiled executable must match the node's processor.
The description, the call in the configuration, the run settings, state, and findings work for an executable as for a script.
The JSON protocol
MMWatchdog starts the executable once for each run of its check, and writes one JSON object to its standard input:
{"protocol": 1,
"inputs": {"queue_file_path": "/var/lib/myapp/pending"},
"file_contents": {},
"state_dir": "/var/lib/mmwatchdog-module-state/myapp_jobs/job_queue_is_short",
"mmwatchdog_ends_the_module_after_milliseconds": 9987}
| Request field | Meaning |
|---|---|
protocol |
Always 1. |
inputs |
The check's inputs, as written. |
file_contents |
The text of each file that an input ending in _file names, under the input's name. Empty without such inputs. |
state_dir |
The check's state directory. The executable keeps files there. mmwatchdog check-now gives a copy. |
mmwatchdog_ends_the_module_after_milliseconds |
The time that the run has left when MMWatchdog starts the executable. MMWatchdog ends the executable then. An executable that waits for a server or a program ends its wait shortly before then, and writes its own result. |
The executable writes one JSON object to its standard output, and any diagnostics to its standard error:
{"protocol": 1,
"status": "ok",
"message": "780 queued jobs",
"measurements": {"queued_jobs": 780}}
- The result holds what
run_checkreturns, and"protocol": 1. A module that reports findings writes them as"findings": [{"id": "nightly_report", "message": "the job nightly_report failed"}]. - Exit status 0 means that the executable produced a result, also a
failedor anunknownone. - Each of these is an execution error, which makes the observation
unknown: another exit status, a start that failed, a run past the time limit, an end by a signal, standard output that is not exactly one JSON object, a key written twice, another protocol number, or more than 64 KiB of standard output. MMWatchdog keeps what the executable wrote, for diagnosis. For another exit status, the message isthe module file_age ended with exit status 1:, followed by the first line that the executable wrote to its standard error. - MMWatchdog keeps the first 16 KiB of standard error.
A destination module that is an executable gets alert, inputs,
file_contents, and mmwatchdog_ends_the_module_after_milliseconds, and
answers delivered or failed with a message:
{"protocol": 1, "status": "delivered", "message": "the alarm system accepted the alert"}
The example queued_jobs_in_go
is the module queued_jobs as a Go program, which uses Go's JSON library.
Build it for the nodes' processor, under the name module_executable.
CGO_ENABLED=0 makes a program that needs no C library on the node:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o module_executable queued_jobs_in_go.go
Use GOARCH=arm64 for ARM64 nodes. A project serves nodes of one
processor kind with one build.
The module-json helper for Bash
The mmwatchdog program includes a JSON helper, so an executable that is a
Bash script needs no jq and no Python. MMWatchdog sets the environment
variable MMWATCHDOG_MODULE_HELPER to the path of the mmwatchdog
program, for executables only.
The example file_age
measures how old a file is, in minutes. A program that must run on time,
such as a backup, writes the file after it succeeded, and the check states
how old the file may be. Its main steps:
request=$("$MMWATCHDOG_MODULE_HELPER" module-json request --inputs file_path) || exit 1
file_path=$(printf '%s' "$request" | "$MMWATCHDOG_MODULE_HELPER" module-json get --type string /inputs/file_path) || exit 1
if [[ ! -f $file_path ]]; then
"$MMWATCHDOG_MODULE_HELPER" module-json result --status failed \
--message 'the file does not exist or is not a regular file'
exit $?
fi
modified=$(stat -L -c %Y -- "$file_path") || exit 1
now=$(date +%s) || exit 1
age_minutes=$(( (now - modified) / 60 ))
"$MMWATCHDOG_MODULE_HELPER" module-json result --status ok \
--message "the file is $age_minutes minutes old" \
--measurement age_minutes "$age_minutes"
# files/hub_and_standalone/check_groups/backup.yml
check_groups:
backup:
check_every: 5m
time_limit_per_run: 10s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
last_backup_is_fresh:
file_age:
file_path: /var/lib/backup/last-successful-backup
fail_when_measurement_above:
age_minutes: 1560 # 26 hours
alert_severity: warn
The module answers failed when the file does not exist. Otherwise it
answers ok with the file's age as age_minutes, and the check fails when
the file is more than 1,560 minutes old, which is 26 hours. The module
needs Bash and the programs stat and date of GNU coreutils on every
node, and the module's account needs access to the file.
| Command | What it does |
|---|---|
module-json request [--inputs NAME,...] |
Reads the request from standard input, checks its protocol and that it holds only the listed inputs, and prints it for the get calls that follow. |
module-json get [--type TYPE] POINTER |
Reads one value from the request on standard input by JSON Pointer, such as /inputs/file_path or /alert/title. TYPE is string, integer, number, boolean, array, object, null, or json. A missing value or a value of another type is an error. |
module-json result --status STATUS (--message TEXT | --message-stdin) [--measurement NAME VALUE]... [--finding ID MESSAGE]... |
Writes a valid result. STATUS is ok, failed, or unknown for a check, and delivered or failed for a destination. |
module-json object [--string NAME TEXT]... [--string-stdin NAME] [--json NAME JSON]... |
Writes one JSON object with the members in the order given, for a module that sends JSON to a service. |
- A text needs no escaping by hand. The helper writes quotes, backslashes, and line breaks as JSON.
- A
stringis printed exactly, with no line break added. Command substitution, as above, is right for ordinary values. When line breaks at the end of a value matter, send the output straight to a file. - Each measurement has its own
--measurement NAME VALUE, and each finding its own--finding ID MESSAGE. A measurement takes no unit: the unit is in the description. --message-stdinreads the message from standard input, for a long message. Use exactly one of--messageand--message-stdin.- Keep a secret out of the arguments. Every account of a machine can
read the arguments of a running program.
module-json object --string-stdin NAMEreads one text from standard input. - Exit status: 0 when the command worked, 4 for wrong arguments, such
as
--status must be ok, failed, or unknown for a check, and delivered or failed for an alert destination, and 1 for data that is not valid, such asvalue at "/inputs/file_path": expected integer. - Each command reads at most 1 MiB from standard input, which is also the most that MMWatchdog writes in one request.
The helper needs no running MMWatchdog and no configuration. An executable in another language uses that language's own JSON library.
Try a module by hand
Four commands of mmwatchdog try a module without a deploy and without the
running service. Each runs wherever the mmwatchdog program runs: on a
node, or on a Linux machine with the program of your release. They need no
sudo.
Run the script once. mmwatchdog run-starlark-module MODULE_DIR reads
one request from standard input, runs the script, and writes the result to
standard output:
echo '{"protocol":1,"inputs":{"queue_file_path":"/var/lib/myapp/pending"}}' |
mmwatchdog run-starlark-module /tmp/modules-to-try/queued_jobs
{"measurements":{"queued_jobs":12},"message":"12 queued jobs","protocol":1,"status":"ok"}
- The request has the form of the JSON protocol. A
request that holds
alertcallsdeliver_alert, one with"verify_destination": truecallsverify_destination, and every other request callsrun_check. run-starlark-modulereads no_fileinput. Put afile_contentsdictionary in the request when the module uses one.- A script that uses
read_stateorwrite_stateneeds"state_dir": "/a/directory/that/exists"in the request. - Without
mmwatchdog_ends_the_module_after_milliseconds, the calls of the script have no time limit. With"mmwatchdog_ends_the_module_after_milliseconds": 5000, a call without an answer ends after 4.5 seconds, as in a check with atime_limit_per_runof 5 seconds. - The script runs as your own account. On a node,
sudo -u myapp mmwatchdog run-starlark-module ...runs it as the check's account. - What the script prints goes to standard error.
- A script with an error ends the command with exit status 4 and one line
that names the file and the line, such as
mmwatchdog: /tmp/modules-to-try/queued_jobs/module.star:6: key "queue_file_path" not in dict.
To try an executable by hand, start module_executable with a request on
its standard input, and with MMWATCHDOG_MODULE_HELPER set to the path of
the mmwatchdog program when it uses the helper:
echo '{"protocol":1,"inputs":{"file_path":"/etc/hostname"},"file_contents":{},"state_dir":"/tmp"}' |
MMWATCHDOG_MODULE_HELPER=$(command -v mmwatchdog) /tmp/modules-to-try/file_age/module_executable
{"protocol":1,"status":"ok","message":"the file is 408 minutes old","measurements":{"age_minutes":408}}
Read the description. mmwatchdog describe-module NAME prints the
module's description, or says what is wrong with the description.
--external-module-dir names a directory that holds your modules, in
place of the node's own:
mmwatchdog describe-module queued_jobs --external-module-dir /tmp/modules-to-try
queued_jobs: a module of the project's own (Starlark), used as a check
Counts the jobs that wait in a queue file.
Files: /tmp/modules-to-try/queued_jobs/
Inputs
queue_file_path string required The absolute path of the file that holds the number of waiting jobs.
Measurements
queued_jobs count The number of jobs that wait.
Example: Fail when more than 500 jobs wait.
queued_jobs:
queue_file_path: /var/lib/myapp/pending
fail_when_measurement_above:
queued_jobs: 500
alert_severity: warn
Validate a configuration against the module. mmwatchdog validate
takes --external-module-dir, and --official-extra-module-dir for the
official extra modules. On a Linux machine, in the project:
./mmwatchdog validate --config-only --config-dir files/hub_and_standalone \
--official-extra-module-dir official_extra_modules \
--external-module-dir mmwatchdog_external_modules
The command prints one line that starts with Valid configuration:, or the
error with its file and line.
See which modules a configuration uses. mmwatchdog module-usage
lists every module that the configuration names, with its kind, the
directory of its files, and the file and line of each check or destination
that names it. On a node:
$ sudo mmwatchdog module-usage
Core modules (built into MMWatchdog)
http /etc/mmwatchdog/check_groups/myapp.yml:14
node_filesystem /etc/mmwatchdog/check_groups/node.yml:19, /etc/mmwatchdog/check_groups/node.yml:25
node_resource /etc/mmwatchdog/check_groups/node.yml:10, /etc/mmwatchdog/check_groups/node.yml:13
systemd_unit_state /etc/mmwatchdog/check_groups/myapp.yml:9
Official extra modules (Starlark, shipped with MMWatchdog 0.1.1)
slack /usr/local/lib/mmwatchdog/official_extra_modules/slack/
/etc/mmwatchdog/alert_destinations.yml:3
The project's own modules
alarm_system /usr/local/lib/mmwatchdog/external_modules/alarm_system/ Starlark
/etc/mmwatchdog/alert_destinations.yml:8
queued_jobs /usr/local/lib/mmwatchdog/external_modules/queued_jobs/ Starlark
/etc/mmwatchdog/check_groups/myapp_jobs.yml:10
After a deploy, sudo mmwatchdog check-now CHECK_GROUP on the node runs
the checks of one check group at once and prints each result, without an
alert and without a change to an incident. Commands lists
every flag of these commands.
Limits
| Limit | Value |
|---|---|
| A module's name, an input's name, a measurement's name | Lower-case letters, digits, and underscores, starting with a letter, at most 64 characters |
module_description.yml |
256 KiB |
A measurement's unit |
64 bytes |
module.star |
1 MiB |
| Steps of one script run | 10 million |
| What one script run prints | 8 KiB passed on |
| Measurements in one result | 128 |
| Findings in one result | 500 |
A finding's id |
255 bytes, on one line |
A file that a _file input names |
16 KiB of UTF-8 text |
| The files of one check or one destination together | 64 KiB |
| The inputs of one destination, as JSON | 256 KiB |
| One request to a module | 1 MiB |
| An executable's standard output | 64 KiB |
| A module's standard error kept | 16 KiB |
read_file, run_command standard output, http_request body |
1 MiB each |
run_command standard error |
16 KiB |
tcp_exchange and udp_exchange received |
64 KiB |
trusted_certificate_authority_file of http_request |
1 MiB |
A value stored with write_state, as JSON |
1 MiB |
The copy of a state directory for check-now |
64 MiB and 10,000 files and directories |
This page is docs/reference/writing-modules.md in the MMWatchdog repository.