A web service fleet
This example watches a small web service with MMWatchdog. The service runs on two machines:
- the application server runs the web server, the web application, its caches, its databases, and its mail server, and holds the data;
- the utility host checks the application server over the network, as a visitor and an operator reach it, and watches itself.
Each machine is a standalone node: it runs its own MMWatchdog with its own configuration and sends its own alerts. The two machines also watch each other's SSH server.
The example started as a real company's production Monit setup, which ran a web service for years. The example makes the Monit setup anonymous: the example's names, domains, addresses, and services are invented. The parts that served only the company's own products are left out. Where the setup repeated one kind of check many times, such as the lag of a database replica at five limits or seven Redis servers, the example keeps one check, or one check with the MMWatchdog feature that replaces the repetition. So each check shows a different kind of monitoring work.
A VM suite runs the example on two test guests and causes each failure for real: see Run the example as a test.
The files
examples/web-service-fleet/
application_server/ becomes /etc/mmwatchdog on the application server
mmwatchdog_node_settings.yml
alert_destinations.yml
check_groups/
web.yml the web server and the web application
application.yml the application's job queue, audio encoder, and version
caches.yml two Redis servers and memcached
databases.yml the primary MariaDB server and its replica
dns_cache.yml the local DNS cache
node.yml memory, swap, processors, failed units, out-of-memory kills
storage.yml filesystems, a software RAID mirror, and a ZFS pool
utility_host.yml the utility host's SSH server
utility_host/ becomes /etc/mmwatchdog on the utility host
mmwatchdog_node_settings.yml
alert_destinations.yml
check_groups/
application_server.yml the application server, over the network
utility_host.yml the utility host itself
Install each configuration directory on the corresponding machine as the
install on one node teaches.
The configuration directories can live under files/application_server and
files/utility_host in the deployment project examples/deploy.
Set each node's MMX inventory variable mmwatchdog_config_source_dir to
the node's configuration directory. The example's test copies each configuration
set to files/hub_and_standalone and deploys each machine separately.
The checks also read three files that the directories do not hold. On the application server, two credentials, each readable by root only:
/etc/mmwatchdog/secrets/web-application-status-page-login, the login of the status page, as one lineuser:password;/etc/mmwatchdog/secrets/mariadb-replica-mmwatchdog-monitoring-password, the password of the MariaDB accountmmwatchdog_monitoringon the replica.
On the utility host, /etc/mmwatchdog/web-service-test-certificate-authority.pem
holds the certificate of the certificate authority that signed the site's
certificate. Provision the credentials through your normal secret
deployment, as Install on one node
says for a webhook's token. Deploys never remove files under
/etc/mmwatchdog/secrets/. The example's test writes all
three files in the test's setup scripts.
The timings in the check groups are seconds, so that the test is quick. Timings in production gives the values for a real fleet.
The application server
These services run on the application server:
- nginx, with the site
app.example.com: a static front page, and the web application's/healthand/internal/status, over HTTP and HTTPS. A default site answers every other host name with 404, as a load balancer's catch-all site does. - The web application,
web-application.service, behind nginx on 127.0.0.1:8000, which runs as the accountwebapp. The web application's health command for the job queue is/opt/web-application/bin/check-job-queue. The web application's report job,report-generator.service, may use 32 MiB of memory. The web application uses the audio encoderlamefor media. - Two Redis servers from the Redis package's template unit:
redis-server@sessions.serviceon port 6380 for the sessions, andredis-server@job-queue.serviceon port 6381 for the job queue; and memcached on port 11211. - MariaDB: the primary on 127.0.0.1:3306, and a replica on 127.0.0.1:3307,
mariadb-replica.service, which follows the primary. - dnsmasq, a local DNS cache on 127.0.0.1:53.
- Postfix, the mail server, on port 25.
- Storage: swap space,
/srv/uploadsfor the users' uploads,/srv/shared, a network share that must be mounted, a software RAID 1 mirror/dev/md0, and a ZFS pooldatapool.
web.yml: the web server and the web application
| Check | Why the check is included |
|---|---|
web_server/web_server_is_running, systemd_unit_state of nginx.service, at sos |
nginx serves the site. When nginx fails, MMWatchdog restarts it 3 seconds into the failure, after the alert. When an operator stops nginx with systemctl stop, MMWatchdog leaves it stopped: a stopped unit stays stopped. |
web_server/default_site_answers_not_found, http that expects 404, at warn |
Another answer than 404, such as the distribution's welcome page, means that a change of nginx's configuration went wrong. |
web_application/web_application_is_running, systemd_unit_state of web-application.service, at sos |
The application's process. When the application's process ends, MMWatchdog restarts web-application.service. |
web_application/web_application_answers_health_request, http with the site's Host header, at sos |
An application that hangs keeps the application's unit active, so only a request finds the hang. On a failure, MMWatchdog first runs a recovery command. The recovery command removes a maintenance flag that a deploy can leave behind. Only when the health request still fails after the command does MMWatchdog restart the unit: two recovery actions on one failure, in order. |
web_application/status_page_reports_no_stalled_workers, http with basic_auth_file and response_body_matches_regex, at warn |
A value from a status page behind a login: the application's count of stalled workers must be 0. When the login file is missing, MMWatchdog sends no request without the login file. MMWatchdog sends a cannot-check alert instead of a failure. |
The health request names the two unit checks in
no_alerts_or_recovery_actions_while_these_checks_fail.
The status page names the health request. So while nginx or the application's
unit is down, the unit's alert is the only alert:
one cause, one alert.
application.yml: the application's own work
The three check groups in application.yml do real work, so the check groups
run every 5 seconds instead of every second. In production, the three check groups in application.yml run every 5 minutes.
| Check | Why the check is included |
|---|---|
job_queue/job_queue_health_command_succeeds, shell with run_as_user_account, run_from_directory, and env, at warn |
The application's health command runs as the application runs the command: as webapp, in the application's directory, and with the application's environment. The health command fails when more than 100 jobs wait. A command of the application's own is what a shell check is for. |
audio_encoder/audio_encoder_encodes_a_test_tone, shell, at warn |
A real piece of work proves that a tool works: the check encodes a test tone to MP3 with lame. A missing library or a broken package fails the audio-encoder check, which a look for the package would not notice. |
application_version/deployed_application_is_version_2_4_1, shell, at warn |
The deployed version must be the expected one. A deploy that installed another version fails the deployed-version check. No module compares versions, so a shell check does. The example writes the expected version directly in the check. |
caches.yml: two Redis servers and memcached
| Check | Why the check is included |
|---|---|
sessions_redis/sessions_redis_is_running and sessions_redis/sessions_redis_answers_ping, systemd_unit_state and redis, at warn |
Each instance of one program has a check group of its own, so that an alert names the instance and MMWatchdog restarts only that instance. A Redis server that hangs keeps its unit active, so the PING finds it. |
job_queue_redis/job_queue_redis_is_running and job_queue_redis/job_queue_redis_answers_ping, systemd_unit_state and redis, at warn |
The job queue's Redis has a separate check group, so alerts name the job-queue instance and MMWatchdog restarts only the job-queue instance. A Redis server that hangs keeps the Redis unit active, so the PING detects the hang. |
memcached/memcached_is_running and memcached/memcached_answers_statistics_request, systemd_unit_state and memcached, at warn |
A service that MMWatchdog watches and does not restart: the check group has no restart_systemd_unit. MMWatchdog alerts, and a person decides. |
Each PING and the statistics request name the unit check of their own check group, so a stopped server sends one alert.
databases.yml: the primary MariaDB server and its replica
| Check | Why the check is included |
|---|---|
database_primary/connections_are_below_50_percent, mysql through the local socket, with a limit on connections_used_percent, at warn |
A measurement of the server past a limit. The server allows 20 connections, and an application that leaks connections uses them up. |
database_primary/reporting_database_answers_a_real_query, shell, at warn |
A real query with the expected result: the reports need at least one row of reporting.daily_totals. The mysql module asks the server fixed questions and takes no query of the user's, so a shell check runs the query. The query command logs in through the local socket as mmwatchdog, which MariaDB knows as mmwatchdog@localhost. The account mmwatchdog@localhost has read access to the database reporting only. |
database_replica/replica_is_less_than_5_seconds_behind, mysql of the replica, with a limit of 5 on replication_lag_seconds, at warn |
A replica that falls behind gets a warning first. |
database_replica/replica_is_less_than_15_seconds_behind, mysql of the replica, with a limit of 15 on replication_lag_seconds, at sos |
The replica gets an urgent alert when the replica falls further behind: two checks with two limits and two severities alert again as replication lag gets worse. A replica that stops replicating fails the 5-second and 15-second lag checks. MMWatchdog logs in to the replica as mmwatchdog_monitoring, with the password in a file. |
dns_cache.yml: the local DNS cache
| Check | Why the check is included |
|---|---|
dns_cache/dns_cache_is_running, systemd_unit_state of dnsmasq.service, at info |
The DNS cache's process. |
dns_cache/dns_cache_answers_for_the_site, dns over UDP with expected_record_values, at info |
dnsmasq must answer app.example.com with 192.0.2.10: a UDP service that answers a real request. |
The DNS cache's check group waits 10 seconds for failure and for recovery.
A failure must last 10 seconds before the failure counts, and a recovery
must last 10 seconds before the recovery counts.
So a brief DNS failure sends no alert, and a DNS cache that comes and goes
sends no recovery alert until the DNS cache stays healthy. The DNS cache's alerts are info, which
only the team's chat channel for information gets.
node.yml: the machine itself
| Check | Why the check is included |
|---|---|
node_memory/memory_is_below_75_percent, node_resource: memory, at warn |
A fix when a resource of the whole machine runs out: the application leaks memory, so the check group restarts web-application.service, before the kernel starts to end processes. |
node_swap/swap_is_not_in_use_while_memory_is_nearly_full, shell, at warn |
Swap space in use after a past spike is harmless. Swap in use while memory is nearly full means the machine is about to struggle. A node_resource check has one measurement, and MMWatchdog has no rule that combines two measurements. So a shell check reads memory usage and swap usage from /proc/meminfo. The shell check fails when memory is above 60 percent and swap above 20 percent. |
node_processors/cpu_is_below_90_percent and node_processors/io_wait_is_below_20_percent, node_resource: cpu and io_wait, at warn |
The processors are busy, or wait for a slow disk. |
failed_systemd_units/machine_has_no_failed_systemd_unit, every_systemd_unit, at warn |
Any unit that systemd reports as failed, such as the report job, each as its own incident. The units with a check of their own are in systemd_units_to_ignore, so their failure sends one alert. |
out_of_memory_kills/out_of_memory_kills, out_of_memory_kills, at warn |
The kernel ended a process for lack of memory. One kill is enough: the out-of-memory-kills event check alerts at the first failed run and takes no alert_after_failing_for. |
storage.yml: filesystems, RAID, and ZFS
| Check | Why the check is included |
|---|---|
uploads_filesystem/uploads_filesystem_is_below_90_percent, node_filesystem of /srv/uploads, with limits on space and inodes, at warn |
A warning while there is still time. |
uploads_filesystem/uploads_filesystem_is_below_98_percent, node_filesystem of /srv/uploads, with limits of 98 percent on space and inodes, at sos |
An urgent alert when the filesystem is nearly full: two levels on one filesystem. |
shared_storage/shared_storage_is_mounted, node_filesystem of /srv/shared without a limit, at warn |
A filesystem that must be mounted: the check fails while nothing is mounted at the path. MMWatchdog then runs the recovery command mount /srv/shared, which mounts the shared filesystem again from /etc/fstab. |
raid_mirror/mdadm_array, mdadm_array of /dev/md0, at warn |
A software RAID mirror that loses a disk. |
zfs_data_pool/data_pool_is_healthy_and_below_88_percent, the official extra module zfs_pool, at warn |
A ZFS pool that loses a device or fills up. |
utility_host.yml: the other machine
| Check | Why the check is included |
|---|---|
utility_host/machine_answers_ssh, ssh of utility-host.web-service.internal:22, at warn |
The application server checks the utility host's SSH server, as the utility host checks the application server's. If the utility host stops answering SSH, the application server alerts. If the application server stops answering SSH, the utility host alerts. |
The utility host
The utility host runs no service of the web service. The utility host checks the
application server from the network, by the site's public name
app.example.com and by the machine's name
application-server.web-service.internal.
application_server.yml: the application server, over the network
| Check | Why the check is included |
|---|---|
application_server/https_port_accepts_connections, tcp of app.example.com:443, at sos |
The site's port. The site's other checks name it, so while the port refuses connections, its alert is the one alert of the site. |
application_server/site_answers_from_outside, http of https://app.example.com/, at warn |
The site as a visitor sees it. The site's certificate comes from a certificate authority of the example's own, so the check trusts that authority's certificate with trusted_certificate_authority_file; a site with a certificate of a public certificate authority needs no such file. |
application_server/site_certificate_is_valid_for_21_more_days, tls with fail_when_certificate_expires_within: 21d, at warn |
A warning three weeks before the certificate expires. |
application_server/site_certificate_is_valid_for_7_more_days, tls with fail_when_certificate_expires_within: 7d, at sos |
An urgent alert one week before: two levels on one certificate. |
application_server/machine_answers_ping, ping, at warn |
The machine answers on the network. |
application_server/machine_answers_ssh, ssh, at warn, with raise_to_sos_after_failing_for: 8s |
A machine that stops answering SSH is a warning at once. MMWatchdog sends reminders about the open SSH incident every 4 seconds to team_chat_warn_and_sos (remind_every). After 8 seconds of SSH failure, MMWatchdog raises the incident's severity to sos, so on_call_pager also receives an alert. |
application_server/mail_server_answers_smtp, smtp, at warn, with send_alerts_only_to_destinations: [mail_administrator] |
The mail server belongs to the mail administrator, and its alerts go to that person alone. |
utility_host.yml: the utility host itself
| Check | Why the check is included |
|---|---|
utility_host_machine/memory_is_below_90_percent, node_resource: memory, at warn |
The utility host's own memory. |
utility_host_machine/root_filesystem_is_below_90_percent, node_filesystem of /, at warn |
The utility host's own disk space. |
The destinations
Each check states an alert severity, and each destination states the severities the destination takes, so severity decides who gets an alert. The recovery alert goes to every destination that got the failure: which destination gets an alert.
| Destination | Takes | Stands in for | In a real fleet |
|---|---|---|---|
team_chat_info |
info |
The team's chat channel for information | A chat module, such as slack or mattermost |
team_chat_warn_and_sos |
warn and sos |
The team's chat channel for warnings and urgent alerts. On the utility host, team_chat_warn_and_sos sends reminders for each open incident every 4 seconds (remind_every: 4s) |
The chat module chosen for team_chat_info, for another channel |
on_call_pager |
sos |
The on-call pager | pushover |
mail_administrator, on the utility host only |
No severity: only the SMTP check names mail_administrator |
The person who runs the mail server | email |
In the example, each destination is a webhook to a receiver on the node
that runs the check. Each destination uses a separate path, such as
http://127.0.0.1:18080/events/on_call_pager, so that the test sees which
destination got each alert. A real fleet replaces each webhook with the
destination module for the chat, paging, or email service receiving the alert.
The Monit setup had four failure-alert scripts, each with a separate script that sent recovery notifications. The failure-alert scripts sent information and warnings to two chat channels, urgent alerts to chat and a paging service, and alerts through a script named after a person.
MMWatchdog provides the scripts' routing and recovery behavior through settings:
- Each check's alert severity marks information, warnings, or urgent alerts.
- Each destination's
alert_severitiesselects the alerts for the chat channels and the paging service. - MMWatchdog sends recovery alerts automatically, replacing the recovery scripts.
send_alerts_only_to_destinationssends alerts only to the named person's destination, replacing the script named after a person.
Timings in production
The example's timings are seconds. The Monit setup checked every 30 seconds, and these are its timings in MMWatchdog's form:
| Checks | In production |
|---|---|
| Local services and health requests | check_every: 5m, alert_after_failing_for: 10m |
| The unit checks of systemd services | check_every: 30s, alert_after_failing_for: 5m |
| CPU | alert_after_failing_for: 10m |
| I/O wait and memory | alert_after_failing_for: 20m |
| Swap while memory is nearly full | the shell check with swap above 10 percent while memory is above 95 percent |
| The recovery of the machine's resources | recovered_after_healthy_for: 5m |
| Disk space and inodes | limits of 90 and 98 percent, alert_after_failing_for: 0s |
| Memory that restarts the application | a limit of 95 percent, restart_after_failing_for: 0s |
| Failed systemd units | alert_after_failing_for: 3m |
| Software RAID | alert_after_failing_for: 30m |
| The DNS cache | alert_after_failing_for: 2m30s |
| SSH of the application server from the utility host, and SSH of the utility host from the application server | alert_after_failing_for: 2m, remind_every: 1m on the chat destination, raise_to_sos_after_failing_for: 10m |
| The site and its certificate from outside | alert_after_failing_for: 2m; fail_when_certificate_expires_within: 21d and 7d |
| The mail server | alert_after_failing_for: 30m, send_alerts_only_to_destinations |
| The replica's lag | limits on replication_lag_seconds such as 1800 (warn) and 10800 (sos), with remind_every on the chat destination |
| The application's health command | check_every: 5m, alert_after_failing_for: 10m |
| The encoder test and the version check | check_every: 5m for the encoder test and the version check. alert_after_failing_for: 5m for the encoder test, and alert_after_failing_for: 30m for the version check. |
With the production table's timings, the default of 8 for
max_concurrent_checks serves the application server. The example raises
max_concurrent_checks to 16 because about 45 checks run every second.
What the example leaves out
- A DHCP server. Like a DHCP check, the DNS cache check tests whether a UDP service answers a real request.
- The services that only the company had, such as its audio streaming servers. The example keeps the kinds of checks behind the company services: a value from a status page, and a measurement past a limit that alerts again as the measurement gets worse.
- A central view, and a cron job that restarted the monitor. MMWatchdog runs under systemd. A fleet hub provides a central view. The hub's silence alert and hub-outage alerter node also notice a machine that stops answering. The example runs two standalone nodes, as the original company's production configuration ran two Monit instances.
- The state change of keepalived on a pair of load balancers. In MMWatchdog,
keepalived's notify script runs
mmwatchdog send-notice. - Limits for each role and each machine, and checks generated for every machine of a group. MMX's templates and inventory set role-specific and machine-specific limits, and generate checks for the machines of an inventory group. The example writes each node's files directly.
Run the example as a test
The VM suite web_service_fleet_example runs the example on the two guests
of a job slot, as the
testing guide describes. The guest mmwatchdog-dev is the application server,
and mmwatchdog-peer is the utility host. Each guest's /etc/hosts gives
mmwatchdog-dev the names
application-server.web-service.internal and app.example.com, and
mmwatchdog-peer the name utility-host.web-service.internal.
python3 tests/integration/incus.py --distribution ubuntu2404 --suite web_service_fleet_example
The suite's guests run Ubuntu 24.04 only. The application server's ZFS pool
needs ZFS's kernel module, and among MMWatchdog's target distributions only
Ubuntu's kernel packages hold it; the setup also installs Ubuntu's packages
by their names. The example's checks run the same MMWatchdog code on every
distribution, and the node and services suites test what differs
between distributions.
The test's own files are in
tests/integration/web_service_fleet_example/:
setup/ holds the scripts that install and start each machine's services.
step_commands/ holds helper commands that cause failures and read results.
scenarios.json is the scenario file. The driver
monitoring_setup_scenarios.py
says what each part of the scenario file means. Each test run performs the
setup, the quiet check, and the failure scenarios in order:
- The test driver runs the setup scripts on each guest, then deploys
MMWatchdog through MMX and
examples/deploy. Each guest receives the corresponding configuration directory fromexamples/web-service-fleet. - The quiet check: every check on the application server and the utility host becomes healthy, and then no alert arrives for 15 seconds.
- Each scenario causes a failure for real, then waits for the expected alerts and the destination of each alert, then restores the pre-failure state. Every check must then be healthy again. Any unexpected alert fails the scenario, so a scenario also shows which checks stay quiet.
| Scenario | What the scenario shows |
|---|---|
web_application_killed |
SIGKILL leaves the application's unit failed. The sos alert reaches the chat and the pager. MMWatchdog restarts the unit 3 seconds into the failure, after the alert. The recovery alert reaches the chat and the pager. The health request, which names the unit check, sends nothing. |
web_application_hangs_and_gets_the_fix_command_then_a_restart |
SIGSTOP freezes the application while the application's unit stays active. The health request times out and alerts at sos. MMWatchdog runs the recovery command first, which does not help the frozen application. MMWatchdog restarts the unit only on a failure MMWatchdog sees after the recovery command. mmwatchdog events shows the order. The status page, which names the health request, sends nothing. |
web_server_stopped_by_an_operator |
systemctl stop nginx. MMWatchdog alerts and does not restart a unit that was stopped on request (systemd_unit_restart_withheld); nginx stays stopped until the scenario starts it. On the utility host, the port check sends the one alert of the site; the site and the two certificate checks stay quiet. On the application server, the health request, the status page, and the 404 check stay quiet. |
default_site_answers_200_instead_of_404 |
The default site answers 200; the check that expects 404 sends a warn alert to the chat only. |
status_page_reports_stalled_workers |
The status page, behind its login, says stalled_workers: 2. |
status_page_login_file_missing |
The status-page login file is missing. MMWatchdog sends no request without the login file, so MMWatchdog cannot check: the cannot_check alert, and can_check_again once the login file is back. |
one_of_two_redis_instances_hangs |
SIGSTOP freezes the job queue's Redis. The job-queue Redis PING check alerts and MMWatchdog restarts the job-queue instance. The sessions' instance stays healthy and quiet. |
memcached_killed_and_left_alone |
memcached ends; MMWatchdog alerts and does nothing else, and memcached stays down until the scenario starts it. |
dns_cache_flaps_then_stops_answering |
dnsmasq stops answering for less than the 10 seconds of alert_after_failing_for, which sends nothing; then for longer, which sends the info alert to team_chat_info only. A healthy time shorter than recovered_after_healthy_for sends no recovery; a longer one does. |
report_job_killed_by_the_kernel |
The report job takes 256 MiB in a unit that allows 32 MiB, and the kernel ends the report job. Two alerts: out_of_memory_kills, and the failed unit report-generator.service reported by every_systemd_unit. The failed unit is a finding of every_systemd_unit, with a recovery of the failed unit after systemctl reset-failed. |
application_leaks_memory_and_gets_restarted |
The application takes memory until the machine's memory is 85 percent used. The memory check alerts and restarts the application, which frees the memory. |
swap_in_use_while_memory_is_free_and_then_nearly_full |
A program in a unit of 64 MiB holds 400 MiB. More than 25 percent of swap is in use while memory is free. The swap check stays healthy, which a plain limit on swap would not. A second program then fills memory to 67 percent, and the check alerts. |
processors_kept_busy |
Endless loops on every processor. |
programs_wait_for_a_slow_disk |
dd writes directly to the disk in a unit that systemd limits to 1 MiB per second, so the processors wait for the disk. |
uploads_filesystem_fills_up |
The uploads filesystem fills to 93 percent: the warn alert to the chat. Then the uploads filesystem fills to 99 percent: the sos alert to the chat and the pager. Removing the files recovers the warning and urgent checks. |
uploads_filesystem_runs_out_of_inodes |
The uploads filesystem fills to 93 percent inode usage: the warn alert to the chat. Then inode usage reaches 99 percent: the sos alert to the chat and pager. Removing the files recovers the warning and urgent checks. |
shared_storage_unmounted_and_mounted_again |
umount /srv/shared: the alert, then the recovery command mounts the share again, then the recovery. |
raid_mirror_loses_a_disk |
mdadm --fail of one device of the mirror; the check fails until the device is back and rebuilt. |
zfs_pool_loses_a_device |
zpool offline takes one half of the pool offline. The check fails until the offline half is back. |
replica_stops_replicating |
The replica stops receiving changes. The 5-second and 15-second lag checks fail. |
replica_falls_further_and_further_behind |
The replica applies changes 30 seconds late while orders are written every half second, so the replica falls behind second by second. The warn alert comes above 5 seconds, then the sos alert above 15 seconds. |
database_connections_nearly_used_up |
12 idle connections to a server that allows 20. |
reporting_query_fails |
The table that the reports read is renamed. |
job_queue_backs_up |
The scenario first verifies the health command's healthy output from a run as webapp. Then the scenario places 500 jobs in the queue. |
audio_encoder_loses_its_library |
libmp3lame.so.0 is moved away, so lame cannot start. |
deployed_application_version_differs |
Version 2.5.0 is deployed while the check expects 2.4.1. |
application_server_ssh_unreachable_from_the_utility_host |
The application server drops new SSH connections from the utility host. The warn alert reaches the chat only; a reminder follows; after 8 seconds of failing, the raise to sos reaches the chat and the pager; the recovery reaches both. |
application_server_does_not_answer_ping |
The application server drops ping requests from the utility host. |
utility_host_ssh_unreachable_from_the_application_server |
The utility host drops SSH from the application server, and the application server's check of the utility host alerts. |
site_certificate_expires_in_10_days_and_then_in_3 |
A certificate with 10 days until expiry fails the 21-day check (warn) and passes the 7-day check. A certificate with 3 days until expiry fails the 7-day check (sos). A certificate with 90 days until expiry makes the 21-day and 7-day checks recover. |
mail_server_down |
Postfix stops; the SMTP check's failure and recovery reach mail_administrator and no other destination. |
How the test causes its failures:
- The application and memcached have
Restart=no, so that a killed program stays down until MMWatchdog restarts it, or until the scenario starts it. WithRestart=always, systemd would restart a killed application or memcached process before MMWatchdog saw the failure. The Redis servers keep theirRestart=always: the scenario freezes Redis withSIGSTOP, so the unit stays active while Redis does not answer. - A scenario that needs a restart kills or freezes the program, because
MMWatchdog leaves a unit that was stopped on request stopped. The scenario
signals the main process only (
systemctl kill --kill-whom=main). - Where an action of MMWatchdog must come after an alert, the recovery action waits 3 seconds and the alert 2 seconds. The DNS cache's 10 seconds are far longer than the scenario's own steps. The scenario controls the duration of a short failure, so the duration does not depend on the machine's speed.
- A
shellcheck's message describes the command's exit code, such ascommand exited 1. What the script prints is the run's standard output, which the alert's body shows and which the step commands read. - The utility host's reminders may come in any scenario of the utility host's checks, including reminders MMWatchdog sends while confirming recovery. The scenario file therefore allows the utility host's reminders.
This page is examples/web-service-fleet/README.md in the MMWatchdog repository.