docker_container
docker_container asks Docker whether one container runs, through Docker's
own program, docker. The value of the module is the container's name,
such as myapp-web. A running container whose own health check fails
counts as failed.
MMWatchdog does not start a container. Restarting a container that ended is the job of Docker's own restart policy, as restarting a program is the job of systemd.
Set up
The node needs Docker and its docker program. MMWatchdog runs docker as root, so
the check needs no membership in the group docker.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
docker_container(string; required): The name of the Docker container that must run.
When the check fails, and when it cannot check
A container that runs is healthy. The check fails when the container:
- ended, or was only created:
the Docker container myapp-web does not run: it ended with exit code 137, orthe Docker container myapp-web does not run: Docker reports it as created; - is paused:
the Docker container myapp-web is paused: its programs are stopped where they were, and do no work. Docker itself reports a paused container as running; - is restarted by Docker again and again:
the Docker container myapp-web is restarting again and again; it last ended with exit code 1. Such a container looks as if it runs at almost every moment, and Docker itself reports it as restarting; - runs, and the container's own health check fails:
the Docker container myapp-web runs, and its own health check fails. The body of the alert shows what the health check wrote last; - was removed:
Docker has no container named myapp-web. A container that is gone does not run.
A container without a health check, and one whose health check has no result yet, are healthy while they run.
MMWatchdog cannot check when it cannot ask Docker, for example because the
Docker service is stopped, when the node has no docker program, and when
docker gives an answer that MMWatchdog does not understand. The run is then
unknown.
A failed run starts or continues the check's failing time. An unknown run opens a separate cannot-check incident after the check's waiting times, as the design's failing time describes. The module reports no measurement, so the check takes no limit.
Examples
# /etc/mmwatchdog/check_groups/containers.yml
check_groups:
containers:
check_every: 30s
time_limit_per_run: 10s
alert_after_failing_for: 2m
recovered_after_healthy_for: 1m
checks:
web_container_runs:
docker_container: myapp-web
alert_severity: sos
Every 30 seconds, MMWatchdog on the node runs docker container inspect for
the container myapp-web. While the container runs and the container's
health check, if present, does not fail, the run is healthy. The message is the Docker container myapp-web runs, or the Docker container myapp-web runs, and its own health check passes for a container with a health check.
When the container has ended, is paused, restarts again and again, fails
its health check, or is gone for 2 minutes, MMWatchdog sends an sos alert.
MMWatchdog does not start the container: Docker's restart policy does, if
the container has one.
How it works
During each run, MMWatchdog executes
docker container inspect --format '{{json .State}}' as root. The command uses
the container's name. MMWatchdog reads whether the
container runs, is paused, or restarts, and the container's exit code.
For a container with a health check, MMWatchdog also reads the health
check's status and the output of the health check's last run.
A container whose health check has no result yet is healthy. The message is
the Docker container myapp-web runs, and its own health check has no result yet.
Errors and what they mean
check_groups.containers.checks.web_container_runs.docker_container: must be the name of a Docker container, such as myapp-web. A container's name starts with a letter or a digit, then has letters, digits,_,., and-, and has at most 255 characters.check_groups.containers.checks.web_container_runs.fail_when_measurement_above: the module docker_container reports no measurement, so its check takes no limit.MMWatchdog cannot run docker: executable file not found in $PATH. The node has no Docker. The run is unknown.MMWatchdog cannot ask Docker about the container myapp-web: .... Docker answered with an error, for example because its service is stopped. The message holds whatdockersaid. Start Docker:systemctl start docker.docker gave an answer about the container myapp-web that MMWatchdog does not understand. The run is unknown. Whatdocker container inspectwrote was not the state of a container in JSON.Docker has no container named myapp-web. The container was removed, or the name is wrong.docker ps --alllists the containers of the node.
Tested on
Limits on each distribution says what MMWatchdog cannot do on each distribution, or did not test there, and why.
- Go tests:
TestDockerContainerCheckinserver_checks_test.goruns the check against a stand-in fordocker: every state that Docker reports, a paused container, a failing health check with its last output, a removed container, a Docker that cannot be queried, and a node withoutdocker. - Real Docker, on every target distribution except Rocky Linux 9 and AlmaLinux 10, whose repositories have no Docker (it comes from Docker's own repository): the
servicessuite (service_scenarios.py) installs Docker and runs containers of an image made from the guest's ownbusyboxprogram, so the suite downloads nothing. The suite finds a container that runs, one that ended, one that is paused, one that was removed, and one whose own health check fails and then passes. A stopped Docker makes the check unknown. The latest run passed on those (validation record).
Why it works this way
- MMWatchdog does not start a container. Restarting a container that ended is the job of Docker's own restart policy, as restarting a program is the job of systemd.
- The check asks through the
dockerprogram, as a core module asks a node tool through the tool's own program. - A paused container fails. Docker reports a paused container as running, but the paused container's programs do no work.
- A container that Docker restarts again and again fails. The repeatedly restarted container looks as if the container runs at almost every moment.
- A container whose own health check fails fails. The container with a failed health check runs but does not work, according to Docker.
- A removed container fails, and a run that cannot query Docker is unknown. A removed container does not run. When Docker does not answer, MMWatchdog learns nothing about the container.
This page is docs/reference/modules/docker_container.md in the MMWatchdog repository.