docker_container

docker_container asks Docker whether one container runs, through Docker's own program, docker. The value of the module is the container's name, such as myapp-web. A running container whose own health check fails counts as failed.

MMWatchdog does not start a container. Restarting a container that ended is the job of Docker's own restart policy, as restarting a program is the job of systemd.

Set up

The node needs Docker and its docker program. MMWatchdog runs docker as root, so the check needs no membership in the group docker.

Inputs

The module takes one value, which a check writes after the module's name, and no other input.

  • the value of docker_container (string; required): The name of the Docker container that must run.

When the check fails, and when it cannot check

A container that runs is healthy. The check fails when the container:

  • ended, or was only created: the Docker container myapp-web does not run: it ended with exit code 137, or the Docker container myapp-web does not run: Docker reports it as created;
  • is paused: the Docker container myapp-web is paused: its programs are stopped where they were, and do no work. Docker itself reports a paused container as running;
  • is restarted by Docker again and again: the Docker container myapp-web is restarting again and again; it last ended with exit code 1. Such a container looks as if it runs at almost every moment, and Docker itself reports it as restarting;
  • runs, and the container's own health check fails: the Docker container myapp-web runs, and its own health check fails. The body of the alert shows what the health check wrote last;
  • was removed: Docker has no container named myapp-web. A container that is gone does not run.

A container without a health check, and one whose health check has no result yet, are healthy while they run.

MMWatchdog cannot check when it cannot ask Docker, for example because the Docker service is stopped, when the node has no docker program, and when docker gives an answer that MMWatchdog does not understand. The run is then unknown.

A failed run starts or continues the check's failing time. An unknown run opens a separate cannot-check incident after the check's waiting times, as the design's failing time describes. The module reports no measurement, so the check takes no limit.

Examples

# /etc/mmwatchdog/check_groups/containers.yml
check_groups:
  containers:
    check_every: 30s
    time_limit_per_run: 10s
    alert_after_failing_for: 2m
    recovered_after_healthy_for: 1m
    checks:
      web_container_runs:
        docker_container: myapp-web
        alert_severity: sos

Every 30 seconds, MMWatchdog on the node runs docker container inspect for the container myapp-web. While the container runs and the container's health check, if present, does not fail, the run is healthy. The message is the Docker container myapp-web runs, or the Docker container myapp-web runs, and its own health check passes for a container with a health check.

When the container has ended, is paused, restarts again and again, fails its health check, or is gone for 2 minutes, MMWatchdog sends an sos alert. MMWatchdog does not start the container: Docker's restart policy does, if the container has one.

How it works

During each run, MMWatchdog executes docker container inspect --format '{{json .State}}' as root. The command uses the container's name. MMWatchdog reads whether the container runs, is paused, or restarts, and the container's exit code. For a container with a health check, MMWatchdog also reads the health check's status and the output of the health check's last run.

A container whose health check has no result yet is healthy. The message is the Docker container myapp-web runs, and its own health check has no result yet.

Errors and what they mean

  • check_groups.containers.checks.web_container_runs.docker_container: must be the name of a Docker container, such as myapp-web. A container's name starts with a letter or a digit, then has letters, digits, _, ., and -, and has at most 255 characters.
  • check_groups.containers.checks.web_container_runs.fail_when_measurement_above: the module docker_container reports no measurement, so its check takes no limit.
  • MMWatchdog cannot run docker: executable file not found in $PATH. The node has no Docker. The run is unknown.
  • MMWatchdog cannot ask Docker about the container myapp-web: .... Docker answered with an error, for example because its service is stopped. The message holds what docker said. Start Docker: systemctl start docker.
  • docker gave an answer about the container myapp-web that MMWatchdog does not understand. The run is unknown. What docker container inspect wrote was not the state of a container in JSON.
  • Docker has no container named myapp-web. The container was removed, or the name is wrong. docker ps --all lists the containers of the node.

Tested on

Limits on each distribution says what MMWatchdog cannot do on each distribution, or did not test there, and why.

  • Go tests: TestDockerContainerCheck in server_checks_test.go runs the check against a stand-in for docker: every state that Docker reports, a paused container, a failing health check with its last output, a removed container, a Docker that cannot be queried, and a node without docker.
  • Real Docker, on every target distribution except Rocky Linux 9 and AlmaLinux 10, whose repositories have no Docker (it comes from Docker's own repository): the services suite (service_scenarios.py) installs Docker and runs containers of an image made from the guest's own busybox program, so the suite downloads nothing. The suite finds a container that runs, one that ended, one that is paused, one that was removed, and one whose own health check fails and then passes. A stopped Docker makes the check unknown. The latest run passed on those (validation record).

Why it works this way

  • MMWatchdog does not start a container. Restarting a container that ended is the job of Docker's own restart policy, as restarting a program is the job of systemd.
  • The check asks through the docker program, as a core module asks a node tool through the tool's own program.
  • A paused container fails. Docker reports a paused container as running, but the paused container's programs do no work.
  • A container that Docker restarts again and again fails. The repeatedly restarted container looks as if the container runs at almost every moment.
  • A container whose own health check fails fails. The container with a failed health check runs but does not work, according to Docker.
  • A removed container fails, and a run that cannot query Docker is unknown. A removed container does not run. When Docker does not answer, MMWatchdog learns nothing about the container.

This page is docs/reference/modules/docker_container.md in the MMWatchdog repository.