node_resource

node_resource measures one resource of the whole node: its CPU, its memory, its swap space, its load, the time that its processors wait for a disk, its memory pressure, or its open files. MMWatchdog reads the resource measurements from the kernel's own files below /proc and runs no program to obtain the resource measurements. The value of the check is the name of the resource, such as node_resource: memory.

A node_resource check only measures. The limit is written beside the module, in fail_when_measurement_above or fail_when_measurement_below, and MMWatchdog applies it in the same way as for every module that measures (checks that measure a number). With only_record_readings: true, the check only records its readings.

Use a related module for other resources: node_filesystem for the space of one filesystem, node_disk_device for one disk, node_network_interface for one network interface, and systemd_unit_resource for the CPU, the memory, or the processes of one systemd unit.

Inputs

The module takes one value, which a check writes after the module's name, and no other input.

  • the value of node_resource (cpu, memory, swap, load, io_wait, memory_pressure, or open_files; required): The resource to measure.

Measurements

  • cpu_used_percent (percent): For cpu: the share of the node's processor time that was in use since the run before.
  • memory_used_percent (percent): For memory: the share of the node's memory that is not available to programs.
  • swap_used_percent (percent): For swap: the share of the swap space that is in use. A node without swap space uses 0.
  • load_average_1m (load): For load: the kernel's load average over 1 minute. load_average_5m and load_average_15m are the same over 5 and 15 minutes.
  • load_average_1m_per_core (load per core): For load: the load average over 1 minute for each processor of the node. load_average_5m_per_core and load_average_15m_per_core are the same over 5 and 15 minutes.
  • io_wait_percent (percent): For io_wait: the share of the processors' time since the run before in which they had nothing to run because programs waited for a disk.
  • memory_pressure_percent (percent): For memory_pressure: the share of the last 60 seconds in which at least one program had to wait for memory.
  • open_files_used_percent (percent): For open_files: the open files of the whole machine, as a share of the kernel's limit.

A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check that only records its readings states only_record_readings: true, no limit, and no alert_severity; it never alerts. The configuration files reference explains limits in checks that measure a number.

When the check fails, and when it cannot check

  • The check fails when a reading passes one of its limits. A reading strictly above its fail_when_measurement_above number, or strictly below its fail_when_measurement_below number, fails the check; an equal reading passes. The check then fails at its alert_severity. The check's incident follows the check group's failing time.
  • A check with only_record_readings: true only records, never fails, never alerts, and takes no alert_severity (checks that only record). A reading that MMWatchdog cannot make shows as unknown in mmwatchdog status, and opens no incident.
  • MMWatchdog cannot check when MMWatchdog cannot make a reading. The run's result, called an observation, is then unknown, with the reason, and never a reading of zero. An unknown result of a check that can alert opens a separate cannot-check incident after alert_after_failing_for (scheduling and observed state). The causes are listed under errors.
  • A limit on a measurement that this run could not make makes the observation unknown, with the reason. The one case in which a resource has only some of its measurements is load: without the number of processors, the three _per_core measurements are missing.

The commands that use the most of a resource

When a check of cpu, memory, swap, or load fails, the body of its alert names the five commands that use the most of that resource:

memory_used_percent is 94.2 (fails above 90)
failing since 14:02 UTC
the commands that use the most memory:
  3051 MiB: /usr/sbin/mysqld (process 812)
  640 MiB: /usr/bin/python3 /srv/myapp/worker.py (process 2210)
  212 MiB: /usr/lib/systemd/systemd-journald (process 301)
  96 MiB: nginx: worker process (process 955)
  95 MiB: nginx: worker process (process 956)
Resource The first line of the list What each line gives
memory the commands that use the most memory: The memory of the process that is in physical memory, in MiB
swap the commands that use the most swap space: The memory of the process that is in swap space, in MiB, from VmSwap in /proc/PROCESS_ID/status
cpu the commands that used the most CPU between the two CPU readings: The processor time that the process used in the time of the reading, as a percent of the whole node, like cpu_used_percent itself, such as 41.5 percent
load the commands with the most threads that run or wait for a disk now: The process's count of threads that run or wait for a disk now, such as 3 threads. The load average counts threads that run or wait for a disk, over a longer time
  • Each line is one process, with its command line as systemctl status shows it, cut after 120 characters, and its process ID. A process without a command line, such as a thread of the kernel, shows its name in brackets, such as [kswapd0]. The highest amount comes first. MMWatchdog leaves out processes that use none of the resource or end in the meantime.
  • MMWatchdog reads the commands in every run in which the check fails, so the alert has the commands from the run that confirmed the failure. mmwatchdog status CHECK_GROUP and mmwatchdog check-now CHECK_GROUP also show the resource-use command list for a failed check. JSON has the resource-use command list as details of the observation (title and body of an alert).
  • A check that only records, and a check that is within its limit, names no command. A check of io_wait, memory_pressure, or open_files names none either.

Examples

Limits on the CPU, the memory, the swap space, and the load

# /etc/mmwatchdog/check_groups/machine.yml
check_groups:
  machine:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 10m
    recovered_after_healthy_for: 5m
    checks:
      cpu_use_is_recorded:
        only_record_readings: true
        node_resource: cpu                        # no limit: records only, never alerts
      memory_is_below_90_percent:
        node_resource: memory
        fail_when_measurement_above:
          memory_used_percent: 90
        alert_severity: warn
      swap_is_below_50_percent:
        node_resource: swap
        fail_when_measurement_above:
          swap_used_percent: 50
        alert_severity: info
      load_average_5m_per_core_is_below_1_5:
        node_resource: load
        fail_when_measurement_above:
          load_average_5m_per_core: 1.5
        alert_severity: warn

Every 30 seconds, MMWatchdog on the node runs the four checks:

  1. cpu_use_is_recorded measures the share of the node's processor time that was in use since its run before, and stores the reading in the node's measurement history. cpu_use_is_recorded never alerts.
  2. memory_is_below_90_percent fails when more than 90 percent of the node's memory is in use. Once memory_is_below_90_percent has failed for 10 minutes, MMWatchdog sends a warn alert whose body names the five commands that use the most memory.
  3. swap_is_below_50_percent fails when more than half of the swap space is in use, and alerts at info.
  4. load_average_5m_per_core_is_below_1_5 fails when the 5-minute load average, divided by the number of processors, is above 1.5. A limit of 1.5 on load_average_5m_per_core suits a node with 2 processors and a node with 64.

Each check is healthy again after 5 minutes within the check's limit, and MMWatchdog then sends the recovery alert.

Signs that the machine is in trouble

Three more resources say that the machine is in trouble before a program fails:

# /etc/mmwatchdog/check_groups/machine_health.yml
check_groups:
  machine_health:
    check_every: 30s
    time_limit_per_run: 5s
    alert_after_failing_for: 10m
    recovered_after_healthy_for: 5m
    checks:
      io_wait_is_below_30_percent:
        node_resource: io_wait
        fail_when_measurement_above:
          io_wait_percent: 30
        alert_severity: warn
      memory_pressure_is_below_10_percent:
        node_resource: memory_pressure
        fail_when_measurement_above:
          memory_pressure_percent: 10
        alert_severity: warn
      open_files_are_below_80_percent:
        node_resource: open_files
        fail_when_measurement_above:
          open_files_used_percent: 80
        alert_severity: warn
  • io_wait_percent is the usual first sign of a slow or failing disk. The check fails when the processors spent more than 30 percent of their time since the run before waiting for a disk, with nothing else to run.
  • memory_pressure_percent says that the machine is short of memory right now. The check fails when, in more than 10 percent of the last 60 seconds, at least one program had to wait for memory. Swap space that was filled during an earlier peak and is still in use does not raise memory_pressure_percent. Limit memory_pressure_percent where a limit on swap would alert for no reason.
  • open_files_used_percent fails above 80 percent of the kernel's limit on open files. A machine that reaches the limit cannot open a file or accept a connection.

How it works

Each run reads one resource from the kernel's files:

Resource Measurements Where MMWatchdog reads each resource, and how it counts
cpu cpu_used_percent From /proc/stat: busy time divided by total time across all processors since the previous run. Idle, I/O wait, and stolen time are not busy. Guest time is not counted twice. A decrease in the I/O wait counter contributes zero to the interval, as the kernel's documentation of /proc/stat allows. Any other backward counter needs a new sample. 0 to 100
memory memory_used_percent /proc/meminfo: 100 × (1 − MemAvailable / MemTotal), the share of the physical memory that is not available to programs. 0 to 100
swap swap_used_percent /proc/meminfo: 100 × (1 − SwapFree / SwapTotal). A node without swap space reads 0. 0 to 100
load load_average_1m, load_average_5m, load_average_15m, and each with _per_core /proc/loadavg: the kernel's load average over 1, 5, and 15 minutes, 0 or more. A _per_core measurement, such as load_average_5m_per_core, divides the load average by the number of online logical CPUs listed in /proc/stat
io_wait io_wait_percent The same counters of /proc/stat as cpu: the share of the processors' time since the run before in which they had nothing to run because programs waited for a disk. 0 to 100
memory_pressure memory_pressure_percent /proc/pressure/memory: the avg60 value of the line some, the share of the last 60 seconds in which at least one program had to wait for memory. 0 to 100
open_files open_files_used_percent /proc/sys/fs/file-nr: subtract the unused kernel file handles from the allocated kernel file handles, then divide by the kernel's file-handle limit. Multiply by 100 to get the percentage of kernel file handles in use. 0 to 100
  • cpu and io_wait measure over a time. A reading covers the time since the check's run before. The first run takes a fresh sample of 200 ms. A run after a counter went backwards or after the number of processors changed also takes a fresh sample. A fresh sample is also needed after a gap longer than 2 × check_every + time_limit_per_run between samples. So the check group's time_limit_per_run must be longer than 200ms.

    The message ends with the time that the reading covers, such as cpu_used_percent is 12.5 (fails above 80) over 30.000s. The CPU sampling section of the configuration files reference gives the rules that every module that measures a rate shares.

  • memory, swap, load, memory_pressure, and open_files are read at the moment of the run. The load averages and the memory pressure are averages that the kernel keeps itself; MMWatchdog does not average them again.

  • The message names each measurement that has a limit, with its reading and its limit, such as memory_used_percent is 41.27 (fails above 90). A check without a limit names every reading: a check of load names all six.

  • The commands for the alert are read only by a check that has a limit and an alert_severity. For memory, swap, and load, MMWatchdog reads the resource-use command list only in a run that fails. For cpu, MMWatchdog reads the processor time of every process at the start and end of every reading. The command list covers the time that cpu_used_percent covers; a process that started during the reading counts with all the processor time that it has used.

Errors and what they mean

Each validation error starts with the file, the line, and the path of the setting, such as check_groups.machine.checks.memory_is_below_90_percent.node_resource:.

  • must be one of cpu, memory, swap, load, io_wait, memory_pressure, open_files. The value names no resource that MMWatchdog measures. The space of a filesystem is a node_filesystem check.

  • this check has no such measurement; it measures memory_used_percent. A limit names a measurement of another resource. Each resource has only the measurements listed in the resource's row of the table above.

  • expected a number from 0 to 100 for a limit on a _percent measurement, and expected a number of 0 or more for a limit on a load average.

  • a node_resource check needs a limit to alert. Add a measurement limit, or replace the severity with only_record_readings: true.

  • time_limit_per_run: must exceed 200ms for the initial CPU sample. A check of cpu or io_wait needs more time than its first sample.

When MMWatchdog cannot make a reading, the observation is unknown with one of these messages:

  • the kernel of this machine does not provide the memory pressure (/proc/pressure/memory). The kernel was built or started without memory pressure accounting. If the kernel supports memory pressure accounting but disables memory pressure accounting by default, the boot option psi=1 enables memory pressure accounting. Otherwise limit memory_used_percent on such a machine instead. The kernels of Rocky Linux 9 and AlmaLinux 9 are built this way: run grubby --update-kernel=ALL --args=psi=1 and restart the machine once.
  • CPU counters changed or did not advance between samples; MMWatchdog waits for a new sample. Even the fresh sample of 200 ms gave no reading: the counters of /proc/stat went backwards, did not move, or the number of processors changed. The next run takes a new sample.
  • the number of processors is unavailable: followed by the reason. A check of load has a limit on a _per_core measurement, and MMWatchdog could not read /proc/stat.
  • A message that names a file of /proc, such as /proc/meminfo: MemAvailable unavailable, or /proc/sys/fs/file-nr: expected three numbers. The kernel's file is missing or has a form that MMWatchdog does not know.
  • cpu_used_percent measurement is invalid, with the name of the measurement. A reading was negative or not a number, which MMWatchdog never reports as a value.

Tested on

  • Go tests in internal/checksandalerts: resources_test.go covers the limits, the CPU counters and sampling, the readings of swap and load, and a check whose limit was removed; node_measurements_test.go covers the readings of the I/O wait, the memory pressure, and the open files, also on a kernel without memory pressure. processes_test.go in internal/primitives covers the commands in the alert of each resource. On Linux, resources_linux_test.go in internal/daemon verifies that mmwatchdog check-now leaves the CPU baseline of the scheduled runs alone.

  • On real guests, the suite resources.py tests recording the node's CPU, memory, swap, and load, including per-core load, without limits. Limits on the CPU and memory open incidents.

    node_scenarios.py measures the I/O wait and the memory pressure of the guest's kernel, compares the open files with /proc/sys/fs/file-nr, and checks the message of a kernel without memory pressure where the guest has none. The suite node_scenarios.py also fails checks of the memory and the CPU with a program that holds 96 MiB and a program that keeps a processor busy. The suite finds the memory program in the alert body and the CPU program in an observation's command list. A check of the load that fails in the same moment names the busy program too.

  • The suites resources.py and node_scenarios.py and the Go tests passed on every target distribution in the latest run of the whole catalog (validation record).

Why it works this way

  • Recording readings is an explicit choice. State only_record_readings: true to record readings without alerts or recovery. A missing severity must not silently turn an intended alert into readings only.

  • MMWatchdog applies the limit, and the module only measures. Every module takes limits in a uniform way, and every reading reaches the measurement history (check group decisions).

  • A check without a limit never alerts. A busy machine is not a failure until the operator sets a limit.

  • A reading that cannot be made is unknown, never zero. A zero would look like a healthy machine.

  • Levels are two checks. A warn at 80 percent and an sos at 95 percent are two checks with two limits. MMWatchdog has no per-measurement setting that raises alert severity when a reading crosses a second limit.

  • The load is also measured per core. One limit then suits machines of every size.

  • The memory pressure is a resource of its own. A limit on the swap space alerts on a machine that filled its swap in an earlier peak and is healthy now; the memory pressure only rises while programs wait.

  • io_wait keeps the kernel's name. io_wait is the word that operators know and search for.

  • The alert names the commands. The first question about full memory or a busy processor is which program caused the high resource use. MMWatchdog includes the resource-use command list in the alert so operators can identify the programs without logging in to the node. The command list has no setting and is not given for filesystems.

This page is docs/reference/modules/node_resource.md in the MMWatchdog repository.