node_resource
node_resource measures one resource of the whole node: its CPU, its
memory, its swap space, its load, the time that its processors wait for a
disk, its memory pressure, or its open files.
MMWatchdog reads the resource measurements from the kernel's own files
below /proc and runs no program to obtain the resource measurements.
The value of the check is the name of the resource, such as
node_resource: memory.
A node_resource check only measures. The limit is written beside the
module, in fail_when_measurement_above or fail_when_measurement_below,
and MMWatchdog applies it in the same way as for every module that measures
(checks that measure a number).
With only_record_readings: true, the check only records its readings.
Use a related module for other resources:
node_filesystem for the space of one
filesystem, node_disk_device for one disk,
node_network_interface for one network
interface, and systemd_unit_resource for
the CPU, the memory, or the processes of one systemd unit.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
node_resource(cpu,memory,swap,load,io_wait,memory_pressure, oropen_files; required): The resource to measure.
Measurements
cpu_used_percent(percent): For cpu: the share of the node's processor time that was in use since the run before.memory_used_percent(percent): For memory: the share of the node's memory that is not available to programs.swap_used_percent(percent): For swap: the share of the swap space that is in use. A node without swap space uses 0.load_average_1m(load): For load: the kernel's load average over 1 minute. load_average_5m and load_average_15m are the same over 5 and 15 minutes.load_average_1m_per_core(load per core): For load: the load average over 1 minute for each processor of the node. load_average_5m_per_core and load_average_15m_per_core are the same over 5 and 15 minutes.io_wait_percent(percent): For io_wait: the share of the processors' time since the run before in which they had nothing to run because programs waited for a disk.memory_pressure_percent(percent): For memory_pressure: the share of the last 60 seconds in which at least one program had to wait for memory.open_files_used_percent(percent): For open_files: the open files of the whole machine, as a share of the kernel's limit.
A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check that only records its readings states only_record_readings: true, no limit, and no alert_severity; it never alerts. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
- The check fails when a reading passes one of its limits. A reading
strictly above its
fail_when_measurement_abovenumber, or strictly below itsfail_when_measurement_belownumber, fails the check; an equal reading passes. The check then fails at itsalert_severity. The check's incident follows the check group's failing time. - A check with
only_record_readings: trueonly records, never fails, never alerts, and takes noalert_severity(checks that only record). A reading that MMWatchdog cannot make shows as unknown inmmwatchdog status, and opens no incident. - MMWatchdog cannot check when MMWatchdog cannot make a reading.
The run's result, called an observation, is then unknown, with the reason,
and never a reading of zero. An unknown result of a check that can alert
opens a separate cannot-check incident after
alert_after_failing_for(scheduling and observed state). The causes are listed under errors. - A limit on a measurement that this run could not make makes the
observation unknown, with the reason. The one case in which a resource
has only some of its measurements is
load: without the number of processors, the three_per_coremeasurements are missing.
The commands that use the most of a resource
When a check of cpu, memory, swap, or load fails, the body of its
alert names the five commands that use the most of that resource:
memory_used_percent is 94.2 (fails above 90)
failing since 14:02 UTC
the commands that use the most memory:
3051 MiB: /usr/sbin/mysqld (process 812)
640 MiB: /usr/bin/python3 /srv/myapp/worker.py (process 2210)
212 MiB: /usr/lib/systemd/systemd-journald (process 301)
96 MiB: nginx: worker process (process 955)
95 MiB: nginx: worker process (process 956)
| Resource | The first line of the list | What each line gives |
|---|---|---|
memory |
the commands that use the most memory: |
The memory of the process that is in physical memory, in MiB |
swap |
the commands that use the most swap space: |
The memory of the process that is in swap space, in MiB, from VmSwap in /proc/PROCESS_ID/status |
cpu |
the commands that used the most CPU between the two CPU readings: |
The processor time that the process used in the time of the reading, as a percent of the whole node, like cpu_used_percent itself, such as 41.5 percent |
load |
the commands with the most threads that run or wait for a disk now: |
The process's count of threads that run or wait for a disk now, such as 3 threads. The load average counts threads that run or wait for a disk, over a longer time |
- Each line is one process, with its command line as
systemctl statusshows it, cut after 120 characters, and its process ID. A process without a command line, such as a thread of the kernel, shows its name in brackets, such as[kswapd0]. The highest amount comes first. MMWatchdog leaves out processes that use none of the resource or end in the meantime. - MMWatchdog reads the commands in every run in which the check fails, so the alert
has the commands from the run that confirmed the failure.
mmwatchdog status CHECK_GROUPandmmwatchdog check-now CHECK_GROUPalso show the resource-use command list for a failed check. JSON has the resource-use command list asdetailsof the observation (title and body of an alert). - A check that only records, and a check that is within its limit, names no
command. A check of
io_wait,memory_pressure, oropen_filesnames none either.
Examples
Limits on the CPU, the memory, the swap space, and the load
# /etc/mmwatchdog/check_groups/machine.yml
check_groups:
machine:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
cpu_use_is_recorded:
only_record_readings: true
node_resource: cpu # no limit: records only, never alerts
memory_is_below_90_percent:
node_resource: memory
fail_when_measurement_above:
memory_used_percent: 90
alert_severity: warn
swap_is_below_50_percent:
node_resource: swap
fail_when_measurement_above:
swap_used_percent: 50
alert_severity: info
load_average_5m_per_core_is_below_1_5:
node_resource: load
fail_when_measurement_above:
load_average_5m_per_core: 1.5
alert_severity: warn
Every 30 seconds, MMWatchdog on the node runs the four checks:
cpu_use_is_recordedmeasures the share of the node's processor time that was in use since its run before, and stores the reading in the node's measurement history.cpu_use_is_recordednever alerts.memory_is_below_90_percentfails when more than 90 percent of the node's memory is in use. Oncememory_is_below_90_percenthas failed for 10 minutes, MMWatchdog sends awarnalert whose body names the five commands that use the most memory.swap_is_below_50_percentfails when more than half of the swap space is in use, and alerts atinfo.load_average_5m_per_core_is_below_1_5fails when the 5-minute load average, divided by the number of processors, is above 1.5. A limit of 1.5 onload_average_5m_per_coresuits a node with 2 processors and a node with 64.
Each check is healthy again after 5 minutes within the check's limit, and MMWatchdog then sends the recovery alert.
Signs that the machine is in trouble
Three more resources say that the machine is in trouble before a program fails:
# /etc/mmwatchdog/check_groups/machine_health.yml
check_groups:
machine_health:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
io_wait_is_below_30_percent:
node_resource: io_wait
fail_when_measurement_above:
io_wait_percent: 30
alert_severity: warn
memory_pressure_is_below_10_percent:
node_resource: memory_pressure
fail_when_measurement_above:
memory_pressure_percent: 10
alert_severity: warn
open_files_are_below_80_percent:
node_resource: open_files
fail_when_measurement_above:
open_files_used_percent: 80
alert_severity: warn
io_wait_percentis the usual first sign of a slow or failing disk. The check fails when the processors spent more than 30 percent of their time since the run before waiting for a disk, with nothing else to run.memory_pressure_percentsays that the machine is short of memory right now. The check fails when, in more than 10 percent of the last 60 seconds, at least one program had to wait for memory. Swap space that was filled during an earlier peak and is still in use does not raisememory_pressure_percent. Limitmemory_pressure_percentwhere a limit on swap would alert for no reason.open_files_used_percentfails above 80 percent of the kernel's limit on open files. A machine that reaches the limit cannot open a file or accept a connection.
How it works
Each run reads one resource from the kernel's files:
| Resource | Measurements | Where MMWatchdog reads each resource, and how it counts |
|---|---|---|
cpu |
cpu_used_percent |
From /proc/stat: busy time divided by total time across all processors since the previous run. Idle, I/O wait, and stolen time are not busy. Guest time is not counted twice. A decrease in the I/O wait counter contributes zero to the interval, as the kernel's documentation of /proc/stat allows. Any other backward counter needs a new sample. 0 to 100 |
memory |
memory_used_percent |
/proc/meminfo: 100 × (1 − MemAvailable / MemTotal), the share of the physical memory that is not available to programs. 0 to 100 |
swap |
swap_used_percent |
/proc/meminfo: 100 × (1 − SwapFree / SwapTotal). A node without swap space reads 0. 0 to 100 |
load |
load_average_1m, load_average_5m, load_average_15m, and each with _per_core |
/proc/loadavg: the kernel's load average over 1, 5, and 15 minutes, 0 or more. A _per_core measurement, such as load_average_5m_per_core, divides the load average by the number of online logical CPUs listed in /proc/stat |
io_wait |
io_wait_percent |
The same counters of /proc/stat as cpu: the share of the processors' time since the run before in which they had nothing to run because programs waited for a disk. 0 to 100 |
memory_pressure |
memory_pressure_percent |
/proc/pressure/memory: the avg60 value of the line some, the share of the last 60 seconds in which at least one program had to wait for memory. 0 to 100 |
open_files |
open_files_used_percent |
/proc/sys/fs/file-nr: subtract the unused kernel file handles from the allocated kernel file handles, then divide by the kernel's file-handle limit. Multiply by 100 to get the percentage of kernel file handles in use. 0 to 100 |
-
cpuandio_waitmeasure over a time. A reading covers the time since the check's run before. The first run takes a fresh sample of 200 ms. A run after a counter went backwards or after the number of processors changed also takes a fresh sample. A fresh sample is also needed after a gap longer than2 × check_every + time_limit_per_runbetween samples. So the check group'stime_limit_per_runmust be longer than200ms.The message ends with the time that the reading covers, such as
cpu_used_percent is 12.5 (fails above 80) over 30.000s. The CPU sampling section of the configuration files reference gives the rules that every module that measures a rate shares. -
memory,swap,load,memory_pressure, andopen_filesare read at the moment of the run. The load averages and the memory pressure are averages that the kernel keeps itself; MMWatchdog does not average them again. -
The message names each measurement that has a limit, with its reading and its limit, such as
memory_used_percent is 41.27 (fails above 90). A check without a limit names every reading: a check ofloadnames all six. -
The commands for the alert are read only by a check that has a limit and an
alert_severity. Formemory,swap, andload, MMWatchdog reads the resource-use command list only in a run that fails. Forcpu, MMWatchdog reads the processor time of every process at the start and end of every reading. The command list covers the time thatcpu_used_percentcovers; a process that started during the reading counts with all the processor time that it has used.
Errors and what they mean
Each validation error starts with the file, the line, and the path of the
setting, such as check_groups.machine.checks.memory_is_below_90_percent.node_resource:.
-
must be one of cpu, memory, swap, load, io_wait, memory_pressure, open_files. The value names no resource that MMWatchdog measures. The space of a filesystem is anode_filesystemcheck. -
this check has no such measurement; it measures memory_used_percent. A limit names a measurement of another resource. Each resource has only the measurements listed in the resource's row of the table above. -
expected a number from 0 to 100for a limit on a_percentmeasurement, andexpected a number of 0 or morefor a limit on a load average. -
a node_resource check needs a limit to alert. Add a measurement limit, or replace the severity withonly_record_readings: true. -
time_limit_per_run: must exceed 200ms for the initial CPU sample. A check ofcpuorio_waitneeds more time than its first sample.
When MMWatchdog cannot make a reading, the observation is unknown with one of these messages:
the kernel of this machine does not provide the memory pressure (/proc/pressure/memory). The kernel was built or started without memory pressure accounting. If the kernel supports memory pressure accounting but disables memory pressure accounting by default, the boot optionpsi=1enables memory pressure accounting. Otherwise limitmemory_used_percenton such a machine instead. The kernels of Rocky Linux 9 and AlmaLinux 9 are built this way: rungrubby --update-kernel=ALL --args=psi=1and restart the machine once.CPU counters changed or did not advance between samples; MMWatchdog waits for a new sample. Even the fresh sample of 200 ms gave no reading: the counters of/proc/statwent backwards, did not move, or the number of processors changed. The next run takes a new sample.the number of processors is unavailable:followed by the reason. A check ofloadhas a limit on a_per_coremeasurement, and MMWatchdog could not read/proc/stat.- A message that names a file of
/proc, such as/proc/meminfo: MemAvailable unavailable, or/proc/sys/fs/file-nr: expected three numbers. The kernel's file is missing or has a form that MMWatchdog does not know. cpu_used_percent measurement is invalid, with the name of the measurement. A reading was negative or not a number, which MMWatchdog never reports as a value.
Tested on
-
Go tests in
internal/checksandalerts:resources_test.gocovers the limits, the CPU counters and sampling, the readings of swap and load, and a check whose limit was removed;node_measurements_test.gocovers the readings of the I/O wait, the memory pressure, and the open files, also on a kernel without memory pressure.processes_test.goininternal/primitivescovers the commands in the alert of each resource. On Linux,resources_linux_test.goininternal/daemonverifies thatmmwatchdog check-nowleaves the CPU baseline of the scheduled runs alone. -
On real guests, the suite
resources.pytests recording the node's CPU, memory, swap, and load, including per-core load, without limits. Limits on the CPU and memory open incidents.node_scenarios.pymeasures the I/O wait and the memory pressure of the guest's kernel, compares the open files with/proc/sys/fs/file-nr, and checks the message of a kernel without memory pressure where the guest has none. The suitenode_scenarios.pyalso fails checks of the memory and the CPU with a program that holds 96 MiB and a program that keeps a processor busy. The suite finds the memory program in the alert body and the CPU program in an observation's command list. A check of the load that fails in the same moment names the busy program too. -
The suites
resources.pyandnode_scenarios.pyand the Go tests passed on every target distribution in the latest run of the whole catalog (validation record).
Why it works this way
-
Recording readings is an explicit choice. State
only_record_readings: trueto record readings without alerts or recovery. A missing severity must not silently turn an intended alert into readings only. -
MMWatchdog applies the limit, and the module only measures. Every module takes limits in a uniform way, and every reading reaches the measurement history (check group decisions).
-
A check without a limit never alerts. A busy machine is not a failure until the operator sets a limit.
-
A reading that cannot be made is unknown, never zero. A zero would look like a healthy machine.
-
Levels are two checks. A
warnat 80 percent and ansosat 95 percent are two checks with two limits. MMWatchdog has no per-measurement setting that raises alert severity when a reading crosses a second limit. -
The load is also measured per core. One limit then suits machines of every size.
-
The memory pressure is a resource of its own. A limit on the swap space alerts on a machine that filled its swap in an earlier peak and is healthy now; the memory pressure only rises while programs wait.
-
io_waitkeeps the kernel's name.io_waitis the word that operators know and search for. -
The alert names the commands. The first question about full memory or a busy processor is which program caused the high resource use. MMWatchdog includes the resource-use command list in the alert so operators can identify the programs without logging in to the node. The command list has no setting and is not given for filesystems.
This page is docs/reference/modules/node_resource.md in the MMWatchdog repository.