systemd_unit_resource
systemd_unit_resource measures one resource of one .service unit on the
node: its CPU, its memory, or the number of its processes. MMWatchdog reads
the CPU and memory from systemd's resource accounting of the unit's control
group with systemctl show. MMWatchdog counts the processes in the unit's
control group.
The memory is that of the whole unit, with its child processes, not only the
memory of the unit's main process.
Like every module that measures a number, the module only measures. A limit,
written beside the module with fail_when_measurement_above or
fail_when_measurement_below, makes the check fail, and MMWatchdog applies it in the
same way for every module
(checks that measure a number).
A check with only_record_readings: true only records its readings in the node's
measurement history and never
alerts (checks that only record).
To measure the whole node, use
node_resource. To watch one program of a unit by
its command, use
systemd_unit_commands, which
reads the same list of processes as resource: processes.
Set up
The CPU and the memory need systemd's accounting for the unit. When systemd
gives no counter, the check is unknown with a message such as
systemd unit myapp.service has no usable MemoryCurrent value; enable MemoryAccounting=yes in the unit configuration. Turn the accounting on in the unit file, such as the unit that
MMX installs:
[Service]
CPUAccounting=yes
MemoryAccounting=yes
MMWatchdog never changes the accounting settings of a unit. resource: processes
needs no accounting setting.
Inputs
systemd_unit(string; required): The .service unit to measure, such as myapp.service.resource(cpu,memory, orprocesses; required): The resource to measure.
Measurements
cpu_used_percent(percent): For cpu: the share of the node's processor time that the unit used since the run before.memory_used_mib(MiB): For memory: the memory that the unit uses.process_count(processes): For processes: the number of processes that the unit runs, which is the number of commands that "systemctl status" lists for it.
A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check that only records its readings states only_record_readings: true, no limit, and no alert_severity; it never alerts. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
The check fails when a reading passes its limit, with a message such as
memory_used_mib is 612.4 (fails above 512). A CPU reading adds the time it
covers, such as cpu_used_percent is 12.5 (fails above 80) over 30.000s.
A reading that cannot be made is unknown with an explanation, never a zero measurement. MMWatchdog cannot check when:
- the unit does not run. MMWatchdog measures a unit only while systemd reports it
as
activeorreloading(checks of a systemd unit). In any other state, such asactivating,deactivating,inactive, orfailed, the message says the state. Forcpuandmemory:systemd unit myapp.service usage unavailable: the unit is inactive. Forprocesses:systemd unit myapp.service process count unavailable: systemd unit myapp.service is inactive. A stopped unit has no count, as it has no CPU and no memory reading: the count of a stopped unit is not 0 processes of a running one. Use asystemd_unit_statecheck to alert for a stopped unit; - systemd does not know the unit:
systemd unit myapp.service not found, for every resource; - systemd gives no counter for the unit (see Set up), or no
InvocationIDfor a CPU reading:systemd unit myapp.service has no InvocationID; MMWatchdog cannot start measuring CPU usage; systemctlfails: the error message containssystemd inspection:;- a CPU reading cannot be compared with the sample that MMWatchdog took, such as
when the unit started a new invocation within the 200 ms sample:
CPU counters changed or did not advance between samples; MMWatchdog waits for a new sample. The next run takes a new sample.
A check with a limit opens a cannot-check incident after
alert_after_failing_for of unknown runs, as every check does
(the failing time). A check that
only records shows an unknown run in status and opens no incident.
Examples
Limit the CPU and the memory of a unit
# /etc/mmwatchdog/check_groups/myapp_resources.yml
check_groups:
myapp_resources:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
myapp_cpu_is_below_80_percent:
systemd_unit_resource:
systemd_unit: myapp.service
resource: cpu
fail_when_measurement_above:
cpu_used_percent: 80
alert_severity: warn
myapp_memory_is_below_512_mib:
systemd_unit_resource:
systemd_unit: myapp.service
resource: memory
fail_when_measurement_above:
memory_used_mib: 512
alert_severity: warn
myapp_process_count_is_recorded:
only_record_readings: true
systemd_unit_resource: # no limit: records only, never alerts
systemd_unit: myapp.service
resource: processes
On the node, every 30 seconds, MMWatchdog reads the CPU time and the memory of
myapp.service and counts its processes.
- The first check measures the share of the node's processor time that
myapp.serviceused since the previous run of the CPU check. When the unit's share of the node's processor time has been above 80 percent for 10 minutes, MMWatchdog sends awarnalert. - The second check fails while the unit uses more than 512 MiB of memory,
and MMWatchdog sends a
warnalert after 10 minutes of failing. - The third check only records the number of processes in the measurement
history. The process-count check never alerts and takes no
alert_severity. - When a failing check has been within its limit for 5 minutes, MMWatchdog sends the recovery.
Restart the unit when it uses too much memory
# /etc/mmwatchdog/check_groups/myapp.yml
check_groups:
myapp:
check_every: 30s
time_limit_per_run: 5s
alert_after_failing_for: 10m
recovered_after_healthy_for: 5m
checks:
myapp_memory_is_below_900_mib:
systemd_unit_resource:
systemd_unit: myapp.service
resource: memory
fail_when_measurement_above:
memory_used_mib: 900
alert_severity: warn
restart_systemd_unit:
max_attempts: 3
min_time_between_attempts: 1m
reset_attempts_after_healthy_for: 10m
systemd_unit: myapp.service
triggered_by_checks: [myapp_memory_is_below_900_mib]
restart_after_failing_for: 5m
On the node, MMWatchdog restarts myapp.service after memory use has
exceeded 900 MiB for 5 minutes.
If the memory stays below 900 MiB for 5 minutes after the restart, the check
recovers and no alert goes out.
If not, the warn alert goes out after 10 minutes of failing.
Restart a systemd unit says
how restarts work. A check that only records cannot be named in
triggered_by_checks.
How it works
Memory. MMWatchdog runs systemctl show --property=LoadState,ActiveState,InvocationID,CPUUsageNSec,MemoryCurrent -- myapp.service, as MMWatchdog itself, and reads MemoryCurrent.
MemoryCurrent is the memory that systemd counts for the unit's control
group, which includes the unit's child processes.
memory_used_mib is MemoryCurrent converted to MiB. The unit must run:
systemd reports it as active or reloading.
CPU. MMWatchdog reads CPUUsageNSec, the processor time that the unit
has used, from the systemctl show command listed under Memory.
MMWatchdog compares CPUUsageNSec with the reading from the previous run
of the CPU check.
cpu_used_percentis the processor time used between the two readings, divided by the time between them and by the node's number of logical processors. One saturated core on a two-core machine is about 50 percent.cpu_used_percentis a share of the whole machine, not of the unit's CPU quota or of the processors that the unit may use. Brief jitter of the accounting is capped at 100.- The reading covers the time since the check's run before, and is not a
long-term moving average. The first run, and a run whose reading before
cannot be compared, takes its own sample of 200 ms. So the check group's
time_limit_per_runmust be longer than200ms. - A new invocation of the unit, such as after a restart, starts a new
sample. A CPU counter smaller than the previous reading or a change in
the number of processors also starts a new sample.
CPU sampling has the rules that
systemd_unit_resourceshares with the CPU of the node.
Processes. process_count is the number of processes that the unit runs,
which is the number of commands that systemctl status myapp.service lists.
MMWatchdog asks systemd for the unit's control group and counts the processes in it,
and in every nested control group, below /sys/fs/cgroup, without a depth
limit. When a process list is unreadable or invalid, MMWatchdog makes the
run unknown and names the process-list file.
A partial count is never a reading. The unit must
run, as for the CPU and the memory. An active unit that runs no process,
such as a unit that systemd keeps active after its program ended
(RemainAfterExit=yes), counts 0.
See the upstream systemd resource accounting documentation for what systemd counts.
Errors and what they mean
| Message | What to do |
|---|---|
check_groups.myapp.checks.myapp_memory_is_below_512_mib.systemd_unit_resource.systemd_unit: is required: the systemd unit this check is about, such as myapp.service |
Name the unit to measure. |
...systemd_unit_resource.systemd_unit: must be a systemd .service unit name, such as myapp.service |
Write the full name of a .service unit. The unit name may contain only letters, digits, and the characters _@.:\-. |
...systemd_unit_resource.resource: must be cpu, memory, or processes |
Write one of the three resources. |
...fail_when_measurement_above.cpu_used_percent: this check has no such measurement; it measures memory_used_mib |
The limit names a measurement of another resource. Each resource has one measurement. |
...fail_when_measurement_above.cpu_used_percent: expected a number from 0 to 100 |
A percent limit is a number from 0 to 100. Every other limit is 0 or more. |
check_groups.myapp.checks.myapp_memory_is_below_512_mib.alert_severity: a systemd_unit_resource check needs a limit to alert; add fail_when_measurement_above or fail_when_measurement_below, or replace alert_severity with only_record_readings: true |
Add a limit, or remove alert_severity. |
check_groups.myapp.time_limit_per_run: must exceed 200ms for the initial CPU sample |
Give a check group with a resource: cpu check a time_limit_per_run longer than 200ms. |
check_groups.myapp.restart_systemd_unit.triggered_by_checks.0: check myapp_process_count_is_recorded only records its readings and never fails; give it a limit to use it for recovery |
Give the check a limit, or remove it from triggered_by_checks. |
systemd unit myapp.service has no usable MemoryCurrent value; enable MemoryAccounting=yes in the unit configuration (unknown), or the same for CPUUsageNSec and CPUAccounting |
Turn on the accounting in the unit file, as Set up shows. |
Tested on
processes_test.go also checks complete nested lists with
TestSystemdProcessCountRequiresEveryNestedList: 12 levels of control groups,
invalid process IDs, and an unreadable child list.
-
Go tests: In
resources_test.go,TestResourceInputsNeverInventZeroUsagechecks memory and CPU readings of active and reloading units.TestResourceInputsNeverInventZeroUsagealso checks that missing counters and units that do not run produce no invented readings.In
resources_test.go,TestCPUSamplerBootstrapsResetsAndHonorsCancellationchecks the first sample, a CPU limit, and a new sample after a new invocation.In
resources_test.go,TestMeasurementLimitsAreOptionalAndStrictchecks required inputs, resources, and measurement limits.TestMeasurementLimitsAreOptionalAndStrictalso checks thattriggered_by_checksrefuses checks that only record readings.severity_test.gochecks refusal ofalert_severitywithout a limit.In
processes_test.go,TestTheProcessesOfASystemdUnitchecks process counts of active and reloading units, and unknown results for stopped and deactivating units.check_test.gocontainsTestEveryCheckOfAnUnknownSystemdUnitSaysNotFound. -
Suites on real nodes: the
resourcessuite (resources.py): a real unit withCPUAccounting=yesandMemoryAccounting=yeswhose child process holds 48 MiB and keeps a processor busy. Checks of the resource-test unit's CPU and memory only record readings or use a limit to open and close an incident. Thenodesuite (node_scenarios.py): the process count of a real unit with two programs. Thesoaksuite measures the CPU and the memory ofmmwatchdog.serviceitself. -
The
resources,node, andsoaksuites passed on every target distribution. The validation record has the latest run.
Why it works this way
-
Every nested process list must be readable. A partial count could look healthy while the processes that use the most resources are hidden.
-
Recording readings is an explicit choice. State
only_record_readings: trueto record readings without alerts or recovery actions. A missing severity must not silently turn an intended alert into readings only. -
MMWatchdog applies the limit, and the module only measures. There is one way to write a limit for every module, and the readings reach the measurement history and charts.
-
The name of a measurement carries its unit, such as
memory_used_mib, so it can be read alone in a limit, in an alert, and in a chart. -
A check without a limit never alerts. A busy unit is not a failure until the operator sets a limit.
-
A reading that cannot be made is unknown, never zero. A zero would look like a healthy reading, and a stopped unit has no CPU, no memory, and no processes to count.
-
MMWatchdog measures a unit only while it runs, active or reloading. While systemd is still starting or stopping a unit, the unit's numbers describe neither the running program nor the program's absence. Every check of a systemd unit counts
activeandreloadingas running (checks of a systemd unit). -
MMWatchdog reads systemd's accounting and never changes it. systemd supervises the unit, and MMWatchdog only observes it. The memory of the whole control group includes the child processes that the unit starts.
This page is docs/reference/modules/systemd_unit_resource.md in the MMWatchdog repository.