node_temperature
node_temperature measures the temperature of one kind of hardware of the
machine, such as its processors or its NVMe drives. The value of the check
is the name of the kind of sensor, as the kernel names it, such as
coretemp. MMWatchdog reads the kernel's files below /sys/class/hwmon and
needs no sensors program to measure temperatures.
A machine has several sensors of one name: one for each core, or one
for each drive. The check measures the highest reading among the sensors
of the configured name as highest_temperature_celsius. Different hardware
has different limits, so each kind of sensor gets its own check.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
node_temperature(string; required): The name of the kind of sensor, as the kernel names it and as the sensors program prints it in front of the first hyphen, such as coretemp, k10temp, or nvme.
Measurements
highest_temperature_celsius(degrees Celsius): The highest temperature among the sensors matching the configured sensor name.
A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check that only records its readings states only_record_readings: true, no limit, and no alert_severity; it never alerts. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
- The check fails at the check's
alert_severitywhenhighest_temperature_celsiuspasses a limit set for the check. A reading strictly above the check'sfail_when_measurement_abovenumber, or strictly below the check'sfail_when_measurement_belownumber, fails the check. An equal reading passes (checks that measure a number). - A check with
only_record_readings: trueonly records its readings, never fails, and takes noalert_severity(checks that only record). - MMWatchdog cannot check when the machine has no sensor of the name
configured in
node_temperature, which is common on a virtual machine. The observation is then unknown, and the message lists the names of the sensors that the machine has:this machine has no temperature sensor named coretemp; the kernel knows these kinds of sensor: acpitz, nvme. A machine without any sensor givesthis machine has no temperature sensor named coretemp; it has no temperature sensor that the kernel knows. An unknown result of a check that can alert opens a separate cannot-check incident afteralert_after_failing_for(scheduling and observed state).
An unreadable sensor of the requested kind makes the run unknown unless a readable sensor already fails a limit. The result message names the unreadable sensor's file and the reason. If a readable sensor fails a limit, the run stays failed, and the result message also names unreadable sensors.
Negative temperature limits are valid, for hardware that can operate below zero.
Examples
# /etc/mmwatchdog/check_groups/temperature.yml
check_groups:
temperature:
check_every: 1m
time_limit_per_run: 5s
alert_after_failing_for: 5m
recovered_after_healthy_for: 5m
checks:
processors_are_below_85_degrees:
node_temperature: coretemp
fail_when_measurement_above:
highest_temperature_celsius: 85
alert_severity: warn
drives_are_below_60_degrees:
node_temperature: nvme
fail_when_measurement_above:
highest_temperature_celsius: 60
alert_severity: warn
Every minute, MMWatchdog on the node reads every sensor named coretemp, which are
those of the Intel processors, and every sensor named nvme. The first
check fails when the hottest processor sensor is above 85 degrees Celsius,
the second when the hottest NVMe drive is above 60. Each check that has
failed for 5 minutes sends a warn alert. Each check recovers after
5 minutes below the check's limit. The check's result message names
the hottest sensor, such as
highest_temperature_celsius is 61 (fails above 85); the hottest of 9 sensors of coretemp is Package id 0.
How it works
| Name of the sensor | Hardware |
|---|---|
coretemp |
The processors of Intel |
k10temp |
The processors of AMD |
nvme |
NVMe drives |
drivetemp |
SATA drives |
acpitz |
The mainboard |
- The names are those that the
sensorsprogram prints in front of the first hyphen, such ascoretempincoretemp-isa-0000. The kernel shows the sensor names in the files/sys/class/hwmon/*/name. Several devices can have the same name, such as one for each NVMe drive, and the check reads all of them. - Each reading comes from a file such as
temp1_input, which holds thousandths of a degree Celsius. A sensor is named in the message by its label, such asPackage id 0fromtemp1_label. For a sensor without a label, the message uses the sensor's file, such astemp1. An unreadable reading makes the run unknown and names its file, unless a readable sensor already fails a limit. When a readable sensor fails a limit and another sensor cannot be read, the run stays failed and also names the unreadable file. - A reading below zero is valid, such as that of a sensor in a cold room. A limit may also be below zero.
Errors and what they mean
Each validation error starts with the file, the line, and the path of the
setting, such as check_groups.temperature.checks.processors_are_below_85_degrees.node_temperature:.
-
must be the name of a kind of temperature sensor, as the sensors program prints it in front of the first hyphen, such as coretemp, k10temp, or nvme. The value holds a character other than letters, digits,_,., and-, such as a space, or is longer than 64 characters. -
this check has no such measurement; it measures highest_temperature_celsius. -
a node_temperature check needs a limit to alert. Add a measurement limit, or replace the severity withonly_record_readings: true. -
An unknown check with
this machine has no temperature sensor named .... The message lists the names that the machine has; use one of them, or remove the check from a machine without sensors. A value such ascoretemp-isa-0000passes validation but names no sensor: it is the whole name of one device, assensorsprints it; writecoretemp.
Tested on
- Go tests in
internal/checksandalerts/server_checks_test.go:TestNodeTemperatureCheckcovers the hottest of several sensors, several devices of one name, a sensor without a label, a reading below zero, no sensor of that name, and no sensor at all. - On virtual machines running MMWatchdog, the suite
node_scenarios.pycheckscoretempon a guest without acoretempsensor. The check is unknown withthis machine has no temperature sensor named coretemp;, followed by what the guest has. The guests are virtual machines, so no test reads a real temperature sensor. - The suite and the Go tests passed on every target distribution in the latest run of the whole catalog (validation record).
Why it works this way
-
Unreadable sensors do not prove that hardware is cool. A known failed limit still proves failure when another sensor cannot be read.
-
Temperatures and their limits may be negative. Cold hardware can be below zero Celsius. Allowing negative temperatures does not change the allowed limits for non-temperature measurements.
-
Recording readings is an explicit choice. State
only_record_readings: trueto record readings without alerts or recovery. A missing severity must not silently turn an intended alert into readings only. -
One kind of sensor for each check. Different hardware has different limits: a processor runs hotter than a drive.
-
The highest reading counts. A single hot core or drive is the one that is in trouble; an average would hide it.
-
MMWatchdog reads the kernel's files. The node needs no
sensorsprogram, and the names are those thatsensorsprints, so a reader who knows that program knows the names. -
A machine without the sensor is unknown, not healthy. A check that cannot read a temperature must not say that the hardware is cool.
This page is docs/reference/modules/node_temperature.md in the MMWatchdog repository.