mdadm_array
mdadm_array watches one software RAID array of Linux, the kind that the
mdadm program manages. The value of the check is the path of the
array, such as /dev/md0. The check is healthy while the array has all its
devices. The check fails when a device is missing or when the node has not
put the array together at all.
MMWatchdog reads the state of the array from the kernel's own files below
/sys/block, and needs no mdadm program for it. For a ZFS pool, use
zfs_pool.
MMWatchdog provides no check for hardware storage controllers.
A shell check or a module of the team's own
can ask the vendor's tool.
Inputs
The module takes one value, which a check writes after the module's name, and no other input.
- the value of
mdadm_array(string; required): The path of the array, such as /dev/md0.
When the check fails, and when it cannot check
The check states an alert_severity and fails at that severity:
- when the array misses a device:
/dev/md0 (raid1) is degraded: 1 of 2 devices work; the kernel reports as faulty: sdb1. After a faulty device was removed, the message ends after1 of 2 devices work. - while a new device is rebuilt. The array misses a device until the
rebuild is done, and the message says how far it is:
/dev/md0 (raid1) is degraded: 1 of 2 devices work; the array is being rebuilt, 50 percent done. - when the node has not put the array together:
/dev/md0 is not an assembled software RAID array on the node. The message about an array the node has not put together also appears for a stopped array and for a path that names no array. - when the kernel reports that the array does not work, such as an
array that is put together but not started:
/dev/md0 does not work: the kernel reports the array as inactive.
A check of the array that the kernel runs from time to time is no failure.
The healthy check's message reports the kernel's array check: /dev/md0 (raid1) has all expected devices: 2 of 2 devices work; the kernel runs a check of the array, 25 percent done.
A device path that cannot resolve fails and names the full configured path.
MMWatchdog never guesses an array from the path's last component. A missing array
fails; an array whose state or degraded count cannot be read is unknown.
An empty, invalid, or negative degraded count is unknown, never zero.
The check's incident follows the check group's
failing time.
Examples
# /etc/mmwatchdog/check_groups/storage.yml
check_groups:
storage:
check_every: 1m
time_limit_per_run: 10s
alert_after_failing_for: 0s
recovered_after_healthy_for: 1m
checks:
data_array_has_all_its_devices:
mdadm_array: /dev/md0
alert_severity: sos
Every minute, MMWatchdog on the node reads the state of
/dev/md0. While the RAID 1 array's two devices work, the check is healthy with
/dev/md0 (raid1) has all expected devices: 2 of 2 devices work.
When the kernel marks a device as faulty, the check fails.
Because alert_after_failing_for is 0s, MMWatchdog sends the sos alert at
once. The check stays failed while the replacement device is rebuilt. Once the array has been whole for 1
minute, MMWatchdog sends the recovery alert.
How it works
- The kernel exposes the array's state in the directory
/sys/block/NAME/md. A path that is a link to an array, such as/dev/md/data, names the array that the link leads to, such asmd0. - A working array is in one of the states
clean,active,active-idle,write-pending,readonly, orread-auto, as the filearray_statesays. Every other state, such asinactive, fails the check. - A device works when the kernel reports it as
in_sync. The message counts devices markedin_syncagainst the number of devices that the array should have (raid_disks), and names each device that the kernel reports asfaulty. The array misses a device while the kernel's count of missing devices (degraded) is above 0. - The progress of a rebuild or of a kernel check comes from
sync_completed. The kernel reports the sectors processed and the total sectors to process for the current rebuild or kernel check.
Errors and what they mean
alert_severity: required: info, warn, or sos. A validation error: the check states noalert_severity.must be the path of a software RAID array, such as /dev/md0. A validation error: the path does not start with/dev/, or is not written in plain form. The validation error starts with the configuration file's path, the line number in the configuration file, and the setting's YAML key path, such ascheck_groups.storage.checks.data_array_has_all_its_devices.mdadm_array:.is not an assembled software RAID array on the noderight after a start of the machine means that the array was not assembled. Look atcat /proc/mdstatand the configuration ofmdadmon the node. On a node that never had the array, the path in the check is wrong.does not work: the kernel reports the array as .... The array exists but is not in use.mdadm --detail /dev/md0says more.
Tested on
- Go tests in
internal/checksandalerts/server_checks_test.go:TestMdadmArrayCheckcovers an array that is whole, one that the kernel checks, one that is degraded with its faulty device, one that is being rebuilt, one that is inactive, one that the node has not put together, and the validation of the path. - In virtual machines (VMs), the suite
node_scenarios.pyinstallsmdadmfrom the distribution's packages and builds a real RAID 1 array on two loop devices. The check is healthy while the array is whole. When a device fails, the check fails and names the faulty device in the result message. The check stays failed while the device is missing, recovers after the rebuild, and fails when the array is stopped. A check of an array never present in the test VM fails with a message about the missing array. - The suite and the Go tests passed on every target distribution in the latest run of the whole test catalog (validation record). The guests are VMs, because the suite loads the kernel's module for RAID.
Why it works this way
-
A missing array differs from unreadable array data. Absence proves a failure; unreadable data cannot prove that an array is whole.
-
The configured device path must resolve. An unrelated array with the configured device path's last component must not make the check healthy.
-
Each check names its array, and MMWatchdog has no check of "every array that is there". An array that is missing after a start of the machine is the worst case. A check of only present arrays would report an overall healthy result because the missing array would never be checked.
-
The check stays failed during a rebuild. Until the rebuild is done, the array still lacks the redundancy of the missing device.
-
MMWatchdog reads the kernel's files and runs no
mdadm. The check needs no program installed on the node beyond MMWatchdog. The kernel files read by the check do not differ between distributions. -
Software RAID is a core module. Arrays of
mdadmare common on Linux servers. Checks for common server needs are built into themmwatchdogprogram as core modules and tested with MMWatchdog (check group decisions).
This page is docs/reference/modules/mdadm_array.md in the MMWatchdog repository.