pgbackrest
pgbackrest asks pgBackRest, a backup tool for PostgreSQL, about the
backups of one stanza: pgBackRest's name for the backups of one PostgreSQL
cluster, such as main. The check is healthy when pgBackRest reports the
stanza and its repositories without a problem and the limit below holds.
The check measures how many hours ago the latest backup without errors
finished (latest_backup_age_hours).
The check only asks: it runs pgbackrest info and changes nothing.
To measure how long write-ahead log segments wait to be archived between
backups, use the wal_archive_lag_seconds reading of
postgresql.
PostgreSQL, not pgBackRest, knows which segments wait.
Set up
The check needs pgBackRest set up for the stanza on the node, as its own
documentation describes: the program pgbackrest, its settings in
/etc/pgbackrest/pgbackrest.conf or /etc/pgbackrest.conf, and the stanza
created with pgbackrest stanza-create. A node that runs the backups
already has pgbackrest, pgBackRest's settings, and the created stanza.
MMWatchdog runs pgbackrest as the account postgres, which reads
pgBackRest's settings for the backups too.
Inputs
stanza(string; required): The pgBackRest stanza: pgBackRest's name for the backups of one PostgreSQL cluster, such as main.
Measurements
latest_backup_age_hours(hours): The hours since the latest backup of the stanza without errors finished. A backup of every type counts: full, differential, and incremental.
The module fails a check by itself, and it also reports these measurements. A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check with the module always states an alert_severity. The configuration files reference explains limits in checks that measure a number.
When the check fails, and when it cannot check
- The check fails when pgBackRest reports a problem with the stanza or
with one of its repositories, such as a stanza that the repository does
not have, a repository that it cannot read, or a stanza without a backup.
The check also fails when
pgbackrestfails or does not answer within the check group'stime_limit_per_run. The check also fails when every backup of the stanza found damaged pages, or when the reading passes the configured limit. - MMWatchdog cannot check, so the run returns an
unknownobservation, when:- the node has no
pgbackrest, or MMWatchdog cannot run it as the accountpostgres; pgbackrestgives an answer that MMWatchdog does not understand, or no answer for the stanza.
- the node has no
A failed run starts or continues the check's failing time. An unknown run opens a separate cannot-check incident after the check's waiting times, as the design's failing time describes. The measurement takes a limit like every measurement (checks that measure a number).
Examples
# /etc/mmwatchdog/check_groups/backups.yml
check_groups:
backups:
check_every: 10m
time_limit_per_run: 1m
alert_after_failing_for: 0s
recovered_after_healthy_for: 0s
checks:
a_backup_finished_in_the_last_26_hours:
pgbackrest:
stanza: main
fail_when_measurement_above:
latest_backup_age_hours: 26 # a daily backup, with 2 hours to spare
alert_severity: warn
Every 10 minutes, MMWatchdog on the node runs pgbackrest --stanza=main --output=json info as the account postgres. The check is healthy with a
message such as latest_backup_age_hours is 3.2 (fails above 26); the latest backup of stanza main finished 3.2 hours ago: 20261027-020001F_20261028-020002D, a diff backup. When no backup has
finished for 26 hours, or pgBackRest reports a problem, MMWatchdog sends a
warn alert at once: the reading already holds the time.
time_limit_per_run leaves pgBackRest a minute, because pgBackRest reads a
repository on object storage over the network.
How it works
-
pgBackRest exits with 0 also when the stanza has a problem, and says it in the stanza's status in the answer. A problem of a repository has its reason in the repository's status, and the stanza's status then says only
other. So MMWatchdog builds the check's message from pgBackRest's answer. The message names each repository with a problem and the first line of the repository's reason. For example:pgBackRest reports a problem with stanza main: repo1: missing stanza path.pgBackRest puts details and hints on the lines below the reason.
-
A backup of every type counts: full, differential, and incremental. pgBackRest lists only the backups that finished, and a restore can start from any of them.
-
A backup with errors does not count. pgBackRest marks a backup in which it found damaged pages of the database. The message names such a backup when it is newer than the latest one without errors, as in
...; pgBackRest found damaged pages in the newer 20261027-020001F_20261028-020002D, so the damage is seen at once, and the reading grows until a backup without errors finishes. -
A running backup is named at the end of the message,
; a backup is running now. A reader of a late backup can then see whether a new backup is running. -
The reading is the time since the backup finished, from pgBackRest's stop time of the backup and the node's clock, and never below 0.
-
Each run reads the repository. For a repository on object storage, each run makes a few storage requests, so a
check_everyof minutes, not seconds, fits a backup-age limit of hours.
Errors and what they mean
stanza: expected the name of a pgBackRest stanza, one word, such as main.pgBackRest reports a problem with stanza main: repo1: missing stanza path. The repository has no stanza namedmain. Create the stanza withpgbackrest --stanza=main stanza-create, or name the stanza that pgBackRest's settings have.pgBackRest reports a problem with stanza main: no valid backups. The stanza exists and has no backup. Take a backup withpgbackrest --stanza=main backup.pgBackRest reports a problem with stanza main: repo1: [FileOpenError] unable to load info file .... pgBackRest cannot read its repository.sudo -u postgres pgbackrest --stanza=main infoshows the whole reason.stanza main has no backup without errors; pgBackRest found damaged pages in .... Every backup found damaged pages of the database.pgbackrest --stanza=main --set=LABEL infonames the damage.stanza main has no backup. pgBackRest reports the stanza without a problem and without a backup. Take a backup withpgbackrest --stanza=main backup.pgBackRest did not answer the check: ....pgbackrestended with an error, such as settings that it cannot read. The first line of pgBackRest's error message follows the colon.pgBackRest did not answer within the time limit of the run. The repository, or the network to it, is too slow for the check group'stime_limit_per_run.MMWatchdog cannot run pgbackrest: executable file not found in $PATH. The check needs that program on this node. Install pgBackRest, or check the stanza on the node that runs its backups.pgbackrest gave an answer that MMWatchdog does not understand: .... The run is unknown. The answer follows in the message.
Tested on
Limits on each distribution says what MMWatchdog cannot do on each distribution, or did not test there, and why.
-
Go tests in
server_checks_test.gouse a stand-in forpgbackrestthat gives answers of pgBackRest 2.55.1.TestPgBackRestCheckcovers the arguments and the account. The test covers a full backup followed by a newer differential backup, and a missed daily backup. Further cases cover a newer backup with damaged pages, a running backup, and a stanza where every backup has errors.Repository cases cover a missing stanza, an unreadable repository, and a stanza without a backup. Command cases cover a failing
pgbackrestand an answer that MMWatchdog does not understand. The test also covers stanza validation and a node withoutpgbackrest. -
Real servers: the
servicessuite (service_scenarios.py) sets up pgBackRest for the PostgreSQL server as each distribution installs it, with a repository on the guest's disk. The suite covers a stanza with a full backup, a stanza that the repository does not have, and a repository whose archive directory pgBackRest cannot read, and read again. The suite runs on every target distribution except openSUSE Leap 16.1, whose pgBackRest 2.53.1 does not support its PostgreSQL 18 ("unexpected control version = 1800"), and Rocky Linux 9, AlmaLinux 10, and Fedora 45, whose repositories have no pgBackRest.
Why it works this way
- The check is named for the tool, as MMWatchdog's checks of other tools are, and uses pgBackRest's own word for what it checks, the stanza.
- The check reads
pgbackrest info --output=json, the answer that pgBackRest documents for programs, so the check needs no access to the repository of its own and no key of it. - pgBackRest runs as the account
postgres. pgBackRest's settings hold the keys of the repository and are readable by the accountpostgres, which runs the backups. - Any problem that pgBackRest reports fails the check. A repository that pgBackRest cannot read now is one that the next backup cannot write to.
- A backup with errors does not count as the latest. Restoring a backup with errors brings back the damaged pages. The check's message makes the damage visible at once.
- The write-ahead log's archive is a reading of
postgresql, because the time that a segment has waited is known to PostgreSQL alone:pgbackrest infonames the archived segments, not when they arrived.
This page is docs/reference/modules/pgbackrest.md in the MMWatchdog repository.