pgbackrest

pgbackrest asks pgBackRest, a backup tool for PostgreSQL, about the backups of one stanza: pgBackRest's name for the backups of one PostgreSQL cluster, such as main. The check is healthy when pgBackRest reports the stanza and its repositories without a problem and the limit below holds. The check measures how many hours ago the latest backup without errors finished (latest_backup_age_hours). The check only asks: it runs pgbackrest info and changes nothing.

To measure how long write-ahead log segments wait to be archived between backups, use the wal_archive_lag_seconds reading of postgresql. PostgreSQL, not pgBackRest, knows which segments wait.

Set up

The check needs pgBackRest set up for the stanza on the node, as its own documentation describes: the program pgbackrest, its settings in /etc/pgbackrest/pgbackrest.conf or /etc/pgbackrest.conf, and the stanza created with pgbackrest stanza-create. A node that runs the backups already has pgbackrest, pgBackRest's settings, and the created stanza. MMWatchdog runs pgbackrest as the account postgres, which reads pgBackRest's settings for the backups too.

Inputs

  • stanza (string; required): The pgBackRest stanza: pgBackRest's name for the backups of one PostgreSQL cluster, such as main.

Measurements

  • latest_backup_age_hours (hours): The hours since the latest backup of the stanza without errors finished. A backup of every type counts: full, differential, and incremental.

The module fails a check by itself, and it also reports these measurements. A check sets a limit on a measurement with fail_when_measurement_above or fail_when_measurement_below, and fails when a reading passes its limit. A check with the module always states an alert_severity. The configuration files reference explains limits in checks that measure a number.

When the check fails, and when it cannot check

  • The check fails when pgBackRest reports a problem with the stanza or with one of its repositories, such as a stanza that the repository does not have, a repository that it cannot read, or a stanza without a backup. The check also fails when pgbackrest fails or does not answer within the check group's time_limit_per_run. The check also fails when every backup of the stanza found damaged pages, or when the reading passes the configured limit.
  • MMWatchdog cannot check, so the run returns an unknown observation, when:
    • the node has no pgbackrest, or MMWatchdog cannot run it as the account postgres;
    • pgbackrest gives an answer that MMWatchdog does not understand, or no answer for the stanza.

A failed run starts or continues the check's failing time. An unknown run opens a separate cannot-check incident after the check's waiting times, as the design's failing time describes. The measurement takes a limit like every measurement (checks that measure a number).

Examples

# /etc/mmwatchdog/check_groups/backups.yml
check_groups:
  backups:
    check_every: 10m
    time_limit_per_run: 1m
    alert_after_failing_for: 0s
    recovered_after_healthy_for: 0s
    checks:
      a_backup_finished_in_the_last_26_hours:
        pgbackrest:
          stanza: main
        fail_when_measurement_above:
          latest_backup_age_hours: 26             # a daily backup, with 2 hours to spare
        alert_severity: warn

Every 10 minutes, MMWatchdog on the node runs pgbackrest --stanza=main --output=json info as the account postgres. The check is healthy with a message such as latest_backup_age_hours is 3.2 (fails above 26); the latest backup of stanza main finished 3.2 hours ago: 20261027-020001F_20261028-020002D, a diff backup. When no backup has finished for 26 hours, or pgBackRest reports a problem, MMWatchdog sends a warn alert at once: the reading already holds the time. time_limit_per_run leaves pgBackRest a minute, because pgBackRest reads a repository on object storage over the network.

How it works

  • pgBackRest exits with 0 also when the stanza has a problem, and says it in the stanza's status in the answer. A problem of a repository has its reason in the repository's status, and the stanza's status then says only other. So MMWatchdog builds the check's message from pgBackRest's answer. The message names each repository with a problem and the first line of the repository's reason. For example: pgBackRest reports a problem with stanza main: repo1: missing stanza path.

    pgBackRest puts details and hints on the lines below the reason.

  • A backup of every type counts: full, differential, and incremental. pgBackRest lists only the backups that finished, and a restore can start from any of them.

  • A backup with errors does not count. pgBackRest marks a backup in which it found damaged pages of the database. The message names such a backup when it is newer than the latest one without errors, as in ...; pgBackRest found damaged pages in the newer 20261027-020001F_20261028-020002D, so the damage is seen at once, and the reading grows until a backup without errors finishes.

  • A running backup is named at the end of the message, ; a backup is running now. A reader of a late backup can then see whether a new backup is running.

  • The reading is the time since the backup finished, from pgBackRest's stop time of the backup and the node's clock, and never below 0.

  • Each run reads the repository. For a repository on object storage, each run makes a few storage requests, so a check_every of minutes, not seconds, fits a backup-age limit of hours.

Errors and what they mean

  • stanza: expected the name of a pgBackRest stanza, one word, such as main.
  • pgBackRest reports a problem with stanza main: repo1: missing stanza path. The repository has no stanza named main. Create the stanza with pgbackrest --stanza=main stanza-create, or name the stanza that pgBackRest's settings have.
  • pgBackRest reports a problem with stanza main: no valid backups. The stanza exists and has no backup. Take a backup with pgbackrest --stanza=main backup.
  • pgBackRest reports a problem with stanza main: repo1: [FileOpenError] unable to load info file .... pgBackRest cannot read its repository. sudo -u postgres pgbackrest --stanza=main info shows the whole reason.
  • stanza main has no backup without errors; pgBackRest found damaged pages in .... Every backup found damaged pages of the database. pgbackrest --stanza=main --set=LABEL info names the damage.
  • stanza main has no backup. pgBackRest reports the stanza without a problem and without a backup. Take a backup with pgbackrest --stanza=main backup.
  • pgBackRest did not answer the check: .... pgbackrest ended with an error, such as settings that it cannot read. The first line of pgBackRest's error message follows the colon.
  • pgBackRest did not answer within the time limit of the run. The repository, or the network to it, is too slow for the check group's time_limit_per_run.
  • MMWatchdog cannot run pgbackrest: executable file not found in $PATH. The check needs that program on this node. Install pgBackRest, or check the stanza on the node that runs its backups.
  • pgbackrest gave an answer that MMWatchdog does not understand: .... The run is unknown. The answer follows in the message.

Tested on

Limits on each distribution says what MMWatchdog cannot do on each distribution, or did not test there, and why.

  • Go tests in server_checks_test.go use a stand-in for pgbackrest that gives answers of pgBackRest 2.55.1. TestPgBackRestCheck covers the arguments and the account. The test covers a full backup followed by a newer differential backup, and a missed daily backup. Further cases cover a newer backup with damaged pages, a running backup, and a stanza where every backup has errors.

    Repository cases cover a missing stanza, an unreadable repository, and a stanza without a backup. Command cases cover a failing pgbackrest and an answer that MMWatchdog does not understand. The test also covers stanza validation and a node without pgbackrest.

  • Real servers: the services suite (service_scenarios.py) sets up pgBackRest for the PostgreSQL server as each distribution installs it, with a repository on the guest's disk. The suite covers a stanza with a full backup, a stanza that the repository does not have, and a repository whose archive directory pgBackRest cannot read, and read again. The suite runs on every target distribution except openSUSE Leap 16.1, whose pgBackRest 2.53.1 does not support its PostgreSQL 18 ("unexpected control version = 1800"), and Rocky Linux 9, AlmaLinux 10, and Fedora 45, whose repositories have no pgBackRest.

Why it works this way

  • The check is named for the tool, as MMWatchdog's checks of other tools are, and uses pgBackRest's own word for what it checks, the stanza.
  • The check reads pgbackrest info --output=json, the answer that pgBackRest documents for programs, so the check needs no access to the repository of its own and no key of it.
  • pgBackRest runs as the account postgres. pgBackRest's settings hold the keys of the repository and are readable by the account postgres, which runs the backups.
  • Any problem that pgBackRest reports fails the check. A repository that pgBackRest cannot read now is one that the next backup cannot write to.
  • A backup with errors does not count as the latest. Restoring a backup with errors brings back the damaged pages. The check's message makes the damage visible at once.
  • The write-ahead log's archive is a reading of postgresql, because the time that a segment has waited is known to PostgreSQL alone: pgbackrest info names the archived segments, not when they arrived.

This page is docs/reference/modules/pgbackrest.md in the MMWatchdog repository.