pg_auto_failover_node
pg_auto_failover_node manages one PostgreSQL node of a
pg_auto_failover formation.
pg_auto_failover keeps a PostgreSQL primary and its replicas in one
formation and promotes a replica when the primary fails; a monitor on a
server of its own decides, and
pg_auto_failover_monitor
manages it. Each database server runs a keeper, pg_autoctl run, which
manages that server's PostgreSQL: the server is a node of the formation.
The module runs pg_autoctl and psql as the account postgres on the
database server, which is a node of MMX too: a Linux server that MMX
manages. The module manages:
- the node's pg_autoctl instance and PostgreSQL cluster;
- the systemd unit
pgautofailover.service; - two logins in the password file of
postgres; - the node's failover settings and maintenance;
- the address where the node's PostgreSQL listens;
- the local login rules in the data directory's
pg_hba.conf; - with
postgresql_settings, the node's PostgreSQL settings, such asshared_buffers.
The first node of a formation becomes its primary. A later node copies the primary and joins as a replica.
Set up
The database servers need the packages of the monitor's Set up: its first fleet stage runs on every server of the formation, and its second one creates the monitor. The database servers' fleet stage follows:
# mmx run configure, after the fleet stages set_up and monitor
fleet_stages:
- id: database_servers
nodes: database_servers
# One node at a time: the first becomes the primary, and each later one
# copies it.
max_nodes_at_once: 1
node_tasks:
- pg_auto_failover_node:
name: "{{ .node.name }}"
data_directory: "/var/lib/postgresql/{{ .vars.postgresql_major_version }}/main"
postgresql_major_version: "{{ .vars.postgresql_major_version }}"
port: 5432
host_name: "{{ .node.network.default_ipv4_address }}"
listen_address: "{{ .node.network.default_ipv4_address }}"
monitor_host_name: 10.20.0.31
monitor_port: 5432
autoctl_node_password: "{{ .secrets.autoctl_node_password }}"
replication_password: "{{ .secrets.pgautofailover_replicator_password }}"
formation: default
candidate_priority: 50
replication_quorum: true
postgresql_settings:
# A quarter of each server's own memory, in steps of 128 MiB, and
# at least 128 MiB: 2048MB on a server with 8 GiB.
shared_buffers: {percent_of_memory: 25, rounded_down_to_mib: 128, at_least_mib: 128}
work_mem: 64MB
# A replica restarts for shared_buffers; the primary's task refuses
# the restart until another node is the primary.
restart_postgresql_for_new_settings: on_a_replica
# A node that joins copies the primary with pg_basebackup, which
# first waits for a checkpoint on the primary: up to two minutes on
# the test servers. The task also waits for the formation to settle,
# at most five minutes at its start and five at its end.
time_limit: 20m
An address on a WireGuard interface. When host_name and
listen_address are the server's address on a WireGuard network, such as
wg0 of wireguard_interface,
PostgreSQL can bind that address only once the interface is up. A drop-in
of pgautofailover.service starts the service after
wg-quick@wg0.service at boot, on the monitor's server as on each database
server. Place the following systemd_unit task after the
pg_auto_failover_monitor or pg_auto_failover_node task, which writes
the unit that the drop-in changes. Adding the drop-in needs no restart of
the running service:
- systemd_unit:
name: pgautofailover.service
drop_in: 10-after-wireguard.conf
content: |
[Unit]
After=wg-quick@wg0.service
Wants=wg-quick@wg0.service
pg_autoctl tunes PostgreSQL from the server's memory, such as
shared_buffers, in its own file, which the data directory's
postgresql.conf includes at its top. The settings of
postgresql_settings come later and win: the module keeps them in
/etc/postgresql/pg_auto_failover_node_settings.conf, which
postgresql.conf includes at its end. A setting that a line task appends
to postgresql.conf works too, as the page of
pgbackrest_stanza shows for the
archiving settings; for a setting in both, the module's value wins, since
the module keeps its line last.
Inputs
| Input | Meaning | When omitted |
|---|---|---|
data_directory |
The node's PostgreSQL data directory, an absolute path such as /var/lib/postgresql/18/main. The data directory's parent directory must belong to postgres. |
Required. |
state |
present, in_maintenance, absent, or deleted_with_all_its_data. in_maintenance puts a replica into maintenance, and present ends the maintenance. absent stops the node's service, removes the service's unit, then drops the node from its formation. The data directory and pg_autoctl's configuration stay. deleted_with_all_its_data also deletes the data directory and pg_autoctl's configuration (pg_autoctl drop node --destroy). The two removals accept only data_directory. |
present. |
name |
The node's name in the formation, such as the inventory name db01. A dropped node's name cannot be used again at once; a replaced server gets a new name. |
Required for present and in_maintenance. |
postgresql_major_version |
The PostgreSQL version whose programs run the node, such as 18. |
Required for present and in_maintenance. |
port |
The port of the node's PostgreSQL, such as 5432. |
Required for present and in_maintenance. |
host_name |
The host name or IP address with which the other nodes and the monitor reach this node. | Required for present and in_maintenance. |
listen_address |
The IP address of the server on which the node's PostgreSQL listens, such as 10.20.0.21, or * for every address. host_name is the address that the other nodes and the monitor connect to, and listen_address is where PostgreSQL listens: usually the same private address, so that the database is not reachable at the server's other addresses. A changed address takes a restart of pgautofailover.service, which on a primary with replicas makes the monitor fail over: switch the primary first. |
Required for present and in_maintenance. |
monitor_host_name, monitor_port |
Where the monitor answers, as its pg_auto_failover_monitor task gives host_name and port. |
Required for present and in_maintenance. |
autoctl_node_password |
The password of the monitor's login autoctl_node, a secret such as "{{ .secrets.autoctl_node_password }}". |
Required for present and in_maintenance. |
replication_password |
The password of pgautofailover_replicator, with which a replica copies from the primary; the same on every node of the formation. |
Required for present and in_maintenance. |
formation |
The formation the node joins, such as default. |
Required for present and in_maintenance. |
candidate_priority |
A whole number from 0 to 100. The monitor promotes the healthy replica with the highest priority; 0 means never. | Required for present and in_maintenance. |
replication_quorum |
true makes a write on the primary wait for this node while it is a healthy replica: synchronous replication. false lets writes go on without it. |
Required for present and in_maintenance. |
postgresql_settings |
The node's PostgreSQL settings, as a mapping of each setting's name to its value, such as work_mem: 64MB. A name is PostgreSQL's own, in lower case, and may hold one dot, such as pg_stat_statements.max. A value is a text, a number, true or false, which the module writes as on or off, or a share of the server's memory: {percent_of_memory: 25, rounded_down_to_mib: 128, at_least_mib: 128}, as in Set up, is a quarter of the server's memory in MiB, rounded down to a multiple of 128, and at least 128, written such as 2048MB. The module reads the memory from MemTotal in /proc/meminfo, as MMX's fact memory.total_mib does. A value may also be a reference, such as "{{ .vars.work_mem }}". The module writes the settings into its file /etc/postgresql/pg_auto_failover_node_settings.conf, and its values win over pg_autoctl's. {} empties the file. The module refuses a few settings that pg_autoctl or pg_auto_failover keep, such as port: see Why it works this way. |
The module leaves its file alone. |
restart_postgresql_for_new_settings |
What the task does with a setting that takes effect only when PostgreSQL restarts, such as shared_buffers. never only reloads PostgreSQL: the setting waits for a restart, and the output postgresql_settings_waiting_for_a_restart and the reason name it. on_a_replica restarts PostgreSQL on a replica, through maintenance. on_a_replica_or_a_node_alone also restarts a node that is alone in its formation, in place. The task never restarts the primary of a formation with replicas, which would be a failover: it refuses. A node that the task creates restarts with any value, since it serves nobody yet. |
Required with postgresql_settings. |
The two passwords are private values: a secret in either does not hide
the task's result. The module keeps them in the password file of
postgres, ~postgres/.pgpass, mode 0600, where the keeper and the
replica's PostgreSQL find them: *:*:pg_auto_failover:autoctl_node:...
and *:*:*:pgautofailover_replicator:..., with a backslash before each
colon and backslash of a password. The file's other lines stay as they
are. The file reaches install through its standard input, so neither the
passwords nor the file's other lines are in a command line or the result;
install writes it beside the old one as .pgpass.mmx-new, and mv
renames it into place.
On the primary, the replication password reaches psql's \password
through standard input. The psql command sends the server only a
SCRAM-SHA-256 password verifier, the hashed form stored by PostgreSQL.
Outputs
| Output | Meaning |
|---|---|
created |
true when the task created the node. In a dryrun: true when a run would create the node. |
current_state |
The node's state as the monitor reports it after the task, such as single, primary, secondary, catchingup, or maintenance. |
is_primary |
true while the monitor has made the node its formation's primary: its assigned state is single, wait_primary, primary, join_primary, or apply_settings. After a failover, the old primary may still report primary itself; is_primary is then false. |
changed_postgresql_settings |
With postgresql_settings: the sorted names of the settings whose value in the module's file the task changed, added, or removed, such as [shared_buffers]. |
restarted_postgresql |
With postgresql_settings: true when the task restarted the node's PostgreSQL for its settings, or restarted pgautofailover.service because no PostgreSQL ran. |
postgresql_settings_waiting_for_a_restart |
With postgresql_settings: the sorted names of settings in the module's file before or after the task whose values still wait for a restart of PostgreSQL. Settings outside the module's file do not count. |
A later node task reads them as .node_tasks.<id>.is_primary. A dryrun
that would create the node, or start its service, does not know
current_state and is_primary, because the monitor assigns the node its
state only then: a later node task that reads them is not_checked. A
dryrun gives changed_postgresql_settings as a run would change them, and
does not know restarted_postgresql and
postgresql_settings_waiting_for_a_restart.
What a dryrun reports
A dryrun runs pg_autoctl's and psql's read commands and changes nothing.
-
A node to create:
would_change, withwould write the logins of autoctl_node and pgautofailover_replicator into the password file ...; would create the node db01 in DIRECTORY in the formation default; would write the unit ... and start it, andcreated: true, withoutcurrent_stateandis_primary. -
A setting that differs:
would set candidate_priority of the node db01 to 70, and the same forreplication_quorum. -
Maintenance:
would put the node db01 into maintenance, orwould end the maintenance of the node db01. -
New passwords:
would write the logins ... into the password file, and on the primarywould set the password of pgautofailover_replicator. -
initdb's trust lines:
would replace trust in DIRECTORY/pg_hba.conf with peer (local all all, local replication all) and scram-sha-256 (...), with each line that the module would change, andwould reload PostgreSQL in DIRECTORY. -
Another listen address:
would stop pgautofailover.service, set postgresql.listen_addresses of the instance in DIRECTORY from * to 10.20.0.21, and start the service again. -
New PostgreSQL settings:
would set work_mem from 4MB to 64MB in /etc/postgresql/pg_auto_failover_node_settings.conf,would remove ... from ...for a setting that the task no longer names,would add the line include_if_exists = '...' at the end of DIRECTORY/postgresql.confwhere it is missing, andwould reload PostgreSQL in DIRECTORY. A setting whosecontextinpg_settingsispostmastertakes effect only with a restart:would restart PostgreSQL on the replica db02 through maintenance for shared_buffers, or withnevershared_buffers would wait for a restart of PostgreSQL; ....The dryrun uses
postgres -Cto convert the requested text to the setting's PostgreSQL unit, then compares againstpg_settings.setting. A changed setting needs a restart only when the converted value differs from the running value. For example,131072kBequals initdb's128MB, so requesting131072kBdoes not need a restart. Restoring the running value also needs no restart, even when the configuration file currently requests a different value pending restart.An unchanged setting in the module's settings file may still be waiting for a restart after an earlier run. The dryrun names the unchanged setting when reporting the required PostgreSQL restart. While no PostgreSQL runs on the node:
would restart pgautofailover.service so that PostgreSQL starts with them. -
A dryrun does not have PostgreSQL check the settings, and does not look for another file that sets one of them again: a run does both, after it wrote the file, and fails where PostgreSQL refuses a value that its dryrun reported as
would set. -
not_checkedapplies under any of the following conditions:pg_autoctl, the PostgreSQL programs, or the accountpostgresare missing.- The data directory's parent directory does not belong to
postgres. - A cluster managed by Debian's tools uses the requested port.
- The monitor does not answer.
- The formation has not settled:
the formation FORMATION is not settled: ...; a run waits for it, at most 300 seconds. - The task requests
in_maintenanceor a newlisten_addresson the primary. - The settings require a PostgreSQL restart on the primary of a formation with replicas.
- The settings require a PostgreSQL restart on a node alone, and
restart_postgresql_for_new_settingsison_a_replica. - The data directory holds a PostgreSQL cluster, and the formation already has nodes.
-
A failure, as in a run, applies under any of the following conditions:
- An input breaks a rule.
- The task requests a setting that the module refuses.
- A setting that a replica needs at least as high as the primary's value is lower than the primary's value.
- The task requests too few WAL senders or replication slots for the formation.
- The data directory holds a monitor.
- The server's unit runs another instance of
pg_autoctl. - An existing node's name, formation, host name, port, or monitor differs from the task's input.
- The monitor dropped the node.
Examples
A database server that is a replica in normal times, and that the monitor
promotes first when the primary fails. It gives PostgreSQL 2 GiB for
shared_buffers, a quarter of the server's 8 GiB, and the statistics of
pg_stat_statements, which pg_autoctl loads, room for more queries:
- pg_auto_failover_node:
name: db02
data_directory: /var/lib/postgresql/18/main
postgresql_major_version: 18
port: 5432
host_name: 10.20.0.22
listen_address: 10.20.0.22
monitor_host_name: 10.20.0.31
monitor_port: 5432
autoctl_node_password: "{{ .secrets.autoctl_node_password }}"
replication_password: "{{ .secrets.pgautofailover_replicator_password }}"
formation: default
candidate_priority: 90
replication_quorum: true
postgresql_settings:
shared_buffers: 2GB
pg_stat_statements.max: 10000
log_lock_waits: true
restart_postgresql_for_new_settings: on_a_replica
time_limit: 10m
On the replica db02, a new shared_buffers restarts PostgreSQL through
maintenance. While db02 is the primary, as after a failover, the task
refuses that restart, with a reason that says to make another node the
primary first.
The following task puts the replica db01 into maintenance, for example,
for a kernel update. The primary does not wait for db01 during
maintenance, so writes continue. Run the following db01 task with
state: present to end maintenance. After a reboot in maintenance,
db01's PostgreSQL stays stopped until state: present ends maintenance.
- pg_auto_failover_node:
name: db01
state: in_maintenance
data_directory: /var/lib/postgresql/18/main
postgresql_major_version: 18
port: 5432
host_name: 10.20.0.21
listen_address: 10.20.0.21
monitor_host_name: 10.20.0.31
monitor_port: 5432
autoctl_node_password: "{{ .secrets.autoctl_node_password }}"
replication_password: "{{ .secrets.pgautofailover_replicator_password }}"
formation: default
candidate_priority: 50
replication_quorum: true
Remove the node from its formation and keep its data:
- pg_auto_failover_node:
data_directory: /var/lib/postgresql/18/main
state: absent
How it works
- The module writes its two lines into
~postgres/.pgpasswhere they differ. - Without pg_autoctl's configuration for the data directory, the module
first logs in to the monitor once as
autoctl_nodewithpsql(a dryrun does not). The monitor letsautoctl_nodein only from the subnet of its own address (seehost_namein the monitor's inputs), andpg_autoctl create postgrestries a refused login again without end. So the task fails at once when the monitor answersno pg_hba.conf entry. Any other outcome of this login, such as a monitor that does not answer within five seconds, leaves the rest topg_autoctl. Then the module creates the node:pg_autoctl create postgreswith the task's settings,--listen,--auth scram-sha-256, and--ssl-self-signed. The first node of a formation becomes its primary, in the statesingle, in 15 to 20 seconds. A later node copies the primary withpg_basebackupand joins as a replica. - The module writes the unit as
pg_autoctl show systemdprints the unit, and runssystemctl enable --now pgautofailover.service. A configuration without its unit is a creation that stopped partway: the module runspg_autoctl create postgresagain, which finishes it. A first run ofpg_autoctl create postgresthat wrote the node's configuration and failed is run once more at once: for a data directory that pg_autoctl managed before, such as a restored one, its first run refused with "PostgreSQL is already running" on the test servers, and the second one registered the cluster. - It reads the node from the monitor with
pg_autoctl show state --json, once the formation has settled (see "A settled formation" below): it waits for that at most five minutes, and a dryrun does not wait. Withpostgresql_settings, it refuses here a setting that the formation cannot take (see "The PostgreSQL settings" below), and while the node's PostgreSQL does not accept connections, it writes the settings now, so that PostgreSQL starts with them; while no PostgreSQL runs at all, it also restartspgautofailover.service, so that pg_autoctl starts one.presenton a node in maintenance runspg_autoctl disable maintenancenow, before anything that needs the node's PostgreSQL, which pg_autoctl keeps stopped after a reboot in maintenance. Then the module waits, at most a minute, until PostgreSQL accepts connections; a node that stays in maintenance needs no PostgreSQL that runs. - It makes the local logins ask who logs in, on every run (below), and reloads PostgreSQL after a change.
- When pg_autoctl's
postgresql.listen_addressesdiffers fromlisten_address, it stops the service, runspg_autoctl config set postgresql.listen_addresses, starts the service, and waits, at most a minute, untilss -ltnHshows PostgreSQL on the port at that address alone and it accepts connections. On the primary of a formation with other nodes it waits for a switchover instead: the restart would make the monitor fail over. - On the primary it sets the password of
pgautofailover_replicatorwhen the role has none, when the task created the node, or when the password file held another replication password before; then a node that joins later can copy. - It sets
candidate_priorityandreplication_quorumwithpg_autoctl set nodewhere the monitor reports other values. - With
postgresql_settings, it keeps the node's PostgreSQL settings, and restarts PostgreSQL whererestart_postgresql_for_new_settingsallows it (below). in_maintenancerunspg_autoctl enable maintenanceon a replica.- After a change, it waits again, at most five minutes, until the
formation has settled with this node healthy in it: a replica that
joined or left maintenance has caught up as
secondary, and a node put into maintenance reportsmaintenance.
A settled formation. The monitor moves a formation from state to state:
in a failover, a switchover, or while a replica catches up, it assigns each
node its next state and waits for the node to report it. The formation has
settled when every node reports the state that the monitor assigned it,
healthy, and that state is one in which a node rests: single, primary,
secondary, maintenance, or wait_primary while no other healthy node
is out of maintenance. Two kinds of node that the monitor does not see as
healthy have stopped, and the module does not wait for them: a node that
still reports a resting state while the monitor has assigned it another
one, such as a node whose server is down, and a node in maintenance, whose
PostgreSQL may be stopped. Any other node that is not healthy is on its way
between states: while a node's PostgreSQL restarted to rejoin as a replica,
the monitor did not see it as healthy for a few seconds. At its start, a
task also counts its own node as stopped while that node's PostgreSQL does
not answer: the task's later steps handle that, and a node alone in its
formation whose PostgreSQL a setting keeps from starting reports single,
as assigned, and would otherwise never settle. A task
starts on a settled formation, and after a change ends on one, so that in a
fleet stage that runs one node at a time the next node starts on a settled
formation too.
The PostgreSQL settings. With postgresql_settings, the module keeps
the file /etc/postgresql/pg_auto_failover_node_settings.conf, owned by
root and the group postgres, mode 0644.
/etc/postgresql/pg_auto_failover_node_settings.conf holds a comment naming
the module, then one line name = 'value' per setting, sorted by name.
The module doubles each quote and backslash in a value, as PostgreSQL
requires. The data directory's postgresql.conf ends with the line
include_if_exists = '/etc/postgresql/pg_auto_failover_node_settings.conf':
the module adds it, or moves it to the end, whenever another setting line
comes after it. /etc/postgresql/pg_auto_failover_node_settings.conf lies
outside the data directory, so pg_basebackup or pg_rewind copies the
primary's include line but leaves the node's settings file unchanged.
Each node therefore keeps values sized from the node's own memory.
On a node whose PostgreSQL runs, the module:
-
writes the file and the include line where they differ, and keeps their old contents;
-
has PostgreSQL read its whole configuration with
postgres -D DIRECTORY -C port, aspostgres. If PostgreSQL refuses a value, name, or line, the module restores the previous file contents and fails the task (below); -
reads
pg_file_settings, which lists configuration lines in PostgreSQL's reading order. For each requested setting, the module checks which file contains the last line assigning the setting's value. If the last assignment comes from outside/etc/postgresql/pg_auto_failover_node_settings.conf, the module fails the task, naming the overriding file and line. For example,ALTER SYSTEMwritespostgresql.auto.conf, which PostgreSQL reads last. Before failing, the module restores the previous contents of the module's settings file andpostgresql.confwhen changed; -
reloads PostgreSQL with
pg_ctl reload; -
reads
pg_settingsfor settings markedpending_restart. The module checks every setting in the module's settings file before or after the task. The check includes removed settings returning to pg_autoctl's values and unchanged settings left pending by an earlier run withnever. Even with an unchanged settings file, the task restarts PostgreSQL whenrestart_postgresql_for_new_settingsallows a restart.Settings outside the module's settings file do not count. For example, after promotion, pg_autoctl removes the replica's
recovery_target_timeline. PostgreSQL marksrecovery_target_timelinepending until a restart, although no restart is needed; -
restarts PostgreSQL for them where
restart_postgresql_for_new_settingsallows it (below), then checks inpg_settingsthat each of the task's settings comes from the module's file.
The steps run in a run that finds the file unchanged too, apart from the
check with postgres -C and the reload.
A restart of PostgreSQL for a setting. A setting such as
shared_buffers takes effect only when PostgreSQL restarts. pg_autoctl
runs PostgreSQL, so the module restarts PostgreSQL by restarting
pgautofailover.service. Restarting the primary's service causes a
failover: the new keeper reports PostgreSQL as stopped, and the monitor
promotes a replica within two seconds. So the task restarts only where
restart_postgresql_for_new_settings allows it:
- On a replica,
on_a_replicaandon_a_replica_or_a_node_alonewait until the formation has settled with the replica healthy, then runpg_autoctl enable maintenance, wait until the monitor reports the replica in maintenance, restartpgautofailover.service, and runpg_autoctl disable maintenance. In maintenance the primary does not wait for the replica, so writes go on. A restart in maintenance leaves PostgreSQL stopped, anddisable maintenancemakes pg_autoctl start it. The task then waits until PostgreSQL accepts connections and the formation has settled with the replica healthy. A replica that is in maintenance and stays in it, within_maintenance, only restarts; its PostgreSQL starts with the new values whenpresentends the maintenance. - On a node alone in its formation,
on_a_replica_or_a_node_alonerestartspgautofailover.servicein place, and the task waits for PostgreSQL.on_a_replicarefuses. - On the primary of a formation with replicas, the task never
restarts:
on_a_replicaandon_a_replica_or_a_node_alonerefuse. - A refusal puts the module's file and the include line back as they
were, reloads PostgreSQL, and fails the task with a missing
prerequisite, without another try: the earlier settings stay in effect.
MMX reports the task
changed, because it wrote the file and put it back. A dryrun reportsnot_checked. neverreloads only: the settings that need a restart wait, the task succeeds, and its reason says how to apply them, such asshared_buffers waits for a restart of PostgreSQL; run this task with restart_postgresql_for_new_settings: on_a_replica on this node.- A node that the task created restarts in place with every value of
restart_postgresql_for_new_settingsbecause the new node serves nobody yet.
To give a new shared_buffers to every node of a formation, run the
playbook with on_a_replica: each replica restarts, and the primary's task
fails with the reason that says to switch. Make a replica the primary with
pg_auto_failover_primary, then
run the playbook again: the old primary, now a replica, restarts if its
setting still waits. In a fleet stage of one node at a time, the next
node's task waits until the formation has settled after each restart.
Settings that a replica needs as high as its primary's. A replica's
PostgreSQL does not run with max_connections, max_worker_processes,
max_wal_senders, max_prepared_transactions, or
max_locks_per_transaction lower than its primary's. On a replica, the
module reads the primary's values from pg_controldata (the lines
max_connections setting:, max_prepared_xacts setting:, and so on), and
refuses a lower value before it writes anything. On the primary of a
formation with replicas, the module refuses a value lower than the
primary's current value. A lower value would have to reach the primary
first, with a restart that causes a failover. So a higher value goes to the
replicas first, then to the primary; a node alone has no such limit.
max_wal_senders and
max_replication_slots must also be at least the number of nodes in the
formation: each replica takes one of each on the primary.
A setting that PostgreSQL refuses. Before the reload, postgres -C
fails for a value that PostgreSQL refuses, such as shared_buffers: not a size: the task puts the old contents back and fails without another try,
with PostgreSQL's words, such as postgresql_settings.shared_buffers: PostgreSQL refuses it: invalid value for parameter "shared_buffers": "not a size"; .... Nothing changed, though MMX reports the task changed, as
for a refusal. A value that PostgreSQL takes but cannot start with, such
as a shared_buffers larger than the server's memory, passes the check:
then the replica's PostgreSQL does not start after the restart, and the
task fails without another try. The same happens when a value that
PostgreSQL refuses reaches the file another way, such as by hand. The other
nodes' tasks go on: the monitor assigns the replica catchingup while it
still reports secondary, so the formation counts as settled without it.
The node's own task repairs it with a fixed value, on a node alone in its
formation too: while the node's PostgreSQL does not accept connections,
the module writes the file before it waits for PostgreSQL, after the same
check. While no PostgreSQL runs at all, it also restarts
pgautofailover.service, also when the file already holds the fixed
value, and the reason says restarted pgautofailover.service, as the node's PostgreSQL did not run, so that PostgreSQL starts with them:
pg_autoctl may not start PostgreSQL again by itself (see the notes below).
A node whose PostgreSQL does not run serves nobody, a replica or a
primary, so the restart takes nothing from anyone. On a primary of a
formation with replicas, the monitor fails over to a healthy replica, as
for any primary whose PostgreSQL stops, and the node then comes back as a
replica; with no healthy replica, it starts as the primary again.
pg_autoctl may restart PostgreSQL once more as it sets the node up again
as a replica, so the task waits until the formation has settled with the
node healthy before it reads PostgreSQL's settings. In maintenance,
pg_autoctl keeps PostgreSQL stopped until present ends the maintenance,
and PostgreSQL then starts with the file. A fleet stage that runs one
node at a time thus goes on to the next node once the failed node is
repaired.
The local logins. pg_autoctl creates the node with initdb --auth=trust, whose six lines in pg_hba.conf let every account of the
server log in as any role, postgres included, without a password, on the
local socket and on 127.0.0.1 and ::1. For exactly the six
initdb --auth=trust rules, the module replaces trust with Debian's
authentication methods: peer on the socket and scram-sha-256 on the
loopback addresses:
local all all peer
host all all 127.0.0.1/32 scram-sha-256
host all all ::1/128 scram-sha-256
local replication all peer
host replication all 127.0.0.1/32 scram-sha-256
host replication all ::1/128 scram-sha-256
peer lets an account in only as the role of its own name: the keeper and
the module's psql log in as postgres, the account they run as. Every
other line stays as it is, pg_autoctl's own hostssl lines included.
A pg_autoctl hostssl line is
hostssl all "pgautofailover_monitor" <monitor's address>/32 trust: the monitor's health check logs in with it, from the monitor's own
address only, and pg_autoctl gives that role no password. A replica that
copies the primary and a restored cluster also have the six initdb rules
checked and trust replaced with peer or scram-sha-256 as above.
A restored data directory. A data directory that already holds a
PostgreSQL cluster, such as one that
pgbackrest_restore restored, is
registered as it is, as the primary of a formation that has no node.
pg_autoctl create postgres refuses a cluster that runs, so the module
stops one first (pg_ctl stop --mode fast), and the node's service starts
it again. pg_autoctl writes the task's listen_address into the data
directory's postgresql-auto-failover.conf over the address that the
restore brought back from the lost server, so the restored cluster
listens at this server's address. While the formation has nodes, the task
waits: pg_autoctl would replace the cluster with a copy of the primary.
Drop the dead nodes first with the monitor's task.
Synchronous replication needs nothing else: with one replica whose
replication_quorum is true, pg_auto_failover sets
synchronous_standby_names to ANY 1 (pgautofailover_standby_2), and
while that replica is down or in maintenance it clears it, so writes go on.
A database, a login, and pg_hba.conf lines for the applications are
ordinary node tasks: postgresql_database and postgresql_role on the
primary, and line tasks in the data directory's pg_hba.conf, which a
joining node copies from the primary. pg_autoctl adds only its own logins
to pg_hba.conf.
Removal. absent stops and disables the service first, because
pg_autoctl drop node ends a running keeper and systemd would start it
again. The module removes the unit, then drops the node from its formation
where the monitor lists the node. The module's file of PostgreSQL settings
and the data stay. A node that the monitor's task dropped from its side
has a keeper that exits and starts again for ever; absent on that server
stops it. deleted_with_all_its_data runs pg_autoctl drop node --destroy
for a node that the monitor lists. For a node that the monitor no longer
lists, --destroy fails, because it asks the monitor first; the module
then stops the cluster with the pg_ctl of pg_autoctl's configuration and
removes the data directory (rm -r --one-file-system) and pg_autoctl's
configuration and state itself, as --destroy does. It does so only when
the monitor answers: while it does not, a run fails and is tried again.
deleted_with_all_its_data also removes
/etc/postgresql/pg_auto_failover_node_settings.conf.
A monitor built again. When the monitor's server is lost and built
again, its task creates a new, empty monitor at the same address, and the
nodes go on serving without one. A node that the monitor does not list is
then one of two kinds, and the monitor's events tell them apart: a monitor
records events for each node it registers, and keeps those of a node it
dropped, while a new monitor has none of a node it never knew. A node that
the events name is refused as a dropped node. A node absent from the
monitor's events registers through pg_auto_failover's monitor replacement
commands, run as postgres. The module runs
pg_autoctl disable monitor --force, then pg_autoctl enable monitor
with the task's monitor address. The node's
password file already holds the password of autoctl_node.
The primary registers first, and stays the primary: the module asks the
node's PostgreSQL whether it is a replica. After enable monitor it signals
the keeper with pg_autoctl reload and waits up to two minutes for the
keeper's first report to the monitor. A replica registers once the monitor
lists a primary of its formation that has reported, and waits for one up to
five minutes, so that the primary's own task, run at the same time,
registers first; in an empty formation the monitor would make the first node
it registers a primary, and a primary without reports it takes for failed.
MMX starts a fleet stage's nodes in the order of their names, whatever
order the inventory lists them in. So in a fleet stage that runs one node
at a time, such as the one in Set up, a replica whose name comes
before the primary's, as after a failover, starts first: it waits the five
minutes, then fails with a missing prerequisite whose reason names the
address of the primary it receives changes from. With
on_node_failure: continue, the default, the primary's task still runs
later in the playbook run where the replica's task failed. The primary's
task registers the primary. The next playbook run registers the replica.
With stop_fleet_stage or stop_run, the primary's task does not
start. After rebuilding the monitor, use the following run order whether
on_node_failure is continue, stop_fleet_stage, or stop_run.
Run the playbook first with --limit and the primary's node, then run the
whole playbook: the primary
registers without the replica's five-minute wait, and the replica registers
in the second run. A dryrun of the replica does not wait and names the
primary's address. A primary while the monitor lists another
primary fails: two nodes serve as primaries.
A person must choose whether to keep the data from the unregistered
primary or the primary already listed by the monitor.
A node whose pg_autoctl enable monitor failed after
its monitor was disabled keeps PG_AUTOCTL_DISABLED as its monitor; the
next run reads the monitor at the task's address and only enables it.
Errors and what they mean
Every reason starts with pg_auto_failover_node.
| The reason holds | What it means, and what to do |
|---|---|
the program ... is not installed; install the package ... |
The packages of Set up are missing. A run fails without another try. |
the directory DIRECTORY, which holds the data directory, ... |
The data directory's parent is missing or belongs to another account. Create it as a directory of postgres. |
the PostgreSQL cluster VERSION main of Debian's tools uses the port PORT |
Installing postgresql-<version> created Debian's cluster main. Write create_main_cluster = false before the package task, or remove the cluster with pg_dropcluster --stop VERSION main. |
the node NAME is the primary of the formation FORMATION (STATE); maintenance on the primary would cause a failover |
Make another node the primary first with pg_auto_failover_primary, in an earlier fleet stage. A run fails without another try. |
the node NAME is the primary of the formation FORMATION (STATE), and a new listen_address takes a restart of its service |
Make another node the primary with pg_auto_failover_primary in an earlier fleet stage. Then change listen_address on the former primary while the former primary is a replica. |
pg_auto_failover_node.listen_address: must be an IP address of the server |
listen_address is a host name or more than one address. Give one IP address, or *. |
pg_auto_failover_node.postgresql_settings.NAME: PostgreSQL refuses it: ..., or postgresql_settings: PostgreSQL refuses its configuration with these settings: ... |
postgres -C refused the configuration with the task's settings; PostgreSQL's own words follow, such as invalid value for parameter, unrecognized configuration parameter, or is outside the valid range for parameter. The module put its file back, so the earlier settings stay in effect, though MMX reports the task changed. Fix the value. MMX does not try the task again. |
pg_auto_failover_node.postgresql_settings.NAME: FILE sets it again at line LINE, which PostgreSQL reads after /etc/postgresql/pg_auto_failover_node_settings.conf |
Another configuration file sets the setting later, so the task's value would not be live. For postgresql.auto.conf, which ALTER SYSTEM writes, run ALTER SYSTEM RESET NAME in psql on the node; for another file, remove the line there. The module put its file back. |
pg_auto_failover_node.postgresql_settings.NAME: is the module's input ..., ... pg_autoctl writes it ..., ... must be on ... |
The module refuses a setting that pg_autoctl or the module keeps, or a value that pg_auto_failover cannot work with: see Why it works this way. Remove the setting, or use the module's own input. |
pg_auto_failover_node.postgresql_settings.NAME: VALUE is lower than PRIMARY_VALUE, the primary's value, which pg_controldata reports on this replica |
A replica's PostgreSQL does not run with a lower value than its primary's. Keep at least the primary's value. |
pg_auto_failover_node.postgresql_settings.NAME: VALUE is lower than CURRENT, with which the primary NAME runs |
Lowering the value on the primary of a formation with replicas would take a restart of the primary first, which is a failover. Keep the value. |
pg_auto_failover_node.postgresql_settings.max_wal_senders: must be at least COUNT, the number of nodes in the formation FORMATION |
Each replica takes one WAL sender and one replication slot on the primary; the same holds for max_replication_slots. |
the node NAME is the primary of the formation FORMATION, and SETTING takes a restart of PostgreSQL, which on the primary is a failover |
Make another node the primary first with pg_auto_failover_primary, then run the task again, or use never and let the setting wait. The module put its file back; the earlier settings stay in effect. A run fails without another try; a dryrun reports not_checked. |
the node NAME is alone in the formation FORMATION, and SETTING takes a restart of PostgreSQL, which restart_postgresql_for_new_settings: on_a_replica makes only on a replica |
Give on_a_replica_or_a_node_alone to restart a node alone in place. The module put its file back. |
putting the node NAME into maintenance to restart PostgreSQL for SETTING: ... |
The monitor refused maintenance on the replica, such as while the primary waits for a replica (wait_primary); the text behind the colon is pg_autoctl's message. The new file is in place and loaded. MMX tries the task again, and the next run restarts once the formation has settled. |
restarted PostgreSQL ... for SETTING, and a minute later PostgreSQL does not accept connections |
PostgreSQL did not start with the new values, such as a shared_buffers larger than the server's memory. Read the journal that the reason names, fix the value, and run the task again: it writes the file and restarts pgautofailover.service before it waits for PostgreSQL. MMX does not try the task again. |
restarted pgautofailover.service, as the node's PostgreSQL did not run, and a minute later the node's PostgreSQL in DIRECTORY does not accept connections |
After the service restart, PostgreSQL still does not start with /etc/postgresql/pg_auto_failover_node_settings.conf. Alternatively, pg_autoctl needs longer, for example, to copy data from the primary again with pg_basebackup. Read the journal named in the reason. Fix the value, or run the task again once pg_autoctl has copied the data. |
restarted PostgreSQL ..., and PostgreSQL takes SETTING from SOURCE, not from /etc/postgresql/pg_auto_failover_node_settings.conf |
After the restart, a setting of the task has its value from elsewhere, such as from PostgreSQL's command line. Look up the setting named as SETTING in pg_settings and pg_file_settings. MMX does not try the task again. |
set postgresql.listen_addresses to ADDRESS and started pgautofailover.service, and a minute later PostgreSQL listens on port PORT at ... |
PostgreSQL did not take the address, such as an address that is not on the server, or not yet, as a WireGuard address before its interface is up. Read the journal that the reason names. MMX does not try the task again. |
the monitor does not list the node NAME of DIRECTORY in the formation FORMATION: it was dropped from the monitor |
The monitor's events name this node, which it no longer lists: the monitor's task dropped it. pg_autoctl cannot add a dropped node again: remove it with state: deleted_with_all_its_data, and create it again under a new name. |
the monitor at ADDRESS has no record of the replica NAME and lists no primary of the formation FORMATION that has reported to it |
The monitor was built again, and the replica waited five minutes for its primary to register, which in a fleet stage of one node at a time starts after it when its name comes later. The reason names the primary's address when the replica receives changes from it. Run the playbook with --limit and the primary's node, then run it again. See A monitor built again. |
the node NAME is a primary, and the monitor at ADDRESS, which has no record of it, lists OTHER as the primary |
Two nodes serve as primaries, such as after a failover that the old monitor ordered while one node could not hear it. Decide which one keeps the data, and remove the other with state: deleted_with_all_its_data. |
registered the node NAME with the monitor at ADDRESS, but its keeper has sent the monitor no report for 120 seconds |
The keeper did not take its new monitor. Read the journal that the reason names. Until the node reports, the monitor takes the node for failed. |
registering the node NAME with the monitor at ADDRESS: pg_autoctl ... |
pg_autoctl disable monitor or enable monitor failed; the text behind the colon is its message. The next run tries again. |
DIRECTORY holds a PostgreSQL cluster, and the formation FORMATION has the nodes ... |
pg_autoctl would replace the cluster with a copy of the primary. Drop the nodes listed for FORMATION in the reason with the monitor's task first, or empty the directory. |
pg_auto_failover_node.INPUT: the node in DIRECTORY has the ... |
The node was created with another name, formation, host name, port, or monitor, which pg_autoctl does not change. Changing the name, formation, host name, port, or monitor is still refused after state: absent, because absent retains pg_autoctl's configuration. |
DIRECTORY holds a pg_autoctl instance of the role monitor, not a node |
The data directory belongs to a monitor. |
/etc/systemd/system/pgautofailover.service runs the pg_autoctl instance in DIRECTORY |
The server runs another instance of pg_autoctl. One instance runs on a server. |
the formation FORMATION has not settled in 300 seconds: NODE is STATE (assigned STATE, healthy) |
A failover, a switchover, or a replica's catching up went on for more than five minutes, or the formation is stuck, such as a primary that does not reach the state primary, or a replica that does not catch up. Each node that kept the formation from settling is named with the state it reports and the one the monitor assigned it. Read pg_autoctl show state and pg_autoctl show events on the monitor's server. MMX does not try the task again, which would wait as long again; run the playbook again once the formation has settled. |
the monitor at ADDRESS, port PORT refused the login of autoctl_node from this node: its pg_hba.conf has no entry for the node's address ... |
The node shares no subnet with the monitor, such as servers whose addresses are /32 on some cloud providers. The monitor lets nodes in only from the subnet of its own address. Put the monitor's host_name and the nodes on one network that they share, such as a private network or a WireGuard network, as the monitor's inputs describe, and run the task again. The text after psql said: is PostgreSQL's message. MMX does not try the task again. |
the monitor does not answer: ... |
pg_autoctl could not reach the monitor. A run is tried again; a dryrun reports not_checked. |
creating the node ... pg_autoctl wrote the node's configuration; the next run of the playbook runs the creation again, which finishes it |
pg_autoctl create postgres failed after it registered the node, such as during a join's copy. MMX does not try the task again: a new try would report created: false. Act on pg_autoctl's message and run the playbook again. |
created the node, and then this failed: ... |
The task created the node, and a later step failed. MMX does not try the task again. Act on the message and run the playbook again. |
putting the node NAME into maintenance: ..., setting ... of the node ..., dropping the node ... |
A command of pg_autoctl failed; the text behind the colon is its message. MMX tries the task again. |
Tested on
Distributions: Debian 12 and 13, Ubuntu 24.04 and 26.04.
The integration suite tests/integration/pg_auto_failover.py runs
pg_auto_failover_node on two database servers with a monitor on a third,
in system containers on x86_64. The suite covers:
- creation and a join with synchronous replication;
- local logins and replacement of a
trustline that returns; - each node listening at only the node's own address;
- a new
listen_addressand refusal to change the primary's address; - new passwords and failover settings;
- maintenance and refusal of maintenance on the primary;
- tasks ending on a settled formation after a join and after maintenance;
- a restart of the primary's service, followed by a node task during the resulting failover that waits for a settled formation;
- PostgreSQL settings with
neveron each node, withwork_memactive from the module's settings file andshared_bufferswaiting for restart; - a replica restart through maintenance to apply the PostgreSQL settings, and refusal of the settings restart on the primary;
- a value PostgreSQL refuses and a setting overridden by
ALTER SYSTEM; - a replica whose PostgreSQL cannot start with a refused value, then recovery through the replica's task;
- refusal of
max_connectionsbelow the primary's value on a replica; - a node alone refusing
on_a_replica, then restarting in place withon_a_replica_or_a_node_alone; - failover with the old primary rejoining;
- a rebuilt monitor, with the primary registering before the replica;
- a node dropped by the monitor;
absentanddeleted_with_all_its_data.
tests/integration/pgbackrest.py uses pg_auto_failover_node to register a
cluster restored onto a replacement server at the replacement server's
own address, rather than the lost server's address.
| Distribution | pg_auto_failover | PostgreSQL |
|---|---|---|
| Debian 12 | 2.0 | 15 |
| Ubuntu 24.04 | 2.1 | 16 |
| Debian 13 | 2.2 | 17 |
| Ubuntu 26.04 | 2.2 | 18 |
- Not tested: more than two nodes in a formation, Citus formations, and a PostgreSQL that does not come from the distribution.
Why it works this way
-
A node that a new monitor never knew registers, and a dropped one does not. A monitor built again in place of a lost one has no node of the formation, and without its record the database servers ran without automatic failover: the primary went on alone, and the replica's keeper failed on every loop. The monitor's events are the record of what it knew: they name each node that it registered or dropped.
-
The module waits for the keeper's report after
enable monitor.disable monitorandenable monitoreach signal the running keeper to read its configuration. On the test servers the second signal, a second after the first, was lost: the keeper went on without a monitor, the new monitor saw no report from the primary, and minutes later it promoted the replica and made the old primary rewind. So the module waits five seconds between the two, signals once more withpg_autoctl reload, and lets a replica register only beside a primary that has reported. -
A replica without its primary fails, even when the primary comes later in the same fleet stage. A module's task in a run either succeeds or fails. A success would report the formation complete while the replica has no failover, and neither a person nor an agent reading the run would see it. A failure is seen, names the primary's address and the
--limitto run, and happens only when a monitor was built again after a failover put the primary on the later name. MMX's order of a fleet stage's nodes is by name, so a playbook cannot list the primary first. -
A task starts and ends on a settled formation. On two database servers, a playbook restarted the primary's
pgautofailover.serviceto load a setting, which made the monitor fail over. The fleet stage ran one server at a time. In the fleet stage that restarted the primary, the playbook next restarted the replica's service during the replica's promotion. The promotion stopped partway. pg_autoctl started PostgreSQL as a primary on the old timeline, and both servers had written different changes under that one timeline. pg_auto_failover then let the old primary rejoin withoutpg_rewind, and the new primary, which waited for a replica that received nothing, stopped every write for minutes. The wait at a task's start keeps a playbook's later work on a node, such as a restart, from meeting a failover that still goes on. -
A restart of the primary is refused, not turned into a switchover. A switchover is a task of its own,
pg_auto_failover_primary, that a reader of the playbook sees. The module refuses actions that restart PostgreSQL on the primary of a formation with replicas:in_maintenance, a newlisten_address, and a restart forpostgresql_settings. -
The module keeps the PostgreSQL settings, not a playbook's
line,file,assert, andsystemd_servicetasks. Such a recipe restarted a primary in place, which was a failover, and when a setting broke PostgreSQL, the next run waited for the PostgreSQL that the setting kept from starting and could not repair it. -
The settings file lies outside the data directory.
pg_basebackupandpg_rewindcopy the data directory from the primary: a file inside would bring the primary's values, sized from its memory, to every node. -
The include line is the last setting line of
postgresql.conf. On pg_auto_failover 2.0 and 2.2, pg_autoctl included its own files at the top ofpostgresql.conf, so a file included at the end wins over them and over initdb's lines. The copies bring the line along, and the module puts it back at the end on every run. -
PostgreSQL checks the settings with
postgres -C, not withpg_file_settings. On PostgreSQL 15 and 18,postgres -Creadpostgresql.confwith every include andpostgresql.auto.conf, worked while the server ran, and failed for an invalid value, an unknown name, and a syntax error.pg_file_settingssaid "setting could not be applied" both for a valid value that waits for a restart and for an invalid one. -
A setting that another file sets again fails the task. Otherwise the task would report a value that is not live.
ALTER SYSTEMwritespostgresql.auto.conf, which PostgreSQL reads last; there the module's line showed as not applied, without an error. -
The task restarts for every setting of the module's file that waits for a restart, not only for a changed file. A setting can remain pending after a run with
never. A person's configuration change can also leave a setting pending. Removing a setting from the task can require a restart to restore pg_autoctl's value. The next run that allows a restart applies the pending settings. -
A replica restarts through maintenance. With
replication_quorum, the primary waits for the replica unless the replica is in maintenance. On the test servers, a restart in maintenance left PostgreSQL stopped, andpg_autoctl disable maintenancemade pg_autoctl start it. The monitor refused maintenance on a replica while the primary waswait_primary, so the task first waits for a settled formation. -
The module writes the settings before it waits for a PostgreSQL that does not run, and restarts
pgautofailover.servicewhile none runs. Without that order, a replica whose PostgreSQL did not start with a setting could not be repaired by a run. On Debian 13 and Ubuntu 26.04, after ashared_bufferslarger than the server's memory, pg_autoctl started PostgreSQL itself for the crash recovery beforepg_rewind, kept it as a defunct process after it failed, loggedFailed to get Postgres pidevery 30 seconds, and never started PostgreSQL again, with the fixed file too; four seconds after a restart of the service, the replica was healthy.pg_isreadytells a server that does not run (no answer) from one that starts or recovers (it refuses connections for now): only the first gets the restart. pg_autoctl also stops PostgreSQL while it copies the data again withpg_basebackup, which the restart starts over; the task then fails and says to read the journal. -
A replica takes no value lower than its primary's for five settings. PostgreSQL does not run a hot standby whose
max_connections,max_worker_processes,max_wal_senders,max_prepared_transactions, ormax_locks_per_transactionis lower than its primary's. On PostgreSQL 15 and 18,pg_controldataon a replica printed the primary's values. The primary of a formation with replicas lowers none, since a lower value would have to start on the primary first. -
The module refuses settings that pg_autoctl, the monitor, or the module keep:
listen_addressesandport: they are the module's inputslisten_addressandport; pg_autoctl's configuration and the monitor's registration hold them.primary_conninfo,primary_slot_name,recovery_target_timeline, and the otherrecovery_targetsettings: pg_autoctl writes them intopostgresql-auto-failover-standby.confto follow the current primary, and a value from the module's file would win.synchronous_standby_names: pg_autoctl sets it on the primary withALTER SYSTEM, as the monitor decides.hot_standbyother thanon: the keeper and the monitor's health checks connect to a replica's PostgreSQL.wal_levelother thanreplicaorlogical: withminimal, streaming replication andpg_rewindstop working.wal_log_hintsother thanon:pg_rewind, which pg_autoctl runs when an old primary rejoins, needs it.sslother thanon: the replicas and the monitor connect withsslmode=require.password_encryptionother thanscram-sha-256: the module and pg_autoctl createscram-sha-256logins.synchronous_commitset toofforlocal: a commit is then acknowledged before a replica has it, and a failover loses writes that were acknowledged.max_wal_sendersandmax_replication_slotslower than the number of nodes in the formation: each replica takes one of each on the primary.
-
Every other setting is accepted,
shared_preload_librariesincluded. pg_autoctl writespg_stat_statementsintoshared_preload_libraries, but runsCREATE EXTENSION pg_stat_statementsonly when pg_autoctl creates a node. Omittingpg_stat_statementsfromshared_preload_librariesleaves only the views ofpg_stat_statementsunusable. For reference, on pg_auto_failover 2.0 and 2.2,postgresql-auto-failover.confheldshared_preload_libraries,listen_addresses,port,max_wal_senders,max_replication_slots,wal_level,wal_log_hints,wal_sender_timeout,hot_standby_feedback,hot_standby,synchronous_commit, the logging settings,password_encryption, thesslsettings,wal_keep_size, and a block of tuning from the server's memory, such asshared_buffers,work_mem, andeffective_cache_size. -
A share of the server's memory is the module's own form. A task's references in MMX are data, not expressions, so a value cannot work out a quarter of the memory from the fact
memory.total_mib. Memory settings such asshared_buffersare sized from each server's memory, so the module works that out on the node. The value is rounded down to a step, such as 128 MiB, because a new kernel can moveMemTotalby a few MiB, and each new value of a setting that takes a restart is a restart. -
A setting's name is in lower case. PostgreSQL ignores the case of a name, and
pg_settingslists every name in lower case, so one setting has one spelling. -
A dryrun does not check the values.
postgres -Creads the files on the node, and a dryrun writes none. -
The passwords are in
~postgres/.pgpass, not in pg_autoctl's configuration. The keeper reaches the monitor, and a replica's PostgreSQL reaches the primary, with libpq. For each connection, libpq reads~postgres/.pgpassbecause the keeper and PostgreSQL run aspostgres.pg_autoctl config set replication.passwordwould take the password on its command line and keep it in a file that every account of the server can read. -
The replication password is set on the primary without a comparing login. pg_autoctl lets
pgautofailover_replicatorlog in only from the other nodes' addresses, and the loopback addresses need not be open, so the primary cannot log in to itself with the password. The module sets it where it cannot know that the role has the task's password: a role without one, a node that the task created, and a password file that held another one. -
in_maintenanceon the primary is refused. pg_autoctl would fail the primary over to a replica. A planned switchover is its own task,pg_auto_failover_primary, in an earlier fleet stage. -
A new
listen_addresson the primary is refused too. On the test servers the monitor failed over to the replica each time the primary'spgautofailover.servicerestarted. -
is_primaryfollows the monitor's assigned state. After a failover on the test servers, the old primary, whose service had stopped, still reportedprimarywhile the monitor had assigned itdemoted. -
No
truston the local socket and the loopback addresses. With initdb's lines, any account of the server logged in aspostgreson the test servers. The module checks the lines on every run, so a line that comes back is replaced again. -
listen_addresschanges with the service stopped. pg_autoctl gives PostgreSQL the address when it starts it, andpg_autoctl config setwhile the service ran did not keep the value on the test servers: the running pg_autoctl wrote its old value back. pg_autoctl's own connections need no loopback address: with each server's address alone, the keepers, replication, a switchover, and a failover worked. -
The module reads the node's state before
disable maintenance, because a secondpg_autoctl disable maintenancefails. -
presentends the maintenance before it waits for PostgreSQL. After a reboot of a replica in maintenance, as for a kernel update, pg_autoctl kept its PostgreSQL stopped on the test servers untilpg_autoctl disable maintenanceran. -
absentkeeps the data directory. Removing the node from its formation is reversible; deleting the data is spelled out asdeleted_with_all_its_data. -
The unit is written after the creation, never before.
pg_autoctl show systemdprints nothing for a data directory without an instance, and an empty unit file reads as masked.
This page is official_extra_modules/pg_auto_failover_node/README.md in the MMX repository.