Highly available Docker apps - Part 03 - PostgreSQL with Patroni in Docker

With etcd in place (previous post), it's time for the part that actually holds the data. Every HA app gets its own PostgreSQL cluster: one member on each Docker host, synchronous replication between them, and Patroni deciding which member is the primary.

Patroni is a small Python daemon that runs next to PostgreSQL. It starts and stops PostgreSQL, sets up replication, holds the leader key in etcd, and promotes the standby when the primary goes away. It's what most serious PostgreSQL HA setups use, and the nice thing is that it doesn't care whether it runs on bare metal, in a VM or in a container.

One image for all clusters #

There are ready-made Patroni images, but I wanted something very specific: the official postgres:18-alpine image, unchanged, with Patroni added. So that an app that doesn't need HA can use the same image as plain PostgreSQL, and so that every PostgreSQL detail (users, PGDATA, locales) behaves exactly like the image everybody knows.

FROM postgres:18-alpine

ARG PATRONI_VERSION=4.1.5
ARG PSYCOPG_VERSION=3.3.6

RUN apk add --no-cache python3 \
 && python3 -m venv /opt/patroni \
 && /opt/patroni/bin/pip install --no-cache-dir \
      "patroni[etcd3]==${PATRONI_VERSION}" "psycopg[binary]==${PSYCOPG_VERSION}" \
 && /opt/patroni/bin/pip uninstall -y pip \
 && /opt/patroni/bin/patroni --version

COPY docker-ha-db-start /usr/local/bin/docker-ha-db-start
COPY patroni-config.py /usr/local/bin/docker-ha-patroni-config
COPY patroni-callback.py /usr/local/bin/docker-ha-patroni-callback
RUN chmod 755 /usr/local/bin/docker-ha-db-start /usr/local/bin/docker-ha-patroni-config /usr/local/bin/docker-ha-patroni-callback

ENTRYPOINT ["docker-ha-db-start"]
CMD ["postgres"]

• Patroni lives in its own virtual environment in /opt/patroni, so it never touches the image's system Python.
• pip is removed from the venv afterwards: nothing gets installed in this image at run time.
• Versions are pinned. A Patroni upgrade is a conscious decision (change the ARG, rebuild), not something that happens when the base image moves.

I build this image on each Docker host (docker build -t docker-ha-db:pg18 .) instead of pushing it to a registry. One less dependency: if my registry is down, nothing here cares. The only rule that comes with it: when you rebuild, rebuild on both hosts, so a switchover never changes the PostgreSQL version under an app.

The start script #

#!/bin/sh
set -e

if [ "${DOCKER_HA:-off}" != "on" ]; then
  # HA off: exactly the official image (its entrypoint, the compose command and PGDATA unchanged)
  exec docker-entrypoint.sh "$@"
fi

PGDATA="${PGDATA:-/var/lib/postgresql/18/docker}"
export PGDATA
# The second node starts with an empty data folder: Patroni fills it with pg_basebackup, as postgres,
# so the mounted folder and the version folder under it must belong to postgres
mkdir -p "$PGDATA" /var/run/postgresql /run/patroni
chown postgres:postgres /var/lib/postgresql "$(dirname "$PGDATA")" "$PGDATA" /var/run/postgresql /run/patroni
chmod 700 "$PGDATA"

# Patroni's configuration from the environment (it holds passwords: never in a file on the host)
/opt/patroni/bin/python /usr/local/bin/docker-ha-patroni-config > /run/patroni/patroni.yml
chown postgres:postgres /run/patroni/patroni.yml
chmod 600 /run/patroni/patroni.yml

# Give up on a connection attempt after 5 s instead of the kernel's ~2 minutes
export PGCONNECT_TIMEOUT="${PGCONNECT_TIMEOUT:-5}"

echo "[db] starting Patroni as ${HA_NODE_NAME} (${HA_NODE_ADDRESS})"
exec gosu postgres /opt/patroni/bin/patroni /run/patroni/patroni.yml

DOCKER_HA off

• The script hands over to the official entrypoint and nothing else happens. That's my escape hatch: if Patroni ever gives me grief, I set DOCKER_HA=off, and the same data folder runs as plain PostgreSQL on one host.

The configuration file

• Patroni's YAML is generated at every start from environment variables, and it lives in /run/patroni inside the container only. It contains passwords, so it's never written to the host or the data folder.

PGCONNECT_TIMEOUT=5 - gotcha #2

• Without it, any connection attempt to the other host while that host is unreachable (Patroni checking its peer, pg_rewind, the standby's WAL receiver) waits for the kernel's TCP connect timeout, which is around two minutes. In my tests that meant replication resumed two minutes after the other host came back instead of right away. libpq reads this variable, and everything in the container uses libpq, so one line fixes all of them.

The Patroni configuration #

The generator is a short Python script that turns environment variables into Patroni's YAML. The variables are all called HA_* - gotcha #3: Patroni reads every environment variable that starts with PATRONI_ as configuration, so naming your own variables PATRONI_SOMETHING will change Patroni's behaviour in ways you didn't intend. Here are the parts that matter:

"bootstrap": {
    "dcs": {
        "ttl": 30,
        "loop_wait": 10,
        "retry_timeout": 10,
        # A commit is confirmed once the standby has it, and only a standby in sync is ever promoted.
        # Not strict: with the standby gone the primary carries on alone.
        "synchronous_mode": True,
        "synchronous_mode_strict": False,
        "postgresql": {
            "use_pg_rewind": True,
            "use_slots": True,
            "parameters": {
                "wal_level": "replica",
                "hot_standby": "on",
                "wal_log_hints": "on",
                "max_connections": 100,
                "max_wal_senders": 10,
                "max_replication_slots": 10,
                # notice a silent standby in 10 s instead of a minute
                "wal_sender_timeout": "10s",
                "wal_receiver_timeout": "10s",
            },
        },
    },
    "initdb": [{"encoding": "UTF8"}, "data-checksums"],
},
"postgresql": {
    "listen": f"0.0.0.0:{pg_port}",
    "connect_address": f"{address}:{pg_port}",
    "authentication": {
        "superuser": {"username": superuser, "password": superuser_password},
        "replication": {"username": "replicator", "password": replication_password},
    },
    "basebackup": [{"checkpoint": "fast"}],
    "callbacks": {"on_start": CALLBACK, "on_role_change": CALLBACK},
    "pg_hba": [
        "local all all trust",
        "host all all 127.0.0.1/32 trust",
        "local replication all trust",
        # only the other node: replication, and the superuser for pg_rewind after a failover
        f"host replication replicator {peer}/32 scram-sha-256",
        f"host all {superuser} {peer}/32 scram-sha-256",
        # the app on this node, over the compose network the container is on; nothing else
        "host all all samenet scram-sha-256",
    ],
},

Let's go through the decisions:

ttl: 30, loop_wait: 10

• The leader key lives 30 seconds and Patroni renews it every 10. A dead primary is replaced roughly 30 seconds after it stopped renewing. Shorter would mean faster failovers but also more false alarms over a VPN between two sites; 30 seconds is Patroni's default and I left it.

synchronous_mode: true

• This is the one that makes it safe. A commit is confirmed to the app only when the standby has written it too, and Patroni will only ever promote a standby that is in sync. So a failover never loses a committed transaction.
• The price: every write waits for a round trip to the other site. For Authentik, Gitea or a password manager, a few milliseconds per write are nothing.

synchronous_mode_strict: false

• With strict on, a primary whose standby is gone refuses writes altogether (nothing can be confirmed). With it off, Patroni removes the dead standby from the synchronous list after a short wait and the primary carries on alone. I prefer my apps working during a site outage over a guarantee that I already have through backups.
• That short wait is something to know about: in my tests, when the standby disappeared, commits on the primary hung for about 18 seconds until Patroni dropped it. That's not a hang; it's the synchronous gate doing its job.

use_pg_rewind, wal_log_hints

• After a failover, the old primary may have a few transactions that never reached the new one. pg_rewind rolls it back to the point where the two diverged so it can follow the new primary, instead of copying the whole database again. It needs either data checksums or wal_log_hints; I enabled both.

wal_sender_timeout / wal_receiver_timeout: 10s

• The default is 60 seconds. Until a silent standby is noticed, every synchronous commit waits for it, so I want it noticed quickly.

pg_hba

• The other node may connect for replication and as the superuser (which pg_rewind needs), from its own address only, with SCRAM.
• samenet is the app on the same Docker network. Since the published ports keep the client's real address, nothing else matches, even though the port is published on the LAN.

The bootstrap block is read once. When a cluster is first written to etcd, these settings are stored there, and from then on the copy in etcd wins. To change ttl or a PostgreSQL parameter later, use patronictl edit-config, not the environment variables. Changing the variables and restarting does nothing, which is confusing the first time.

Gotcha #4 - an existing database has no replication user #

Patroni creates its replicator role only when it initialises a brand new cluster with initdb. But I was going to move existing apps into these clusters - their databases already existed. Without that role, the standby can never copy the primary.

The fix is a small callback that Patroni runs whenever a node starts or changes role. On the primary it creates the role (or resets its password, so changing the replication password is just "change the variable on both hosts and restart"); on a standby it does nothing, because the role arrives through replication:

def ensure_role(conn, password):
    with conn.cursor() as cur:
        cur.execute("SELECT pg_is_in_recovery()")
        if cur.fetchone()[0]:
            return "standby, nothing to do"
        cur.execute("SELECT 1 FROM pg_roles WHERE rolname = %s", ("replicator",))
        verb = "ALTER" if cur.fetchone() else "CREATE"
        cur.execute(sql.SQL("{} ROLE {} WITH LOGIN REPLICATION PASSWORD {}").format(
            sql.SQL(verb), sql.Identifier("replicator"), sql.Literal(password)))
        return f"replication role {'updated' if verb == 'ALTER' else 'created'}"

Right after a promotion PostgreSQL may still be leaving recovery, so the real script retries for up to a minute. On the very first start you may see a couple of role "replicator" does not exist lines in the log; they're harmless.

A database service in Docker Compose #

Here's the database part of my Gitea stack. Every HA app looks the same, apart from the names and ports:

services:
  gitea-pg:
    image: docker-ha-db:pg18
    container_name: gitea-pg
    restart: unless-stopped
    stop_signal: SIGTERM
    stop_grace_period: 2m
    env_file: .env.ha
    environment:
      POSTGRES_USER: gitea
      POSTGRES_PASSWORD: ${GITEA_DB_PASSWORD}
      POSTGRES_DB: gitea
      PATRONI_POSTGRESQL_LISTEN: 0.0.0.0:5432
      PATRONI_RESTAPI_LISTEN: 0.0.0.0:8008
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U gitea -d gitea"]
      interval: 10s
      retries: 5
      timeout: 5s
    volumes:
      - ./database-pg18:/var/lib/postgresql
    ports:
      - "${HA_NODE_ADDRESS}:5442:5432"
      - "${HA_NODE_ADDRESS}:8024:8008"
    networks:
      gitea-db-net:

DOCKER_HA=on
HA_SCOPE=gitea
HA_NODE_NAME=docker-01
HA_NODE_ADDRESS=10.10.0.81
HA_PEER_ADDRESS=10.20.0.81
HA_ETCD_HOSTS=10.10.0.81:2379,10.20.0.81:2379,10.99.0.1:2389
HA_PG_PORT=5442
HA_REST_PORT=8024
HA_REPLICATION_PASSWORD=<long random value, the same on both hosts>
HA_REST_PASSWORD=<long random value, the same on both hosts>

• HA_SCOPE is the cluster's name in etcd. One scope per app: authentik, vaultwarden, gitea, linkwarden.
• Only three values differ between the hosts: HA_NODE_NAME, HA_NODE_ADDRESS and HA_PEER_ADDRESS. Everything else, including both passwords and the compose file itself, must be identical.
• Each cluster gets its own pair of ports on the host: PostgreSQL 5440, 5441, 5442, 5443 and the Patroni REST API 8020, 8021, 8024, 8025 (8022 is taken by something else, you'll see in part 5). Inside the container it's always 5432 and 8008; connect_address tells the other node the published port.
• stop_grace_period: 2m - Docker's default is 10 seconds before it kills a container. A primary that's being stopped should get the chance to shut down cleanly.
• env_file: .env.ha, not .env. Every other service of the stack uses .env. The database deliberately doesn't, and there's a story behind that, which I'll tell in the next post because it's the follower's fault.

Gotcha #5 - POSTGRES_USER must be there #

The official image defaults POSTGRES_USER to postgres when it's not set, and some of my old compose files relied on that. The Patroni configuration needs the superuser's name and password explicitly (Patroni connects as it, and so does pg_rewind), so the generator stops with [db] HA: POSTGRES_USER is not set. Set it explicitly in every database service.

Bringing up a cluster #

The first host starts with the existing data folder (or an empty one for a new app), the second one with no data folder at all:

# on docker-01: the first member, which becomes the leader
cd /srv/dockers/21-gitea && docker compose up -d gitea-pg
docker logs -f gitea-pg          # wait for "no action. I am (docker-01), the leader with the lock"

# on docker-02: make sure database-pg18 does NOT exist, then
cd /srv/dockers/21-gitea && docker compose up -d gitea-pg

The second member finds the leader key in etcd, copies the primary with pg_basebackup (that's what checkpoint: fast is for: it starts immediately instead of waiting for the next checkpoint) and starts streaming. For a small database that's seconds; my Authentik database is 2.8 GB, and copying it from site A to site B over the 28 Mbit/s uplink takes about 15 minutes.

Looking at a cluster #

patronictl is your friend. It runs inside the database container and reads the generated configuration:

docker exec gitea-pg /opt/patroni/bin/patronictl -c /run/patroni/patroni.yml list

+ Cluster: gitea ---------------+--------------+-----------+----+-------------+-----+------------+-----+
| Member    | Host            | Role         | State     | TL | Receive LSN | Lag | Replay LSN | Lag |
+-----------+-----------------+--------------+-----------+----+-------------+-----+------------+-----+
| docker-01 | 10.10.0.81:5442 | Sync Standby | streaming |  7 |  0/9A2B3C40 |   0 |  0/9A2B3C40 |   0 |
| docker-02 | 10.20.0.81:5442 | Leader       | running   |  7 |             |     |            |     |
+-----------+-----------------+--------------+-----------+----+-------------+-----+------------+-----+

This is what healthy looks like: one Leader, one Sync Standby that is streaming with a lag of 0. If the second line says Replica instead of Sync Standby, it's replicating but not synchronously, and Patroni will not promote it in a failover. Find out why before you rely on it.

The other commands you'll need:

And the REST API, which the rest of this series relies on: GET http://<host>:8024/primary answers 200 on the primary and 503 everywhere else (and /replica the other way round). Reading needs no password; the calls that change something use HA_REST_PASSWORD. That one URL is all the follower in the next post needs.

How it behaves #

Measured on my setup and on an earlier build of the same design:

Note that Patroni never fails back by itself. If site A died and its apps moved to site B, they stay at site B until you switch them back. That's a feature: you decide when to move things again, after you know why site A died.

Conclusion #

We now have a database per app that survives losing a host, with a clear answer to "who is the primary right now". What we don't have yet is the app following it: the app container still runs wherever I started it, and if it's on the host with the standby, it's talking to a read-only database. That's the job of the follower, and it's the next post.

Until next time, stay safe and see you soon!