Highly available Docker apps - Part 04 - Making the apps follow their database

After the previous post, each app has a database that lives on both hosts and knows which copy is the primary. Now the app itself has to follow it: it must run on the host where its database is the primary, and only there. A standby database is read-only, so an app running next to it would fail on its first write - or worse, half-work.

Two small Python programs do this on each host: a follower that starts and stops the apps, and a sync that copies their files. Both use only Python's standard library, both are about 250 lines, and both turned out to be much simpler than I expected - once I found the right trick.

The trick: Compose profiles in .env #

Docker Compose has profiles: a service with profiles: ["primary"] is only started when the profile primary is active. And which profiles are active can be set with COMPOSE_PROFILES in the stack's .env file. So I split every HA stack in two:

• the database service has no profile: it runs on both hosts all the time (one primary, one standby);
• every other service gets profiles: ["primary"]: the web app, the worker, the cache.

  gitea-app:
    image: gitea/gitea:1.23.3
    container_name: gitea-app
    profiles: ["primary"]
    restart: unless-stopped
    environment:
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=gitea-pg:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=${GITEA_DB_PASSWORD}
    ports:
      - "${HA_NODE_ADDRESS}:18022:22"
      - "${HA_NODE_ADDRESS}:18030:3000"
    volumes:
      - ./gitea:/data
    labels:
      - traefik.enable=false
    depends_on:
      - gitea-pg
    networks:
      gitea-db-net:

The follower's whole job is then: on the host where the database is the primary, write COMPOSE_PROFILES=primary into .env and run docker compose up -d; on the other host, remove the line and stop the profile's services.

The beautiful side effect: because the profile lives in .env, a plain docker compose up -d typed by a person does the right thing on either host. On the primary it starts the app, on the standby it only touches the database. Nobody has to remember which host is which when they update an app.

Also notice that the app's port is published on ${HA_NODE_ADDRESS} - the host's own LAN address. On the host where the app doesn't run, nothing listens on that port. That's what Traefik's health checks use in part 5 to find the active host.

apps.conf #

Both programs read one small file per host:

[node]
address = 10.10.0.81          # this host
peer = 10.20.0.81             # the other host
interval = 5                  # seconds between rounds
unknown_after = 6             # rounds without an answer before "unknown" counts as standby

[app authentik]
dir = /srv/dockers/24-authentik
rest_port = 8020
sync = media certs custom-templates

[app vaultwarden]
dir = /srv/dockers/06-vaultwarden
rest_port = 8021
sync = data

[app gitea]
dir = /srv/dockers/21-gitea
rest_port = 8024
sync = gitea

[app linkwarden]
dir = /srv/dockers/07-linkwarden
rest_port = 8025
sync = data

# no database of their own: they follow Authentik's cluster (its REST port 8020)
[app ntfy]
dir = /srv/dockers/34-ntfy
rest_port = 8020
sync = config

[app linkstack]
dir = /srv/dockers/39-linkstack
rest_port = 8020
sync = /srv/docker/volumes/39-linkstack_linkstack_data/_data

# static sites run on both hosts; only their files follow Authentik's role
[app static-site]
dir = /srv/dockers/22-static-site
rest_port = 8020
sync = site
manage = no

Two things to point out:

• ntfy and Linkstack have no database, so they can't follow one of their own. Instead they point at Authentik's Patroni REST port. Whenever Authentik moves, they move with it. Any app can follow any cluster this way, which makes "HA" possible for apps that only keep files.
• manage = no means: don't start or stop anything, only use the role to decide which host sends the files. The static sites run on both hosts all the time; the files are copied from the host where Authentik is the primary.

The follower #

Every 5 seconds, for each app, the follower asks the local Patroni whether this host is the primary:

def read_role(address: str, port: int) -> str:
    """primary | standby | unknown, from Patroni's REST API on this node."""
    try:
        with urllib.request.urlopen(f"http://{address}:{port}/primary", timeout=3) as resp:
            return "primary" if resp.status == 200 else "standby"
    except urllib.error.HTTPError as e:
        return "standby" if e.code == 503 else "unknown"
    except (urllib.error.URLError, OSError):
        return "unknown"

Then the rules:

1. A role must be seen twice in a row before anything happens. One odd answer (Patroni restarting, a slow round) never starts or stops an app.
2. unknown counts as standby after 6 rounds (about 30 seconds). If Patroni on this host doesn't answer at all, the app stops here - a primary whose Patroni is gone loses its leader key to the other host anyway.
3. primary: add primary to COMPOSE_PROFILES in .env if it isn't there, then docker compose up -d --no-build, once. As long as nothing changes, nothing more happens.
4. standby: remove the profile from .env and docker compose stop the profile's services. It also stops them if somebody started them by hand on the standby.

Which services belong to the profile? The follower asks Compose itself, with and without the profile, and takes the difference - so it never needs a list of service names:

def profiled_services(self) -> list[str]:
    with_p = set(self._compose("config", "--services", profiles="primary").split())
    without = set(self._compose("config", "--services", profiles="").split())
    return sorted(with_p - without)

And editing .env keeps every other line and only touches COMPOSE_PROFILES. The file is written in place, so it keeps its owner and permissions (it holds the app's secrets):

def set_profile_line(text: str, on: bool, profile: str = "primary") -> str:
    profiles = [p for p in profiles_of(text) if p != profile] + ([profile] if on else [])
    lines = [ln for ln in text.splitlines() if not ln.startswith("COMPOSE_PROFILES=")]
    if profiles:
        lines.append("COMPOSE_PROFILES=" + ",".join(profiles))
    return "\n".join(lines) + "\n"

It only logs when something changes, so a quiet follower is a healthy one:

sudo journalctl -u docker-ha-follow -n 20 --no-pager

gitea: standby (Patroni standby): profile off, stopped gitea-app
gitea: primary (Patroni primary): profile on, started gitea-app

Gotcha #6 - the follower recreated my brand new primary #

This is the story I promised in the last post, and it's the best one of the whole project.

In my first version, the database service loaded .env like every other service of the stack. On my first real switchover, the database moved, the follower on the new primary wrote COMPOSE_PROFILES=primary into .env and ran docker compose up -d... and Docker Compose noticed that the database service's environment had changed (the .env it loads now had a new line). So it did what Compose does: it recreated the database container. Seconds after it had been promoted. Patroni lost its fresh primary, the other host took the leader key back, and the cluster fell back to where it came from.

The fix is the one you've already seen: the database service loads .env.ha, a file with only the HA values that the follower never touches, and the app services load .env. Two rules come out of it:

• Never put the database service on .env. If a value is needed by both (the app's published port uses HA_NODE_ADDRESS), put it in both files.
• docker compose up -d is always run for the whole project. Any service whose definition changed since it was created is recreated at that moment - including the database. If you edit .env.ha or a database image, that recreate happens the next time the follower starts the app. Plan such changes on the standby first.

A cheap way to check what a change would recreate before you make it: docker compose config --hash='*' before and after the edit, and diff them. Only the services you meant to change should differ.

Gotcha #7 - Requires=docker.service #

The follower runs as a systemd service:

[Unit]
Description=docker apps follow their database's HA role
After=docker.service network-online.target
Wants=network-online.target
# Wants, not Requires: with Requires a docker stop (maintenance) stops the follower too and nothing starts it again
Wants=docker.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 -I /usr/local/sbin/docker-ha-follow --config /etc/docker-ha/apps.conf
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

The comment says it all. With Requires=docker.service, systemd stops the follower when Docker stops. That sounds logical, until you stop Docker for maintenance and start it again: Docker comes back, the follower doesn't, and nothing ever starts the apps again after the next switchover. Wants= orders the two but doesn't tie their lives together, and Restart=always takes care of the rest.

Gotcha #8 - the follower never pulls images #

The follower runs docker compose up -d --no-build, and up only pulls an image that's completely missing. That's deliberate: a failover should start what's already there, not wait for a download that may not even work while half of your network is down.

The consequence: when you upgrade an app, pull the new image on both hosts, not only on the one where it runs. Otherwise the next failover either waits for a download or starts the old version against a database the new version has already migrated.

# on BOTH hosts, before upgrading
cd /srv/dockers/21-gitea && COMPOSE_PROFILES=primary docker compose pull gitea-app

(COMPOSE_PROFILES=primary on the command line is needed on the standby, where the profile is off and Compose would otherwise ignore the app's services.)

The sync #

The database is replicated by PostgreSQL, but apps also keep files: Gitea's repositories, Vaultwarden's attachments, Authentik's uploaded icons, Linkwarden's archived pages. Those are copied by the second program, docker-ha-sync, from the primary to the other host:

• It runs from a systemd timer, 5 minutes after the previous run ended (OnUnitInactiveSec=5min). Not "every 5 minutes": a long copy never overlaps the next one. The very first copy of my 7.2 GB of git repositories took over an hour and a half at the bandwidth limit, and the timer simply waited.
• For each app it asks the role first. Only the primary sends; on the standby it logs standby here, nothing to send.
• Each folder is copied with rsync -a --delete --numeric-ids --mkpath --bwlimit=1250, one after the other. 1250 KiB/s is 10 Mbit/s - a third of site A's uplink, so a big sync never eats the whole line.
• An empty source folder is never copied. If a mount is missing on the primary, the folder looks empty, and rsync --delete would happily empty the standby too. One if prevents that disaster.

if not any(os.scandir(src)):
    print(f"{app.name}: {src} is empty - not copied (a missing mount must not empty the standby)")
    continue
cmd = ["rsync", "-a", "--delete", "--numeric-ids", "--mkpath", f"--bwlimit={args.bwlimit}",
       "-e", f"ssh -i {KEY} -o BatchMode=yes -o ConnectTimeout=10",
       "--stats", src + "/", f"root@{cfg.peer}:{dest}/"]

A key that can only do one thing #

The sync connects to the other host as root over SSH, which sounds scary. That's why the key is restricted on the receiving side to exactly one thing:

from="10.10.0.81",command="/usr/bin/rrsync -no-lock /srv",restrict ssh-ed25519 AAAA... docker-ha-sync@docker-01

• from= - only from the other Docker host's address;
• command= - whatever the client asks for, the only thing that runs is rrsync, the restricted rsync that ships with the rsync package, and it only allows paths under /srv;
• restrict - no shell, no port forwarding, no agent forwarding, no TTY.

Gotcha #9: without -no-lock, rrsync takes a lock on /srv for every connection, and the copies failed in my first tests. With one sync at a time, the lock buys nothing.

Gotcha #10 - the sync is not a backup, and it can overwrite your newest files #

Two things to keep in mind:

• --delete mirrors deletions. If somebody deletes a repository at 10:00, it's gone from the standby by 10:05. Replication is not a backup - keep your normal backups.
• After an unplanned failover, files written on the dead host since the last sync exist only there. When it comes back, the new primary's sync copies to it with --delete and overwrites those newer files. If that matters (a push to Gitea in the last five minutes, an upload), stop the timer on the survivor before the dead host comes back: sudo systemctl stop docker-ha-sync.timer, rescue the files, then start it again.

For a planned switchover, run the sync by hand on the current primary first, so the target has everything:

sudo systemctl start docker-ha-sync.service      # runs once and waits until it's done
sudo journalctl -u docker-ha-sync --no-pager -n 15

A planned switchover, end to end #

With the follower and the sync in place, moving an app to the other host is one command plus a few checks:

1. On the current primary: run the sync by hand (above).
2. Make sure the image is pulled on the target (gotcha #8).
3. Switch the database:
   Bash Copy

   docker exec gitea-pg /opt/patroni/bin/patronictl -c /run/patroni/patroni.yml \
     switchover gitea --leader docker-02 --candidate docker-01 --force

4. Watch the followers on both hosts (journalctl -u docker-ha-follow -f). The old primary's follower sees "standby" twice and stops the app; the new primary's sees "primary" twice, writes the profile and starts the app.

What I measured on the real apps, end to end (from the switchover command until the app answered on the other host), both directions:

The database part is always a few seconds. The rest is the two follower rounds (10 seconds) plus however long the app takes to start - which is why Linkwarden, the slowest starter, is the slowest to switch.

Things that surprise people #

• Don't start an HA app by hand on the standby. The follower stops it again within a round or two - and it would fail on the read-only database anyway.
• Edit files on the primary. If you change a file in a synced folder on the standby, the next sync overwrites it.
• A database restart on the primary that takes longer than about 30 seconds stops the app (Patroni doesn't answer, "unknown" becomes "standby"). The follower starts it again as soon as the role is seen again.

Conclusion #

The apps now live wherever their database is the primary, their files follow them, and moving one is a single command. What's left is the traffic: a browser that asks for git.lab.networktechguy.com has to end up at whichever host runs Gitea right now. That's Traefik, split-horizon DNS, and for the public names a little HAProxy on the VPS - next post.

Until next time, stay safe and see you soon!