Highly available Docker apps - Part 06 - Moving the apps in, and everything I learned

The machinery is complete (previous post): etcd decides, Patroni moves the database, the follower moves the app, the sync moves the files, Traefik and DNS move the traffic. Now for the part that actually matters - moving real apps into it. Four apps with a database, three different starting points: PostgreSQL 16, SQLite and MySQL. Plus two apps without any database, and a long list of things I learned about the hosts themselves.

I did the whole move in one night, one app at a time, with a backup and a rollback plan for each. That's the only way I'd recommend doing it.

Before touching any app #

Every app got the same treatment before I changed anything:

D=/srv/backups/$(date +%Y%m%d)-gitea
sudo install -d -m 700 $D
# the database, in PostgreSQL's custom format
docker exec gitea-pg sh -c 'pg_dump -U "$POSTGRES_USER" -Fc gitea' | sudo tee $D/gitea.pgdump >/dev/null
# the files
sudo tar -C /srv/dockers/21-gitea -czf $D/gitea-data.tgz gitea
# the compose file and the env files (they hold secrets: the folder is root-only)
sudo cp -p /srv/dockers/21-gitea/docker-compose.yml /srv/dockers/21-gitea/.env /srv/dockers/21-gitea/.env.ha $D/
# check that the dump is readable
sudo cat $D/gitea.pgdump | docker exec -i gitea-pg pg_restore -l | head

And three rules for the move itself:

1. Keep a copy of the old compose file (docker-compose.yml.pre-ha-<date>) and the old data folder, stopped but untouched, for a week. The way back must stay open until the new setup has proven itself.

2. Compare counts, not feelings. After every conversion I counted the rows of every table, old versus new. "The app starts and looks fine" isn't proof.

3. Switch over both ways before calling it done, with a real user test each time (sign in, open an item, push a commit).

Authentik: PostgreSQL 16 to 18 #

Authentik already used PostgreSQL (16), so this one was "only" a major version jump. An in-place pg_upgrade inside a two-node Patroni cluster is more trouble than it's worth for a database this size, so I went with dump and restore into the new cluster:

# stop the app (its server and worker services), then dump the old database
docker compose stop <server> <worker>
docker exec <old postgres container> sh -c 'pg_dump -U "$POSTGRES_USER" -Fc <database>' | sudo tee $D/authentik.pgdump >/dev/null
# the new cluster's leader is running and empty: restore into it
sudo cat $D/authentik.pgdump | docker exec -i auth-db sh -c 'pg_restore -U "$POSTGRES_USER" -d <database> --no-owner'

The 245 MB dump restored in a few minutes. Then the row counts, every table, old against new - all equal. Then the second member started with an empty folder and copied the 2.8 GB database across the VPN (about 15 minutes from site A).

Three Authentik-specific things to know:

• Gotcha #15 - AUTHENTIK_SECRET_KEY must be identical on both hosts. Authentik encrypts parts of its data with it. Start the app on the other host with a different key and it can't read its own database. Copy .env to the second host, don't generate a new one.

• The outposts (LDAP, RADIUS, remote access) run on both hosts, all the time. They're stateless clients of the Authentik server, so they don't need the profile. LDAP and RADIUS clients can use either host. I also switched off Authentik's Docker integration for them, so that Compose, not Authentik, decides where they run.

• Server, worker and outposts must run the same version. The version is written in the compose file in four places (the server's default tag and three outpost images). One of them is easy to forget during an upgrade.

Upgrading Authentik in this setup goes like this: change the tag on both hosts, pull on both hosts, run docker compose up -d on the standby (that only recreates the outposts there), then on the primary (that recreates the server and worker, which migrate the database on their first start). And because the migration changes the schema, an image rollback is not a rollback - the old version won't start against the new schema. The dump from before the upgrade is the real way back.

Vaultwarden: SQLite to PostgreSQL #

Vaultwarden had been running on SQLite since day one. SQLite can't be replicated by PostgreSQL, so it had to be converted - and this is the conversion I was most nervous about, because it's my password manager.

What worked:

1. Let Vaultwarden create its own schema. Point it at the new, empty PostgreSQL database (DATABASE_URL=postgresql://...) and start it once. It creates every table, index and constraint exactly the way it expects them. Stop it again.

2. Take a consistent copy of the SQLite file - not a cp of a file that's in use, but SQLite's own backup:

   Bash Copy

   sqlite3 data/db.sqlite3 ".backup '/srv/backups/$(date +%Y%m%d)-vaultwarden/db.sqlite3'"

3. Load the data only with pgloader, into the schema Vaultwarden made. A load file along these lines (check the options against pgloader's documentation for your version):

   Bash vaultwarden.load (an example) Copy

   LOAD DATABASE
     FROM sqlite:///work/db.sqlite3
     INTO postgresql://vaultwarden:[email protected]:5441/vaultwarden
   WITH data only, reset sequences, include no drop, create no tables, create no indexes
   EXCLUDING TABLE NAMES LIKE '__diesel_schema_migrations';

4. Compare everything: the row count of every table, and for the tables that matter (the vault items and the keys) the total length of the encrypted contents. Every byte of the encrypted vault had to arrive, and it did.

Why data only? Because pgloader's idea of the schema (types it guesses from SQLite) is not Vaultwarden's idea of the schema, and the next Vaultwarden upgrade runs migrations that expect Vaultwarden's own tables. reset sequences matters too - see gotcha #17 below. And the migrations table is excluded because Vaultwarden already filled it when it created the schema.

The old db.sqlite3 is still in the data folder as a last-resort rollback.

Gotcha #16 - moving image tags. I run Vaultwarden's testing tag. It moves. If I pull on one host today and on the other next week, the two hosts have different versions, and a failover silently downgrades the app against a database the newer version has already migrated. Whatever tag you use: pull on both hosts at the same time and compare the image IDs:

docker image ls vaultwarden/server --format '{{.Tag}} {{.ID}}'      # must be the same on both hosts

Gitea: MySQL to PostgreSQL #

Gitea was on MySQL 8. For this one I trusted the application's own tool more than any generic converter:

docker exec -u git gitea-app gitea dump --database postgres --file /tmp/gitea-dump.zip

That produces a zip with the repositories and a PostgreSQL-flavoured SQL dump of the MySQL database. I loaded the SQL into the new cluster, pointed Gitea at it (GITEA__database__DB_TYPE=postgres and friends) and checked all 111 tables and their sequences.

Gotcha #17 - sequences. After loading data into PostgreSQL by any route other than a pg_dump/pg_restore, check that every sequence is at least as high as the highest ID in its table. If it isn't, everything looks perfect until the first new row, which fails with duplicate key value violates unique constraint. For each table with an id column:

SELECT setval(pg_get_serial_sequence('repository', 'id'), COALESCE(MAX(id), 1)) FROM repository;

The old MySQL container is still in the compose file - in its own profile, mysql-old, so it never starts - with its data folder next to it. A week of rollback insurance, then delete.

Gotcha #18 - an image you can't get back. One of my hosts was running Gitea from an image whose digest is no longer available on Docker Hub (it came from an older host). The same version tag exists, but not that exact build. On that host, docker image prune -a would have deleted something that can't be pulled again. Pin explicit version tags, and pull the same one on both hosts before you rely on a failover.

Two Gitea details that HA makes nice:

• The sync copies Gitea's whole data folder, including its SSH host keys, so both hosts have the same SSH identity - no "host key changed" warning after a failover.

• The first sync of my 7.2 GB of repositories took over an hour and a half at 10 Mbit/s. Later runs only send what changed.

Linkwarden: PostgreSQL 16 to 18 (and the slowest switch) #

Linkwarden was the same story as Authentik: dump and restore, all 18 tables equal. Two small surprises:

• Linkwarden keeps its data in the default postgres database, not in a database of its own. You can't drop that one, so a restore means emptying it table by table (or using pg_restore --clean), not "drop and create".

• It's the slowest to switch: 53 and 86 seconds. The app waits for its search engine (Meilisearch) to start, which waits for its index. If you don't use Linkwarden's search, you can make this faster by removing the dependency.

ntfy and Linkstack: HA without a database #

These two keep only files. ntfy's users, tokens and access rules live in a small SQLite file (auth.db) next to its server.yml; Linkstack is a whole site in one Docker volume. As explained in part 4, they follow Authentik's cluster, and the sync copies their folders every five minutes.

The trade-offs:

• A user change in ntfy reaches the other host up to five minutes later. Before a planned switchover, run the sync by hand.

• ntfy's message cache is not copied. A notification that was waiting to be delivered when the host died is lost. For my use (alerts that also go elsewhere) that's fine.

• They can't be moved on their own: they go where Authentik goes.

Lessons from the hosts themselves #

Not everything that went wrong was about HA. Building two new Docker hosts and moving four old ones into them in one night produced its own collection.

Gotcha #19 - Docker 29 keeps images in containerd, not in Docker's data-root. I had moved Docker's data-root to the big /srv disk like I always do, and the small system disk filled up anyway: 17 GB of images on a 64 GB disk. Recent Docker versions use containerd's image store, which lives under containerd's root. The fix:

root = "/srv/containerd"

[Unit]
RequiresMountsFor=/srv

The drop-ins make sure neither starts before /srv is mounted - otherwise they'd happily create a fresh, empty store on the system disk.

Gotcha #20 - live-restore and moving folders. With "live-restore": true in daemon.json, stopping Docker leaves the containers running - on the old paths. I stopped Docker, moved the folder, deleted the old one... and broke the containers that were still using it. Two outages of about four minutes each before I understood. If you move Docker's or containerd's storage: turn live-restore off first, check with lsof +D <old folder> that nothing uses the old folder, and recreate broken containers with docker compose up -d --force-recreate (a plain restart fails with "wandered into deleted directory").

Gotcha #21 - changing a VM's address. Removing a host's primary IPv4 address with the network tool also removed its secondary address and the default route, and the host was unreachable for four minutes (I got back in through its IPv6 link-local address from the other host). Change an address in one step (nmcli con mod ... && nmcli con up ...) or from the hypervisor's console.

Gotcha #22 - slow storage looks like a network problem. One of my hosts sits on spinning disks, and when the hypervisor was busy, synced writes took 99 to 280 ms instead of 2.6. etcd and Patroni both fsync on every write, so they're the first to complain, and their timeouts look exactly like a flaky link. If you see etcd or Patroni timeouts on one host only, look at the storage before you look at the VPN.

Gotcha #23 - after moving anything, flush caches. Firewalls with a DNS proxy keep the old answer for the TTL; reverse proxies keep keep-alive connections to the old backend; UDP sessions on a firewall keep their old translation. After every move: clear the DNS cache, restart the proxy in front, clear stale sessions.

Living with it #

Four commands tell me almost everything, from either host:

# 1. who leads each database cluster
for c in auth-db vaultwarden-db gitea-pg lw-db; do
  docker exec $c /opt/patroni/bin/patronictl -c /run/patroni/patroni.yml list
done
# 2. the three etcd members
docker exec docker-ha-etcd etcdctl --endpoints=http://10.10.0.81:2379,http://10.20.0.81:2379,http://10.99.0.1:2389 endpoint status -w table
# 3. what the follower decided, and when
sudo journalctl -u docker-ha-follow -n 30 --no-pager
# 4. whether the last file sync worked
sudo journalctl -u docker-ha-sync -n 20 --no-pager

And a few habits:

• Maintenance on a host: switch every app it runs to the other host first, then patch and reboot, then wait until its databases are Sync Standby and streaming again before touching the other host.

• Patroni never fails back by itself. After a failover, apps stay where they landed until I move them back. I keep the public-facing apps at site B normally (that's where the fast uplink is), so after a site B outage, I move them back by hand once I know what happened.

• Replication is not a backup. The sync's --delete and PostgreSQL's replication copy mistakes just as faithfully as data. Backups still run, as before.

• Write it down. Every procedure in this series lives in a manual in the same git repository as the scripts. The night you need it is not the night to figure it out.

All the gotchas in one place #

1. Start all three etcd members before the first Patroni, and check the cluster version (/version, etcdcluster 3.7).

2. Set PGCONNECT_TIMEOUT, or every connection to an unreachable peer waits two minutes.

3. Patroni reads every PATRONI_* variable as configuration: name your own variables differently.

4. A database Patroni didn't create has no replication role: create it in a callback.

5. Set POSTGRES_USER explicitly in every database service.

6. The database service must never load the .env the follower writes, or the follower recreates the new primary.

7. Wants=docker.service, not Requires=, for the follower.

8. The follower never pulls images: pull on both hosts before upgrades.

9. Restricted rrsync needs -no-lock.

10. The sync is not a backup, and after an unplanned failover it can overwrite the newest files on the returning host: stop its timer first.

11. Technitium's app is called Split Horizon, with a space; create A records before CNAMEs.

12. HTTP-01 doesn't work for split-horizon names: use DNS-01, without CNAME following and with the propagation check off.

13. systemctl reload haproxy keeps the old start arguments: restart after changing them.

14. Traefik v3 cuts slow requests after 60 seconds: raise readTimeout.

15. Authentik's secret key must be identical on both hosts.

16. Moving image tags give the two hosts different versions: pull together, compare image IDs.

17. After loading data into PostgreSQL, check the sequences.

18. An image digest that's gone from the registry can't be pulled again: don't prune it, pin versions.

19. Docker 29 stores images under containerd's root: move that too.

20. With live-restore on, containers keep using old paths after Docker stops.

21. Change a VM's address in one step, or you may lose the route with it.

22. etcd and Patroni timeouts on one host: check the storage first.

23. After every move: flush DNS caches, restart proxies, clear stale sessions.

Conclusion #

Every app in this setup was switched to the other host and back before I called it done, and since then the apps have moved between the sites without anyone having to log in anywhere. It's not Kubernetes, and it doesn't pretend to be. It's two Docker hosts, a small VPS, a few hundred lines of configuration and two short scripts - and every part of it is something you can read, understand and fix at 3 in the morning.

If you build something similar, or you have a better solution to one of the gotchas above, leave a comment below. I'd love to hear about it.

Until next time, stay safe and see you soon!