Highly available Docker apps - Part 01 - The plan: two hosts, two sites and a witness

Everything I self-host runs in docker compose, and for years "high availability" in my homelab meant "I have backups and I know how to restore them". That works right up to the moment the host with your identity provider, your password manager and your git server decides to die while you're on vacation. Every other service that signs in through the identity provider goes down with it, and suddenly a single VM is a single point of failure for half of your network.

So I finally did something about it. In this series I'll show you how I made the important apps highly available across two Docker hosts at two different sites, with automatic failover, without Kubernetes and without Docker Swarm. Just Docker Compose, PostgreSQL with Patroni, a tiny witness VPS, and two small Python scripts. And, as always, every single thing that went wrong along the way.

If you'd rather go the Kubernetes way, I have a separate series on that: Kubernetes in the homelab. This one is for the apps that are perfectly happy in Docker Compose and should stay there.

What "highly available" means here #

Let's be precise, because "HA" means very different things to different people. Here's what I wanted:

• Each app runs on exactly one host at a time (active/passive). Most self-hosted apps were never written to run twice against the same database, and I'm not going to fight that.

• Its database is replicated to the other host, synchronously. When a write is confirmed, it's on both hosts. A failover loses nothing that was committed.

• If the active host dies, the other one takes over by itself, without me logging in anywhere. Seconds to a minute or two of interruption is fine.

• Planned moves are one command, so that I can patch and reboot a host in the middle of the day.

• The two hosts are at different sites. A power cut, an ISP outage or a dead hypervisor at one site shouldn't take everything down.

What it's not: zero downtime. Planned switchovers measured between 13 and 86 seconds depending on the app, and an unplanned failover adds about 30 seconds for the cluster to notice that a host is gone. For a homelab (and honestly, for plenty of small businesses) that's a great deal.

The apps #

Notice the "moved to PostgreSQL" lines. One of the decisions I made early: every HA app gets PostgreSQL, because that's the database I know how to replicate properly. Converting SQLite and MySQL databases was a whole adventure on its own, and it gets its own part in this series.

Everything else (Uptime Kuma, draw.io, Joplin, PrivateBin, Stirling PDF and friends) stays on one host. If they're down for an hour, nobody cares, and HA has a price in complexity that not every app is worth.

The picture #

                       Internet clients
                              |
                       Cloudflare (proxied records)
                              |
              +---------------------------------+
              |  VPS (203.0.113.10)             |
              |  - HAProxy: TLS passthrough     |
              |    by SNI, PROXY protocol v2    |
              |  - etcd member #3 (the witness) |
              +---------------------------------+
                 |  WireGuard          |  WireGuard
                 |  10.99.0.2          |  10.99.0.3
   +--------------------------+   +--------------------------+
   | Site A: docker-01        |   | Site B: docker-02        |
   | 10.10.0.81               |   | 10.20.0.81               |
   | - Traefik                |   | - Traefik                |
   | - etcd member #1         |   | - etcd member #2         |
   | - Postgres + Patroni     |<=>| - Postgres + Patroni     |
   |   (one per app)          |   |   (one per app)          |
   | - the apps whose DB is   |   | - the apps whose DB is   |
   |   the leader HERE        |   |   the leader HERE        |
   +--------------------------+   +--------------------------+
          site-to-site VPN between site A and site B

Let's start by identifying the key components, each of which gets its own post:

• Two Docker hosts, docker-01 at site A (my home) and docker-02 at site B. Debian 13 VMs, Docker 29, the same folder layout on both. They see each other over the existing site-to-site VPN.

• A small VPS at a cloud provider. It has two jobs: it's the third vote that decides which host wins (the witness), and it's the public entrance for the few apps that are reachable from the internet.

• A WireGuard link from each Docker host to the VPS.

• etcd, three members: one per Docker host and one on the VPS. It holds one thing: who is the database leader of each app right now.

• PostgreSQL 18 with Patroni, in one Docker image I build myself. Every HA app gets its own two-node cluster: one member on each host, synchronous replication, automatic failover through etcd.

• A follower: a small Python service on each host that watches the database role of every app and starts the app where its database is the leader (and stops it where it isn't).

• A sync job that copies each app's files (uploads, repositories, attachments) from the active host to the other one every five minutes.

• Traefik on both hosts with health checks, so that a request always ends up at the host where the app is running, plus split-horizon DNS so that clients at each site use their own site's host.

Why not Swarm or Kubernetes? #

Fair question, and I did think about it. Three reasons:

1. Storage. The hard part of HA is never the container, it's the data. Swarm has no answer for replicated volumes. Kubernetes does (Longhorn, Ceph...), but replicating block storage over a site-to-site VPN with a 28 Mbit/s uplink at one end is asking for trouble. Replicating at the database level (PostgreSQL streaming replication) sends only the changes, and it was designed for exactly this.

2. Two sites, not one. Kubernetes wants its nodes close to each other: etcd and the API server don't like a VPN in the middle. Here, the only thing that needs a quorum is a tiny etcd with a handful of keys in it.

3. The apps stay what they are. Every app keeps its own docker-compose.yml, upgrades work the way the app's documentation says, and if this whole construction ever breaks, each host can still run its stacks the old way.

Why a witness? #

With two hosts, there's a problem you can't get around: when host A can't reach host B, it has no way of knowing whether B is dead or whether only the link between them is down. If both decide "the other one is dead, I'm the primary now", you have two primaries writing to two copies of the same database - split brain - and that's the one thing worse than downtime.

The answer is a third vote. etcd needs a majority (two of three) to change anything. If host A loses the link to host B but still sees the witness, A and the witness are a majority and A may stay primary. B, alone, can't take over. If host A really dies, B and the witness are a majority and B takes over. The witness never runs an app or a database; it only votes. That's why a tiny VPS (in my case 2 vCPUs and 4 GB RAM, which is far more than it needs) is enough.

The witness should be somewhere else than both sites. If it lived at site A, a power cut at site A would take two of three votes with it, and site B could never take over.

The three kinds of service #

Once everything is in place, the apps fall into three groups, and they behave differently when a host fails:

And a fourth group, the one-host apps, which are simply down until the host comes back.

Names and addresses used in this series #

So that you can follow along, here's the cast:

Site A has a dynamic public address and an upload of about 28 Mbit/s. Site B has about 466 Mbit/s up. That difference will come back a few times: it decides where things should run normally, and how fast files can be copied from A to B.

What's coming #

• Part 01 - The plan (this post)

• Part 02 - The witness: WireGuard and a three-member etcd across two sites and a VPS

• Part 03 - PostgreSQL with Patroni in Docker

• Part 04 - Making the apps follow their database

• Part 05 - Getting the traffic to the right host: Traefik, split-horizon DNS and a public entrance

• Part 06 - Moving the apps in: Authentik, Vaultwarden, Gitea and Linkwarden, plus everything I learned

The order matters, by the way. I built it in exactly this order, and the first gotcha of the whole project is in the next post: if you start Patroni before etcd is complete, it waits forever.

Until next time, stay safe and see you soon!