Highly available Docker apps - Part 02 - The witness: WireGuard and a three-member etcd

In the previous post I explained the plan: two Docker hosts at two sites, a witness VPS somewhere else, and PostgreSQL with Patroni deciding which host runs each app. Before a single database can be replicated, though, we need the thing that makes the decisions: a three-member etcd cluster, and a network that lets all three members talk to each other.

This is the least exciting part of the whole project and the most important one. If etcd is wrong, every failover is wrong.

What etcd does here #

etcd is a small, consistent key-value store. Patroni (which we'll set up in part 3) uses it for one thing: a leader key per database cluster, with a time to live of 30 seconds. The database member that holds the key is the primary; it renews the key every 10 seconds. If the primary dies, the key expires and the other member, if it's in sync, takes it.

etcd only accepts a write when a majority of its members agree. With three members, that's two. Here's what that means in practice:

That last line is the rule for maintenance: never take two of the three members down at the same time. Rebooting the VPS while a Docker host is being patched is exactly how you get an outage you caused yourself.

The WireGuard link #

The two Docker hosts already reach each other over my site-to-site VPN, so they don't need anything new between them. The VPS, on the other hand, is out on the internet. I gave it a small WireGuard network of its own, 10.99.0.0/29, with the VPS in the middle:

Both of my sites are behind NAT, so the Docker hosts dial out to the VPS and keep the connection alive; the VPS learns their addresses from the handshake. First, the keys. On each machine (as root, and without ever printing the private key):

sudo apt install wireguard-tools
sudo sh -c 'umask 077; wg genkey > /etc/wireguard/dkha0.key; wg pubkey < /etc/wireguard/dkha0.key > /etc/wireguard/dkha0.pub'
sudo cat /etc/wireguard/dkha0.pub

The public keys go into the other side's configuration. On the VPS:

[Interface]
Address = 10.99.0.1/29
ListenPort = 51841
PostUp = wg set %i private-key /etc/wireguard/dkha0.key

[Peer]
# docker-01 (site A)
PublicKey = <public key of docker-01>
AllowedIPs = 10.99.0.2/32, 10.10.0.81/32

[Peer]
# docker-02 (site B)
PublicKey = <public key of docker-02>
AllowedIPs = 10.99.0.3/32, 10.20.0.81/32

And on docker-01 (docker-02 is the same with 10.99.0.3):

[Interface]
Address = 10.99.0.2/29
PostUp = wg set %i private-key /etc/wireguard/dkha0.key

[Peer]
# the VPS
PublicKey = <public key of the VPS>
Endpoint = 203.0.113.10:51841
AllowedIPs = 10.99.0.1/32
PersistentKeepalive = 25

A few things worth explaining:

• PostUp = wg set %i private-key ... - the private key stays in its own file instead of inside the .conf. I like that because I can show a config file to someone (or paste it into a blog post) without thinking twice.
• The VPS allows each host's LAN address too (10.10.0.81/32, 10.20.0.81/32). The etcd members on the Docker hosts listen on their LAN addresses, and the witness talks to them there. Packets from 10.10.0.81 arrive through the tunnel, and WireGuard only accepts a source address that's in that peer's AllowedIPs.
• The Docker hosts allow only 10.99.0.1. The tunnel isn't a general-purpose VPN, it's a wire to one machine.
• PersistentKeepalive = 25 keeps the NAT mapping at each site open, so the VPS can reach the hosts at any time, not only after they've sent something.

Start it everywhere and check the handshake:

sudo systemctl enable --now wg-quick@dkha0
sudo wg show dkha0
ping -c 2 10.99.0.1

latest handshake should be under two minutes, and the ping should answer.

The VPS firewall #

The VPS is on the internet, so its firewall is drop by default. In nftables, in a table of its own:

iif lo accept
ct state established,related accept
meta l4proto icmp accept
iifname "dkha0" ip saddr 10.99.0.0/29 accept     # everything from the two Docker hosts
udp dport 51841 accept                            # WireGuard
tcp dport { 80, 443 } accept                      # the public entrance (part 5)
tcp dport 22 ip saddr @ssh_allowed accept         # SSH only from my own addresses

One rule I follow on this box: never nft flush ruleset. The VPS runs other things too, which own their own tables. Change only your own table.

The etcd members on the Docker hosts #

On the Docker hosts etcd runs as a container. One compose file, the same on both hosts, with the per-host values in a small .env:

services:
  etcd:
    image: quay.io/coreos/etcd:v3.7.2
    container_name: docker-ha-etcd
    restart: unless-stopped
    command:
      - /usr/local/bin/etcd
      - --name=${HA_NODE_NAME:?set HA_NODE_NAME}
      - --data-dir=/etcd-data
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-advertise-peer-urls=http://${HA_NODE_ADDRESS:?set HA_NODE_ADDRESS}:2380
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://${HA_NODE_ADDRESS}:2379
      - --initial-cluster=docker-01=http://10.10.0.81:2380,docker-02=http://10.20.0.81:2380,witness=http://10.99.0.1:2390
      - --initial-cluster-token=docker-ha
      - --initial-cluster-state=${HA_ETCD_CLUSTER_STATE:-new}
      - --auto-compaction-retention=1
    volumes:
      - ./data:/etcd-data
    ports:
      - "${HA_NODE_ADDRESS}:2379:2379"
      - "${HA_NODE_ADDRESS}:2380:2380"

HA_NODE_NAME=docker-01
HA_NODE_ADDRESS=10.10.0.81

• Ports published on the LAN address only (${HA_NODE_ADDRESS}:2379), not on every interface. Docker's port publishing goes around the host firewall, so "where does it listen" is the only control you really have.
• --initial-cluster lists all three members by name and peer URL. It's only used the first time a member starts; after that, the membership lives in the data folder.
• --initial-cluster-state is new for the first start of a brand new cluster. If you ever replace a member later, it must be existing - that's why it's a variable.
• --auto-compaction-retention=1 keeps one hour of history. Patroni writes small keys every 10 seconds, forever; without compaction, the database grows until it hits its quota.

The witness #

On the VPS I didn't want Docker at all - it's a small box with a few important jobs, and fewer moving parts is better. etcd is a single static binary, so it runs as a plain systemd service:

ETCD_VER=v3.7.2
curl -fsSLO https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz
curl -fsSLO https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar xzf etcd-${ETCD_VER}-linux-amd64.tar.gz
sudo install -m 755 etcd-${ETCD_VER}-linux-amd64/etcd /usr/local/bin/etcd
sudo useradd --system --home /var/lib/docker-ha-etcd --shell /usr/sbin/nologin docker-ha-etcd
sudo install -d -m 700 -o docker-ha-etcd -g docker-ha-etcd /var/lib/docker-ha-etcd

[Unit]
Description=etcd for docker-ha (witness)
After=network-online.target [email protected]
Wants=network-online.target [email protected]

[Service]
Type=notify
User=docker-ha-etcd
Group=docker-ha-etcd
ExecStart=/usr/local/bin/etcd \
  --name=witness \
  --data-dir=/var/lib/docker-ha-etcd \
  --listen-peer-urls=http://10.99.0.1:2390 \
  --initial-advertise-peer-urls=http://10.99.0.1:2390 \
  --listen-client-urls=http://10.99.0.1:2389,http://127.0.0.1:2389 \
  --advertise-client-urls=http://10.99.0.1:2389 \
  --initial-cluster=docker-01=http://10.10.0.81:2380,docker-02=http://10.20.0.81:2380,witness=http://10.99.0.1:2390 \
  --initial-cluster-token=docker-ha \
  --initial-cluster-state=new \
  --auto-compaction-retention=1
# etcd reports ready only once the cluster has a majority
TimeoutStartSec=0
Restart=always
RestartSec=5
LimitNOFILE=65536
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/docker-ha-etcd
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

• It listens only on the WireGuard address (and on localhost for etcdctl). Nothing on the public side can reach it.
• Ports 2389 and 2390 instead of the usual 2379 and 2380, because my VPS already runs another etcd for a different project. Two separate clusters, two users, two data folders, no shared state. If your VPS is empty, the standard ports are fine.
• Type=notify with TimeoutStartSec=0 - etcd tells systemd "ready" only when the cluster has a majority. On the very first start that means the unit waits until a second member is up, which is exactly what you want (and why the timeout is off).
• The hardening lines (ProtectSystem=strict and friends) cost nothing: etcd only ever writes into its own data folder.

Gotcha #1 - start all three members before Patroni #

This one is subtle and it cost me seven minutes of outage on an earlier build of this same design, which is why it's the first gotcha of the series.

When an etcd cluster is created, it reports its cluster version as 3.0 until every member listed in --initial-cluster has joined. Patroni looks at that version to decide which etcd API to use, and for 3.0 it picks the old /v3alpha API - which etcd 3.7 doesn't have anymore. Patroni doesn't crash; it just waits. Forever. And it never starts PostgreSQL.

So the order is: WireGuard, then all three etcd members, then check that the cluster is complete, and only then the first Patroni. Start the two Docker hosts' members and the witness:

# on docker-01 and docker-02
cd /srv/ha/etcd && docker compose up -d
# on the VPS
sudo systemctl enable --now docker-ha-etcd

Checking the cluster #

E=http://10.10.0.81:2379,http://10.20.0.81:2379,http://10.99.0.1:2389
docker exec docker-ha-etcd etcdctl --endpoints=$E endpoint health -w table
docker exec docker-ha-etcd etcdctl --endpoints=$E endpoint status -w table
docker exec docker-ha-etcd etcdctl --endpoints=http://10.10.0.81:2379 member list -w table

What you want to see:

• endpoint health: three rows, all true.
• endpoint status: exactly one IS LEADER true and the same RAFT TERM on all three.
• member list: docker-01, docker-02 and witness, all started, none of them a learner.

And the check for gotcha #1. The VERSION column of endpoint status is each member's own version; the cluster version is a separate thing, and you can ask any member for it:

curl -s http://10.10.0.81:2379/version

{"etcdserver":"3.7.2","etcdcluster":"3.7.0","storage":"3.7.0"}

etcdcluster must say 3.7. If it still says 3.0, one of the members hasn't joined yet - don't start Patroni.

If you want to see what Patroni keeps in there later, it's all under /service/:

docker exec docker-ha-etcd etcdctl --endpoints=http://10.10.0.81:2379 get --prefix /service/ --keys-only

Reading is safe; don't write or delete keys by hand, ever. Patroni's own tool (patronictl, part 3) is the way to change anything there.

A word about security #

As configured here, etcd speaks plain HTTP and has no authentication. Anyone who can reach port 2379 on a Docker host, or 2389 on the link, can read and change who the database leader is. The only protection is the network:

• the etcd ports are published on the LAN address only, not on 0.0.0.0;
• the witness listens only on the WireGuard address, and the VPS firewall accepts that link only from the two hosts;
• the site firewalls should allow 2379 and 2380 only between the two Docker hosts and the VPS.

For a homelab where those networks are mine, that's an acceptable trade-off. If your Docker hosts share a network with things you don't trust, add client and peer TLS to etcd (it's well documented) and give Patroni the certificates. It's more work, but it closes that door properly.

Maintenance rules #

Three habits that follow from all of the above:

1. One member at a time. Restart or upgrade one etcd member, wait until endpoint health shows three healthy members again, then the next. The Docker hosts first, the witness last.
2. Never two of three. Don't reboot the VPS while a Docker host is down or being worked on.
3. Never delete the data folder of a running member (docker compose down -v included). A member that lost its data has to be removed and added again with etcdctl member remove / member add and --initial-cluster-state=existing.

Conclusion #

We now have the referee: a three-member etcd across two sites and a VPS, connected by a small WireGuard network, with a clear rule of who may decide what. In the next post we'll put the players on the field: a PostgreSQL 18 image with Patroni, one cluster per app, synchronous replication between the two sites, and the first switchover.

Until next time, stay safe and see you soon!