Highly available Docker apps - Part 05 - Getting the traffic to the right host

At this point (previous post), every app runs on whichever host holds its database leader, and moves when the leader moves. The last piece of the puzzle is the traffic: when I open https://git.lab.networktechguy.com, my browser has no idea which host runs Gitea right now, and it shouldn't have to.

There are two kinds of visitors, and they take different paths:

In both cases, the request ends up at a Traefik, and it's Traefik's job to find the host where the app actually runs. Let's start there.

Traefik: two servers and a health check #

Both hosts run Traefik with exactly the same configuration, so either host can serve any name. For each HA app there's a file route (not Docker labels - the app's container exists on only one host at a time, but the route must exist on both). Here's Gitea's:

http:
  routers:
    ha-gitea:
      rule: "Host(`git.lab.networktechguy.com`)"
      service: ha-gitea
      entryPoints: [websecure]
      tls:
        certResolver: letsencrypt
  services:
    ha-gitea:
      loadBalancer:
        passHostHeader: true
        healthCheck:
          path: /api/healthz
          interval: 5s
          timeout: 3s
        servers:
          - url: "http://10.20.0.81:18030"
          - url: "http://10.10.0.81:18030"

Remember from the last post that the app publishes its port on the host's LAN address only while it runs there. So on the host where Gitea is stopped, nothing listens on 18030, the health check fails, and Traefik sends no traffic there. Exactly one of the two servers passes at any time, and that's where every request goes - from either host's Traefik. A request that lands at site A for an app that runs at site B simply crosses the site-to-site VPN.

During a switchover there's a short window in which neither server passes (the app is stopped on the old host and not yet up on the new one). Traefik answers 503 for those seconds. That's the interruption you measured in the switchover table, and there's no way around it with active/passive apps.

Each app needs a cheap health URL that only answers 200 when the app really works:

Internal clients: split-horizon DNS #

At home I don't want my traffic to go to Cloudflare and back, and the internal apps (Gitea, Vaultwarden, Linkwarden, ntfy) aren't on the internet at all. So internally, the names resolve to a Docker host directly - but which one depends on where I am:

• clients at site A get docker-01 (10.10.0.81),
• clients at site B get docker-02 (10.20.0.81).

My internal DNS is Technitium, which has an app for exactly this: Split Horizon. You add an APP record that answers differently per client network. I use its SimpleCNAME variant, whose record data maps client networks to the name they should get - roughly like this:

{
  "10.10.0.0/16": "docker-01.lab.networktechguy.com",
  "10.20.0.0/16": "docker-02.lab.networktechguy.com"
}

The nice part: internal clients never touch the VPS. If the VPS or the internet connection is down, everything still works at home, because my own site's Traefik reaches the other site over the VPN.

Gotcha #11 - "Split Horizon" has a space in it #

Two small things that bit me when I created the records through Technitium's API:

• The application name is Split Horizon, with a space. With SplitHorizon, the record was accepted and the name answered SERVFAIL for about two minutes until I noticed.
• Create the target's A record first (docker-02.lab.networktechguy.com), and only then point CNAMEs at it. In my first attempt I pointed seven names at a host record that didn't exist yet, and for about 50 seconds all seven resolved to nothing.

Gotcha #12 - HTTP-01 and split horizon don't mix #

For internal names I normally use my own CA with ACME (I wrote about that in my PKI series), with the HTTP-01 challenge. It didn't work here, and once you see why, it's obvious: with HTTP-01, the CA connects to the name and asks for a token. But the name resolves to a different host depending on where the CA is. My CA lives at one of the two sites, so it always reached that site's host - even when the certificate had been requested by the other host's Traefik, which then never saw the challenge.

The fix: these routes use Let's Encrypt with the DNS-01 challenge through Cloudflare, which doesn't care where a name resolves - it proves ownership with a TXT record in the public zone. Two more details made it work:

[certificatesResolvers.letsencrypt.acme]
  storage = "/acme.json"
  [certificatesResolvers.letsencrypt.acme.dnsChallenge]
    provider = "cloudflare"
    resolvers = ["<internal DNS 1>:53", "<internal DNS 2>:53"]
    [certificatesResolvers.letsencrypt.acme.dnsChallenge.propagation]
      delayBeforeChecks = "60s"
      disableChecks = true

environment:
  CLOUDFLARE_DNS_API_TOKEN: ${CLOUDFLARE_DNS_API_TOKEN}
  # internal DNS answers with CNAMEs into zones Cloudflare does not hold: follow nothing, ask Cloudflare directly
  LEGO_DISABLE_CNAME_SUPPORT: "true"

• LEGO_DISABLE_CNAME_SUPPORT: the ACME library in Traefik (lego) follows CNAMEs when it looks for the zone to put the TXT record in. Internally, my names are CNAMEs (to docker-01/docker-02), so it went looking in the wrong place. With this off, it puts the record where the name itself lives in Cloudflare.
• disableChecks with a 60-second delay: before telling Let's Encrypt to look, Traefik normally checks that the TXT record is visible. Through my internal resolvers it never is - internally, the zone answers with its own records. So Traefik waits 60 seconds and lets Let's Encrypt check from the outside.

Public names: HAProxy on the VPS #

A few names must be reachable from the internet - Authentik, for example, because I sign in from my phone and from apps outside my network. For those, the public DNS record (at Cloudflare, proxied) points at the VPS, and HAProxy there passes the connection on to one of the two hosts.

The interesting part: HAProxy never decrypts anything. It reads only the server name from the TLS handshake (SNI), picks a backend, and passes the encrypted stream through. The certificates stay on the Docker hosts, and the VPS never sees a password or a cookie.

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5s
    timeout client 1h
    timeout server 1h
    timeout tunnel 1h

frontend https_in
    bind :443
    bind :::443 v6only
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    tcp-request content reject
    use_backend %[req.ssl_sni,lower,map(/etc/haproxy/docker-ha.map)] if { req.ssl_sni,lower,map(/etc/haproxy/docker-ha.map) -m found }
    default_backend refuse

backend refuse
    tcp-request content reject

frontend http_in
    mode http
    bind :80
    bind :::80 v6only
    timeout client 10s
    timeout http-request 10s
    http-request redirect scheme https code 301

backend dk_b_first
    balance first
    default-server check inter 3s fall 2 rise 2 send-proxy-v2 on-marked-down shutdown-sessions
    server docker-02 10.99.0.3:443
    server docker-01 10.99.0.2:443

backend dk_a_first
    balance first
    default-server check inter 3s fall 2 rise 2 send-proxy-v2 on-marked-down shutdown-sessions
    server docker-01 10.99.0.2:443
    server docker-02 10.99.0.3:443

idp.lab.networktechguy.com dk_b_first
www.lab.networktechguy.com dk_b_first      # a static site

How it works:

• tcp-request inspect-delay and req.ssl_hello_type 1 - wait until the TLS hello has arrived, so the server name can be read. Anything that isn't TLS is rejected.
• The map decides which names are allowed at all. A name that's not in it goes to refuse, and the connection is dropped right after the hello. So pointing a random DNS record at the VPS doesn't expose anything.
• balance first sends everything to the first healthy server. Since both hosts' Traefik can serve every app (they forward to the active host themselves), the order only saves a hop across the VPN. Site B has the much faster uplink, so most names go there first.
• check inter 3s fall 2 - a TCP check of Traefik's port 443 every 3 seconds; a host that fails twice is out after about 6 seconds, and on-marked-down shutdown-sessions closes its open connections so clients reconnect to the other one.
• send-proxy-v2 - see the next section.
• The connections go over the WireGuard link from part 2 (10.99.0.2/.3), not over the internet. Neither site needs a single port forward for this.

Gotcha #13 - reload versus restart #

I keep my backends in their own file, docker-ha.cfg, which HAProxy loads with a second -f option in /etc/default/haproxy:

EXTRAOPTS="-S /run/haproxy-master.sock -f /etc/haproxy/docker-ha.cfg"

After adding that line, I ran systemctl reload haproxy... and nothing changed. A reload starts new workers with the start arguments the master process already has. A change to the map or to the config files only needs a reload; a change to the start arguments needs a restart - which also cuts every open connection, so pick a quiet moment.

The client's real address #

With two proxies in front of an app (Cloudflare and HAProxy), the app would see HAProxy's address for every request, which makes logs useless and IP-based rules dangerous. Two mechanisms carry the real address through, and Traefik has to be told exactly whom to believe:

[entryPoints.websecure]
  address = ":443"
  # who may tell us the client address: Cloudflare's published ranges (and localhost)
  [entryPoints.websecure.forwardedHeaders]
    trustedIPs = ["127.0.0.1/32", "173.245.48.0/20", "103.21.244.0/22", "...the rest of Cloudflare's list..."]
  # PROXY protocol only from the public entrance, nobody else
  [entryPoints.websecure.proxyProtocol]
    trustedIPs = ["127.0.0.1/32", "10.99.0.1/32"]
  # long uploads through here
  [entryPoints.websecure.transport.respondingTimeouts]
    readTimeout = "3600s"

• PROXY protocol is the header HAProxy puts in front of the TLS stream (send-proxy-v2). It tells Traefik "this connection really came from address X". Traefik accepts it only from the VPS's link address; from anyone else it would be a way to fake your address.
• Because the public names are proxied by Cloudflare, X is a Cloudflare address. Cloudflare then sends the visitor's address in X-Forwarded-For, and Traefik believes that header only from Cloudflare's published ranges.

Two things I've learned to never do here: never set forwardedHeaders.insecure = true, and never trust "all of 10.0.0.0/8" because it's easier. Either one lets anybody on that network claim any address they like.

Gotcha #14: the readTimeout. Traefik v3 cuts a request whose body takes longer than 60 seconds to arrive. That's invisible until somebody uploads a big attachment over a slow line, or pushes a big repository, and gets a reset halfway. An hour is plenty for my uses.

Keeping internal names internal #

One risk with all this: Traefik serves every name it has a route for, on every connection that reaches it. If someone guesses an internal name and sends a request for it to the VPS... HAProxy's map stops them. But I also wanted a second wall on the hosts themselves.

So I wrote a small script that runs every five minutes on both hosts. It reads every host name from Traefik's API and asks two public resolvers (1.1.1.1 and 8.8.8.8) about each one. For every name without a public DNS record, it writes a router with a very high priority that answers 403 to every client outside the private address ranges. A name becomes reachable from the internet only when I give it a public record on purpose. If a lookup fails, the previous decision stays, so a DNS hiccup never opens anything.

Bonus: git over SSH #

Gitea also speaks SSH, which Traefik (an HTTP proxy) doesn't help with. Same trick in miniature: a tiny HAProxy container on both hosts listens on port 8022 and forwards to whichever host has Gitea's SSH port (18022) open:

defaults
    mode tcp
    timeout connect 5s
    timeout client 1h
    timeout server 1h
frontend ssh_in
    bind :8022
    default_backend gitea_ssh
backend gitea_ssh
    balance first
    default-server check inter 3s fall 2 rise 2 on-marked-down shutdown-sessions
    server docker-02 10.20.0.81:18022
    server docker-01 10.10.0.81:18022

Because the sync copies Gitea's whole data folder, including its SSH host keys, both hosts present the same host key - so after a failover, my git clients don't throw a "host key changed" warning at me.

Tracing a request that fails #

When a name doesn't work, I go through the path in this order:

1. What does the name resolve to, for this client? dig +short git.lab.networktechguy.com from the client, and dig +short @1.1.1.1 ... for the public view.
2. Public names: does HAProxy know it? grep the map on the VPS. A reset right after the TLS hello means the name isn't in it.
3. Does Traefik answer? On a host: curl -sk --resolve git.lab.networktechguy.com:443:127.0.0.1 https://git.lab.networktechguy.com/ -o /dev/null -w '%{http_code}\n'.
4. Which answer? 403 = the internal-names guard (or the app); 404 = no route matches the name; 502/503 = no server passes the health check, so look at patronictl list and the follower log.
5. Is the app itself up? curl http://<active host>:18030/api/healthz - the active host answers 200, the other one refuses the connection.

Conclusion #

That's the whole machine: etcd decides, Patroni moves the database, the follower moves the app, the sync moves the files, and Traefik plus DNS move the traffic. In the last post of the series, we'll put real apps into it - Authentik, Vaultwarden (from SQLite!), Gitea (from MySQL!) and Linkwarden - and I'll go through everything else that went wrong along the way.

Until next time, stay safe and see you soon!