Kubernetes in the homelab - Part 04 - Replicated storage with Longhorn

So far, our cluster can run containers and get traffic to them (previous post). What it can't do is keep data. A pod that writes to its own filesystem loses everything when it's rescheduled, and in a cluster that's designed to move pods around, that happens all the time. Applications that need to keep data (databases, uploads, configuration) ask for a PersistentVolumeClaim, and something has to provide the volume. That something is Longhorn.

Why Longhorn #

In a homelab, the usual options are:

• NFS from a NAS - simple, and probably what you already have. But the NAS is now a single point of failure for the whole cluster, and some applications (databases especially) don't like NFS at all.

• Local volumes - fast, but the data lives on one node, so the pod can only run on that node. That's the opposite of what we want.

• Ceph (with Rook) - the heavyweight. Excellent, but it wants more nodes, more disks and more RAM than three small VMs.

• Longhorn - every volume is replicated across the nodes' own disks. Lose a node and the volume keeps working from the other replicas. It's light enough for small clusters, and it has a nice UI.

For three nodes with one data disk each, Longhorn is the sweet spot. We already prepared everything it needs in part 1: open-iscsi and nfs-common installed, iscsid running, the iscsi_tcp module loaded and the empty disk mounted on /var/lib/longhorn. Let's quickly double-check it on every node before installing anything:

for n in 01 02 03; do
  ssh knetsam-kb-$n 'echo "$(hostname): iscsid=$(systemctl is-active iscsid) multipathd=$(systemctl is-active multipathd) disk=$(findmnt -no SOURCE,SIZE /var/lib/longhorn)"'
done

knetsam-kb-01: iscsid=active multipathd=inactive disk=/dev/sda 48.9G
knetsam-kb-02: iscsid=active multipathd=inactive disk=/dev/sdb 48.9G
knetsam-kb-03: iscsid=active multipathd=inactive disk=/dev/sdb 48.9G

Notice /dev/sda on the first node and /dev/sdb on the other two? That's gotcha #2 from part 1 in action - which is why the disk is mounted by UUID.

Installing Longhorn #

Same approach as MetalLB - a HelmChart in the manifests directory on knetsam-kb-01:

apiVersion: v1
kind: Namespace
metadata:
  name: longhorn-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: longhorn
  namespace: kube-system
spec:
  repo: https://charts.longhorn.io
  chart: longhorn
  version: 1.13.0
  targetNamespace: longhorn-system
  valuesContent: |-
    defaultSettings:
      defaultDataPath: /var/lib/longhorn
      defaultReplicaCount: 3
      storageMinimalAvailablePercentage: 15
      storageOverProvisioningPercentage: 200
      nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod
    persistence:
      defaultClass: true
      defaultClassReplicaCount: 3
      reclaimPolicy: Delete
    preUpgradeChecker:
      jobEnabled: false

defaultSettings

• defaultDataPath - where Longhorn keeps its replicas on each node: our dedicated disk. This is Longhorn's default anyway, but I like to have it written down.

• defaultReplicaCount: 3 - every volume gets a replica on each of the three nodes. That's maximum safety; the price is that every gigabyte you store takes three gigabytes of disk. With 50 GB per node, the cluster can hold a bit under 50 GB of data, not 150.

• storageMinimalAvailablePercentage: 15 - Longhorn stops scheduling new replicas on a disk that's below 15% free. A full disk under a running database is a very bad day, so I'd rather be told "no" early.

• storageOverProvisioningPercentage and nodeDownPodDeletionPolicy - I did not have these two in my first install. I added them after two things went wrong later on, and both are explained in their own section at the end of this post (gotchas #9 and #10). If you're installing now, keep them in.

persistence

• defaultClass: true - the longhorn storage class becomes the cluster's default, so any PersistentVolumeClaim that doesn't name a class gets a Longhorn volume. RKE2 doesn't come with a default storage class, so this doesn't conflict with anything.

• reclaimPolicy: Delete - when you delete the claim, the volume is deleted too. For a lab that's what I want; for data you really care about, you may prefer Retain.

preUpgradeChecker

• The chart runs a check job before upgrades. On a brand new install with the Helm controller it only gets in the way, so it's off.

Longhorn is a lot bigger than MetalLB - around 30 pods - so give it a few minutes:

kubectl -n longhorn-system rollout status ds/longhorn-manager
kubectl get storageclass
kubectl -n longhorn-system get nodes.longhorn.io

NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION
longhorn (default)   driver.longhorn.io   Delete          Immediate           true
longhorn-static      driver.longhorn.io   Delete          Immediate           true

NAME            READY   ALLOWSCHEDULING   SCHEDULABLE
knetsam-kb-01   True    true              True
knetsam-kb-02   True    true              True
knetsam-kb-03   True    true              True

Gotcha #7 - the managers restart on the first install #

While watching the rollout, I noticed this:

kubectl -n longhorn-system get pods -l app=longhorn-manager

NAME                     READY   STATUS    RESTARTS      AGE
longhorn-manager-hnmvx   2/2     Running   0             94s
longhorn-manager-p5wkx   2/2     Running   3 (64s ago)   94s
longhorn-manager-xd6jd   2/2     Running   1 (72s ago)   94s

Restarts on a fresh install never look good, so I checked the logs of the previous run:

kubectl -n longhorn-system logs longhorn-manager-p5wkx -c longhorn-manager --previous | grep -i fatal

level=fatal msg="Error starting manager: Operation cannot be fulfilled on settings.longhorn.io \"default-replica-count\": the object has been modified; please apply your changes to the latest version and try again"

All three managers start at the same time and all three try to write the default settings - the ones we gave in defaultSettings. One of them wins, the others get a conflict, exit and start again, and on the next start the setting is already there. It's a race on the very first start only and it fixes itself. If the restart counter keeps going up after the first couple of minutes, though, that's a real problem - look at the logs.

Testing a volume #

Let's prove it works - a claim and a pod that writes into it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lh-test
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: lh-test
spec:
  containers:
    - name: test
      image: busybox:1.37
      command: ["sh", "-c", "echo hello-longhorn > /data/f && sync && cat /data/f && sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: lh-test

kubectl apply -f lh-test.yaml
kubectl wait --for=condition=Ready pod/lh-test --timeout=180s
kubectl logs lh-test

hello-longhorn

And where did the data go?

kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n longhorn-system get replicas.longhorn.io -o custom-columns=NODE:.spec.nodeID,STATE:.status.currentState

NAME                                       STATE      ROBUSTNESS
pvc-160504a4-9f7e-4c2f-bb2f-f4fd4ef22983   attached   degraded

NODE            STATE
knetsam-kb-02   running
knetsam-kb-01   running
knetsam-kb-03   running

Three replicas, one on each node - but degraded? Don't panic. Right after a volume is created, its replicas are still being built and synchronised, and until all three are in sync, Longhorn reports the volume as degraded. 20 seconds later:

NAME                                       STATE      ROBUSTNESS
pvc-160504a4-9f7e-4c2f-bb2f-f4fd4ef22983   attached   healthy

If a volume stays degraded, then it's worth a look - usually a node that's down, or a disk that's too full for another replica (remember storageMinimalAvailablePercentage).

Clean up the test:

kubectl delete -f lh-test.yaml

The Longhorn UI #

Longhorn comes with a really nice web UI: volumes, replicas, nodes, disks, snapshots, backups. The chart creates a service for it (longhorn-frontend), but no ingress.

Gotcha #8 - the Longhorn UI has no login #

None at all. Whoever can reach it can delete every volume in your cluster. Even on a home network, I'm not putting that on an address without a password. Traefik has a basicAuth middleware, so let's use it.

First, a password hash. You don't need to install apache2-utils for htpasswd, the httpd image has it:

docker run --rm httpd:2.4-alpine htpasswd -nbB admin 'YourPasswordHere'

That gives you a line like admin:$2y$05$.... Then:

apiVersion: v1
kind: Secret
metadata:
  name: longhorn-ui-auth
  namespace: longhorn-system
type: Opaque
stringData:
  users: 'admin:$2y$05$...'
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: longhorn-ui-auth
  namespace: longhorn-system
spec:
  basicAuth:
    secret: longhorn-ui-auth
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ui
  namespace: longhorn-system
  annotations:
    cert-manager.io/cluster-issuer: lab-acme
    traefik.ingress.kubernetes.io/router.middlewares: longhorn-system-longhorn-ui-auth@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: longhorn.k8s.lab.networktechguy.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: longhorn-frontend
                port:
                  number: 80
  tls:
    - secretName: longhorn-ui-tls
      hosts:
        - longhorn.k8s.lab.networktechguy.com

Secret

• The key has to be called users - that's where Traefik's basicAuth looks. Use single quotes around the value in YAML, the hash is full of $ signs.

Middleware

• A Traefik Middleware resource - the Kubernetes equivalent of the middleware labels you may know from Traefik in docker.

Ingress

• The middleware is attached with an annotation, and its format is the one I always get wrong: <namespace>-<middleware name>@kubernetescrd. So longhorn-system-longhorn-ui-auth@kubernetescrd. Not longhorn-ui-auth, not longhorn-system/longhorn-ui-auth. If you get it wrong, Traefik doesn't fail loudly - it just doesn't apply the router, and you get a 404.

• The cert-manager.io/cluster-issuer annotation and the tls section request a certificate for the name. We set up cert-manager in the next post, so if you're following along in order, leave those two out for now (and expect a certificate warning) or come back and add them afterwards.

Test it without and with the password:

curl -s -o /dev/null -w "%{http_code}\n" https://longhorn.k8s.lab.networktechguy.com/
curl -s -o /dev/null -w "%{http_code}\n" -u 'admin:YourPasswordHere' https://longhorn.k8s.lab.networktechguy.com/

401
200

Exactly what we want. Open https://longhorn.k8s.lab.networktechguy.com in the browser, log in, and you'll see the three nodes, their disks and how much space is left.

Two settings I learned the hard way #

Everything above worked fine with test volumes. Then I installed a real application on the cluster - a web app with a PostgreSQL database and a shared volume for its files - and Longhorn taught me two more lessons.

Gotcha #9 - Longhorn counts the size you ask for, not the size you use #

The application asked for two 20 GiB volumes, one for the database and one for its files. Both came up fine. Then I installed a second copy of the same application next to it (for testing), and its volumes went straight to faulted, while the first copy's volumes turned degraded. The disks were practically empty, so what happened?

kubectl -n longhorn-system get nodes.longhorn.io -o json | jq -r '.items[] | .metadata.name as $n | .status.diskStatus | to_entries[] |
  "\($n) max=\(.value.storageMaximum/1073741824|floor)G scheduled=\(.value.storageScheduled/1073741824|floor)G"'

knetsam-kb-01 max=48G scheduled=40G
knetsam-kb-02 max=48G scheduled=20G
knetsam-kb-03 max=48G scheduled=20G

scheduled is the thing to look at. When Longhorn places a replica on a disk, it reserves the full size of the volume on that disk, no matter how much data is actually in it. With the 15% we keep free and the default over-provisioning of 100%, a 48 GB disk can take about 40 GB worth of replicas - so two 20 GiB volumes, and that's it. The first application took all of it, and the second one had nowhere to go.

Longhorn volumes are thin-provisioned (a 20 GiB volume with 1 GB of data uses 1 GB of disk), so you can safely let it reserve more than the disk's size, as long as you keep an eye on the real usage. That's storage-over-provisioning-percentage: at 200, a disk can take replicas worth twice its size. You can change it on a running cluster too:

kubectl -n longhorn-system patch settings.longhorn.io storage-over-provisioning-percentage --type merge -p '{"value":"200"}'

Within a minute, the degraded volumes rebuilt their third replica, and after I removed the stuck second copy and installed it again, it came up without a problem. The other half of the lesson: ask for volume sizes that make sense. On a small lab cluster, 5 GiB is plenty for most applications, and Longhorn volumes can be expanded later anyway.

Gotcha #10 - nothing moves off a dead node #

The whole point of three nodes and three replicas is surviving a dead node, so of course I tested it: I killed the node running the database (rke2-killall.sh, as in part 2). The application's other pods moved to the remaining nodes after a few minutes. The database didn't. It sat there as Terminating on the dead node, forever.

This is Kubernetes being careful, not broken. A database runs as a StatefulSet, and Kubernetes guarantees that there is never more than one copy of a StatefulSet pod. When a node stops answering, Kubernetes can't know whether the pod on it is really dead or just cut off from the network and still writing to its volume. So it waits for the node to confirm, and a dead node never confirms anything. The volume stays attached to the dead node and the replacement pod can't start.

Longhorn can make that decision for you: when it sees that a node is down, it force-deletes the pods there that use its volumes, detaches the volumes and lets them start elsewhere. That's what nodeDownPodDeletionPolicy does (or, on a running cluster):

kubectl -n longhorn-system patch settings.longhorn.io node-down-pod-deletion-policy --type merge \
  -p '{"value":"delete-both-statefulset-and-deployment-pod"}'

With it, the same test looked like this, measured from the moment I killed the node:

Five minutes isn't instant, but it happened on its own, at 3 in the morning if it had to, and nothing was lost - the volume had replicas on the other two nodes. When the dead node came back, Longhorn rebuilt its replicas and everything went back to healthy.

A word of caution: this setting trades Kubernetes' caution for availability. If a node isn't really dead but only cut off from the others (a broken switch port, for example), the old pod may still be running there. Longhorn only lets one node attach a volume at a time, which is what keeps this safe, but it's worth knowing what you switched on.

Conclusion #

The cluster now has storage that survives losing a node: every volume lives on all three disks, and the UI shows you what's where. In the next (and last of the base setup) post, we'll get proper certificates for everything from our own CA with cert-manager, and install Headlamp - a GUI for the whole cluster, so you don't have to live in kubectl.

Until next time, stay safe and see you soon!