Kubernetes in the homelab - Part 02 - A highly available control plane with RKE2 and kube-vip

In the previous post we prepared three Debian nodes. In this one, we will turn them into a Kubernetes cluster with a highly available control plane: three RKE2 servers with embedded etcd, and one floating IP address in front of the Kubernetes API that moves to another node when the one holding it dies. And then we will actually kill a node to see if it works - because a HA setup you haven't tested is just a hope.

How the pieces fit #

Every RKE2 server runs the full control plane: the Kubernetes API server, the scheduler, the controller manager and an etcd member. So in theory you can talk to any of the three nodes. In practice, you want one address that always works, for two reasons:

• your kubeconfig, scripts and tools need one address to point at

• nodes joining the cluster need one address to register with

That's what kube-vip does. It runs as a pod on every control plane node, the pods elect a leader among themselves, and the leader adds the cluster address (10.5.0.150) to its network interface and announces it with ARP. When the leader goes away, another pod wins the election and takes the address over.

RKE2 itself uses two ports that matter here:

• 6443 - the Kubernetes API

• 9345 - the RKE2 supervisor, which new nodes use to join the cluster

The first server #

RKE2 reads its configuration from /etc/rancher/rke2/config.yaml. First, we need a shared secret that every node will use to join. Generate one and keep it somewhere safe (a password manager is a good place):

openssl rand -hex 32

Then, on knetsam-kb-01:

token: <your token>
node-name: knetsam-kb-01
tls-san:
  - 10.5.0.150
  - k8s.lab.networktechguy.com
  - 10.5.0.151
  - 10.5.0.152
  - 10.5.0.153
write-kubeconfig-mode: "0600"

token

• The shared secret. Since it is sitting in this file, the file should be readable by root only: sudo chmod 600 /etc/rancher/rke2/config.yaml.

tls-san

• The names and addresses that go into the API server's certificate. This is the important one: the cluster address and its name must be in there from the very beginning. The address doesn't even exist yet - kube-vip will bring it up in a few minutes - but the certificate is created now. If you forget it, your kubectl will complain that the certificate isn't valid for 10.5.0.150, and fixing certificates on a running control plane is not how you want to spend your evening.

write-kubeconfig-mode

• RKE2 writes an admin kubeconfig to /etc/rancher/rke2/rke2.yaml. That file is full cluster admin, so 0600 - root only.

Now install RKE2. I always pin the version, so that the other two nodes get exactly the same one, even if a new release comes out while I'm working:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 sh -
sudo systemctl enable --now rke2-server.service

The first start takes a minute or two - it unpacks its images, creates certificates, starts etcd and then the rest. You can follow along with:

sudo journalctl -u rke2-server -f

RKE2 puts kubectl in its own directory, and the kubeconfig is root only, so on the nodes I use this alias:

alias k='sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml'
k get nodes

NAME            STATUS   ROLES                AGE   VERSION
knetsam-kb-01   Ready    control-plane,etcd   39s   v1.36.5+rke2r1

One node, Ready. Before adding the other two, we need the cluster address.

kube-vip #

RKE2 has a very handy feature: every YAML file in /var/lib/rancher/rke2/server/manifests/ is applied to the cluster automatically, and re-applied when the file changes. So that's where kube-vip goes.

kube-vip needs two things: permissions (a service account and a role) and the DaemonSet itself. The permissions are published by the project:

curl -s https://kube-vip.io/manifests/rbac.yaml -o kube-vip-rbac.yaml

The DaemonSet is generated by kube-vip itself. You need the name of the node's network interface first - in my case (Proxmox, virtio) it's ens18:

ip -br a

The easiest way to run the generator is with docker on your workstation (or any machine that has it):

docker run --rm ghcr.io/kube-vip/kube-vip:v1.3.0 manifest daemonset \
  --interface ens18 \
  --address 10.5.0.150 \
  --inCluster \
  --taint \
  --controlplane \
  --arp \
  --leaderElection > kube-vip-ds.yaml

• --controlplane - this instance is for the Kubernetes API (kube-vip can also do LoadBalancer services, but we'll leave that to MetalLB in the next post - don't let both of them do services)

• --arp - announce the address with ARP, which is what you want on a flat L2 network like most homelabs (the alternative is BGP)

• --leaderElection - only the elected leader holds the address

• --taint - run only on control plane nodes

Gotcha #3 - check the image tag #

Take a look at the generated file before you use it:

grep image: kube-vip-ds.yaml

In my case, the v1.3.0 generator wrote image: ghcr.io/kube-vip/kube-vip:v1.2.4 into the manifest - the previous release. Not dangerous, but not what I asked for either, so I fixed it:

sed -i 's#kube-vip:v1.2.4#kube-vip:v1.3.0#' kube-vip-ds.yaml

The most interesting part of the generated manifest is the leader election:

        - name: vip_leaseduration
          value: "15"
        - name: vip_renewdeadline
          value: "10"
        - name: vip_retryperiod
          value: "2"

The leader has to renew its lease, and if it doesn't do it within 15 seconds, the others consider it gone. Keep that number in mind for the failover test at the end.

Now put both files together into the manifests directory on knetsam-kb-01:

(cat kube-vip-rbac.yaml; echo "---"; cat kube-vip-ds.yaml) | ssh knetsam-kb-01 'sudo tee /var/lib/rancher/rke2/server/manifests/kube-vip.yaml >/dev/null'

A few seconds later:

ip -br a show ens18

ens18            UP             10.5.0.151/24 10.5.0.150/32 fe80::be24:11ff:fe10:395a/64

There it is - 10.5.0.150/32 next to the node's own address. And the API answers on it:

curl -k https://10.5.0.150:6443/version

Don't worry when that returns a 401 / "status": "Failure" - we didn't send any credentials, so that's the API politely telling us to go away. What matters is that something answered.

Joining the other two servers #

The configuration of the other two nodes is the same, plus one line: server, the address to join. And this is where the cluster address pays off for the first time - we point them at 10.5.0.150, not at knetsam-kb-01:

server: https://10.5.0.150:9345
token: <your token>
node-name: knetsam-kb-02
tls-san:
  - 10.5.0.150
  - k8s.lab.networktechguy.com
  - 10.5.0.151
  - 10.5.0.152
  - 10.5.0.153
write-kubeconfig-mode: "0600"

Same thing on knetsam-kb-03 with its own node-name. Install RKE2 with the same version and start it - one node at a time, and wait for each to be Ready before starting the next. Every new server is a new etcd member, and adding members one by one is a lot gentler on etcd than adding two at once.

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.36.5+rke2r1 sh -
sudo systemctl enable --now rke2-server.service

A few minutes later:

NAME            STATUS   ROLES                AGE     VERSION
knetsam-kb-01   Ready    control-plane,etcd   6m17s   v1.36.5+rke2r1
knetsam-kb-02   Ready    control-plane,etcd   2m9s    v1.36.5+rke2r1
knetsam-kb-03   Ready    control-plane,etcd   83s     v1.36.5+rke2r1

Three servers, three etcd members. Since the kube-vip DaemonSet runs on every control plane node, the two new nodes got a kube-vip pod as well, waiting to take over.

kubectl on your workstation #

Working on the nodes with sudo gets old quickly. Copy the admin kubeconfig to your workstation and point it at the cluster name instead of 127.0.0.1:

mkdir -p ~/.kube && chmod 700 ~/.kube
ssh knetsam-kb-01 'sudo cat /etc/rancher/rke2/rke2.yaml' \
  | sed 's#https://127.0.0.1:6443#https://k8s.lab.networktechguy.com:6443#' > ~/.kube/homelab.yaml
chmod 600 ~/.kube/homelab.yaml
export KUBECONFIG=~/.kube/homelab.yaml
kubectl get nodes

This works only because we put k8s.lab.networktechguy.com into tls-san at the beginning. Remember - this file is full cluster admin. Treat it like a root password.

Testing the failover #

Now the fun part. knetsam-kb-01 holds the cluster address. Let's take it away and see what happens.

Gotcha #4 - stopping the service is not a failover test #

My first attempt was the obvious one:

sudo systemctl stop rke2-server

The address didn't move. The API kept answering on 10.5.0.150 without a single dropped request. Success? Nope. RKE2 is a supervisor - stopping the service stops the supervisor, but the containers it started (the API server, etcd, kube-vip) keep running under containerd. Nothing actually failed, so there was nothing to fail over. This is by design (it lets you restart RKE2 without taking down the workloads), but it means that systemctl stop tests exactly nothing.

RKE2 comes with a script that really stops everything on a node:

sudo /usr/local/bin/rke2-killall.sh

And with that, from my workstation, polling the API every second:

API answers on .150 after 27 s
VIP now on kb-03

27 seconds. Where does that come from? The kube-vip lease is 15 seconds, so the other nodes wait up to that long before deciding that the leader is gone. On top of that, the node we killed was also running an etcd member - quite possibly the etcd leader - so etcd has to elect a new leader too, and the API server on the new node has to get its connections back. 27 seconds of no API is perfectly fine for a homelab - and keep in mind that it's only the API that was gone; pods running on the other two nodes didn't notice a thing.

Gotcha #5 - the old node keeps the address #

Here's the one that surprised me. While knetsam-kb-03 was happily serving 10.5.0.150, I looked at knetsam-kb-01:

ens18            UP             10.5.0.151/24 10.5.0.150/32 fe80::be24:11ff:fe10:395a/64

It still had the address! kube-vip was killed, so it never got the chance to remove it. For a moment, two machines on the network had the same IP address. The new leader announces itself with gratuitous ARP, so switches and most clients follow it, but the old node will still answer ARP requests for that address, and whoever asks at the wrong moment ends up talking to a node with no API behind that address.

The good news: as soon as RKE2 started again on knetsam-kb-01, kube-vip came back, saw that it wasn't the leader and removed the address within a few seconds. And if a node really crashes (power off, kernel panic), the address goes away with it. The lesson is about planned work: when you want to take a node down, do it properly:

kubectl drain knetsam-kb-01 --ignore-daemonsets --delete-emptydir-data
ssh knetsam-kb-01 'sudo systemctl stop rke2-server && sudo rke2-killall.sh && sudo poweroff'

and when it's back up, kubectl uncordon knetsam-kb-01.

Conclusion #

We now have a three-node Kubernetes cluster that survives losing any one of its nodes, one address to talk to it and a kubeconfig on the workstation. What we can't do yet is reach anything running inside the cluster from the outside. That's what the next post is about: MetalLB, Traefik, and the reason most Kubernetes guides you'll find are now out of date.

Until next time, stay safe and see you soon!