Kubernetes in the homelab - Part 01 - Planning and preparing the nodes

I've been running pretty much everything in my homelab with docker compose for years now, and honestly, for most of the stuff I self-host, that's all you really need. But every now and then somebody asks me "can I run this on Kubernetes?" and up until now my answer was always some variation of "probably, I haven't tried". So I decided to finally build a proper cluster - not a single-node toy, but something that actually survives losing a node - and document every step of it, including all the things that didn't go as planned.

This is the first post in a series. By the end of it, we will have a three-node cluster with a highly available control plane, our own load balancer addresses, an ingress controller, replicated storage, certificates from our own CA and a nice GUI to click around in. In this post, we will plan the whole thing and prepare the nodes.

The plan #

Let's start by identifying key components:

• Three Debian 13 VMs on Proxmox - knetsam-kb-01, knetsam-kb-02 and knetsam-kb-03
• RKE2 as the Kubernetes distribution - all three nodes will be servers (control plane + etcd) and they will also run workloads
• kube-vip for one floating IP address in front of the Kubernetes API
• MetalLB for LoadBalancer services (the thing you get for free in the cloud and have to build yourself at home)
• Traefik as the ingress controller - it comes with RKE2 these days, more on that in part 3
• Longhorn for replicated block storage on the nodes' own disks
• cert-manager requesting certificates from my own step-ca through ACME
• Headlamp as the web GUI

Why three nodes and not two? Because of etcd. etcd is the database that holds the whole state of the cluster and it needs a majority (a quorum) to work. With three members you can lose one and the remaining two are still a majority. With two members, losing one means losing the majority - so two nodes are actually less available than one, because now either of them can take the cluster down. Three is the smallest number that makes sense.

Why RKE2? There are plenty of options (kubeadm, k3s, Talos, MicroK8s...), and I'm not going to pretend that RKE2 is the only right answer. I picked it because it is a single binary with everything included (containerd, the CNI, CoreDNS, an ingress controller), it is CIS-hardened out of the box, it uses embedded etcd and - this is the part I like the most - it has a built-in Helm controller, so you can install applications by dropping a YAML file into a directory on the node. No helm binary needed anywhere. We'll use that a lot.

Addressing #

Here's how I split the addresses in my lab network (10.5.0.0/24):

For names, I'm using the lab.networktechguy.com zone on my internal DNS server: k8s.lab.networktechguy.com for the cluster address and a wildcard *.k8s.lab.networktechguy.com pointing to the ingress address, so every new application just needs an ingress and no new DNS record. We'll add those records in part 3.

Please make sure that whatever range you pick for kube-vip and MetalLB is outside of your DHCP scope. Both of them simply start answering ARP for those addresses and they will happily fight with whatever device your DHCP server gave the same address to.

The VMs #

Each of my VMs has:

• 4 vCPUs
• 8 GB of RAM
• a 60 GB disk for the operating system
• a second, empty 50 GB disk for Longhorn
• a static IP address

That is more or less the minimum I'd recommend. RKE2 server itself wants 2 CPUs and 4 GB of RAM, and once you add Longhorn, the monitoring that comes with it and a couple of real applications, 8 GB fills up faster than you'd think (yes, it's always RAM!).

I installed the first VM with a minimal Debian 13 (SSH server and standard system utilities only), set it up the way I like it and then cloned it twice in Proxmox. And that's where the first gotchas showed up.

Gotcha #1 - cloned VMs share more than you want #

When you clone a VM, you clone everything on its disk, including things that are supposed to be unique per machine:

• SSH host keys - all three nodes would present the same host key. Not the end of the world in a lab, but it defeats the whole point of host keys.
• /etc/machine-id - systemd uses it for all sorts of things, including the DHCP client identifier. Two VMs with the same machine-id can end up with the same DHCP lease, which is a fun one to troubleshoot.

Proxmox gives every clone a new MAC address and a new SMBIOS UUID, but it does not touch the inside of the disk. So on each clone (from the console, not over SSH - you'll see why in a second):

sudo rm /etc/ssh/ssh_host_*
sudo dpkg-reconfigure openssh-server
sudo rm -f /etc/machine-id /var/lib/dbus/machine-id
sudo systemd-machine-id-setup
sudo reboot

After that, you can quickly check that all three are really different:

cat /etc/machine-id
sudo cat /sys/class/dmi/id/product_uuid
ip link show ens18 | grep ether

The flip side: if you already connected to the VM before regenerating the keys, your workstation will greet you with the big REMOTE HOST IDENTIFICATION HAS CHANGED! warning the next time you SSH in. In this case that is expected, but please don't just blindly delete the old entry. Check the new fingerprint on the VM's console first:

ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

and only when it matches what your SSH client shows you, remove the old entry on your workstation:

ssh-keygen -R 10.5.0.151

This is exactly the warning that is supposed to make you stop and think, so it's a good habit not to train yourself to ignore it.

Gotcha #2 - disk names are not stable #

On knetsam-kb-01 my empty 50 GB disk showed up as /dev/sda and the system disk as /dev/sdb. On the two clones it was the other way around. Same hardware configuration, different order. If I had written a script that formats /dev/sdb on every node, I would have wiped the operating system of the first one.

So two rules for the storage disk:

• find the disk by looking for one that has no partitions and no filesystem, never by its name
• mount it by UUID in /etc/fstab, never by /dev/sdX

You can see what you're working with like this:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT

Preparing the nodes #

Kubernetes has a few requirements that a fresh Debian install doesn't meet, and Longhorn adds a few more. Since I had to do the same thing on three machines, I put it all in one script. It's safe to run more than once.

#!/bin/bash
# Base preparation of a Kubernetes node. Run as root: ./prep-node.sh knetsam-kb-01
set -euo pipefail
NAME="$1"

# host name, and all nodes plus the cluster address in /etc/hosts
hostnamectl set-hostname "$NAME"
sed -i '/knetsam-kb-0[123]\|k8s\.lab\.networktechguy\.com/d' /etc/hosts
cat >> /etc/hosts <<'H'
10.5.0.150 k8s.lab.networktechguy.com
10.5.0.151 knetsam-kb-01.lab.networktechguy.com knetsam-kb-01
10.5.0.152 knetsam-kb-02.lab.networktechguy.com knetsam-kb-02
10.5.0.153 knetsam-kb-03.lab.networktechguy.com knetsam-kb-03
H

# no swap (kubelet)
swapoff -a
sed -i -E 's/^([^#].*[[:space:]]swap[[:space:]].*)$/# \1  # off: Kubernetes/' /etc/fstab

# packages: Longhorn needs iSCSI and the NFS client
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq open-iscsi nfs-common curl jq ca-certificates gnupg >/dev/null
systemctl enable --now iscsid >/dev/null 2>&1

# kernel modules and settings for the container network
printf 'overlay\nbr_netfilter\niscsi_tcp\n' > /etc/modules-load.d/k8s.conf
modprobe overlay; modprobe br_netfilter; modprobe iscsi_tcp
cat > /etc/sysctl.d/90-k8s.conf <<'S'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
S
sysctl --system >/dev/null

# the empty disk -> /var/lib/longhorn (found by being empty, mounted by UUID)
DISK=$(lsblk -dnpo NAME,SIZE,TYPE | awk '$3=="disk"{print $1}' | while read d; do
  if [ -z "$(lsblk -no NAME "$d" | tail -n +2)" ] && [ -z "$(blkid -o value -s TYPE "$d" 2>/dev/null || true)" ]; then echo "$d"; fi
done | head -1)
if ! grep -q /var/lib/longhorn /etc/fstab; then
  [ -n "$DISK" ] || { echo "no empty disk"; exit 1; }
  mkfs.ext4 -q -L longhorn "$DISK"
  mkdir -p /var/lib/longhorn
  echo "UUID=$(blkid -o value -s UUID "$DISK") /var/lib/longhorn ext4 defaults,nofail 0 2" >> /etc/fstab
  systemctl daemon-reload
fi
mount -a

echo "$(hostname): swap $(tail -n +2 /proc/swaps | wc -l), longhorn $(df -h --output=size /var/lib/longhorn | tail -1), iscsid $(systemctl is-active iscsid)"

Let's go through it:

Host names

• Each node gets its short host name, and all three nodes plus the cluster address go into /etc/hosts. Your DNS should resolve them too, but this way the cluster doesn't fall apart if the DNS server is the thing that's being rebooted.

Swap

• The kubelet traditionally refuses to start with swap on. Newer versions can live with it, but the memory accounting gets less predictable, so for a cluster like this one it's simply off. The sed line comments out the swap entry in /etc/fstab so it stays off after a reboot.

Packages

• open-iscsi - Longhorn presents its volumes to the node as iSCSI devices, so iscsid has to be installed and running on every node. This is the number one reason Longhorn volumes don't attach.
• nfs-common - only needed for ReadWriteMany volumes, but it's tiny and you'll want it sooner or later.

Kernel modules and sysctl

• overlay is what containerd uses for container filesystems, br_netfilter lets iptables see bridged traffic between pods, iscsi_tcp is for Longhorn.
• ip_forward - a node has to route traffic for its pods.
• The two inotify values are raised because every pod that watches files eats into them, and the Debian defaults run out surprisingly quickly on a busy node ("too many open files" errors from pods that don't even open many files - that's this).

The Longhorn disk

• This is gotcha #2 in practice: the script looks for a disk without partitions and without a filesystem, formats it with the label longhorn and mounts it by UUID on /var/lib/longhorn, which is where Longhorn keeps its data by default.
• nofail means a missing disk won't drop the node into emergency mode at boot. The node will come up and Longhorn will simply complain about the disk, which is a lot easier to fix remotely.

A small side note: the first version of my script had a bug in the disk detection, and because of set -euo pipefail it stopped with an error on the first node instead of carrying on with an empty variable. That is exactly why that line is at the top of every script I write - a script that formats disks should fail loudly, not "mostly work".

Copy the script to each node and run it with that node's name:

scp prep-node.sh knetsam-kb-01:
ssh knetsam-kb-01 'sudo bash prep-node.sh knetsam-kb-01'

The last line of the output should look something like this:

knetsam-kb-01: swap 0, longhorn 49G, iscsid active

One more thing to check before we move on - multipathd. If it's installed and running, it can grab the iSCSI devices Longhorn creates and you will get volumes that refuse to mount. A minimal Debian 13 doesn't have it, but some cloud images and appliances do:

systemctl is-active multipathd

If it says active and you don't actually use multipath, stop and disable it. If you do use it, Longhorn's documentation has a blacklist entry for its devices.

Conclusion #

That's it for the preparation. We have three identical-but-not-too-identical Debian nodes with swap off, the kernel prepared for container networking and an empty disk waiting for Longhorn. Here's what's coming in this series:

• Part 01 - Planning and preparing the nodes (this post)
• Part 02 - A highly available control plane with RKE2 and kube-vip
• Part 03 - Getting traffic in with MetalLB and Traefik
• Part 04 - Replicated storage with Longhorn
• Part 05 - Certificates from your own CA and a GUI with Headlamp

After that, we'll start putting real applications on it. If there's something you'd like me to cover along the way, leave a comment below.

Until next time, stay safe and see you soon!