Kubernetes in the homelab - Part 03 - Getting traffic in with MetalLB and Traefik

At the end of the previous post we had a working three-node cluster with a highly available API. What we didn't have was a way to reach applications running in it. In the cloud, you create a Service of type LoadBalancer and your provider hands you a public IP address. At home, nobody is going to hand you anything - the service just sits there in Pending forever. In this post we'll fix that with MetalLB, and then put the ingress controller behind one of its addresses.

Gotcha #6 - most guides still say ingress-nginx #

If you search for "Kubernetes ingress homelab", the vast majority of what you'll find tells you to install ingress-nginx. That project was retired by the Kubernetes maintainers in March 2026 - no more releases, no more security fixes. RKE2 moved on as well: the version I installed (1.36) ships Traefik as its default ingress controller. You can see it right after installation:

kubectl get ingressclass

NAME                CONTROLLER                      PARAMETERS   AGE
traefik (default)   traefik.io/ingress-controller   <none>       24m

Personally, I'm very happy about this one - Traefik has been my reverse proxy of choice in docker for years (I wrote about using it with my own CA here). But if you follow an older guide, you'll end up with two ingress controllers fighting over ports 80 and 443. So: no ingress-nginx, and every Ingress in this series uses ingressClassName: traefik.

Let's look at how RKE2 runs it out of the box:

kubectl -n kube-system get ds,svc rke2-traefik

NAME                          DESIRED   CURRENT   READY
daemonset.apps/rke2-traefik   3         3         3

NAME                   TYPE        CLUSTER-IP     PORT(S)
service/rke2-traefik   ClusterIP   10.43.110.24   80/TCP,443/TCP

A DaemonSet - one Traefik pod per node - listening with hostPort 80 and 443 on every node's own address. That works, but then which address do you put in DNS? Pick one node and it's a single point of failure again; put all three in and clients get a dead address a third of the time when a node is down. We want one address that moves, just like the API - which is exactly what MetalLB gives us.

MetalLB #

MetalLB watches for LoadBalancer services, gives each one an address from a pool you define and announces that address on the network. In L2 mode it works just like kube-vip did for the API: one node answers ARP for the address, and if that node goes away, another one takes over.

Remember the built-in Helm controller from part 1? Here it is in action. Instead of installing helm anywhere, we describe the chart in a HelmChart resource and drop it into the manifests directory on knetsam-kb-01:

apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: metallb
  namespace: kube-system
spec:
  repo: https://metallb.github.io/metallb
  chart: metallb
  version: 0.16.1
  targetNamespace: metallb-system

Namespace

• MetalLB's speaker needs host networking and raw network access to send ARP. The pod-security labels allow that in its namespace (and only there). We create the namespace ourselves so the labels are on it before the first pod arrives.

HelmChart

• This is the RKE2 way of saying helm install metallb metallb/metallb --version 0.16.1 -n metallb-system. The HelmChart resource itself lives in kube-system (that's where the Helm controller looks), the chart goes into targetNamespace. Pinning version means nothing changes behind your back - you upgrade by changing that line.

Within a minute:

kubectl -n metallb-system get pods

NAME                                             READY   STATUS
metallb-controller-bc9cbb54b-xdt4v               1/1     Running
metallb-frr-k8s-mxdj5                            5/5     Running
metallb-frr-k8s-vcg7w                            5/5     Running
metallb-frr-k8s-zqtjp                            5/5     Running
metallb-speaker-42rvt                            1/1     Running
metallb-speaker-vfw2t                            1/1     Running
metallb-speaker-zqxth                            1/1     Running

If you've used MetalLB before, the frr-k8s pods may be new to you - recent versions run FRR (the routing daemon) for BGP by default. We only use L2 mode, so they just sit there. Don't be alarmed by the 5/5.

MetalLB is installed, but it doesn't know which addresses it may give out. That's a second file:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lab
  namespace: metallb-system
spec:
  addresses:
    - 10.5.0.154-10.5.0.165
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lab
  namespace: metallb-system
spec:
  ipAddressPools:
    - lab

• IPAddressPool - the range from part 1. Again, make sure it's outside your DHCP scope.

• L2Advertisement - announce addresses from that pool with ARP.

Why a separate file? These two resources are MetalLB's own types, and they only exist after the chart is installed. If you put them in the same file as the HelmChart, the first apply fails because Kubernetes doesn't know what an IPAddressPool is yet. RKE2 does retry, so it would sort itself out eventually, but I prefer to wait until kubectl -n metallb-system get pods shows everything running and only then add the pool.

kubectl -n metallb-system get ipaddresspool

NAME   AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
lab    true          false             ["10.5.0.154-10.5.0.165"]

Traefik behind MetalLB #

Now we change the way RKE2 runs Traefik. We don't edit RKE2's own chart - that would be overwritten on the next upgrade. RKE2 has a HelmChartConfig resource for exactly this: values that are merged into a chart RKE2 installs itself.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        hostPort: null
      websecure:
        hostPort: null
    service:
      annotations:
        metallb.io/loadBalancerIPs: 10.5.0.154
      spec:
        type: LoadBalancer
        externalTrafficPolicy: Local
        ipFamilyPolicy: SingleStack

hostPort: null

• Traefik stops listening on the nodes' own addresses. Everything comes in through the MetalLB address from now on, and ports 80 and 443 on the nodes are free again.

metallb.io/loadBalancerIPs

• Without this, MetalLB would hand Traefik the first free address from the pool. Since this is the address our wildcard DNS record will point to, I want it fixed: 10.5.0.154.

externalTrafficPolicy: Local

• With the default (Cluster), traffic arriving at the node holding the address can be forwarded to a Traefik pod on another node, and on the way the source address gets replaced with the node's address - every request in your logs would come from a node. Local sends it only to the Traefik pod on the node that received it, so Traefik sees the real client address.

• The catch with Local: the node holding the address must have a Traefik pod, or the traffic goes nowhere. Since Traefik runs as a DaemonSet (a pod on every node), that's always true here. If you ever run your ingress controller as a Deployment with fewer replicas than nodes, think twice about this setting.

ipFamilyPolicy: SingleStack

• The chart asks for dual-stack by default. This cluster is IPv4 only, so I asked for a single stack to keep things simple.

RKE2 picks up the file, re-runs the Traefik chart with the new values and the DaemonSet rolls through the nodes one by one:

kubectl -n kube-system get svc rke2-traefik

NAME           TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
rke2-traefik   LoadBalancer   10.43.110.24   10.5.0.154    80:31991/TCP,443:32704/TCP

And from the workstation:

curl -s -o /dev/null -w "%{http_code}\n" http://10.5.0.154/

404

A 404 is the best result we could get right now: Traefik answered on the MetalLB address, it just doesn't have any routes yet.

And yes, I tested this one as well: while knetsam-kb-01 was down for the failover test in part 2, Traefik kept answering on 10.5.0.154 from one of the other nodes.

DNS #

Last piece: names. I'm running Technitium as my internal DNS server, but any DNS server will do. Two records in the lab.networktechguy.com zone:

The first one is the API, the wildcard is the ingress. From now on, any application gets a name like whatever.k8s.lab.networktechguy.com just by creating an Ingress for it - no new DNS record, no new IP address.

A small PSA if you have more than one DNS server: the records go to the primary, and the secondaries pick them up at their next zone transfer. In my case, the second resolver didn't know the names for a couple of minutes. If a name doesn't resolve right after you've added it, ask each server directly before you start troubleshooting:

dig +short headlamp.k8s.lab.networktechguy.com @<dns server 1>
dig +short headlamp.k8s.lab.networktechguy.com @<dns server 2>

Conclusion #

Our cluster can now hand out real addresses on the home network, and every web application can be reached through Traefik on one address that survives losing a node. In the next post, we'll give the cluster storage that survives losing a node too: Longhorn.

Until next time, stay safe and see you soon!