Kubernetes in the homelab - Part 05 - Certificates from your own CA and a GUI with Headlamp

This is the last post of the base setup. So far we have a highly available cluster, addresses from MetalLB, Traefik as the ingress controller and replicated storage with Longhorn (previous post). Two things are missing: proper certificates for everything we publish, and a GUI so that you can see what's going on in the cluster without typing kubectl all day. We'll take care of both.

Certificates from your own CA #

A while ago I wrote a whole series about running your own PKI with Smallstep's step-ca, including an ACME provisioner - the same protocol Let's Encrypt uses, but answered by a CA that you run on your own network:

If you don't have a CA yet, start there. If you'd rather use Let's Encrypt, everything below works with it too - but for names that only exist on your internal network, you'd need the DNS-01 challenge (I covered that one in the RFC2136 post), because Let's Encrypt can't reach your lab to check an HTTP challenge.

In Kubernetes, certificates are handled by cert-manager. You describe a CA once (an Issuer or ClusterIssuer), add an annotation to an Ingress, and cert-manager requests the certificate, stores it in a secret, and renews it before it expires. Exactly what Traefik's certificate resolver did for us in docker.

Installing cert-manager #

By now you know the drill - a HelmChart in the manifests directory:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: cert-manager
  namespace: kube-system
spec:
  repo: https://charts.jetstack.io
  chart: cert-manager
  version: v1.21.2
  targetNamespace: cert-manager
  createNamespace: true
  valuesContent: |-
    crds:
      enabled: true
      keep: true

• crds.enabled - cert-manager brings its own resource types (Certificate, ClusterIssuer...), and the chart installs them only when you ask it to.
• crds.keep - if the chart is ever removed, the resource types stay. Without this, uninstalling cert-manager would also delete every Certificate in the cluster - not something you want to find out during a "quick reinstall".

kubectl -n cert-manager get pods

NAME                                       READY   STATUS
cert-manager-76ffbfcbfc-tprnq              1/1     Running
cert-manager-cainjector-6468bc96c7-x2smq   1/1     Running
cert-manager-webhook-558c6d4f4d-4cm4v      1/1     Running

The root certificate #

cert-manager has to talk to step-ca over HTTPS, and step-ca's own certificate is signed by... our own root CA, which nothing in the cluster trusts. So we have to give it the root certificate. You can download it from the CA:

curl -sk https://ca.demo.networktechguy.com/roots.pem -o root_ca.crt

Yes, that's -k - we don't trust the CA yet, that's the whole point. Which means we have to check that we got the right root before we trust it. Compare its fingerprint with the one step-ca printed when you created it (or run step certificate fingerprint $(step path)/certs/root_ca.crt on the CA server):

step certificate fingerprint root_ca.crt

No step on this machine? openssl shows the same value, just in upper case with colons:

openssl x509 -in root_ca.crt -noout -fingerprint -sha256

Only when they match, carry on.

The ClusterIssuer #

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lab-acme
spec:
  acme:
    server: https://ca.demo.networktechguy.com/acme/acme/directory
    caBundle: <base64 of root_ca.crt>
    email: [email protected]
    privateKeySecretRef:
      name: lab-acme-account
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

server

• The ACME directory of your step-ca. The path is /acme/<provisioner name>/directory; mine is called acme, hence the double acme.

caBundle

• The root certificate, base64-encoded on a single line - base64 -w0 root_ca.crt. This is the one people miss when following a Let's Encrypt guide, and the error you get without it is an unhelpful x509: certificate signed by unknown authority in the cert-manager logs, while the ClusterIssuer just sits there not being ready.

privateKeySecretRef

• Where cert-manager keeps the key of its ACME account. It creates it on its own.

solvers

• How to prove that we own a name: http01 through Traefik. For every certificate, cert-manager starts a tiny pod that answers on http://<name>/.well-known/acme-challenge/... and creates an ingress for it. step-ca then fetches that URL - which means step-ca has to resolve the name to 10.5.0.154 and be able to reach port 80 on it. If your CA sits in another network segment, check the firewall between them. And if you ever add a global HTTP to HTTPS redirect in Traefik, test that certificates are still issued afterwards.

A ClusterIssuer is ready when it has registered its account with the CA:

kubectl get clusterissuer lab-acme

NAME       READY   AGE
lab-acme   True    3s

Headlamp #

There are a few GUIs for Kubernetes. The old Kubernetes Dashboard was retired, Lens is a desktop app (with a licence that changed a few times), Rancher is great but it's a big management platform of its own. Headlamp is a web UI that is now an official Kubernetes project, it runs inside the cluster and it's simple to set up. Perfect for what we need.

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: headlamp
  namespace: kube-system
spec:
  repo: https://kubernetes-sigs.github.io/headlamp/
  chart: headlamp
  version: 0.46.0
  targetNamespace: headlamp
  createNamespace: true
  valuesContent: |-
    ingress:
      enabled: true
      ingressClassName: traefik
      annotations:
        cert-manager.io/cluster-issuer: lab-acme
      hosts:
        - host: headlamp.k8s.lab.networktechguy.com
          paths:
            - path: /
              type: Prefix
      tls:
        - secretName: headlamp-tls
          hosts:
            - headlamp.k8s.lab.networktechguy.com

This is where everything from the previous posts comes together:

• the name headlamp.k8s.lab.networktechguy.com resolves through our wildcard record to 10.5.0.154
• that's MetalLB's address for Traefik, on whichever node currently holds it
• the cert-manager.io/cluster-issuer annotation makes cert-manager request a certificate from step-ca into the headlamp-tls secret, and Traefik uses it

kubectl -n headlamp get certificate,ingress

NAME                                       READY   SECRET         AGE
certificate.cert-manager.io/headlamp-tls   True    headlamp-tls   19s

NAME                                 CLASS     HOSTS                                ADDRESS      PORTS
ingress.networking.k8s.io/headlamp   traefik   headlamp.k8s.lab.networktechguy.com  10.5.0.154   80, 443

19 seconds from nothing to a trusted certificate. If you look at the pods while it's happening, you'll see a short-lived cm-acme-http-solver-... pod - that's the HTTP-01 responder, gone as soon as the certificate is issued.

You can check what Traefik is serving:

echo | openssl s_client -connect 10.5.0.154:443 -servername headlamp.k8s.lab.networktechguy.com 2>/dev/null | openssl x509 -noout -issuer -dates

The issuer is your intermediate CA, and the validity is whatever you configured on the ACME provisioner in step-ca. cert-manager renews certificates on its own when two thirds of that time has passed.

As always: on a computer that doesn't trust your root CA, the browser will warn you. Import the root certificate the way I described in the Traefik post - and don't forget that Firefox has its own certificate store.

Gotcha #11 - logging in to Headlamp #

Open https://headlamp.k8s.lab.networktechguy.com and Headlamp asks for a token. Not a username, not a password - a Kubernetes token. Headlamp doesn't have users of its own; it does everything as whoever the token belongs to.

So we create a service account for ourselves. In older guides, you'd create the account and then read its token from the secret Kubernetes created for it automatically. Since Kubernetes 1.24, those secrets aren't created anymore, which is why so many old guides end with "and now the secret doesn't exist". You either ask for a short-lived token (kubectl create token) or create the token secret yourself:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: lab-admin
    namespace: kube-system
---
apiVersion: v1
kind: Secret
metadata:
  name: lab-admin-token
  namespace: kube-system
  annotations:
    kubernetes.io/service-account.name: lab-admin
type: kubernetes.io/service-account-token

kubectl apply -f lab-admin.yaml
kubectl -n kube-system get secret lab-admin-token -o jsonpath='{.data.token}' | base64 -d > ~/.kube/headlamp.token
chmod 600 ~/.kube/headlamp.token

Paste the content of that file into Headlamp and you're in: nodes, workloads, storage, network, events, logs, even a shell into a container - all in the browser.

A word of warning, though. This token is cluster-admin, it never expires, and whoever has it can do anything in the cluster. For a homelab where you're the only admin, that's acceptable - keep it in a password manager, not on a sticky note. Two better options for anything more serious:

• a short-lived token each time: kubectl -n kube-system create token lab-admin --duration=8h
• Headlamp supports OIDC, so you can sign in through your identity provider (I've been using Authentik for that kind of thing) - that's a topic for a future post

What we built #

Let's take a step back and look at the whole thing:

And everything except RKE2 itself lives as YAML files in /var/lib/rancher/rke2/server/manifests/ on the first node. One piece of advice there: those files exist on one node only. The resources they created are in the cluster (in etcd, on all three nodes), so losing that node doesn't lose your setup - but losing the files means you have to remember what was in them. Keep a copy in git - it's the first thing you'll reach for if you ever have to rebuild this from scratch.

Here's the full list of gotchas from the series, in one place:

1. Cloned VMs share SSH host keys and machine-id - regenerate them, and check new host key fingerprints instead of blindly removing old ones.
2. Disk names change between clones - find the data disk by being empty, mount it by UUID.
3. The kube-vip manifest generator wrote an older image tag - check it.
4. systemctl stop rke2-server doesn't stop the containers - use rke2-killall.sh to test a failover.
5. A killed node keeps the cluster address until RKE2 starts again - drain and shut down cleanly for planned work.
6. ingress-nginx is retired and RKE2 ships Traefik - ignore guides that install ingress-nginx.
7. Longhorn's managers restart a couple of times on the first install - a race, not a problem, unless it keeps going.
8. The Longhorn UI has no login - put it behind basicAuth (and get the middleware annotation format right).
9. Longhorn reserves the full size of every replica - raise its over-provisioning on small disks and ask for sensible volume sizes.
10. Pods with Longhorn volumes never leave a dead node by themselves - set Longhorn's node-down pod deletion policy.
11. Service accounts don't get token secrets automatically anymore - create one or use kubectl create token.

Conclusion #

And that's the base setup done! A cluster you can actually lose a node in, with storage, certificates and a GUI - built from five YAML files and a couple of shell commands. In the next posts, we'll start putting real applications on it: Helm charts, databases on Longhorn, and what it takes to move something that runs happily in docker compose today onto Kubernetes.

If you have questions or ran into a gotcha I didn't mention, leave a comment below.

Until next time, stay safe and see you soon!