Using traefik with local CA with ACME support

Smallstep’s step-ca as CA with ACME support

In this blog post, we will go through the basic installation process and install both the step-ca and step-cli tool that will help us manage our CA and certificates issued by the CA.

NetworkTechGuy.comNenad Karlovcec

In previous post we enabled our PKI with step-ca and in this post we will finally utilise it to request and assign certificates through our traefik reverse proxy.

Let's start by identifying key components:

• traefik container
  • docker-compose.yml file
  • .env file
  • traefik.toml configuration file
  • acme.json file for storing certificates
  • /data/logs directory for logging purposes
• step-cli tool to bootstrap the CA on the docker server
• ca.demo.networktechguy.com.crt or root_ca.crt file

First, let's install the step-cli tool:

wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.deb
sudo dpkg -i step-cli_amd64.deb

Next, we will bootstrap the CA:

step ca bootstrap --ca-url https://ca.demo.networktechguy.com --fingerprint e27ae5a0b80320d646a233ec281c47cf4768ba4d011245f9e550ac4a6acdf876

Remember, you can find out the fingerprint of your CA by issuing the following command on your step-ca CA server:

step certificate fingerprint $(step path)/certs/root_ca.crt

The output of bootstrapping should be something like this:

The root certificate has been saved in /home/nenad/.step/certs/root_ca.crt.
The authority configuration has been saved in /home/nenad/.step/config/defaults.json.

Next, we will enable the our root CA globally:

sudo cp /home/nenad/.step/certs/root_ca.crt /usr/local/share/ca-certificates/ca.demo.networktechguy.com.crt
sudo update-ca-certificates

With that out of the way, we can now set up our traefik container. Of course, you need to have docker installed, and if you don't, the easiest way to get the newest version is to run the docker install script:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
sudo usermod -aG docker nenad
sudo mkdir /data/dockers -p
sudo chown nenad:nenad /data -R
exit

As part of the docker installation, I've also added my user to the docker group, as well as created the /data/dockers directory where all of my containers will be and then I changed the ownership to my user. exit command at the end is there so that I get logged out of the system so that next time when I log in, all the changes are applied. I know that there are other ways to do it, but this is the fastest for me.

Once all of it is done, we will create a traefik directory - I tend to number all of them, but that's up to you:

cd /data/dockers
mkdir 01-traefik
cd 01-traefik
touch acme.json
chmod 600 acme.json
touch .env
touch docker-compose.yml
touch traefik.toml
mkdir confs
mkdir /data/logs/traefik -p

As you can see, I've created the directory, and then immediately created necessary files as well. I also tend to forget to create acme.json file if I don't do it immediately, and that then causes issues when I start the traefik container as it tries to create an acme.json directory instead. This way, all the files are created immediately, including two of the directories I reference in the configuration files.

Now is the time to populate the configuration files. Let's start with traefik.toml:

[api]
  dashboard = true

[log]
  level = "info"

[entryPoints]
  [entryPoints.web]
    address = ":80"
    [entryPoints.web.proxyProtocol]
      trustedIPs = ["127.0.0.1/32"]
    [entryPoints.web.forwardedHeaders]
      trustedIPs = ["127.0.0.1/32", "10.0.0.0/8"]
    [entryPoints.web.http.redirections.entryPoint]
      to = "websecure"
      scheme = "https"

  [entryPoints.websecure]
    address = ":443"
    [entryPoints.websecure.proxyProtocol]
      trustedIPs = ["127.0.0.1/32"]
    [entryPoints.websecure.forwardedHeaders]
      trustedIPs = ["127.0.0.1/32", "10.0.0.0/8"]
    [entryPoints.websecure.http.tls]
      certResolver = "step"
      [[entryPoints.websecure.http.tls.domains]]
        main = "traefik-01.demo.networktechguy.com"

[accessLog]
  filePath = "/logs/access.log"
  format = "json"
  [accessLog.filters]
    statusCodes = ["200", "300-302"]
    retryAttempts = true
    minDuration = "10ms"
  [accessLog.fields]
    defaultMode = "keep"
    [accessLog.fields.headers]
      defaultMode = "keep"

[providers]
  [providers.docker]
    endpoint = "unix:///var/run/docker.sock"
    exposedByDefault = false
    network = "ext-net"
  [providers.file]
    directory = "/data"
    watch = true

[certificatesResolvers]
  [certificatesResolvers.step.acme]
    caServer = "https://ca.demo.networktechguy.com/acme/acme/directory"
    email = "[email protected]"
    storage = "acme.json"
    certificatesDuration = 8766
    [certificatesResolvers.step.acme.httpChallenge]
      entryPoint = "web"

[serversTransport]
  insecureSkipVerify = true
  rootCAs = ["/etc/ssl/certs/ca.demo.networktechguy.com.pem"]

API Dashboard

• The api section enables traefik's web dashboard, allowing you to monitor your reverse proxy's traffic and settings through a visual interface.

Logging

• The log section sets the logging level to info, ensuring basic informational messages are captured without overwhelming the logs with too much detail.

Entry Points

• The entryPoints define the ports where traefik listens for incoming requests.
  • HTTP (Port 80): The web entry point listens on port 80 (HTTP), but it redirects all traffic to HTTPS via the websecure entry point, ensuring secure communication.
  • HTTPS (Port 443): The websecure entry point listens on port 443 (HTTPS) and is configured with the step certificate resolver to handle TLS encryption and obtain certificates from a custom CA server.
  • For both entry points, trusted IPs are defined for proxy protocol and forwarded headers to secure traffic from specific trusted sources.

Access Logs

• traefik access logs are stored in JSON format in /logs/access.log, allowing for structured logging, useful for analysis.
• The accessLog section filters log entries, only keeping status codes between 200 and 302, retry attempts, and requests that take longer than 10ms.
• Headers in requests are preserved for easier debugging and tracking.

Providers

• traefik connects to docker through the providers.docker section, using the docker socket to dynamically discover services. The services are not exposed by default unless explicitly configured.
• The providers.file section watches a directory (/data) for dynamic updates to file-based configurations.

Certificates

• The certificatesResolvers section uses ACME protocol to obtain certificates from a custom ACME server (https://ca.demo.networktechguy.com) with the step resolver. Certificates are stored in acme.json, and HTTP challenges are handled via the web entry point to automate certificate management.

Security

• In the serversTransport section, insecureSkipVerify is set to true, which disables certificate verification for back-end servers. Additionally, custom root CAs are loaded to trust certificates from ca.demo.networktechguy.com, ensuring the correct handling of internal certificates.

This configuration optimises security by enforcing HTTPS, integrates with docker for dynamic service discovery, and automates certificate issuance through ACME with a custom CA.

I like to use variables as much as possible, that way I can easily change container parameters without modifying the docker-compose file itself.

Let's take a look at the .env file now:

# SystemStuff
TZ=America/Toronto
DNS_SRV1=10.75.1.1
DOCKER_SOCK_PATH=/var/run/docker.sock
SYS_TIME_PATH=/etc/localtime
CNT_NET1=ext-net
CNT_NET2=all-net
CNT_PATH=/data/dockers/01-traefik
CNT_DN1=demo.networktechguy.com
CNT_RSLV=step

#Container specific stuff - split by app
# traefik-01-app
CNT1_IMG=traefik
CNT1_VER=v3.1.4
CNT1_NAME=traefik-01-app
CNT1_PUID=1000
CNT1_PGID=1000
CNT1_RESTART=unless-stopped
CNT1_PORT1=80
CNT1_PORT2=443
CNT1_PORT3=8080
CNT1_SRVCNAME=traefik-01
CNT1_DMN_NAME1=traefik-01

As you can see here, the file is divided into two sections - one that defines system variables and the second one that is service-specific. This container only has one service (traefik-01-app), but in case there were more, I could easily create another section that would contain variables for that service only.

Finally, there is the docker-compose.yml file itself:

services:
  traefik:
    image: ${CNT1_IMG}:${CNT1_VER}
    container_name: ${CNT1_NAME}
    restart: ${CNT1_RESTART}
    command:
      - --configFile=/etc/traefik/traefik.toml
    environment:
      - PUID=${CNT1_PUID}
      - PGID=${CNT1_PGID}
    ports:
      - ${CNT1_PORT1}:${CNT1_PORT1}
      - ${CNT1_PORT2}:${CNT1_PORT2}
      - ${CNT1_PORT3}:${CNT1_PORT3}
    dns:
      - ${DNS_SRV1}
    labels:
      - traefik.enable=true
      - traefik.http.routers.${CNT1_SRVCNAME}.rule=Host(`${CNT1_DMN_NAME1}.${CNT_DN1}`)
      - traefik.http.routers.${CNT1_SRVCNAME}.tls.certresolver=${CNT_RSLV}
      - traefik.http.routers.${CNT1_SRVCNAME}.entrypoints=websecure
      - traefik.http.routers.${CNT1_SRVCNAME}.service=api@internal

      # Middleware to add an exception for .well-known/acme-challenge/
      - traefik.http.middlewares.no-redirect-acme.headers.customrequestheaders.X-No-Redirect=true

      # Middleware for HTTPS redirection
      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
      - traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true

      # Apply no-redirect-acme only to the specific router
      - traefik.http.routers.acme-http.rule=PathPrefix(`/.well-known/acme-challenge/`)
      - traefik.http.routers.acme-http.middlewares=no-redirect-acme
      - traefik.http.routers.acme-http.entrypoints=web

      # Apply redirection to all other routers
      - traefik.http.routers.${CNT1_SRVCNAME}.middlewares=strip,redirect-to-https

      # Strip prefix middleware
      - traefik.http.middlewares.strip.stripprefix.prefixes=/traefik

    volumes:
      - ${DOCKER_SOCK_PATH}:${DOCKER_SOCK_PATH}
      - ${CNT_PATH}/acme.json:/acme.json
      - ${CNT_PATH}/traefik.toml:/etc/traefik/traefik.toml
      - ${SYS_TIME_PATH}:${SYS_TIME_PATH}:ro
      - ${CNT_PATH}/confs:/data
      - /data/logs/traefik:/logs
      - /data/logs/traefik:/var/log/traefik
      - /etc/ssl/certs/ca.demo.networktechguy.com.pem:/etc/ssl/certs/ca.demo.networktechguy.com.pem:ro
    networks:
      - ${CNT_NET1}
      - ${CNT_NET2}

networks:
  ext-net:
    external: true
    driver: bridge
  all-net:
    driver: bridge
    external: true

traefik Service Setup

• Image and Version: traefik image and its version are dynamically defined by the ${CNT1_IMG} and ${CNT1_VER} environment variables, allowing easy updates.
• Container Name: traefik container is named using the ${CNT1_NAME} variable.
• Restart Policy: The restart behavior is set using ${CNT1_RESTART}, ensuring that the service is resilient and restarts if it encounters an issue (if so configured in the .env file)

command and Configuration File

• traefik command specifies a custom configuration file located at /etc/traefik/traefik.toml, which is mounted from the host.

Environment Variables

• Environment variables for user and group IDs (PUID and PGID) are passed to set appropriate permissions inside the container.

Ports

• traefik exposes several ports, defined by ${CNT1_PORT1}, ${CNT1_PORT2}, and ${CNT1_PORT3}, for external access and service routing.

DNS

• Custom DNS server configuration is set using ${DNS_SRV1} to ensure traefik uses a specific DNS server for resolving domain names.

Labels for Routing and Security

• Routing and Certificates: Several labels are defined to configure routing:
  • traefik is enabled for this container with traefik.enable=true.
  • A router named ${CNT1_SRVCNAME} is defined to route traffic based on the host ${CNT1_DMN_NAME1}.${CNT_DN1} and use HTTPS with a certificate resolver ${CNT_RSLV}.
  • The router uses the secure entry point websecure for encrypted traffic and links to the internal API service.
• ACME Challenge: Special middleware (no-redirect-acme) is used to allow ACME challenges to bypass HTTPS redirection, which is necessary for certificate issuance.
• HTTPS Redirection: Middleware (redirect-to-https) is applied to redirect all non-HTTPS traffic to HTTPS.
• Path Prefix Stripping: Middleware (strip) is configured to strip the /traefik prefix from requests.

Volumes

• Several volumes are mounted:
  • Docker socket (${DOCKER_SOCK_PATH}) is mounted for traefik to dynamically configure itself based on running Docker containers.
  • TLS certificate files (like acme.json and the custom CA certificate) and the traefik configuration file are mounted to provide necessary access to these resources.
  • System time (${SYS_TIME_PATH}) is mounted as read-only for accurate time synchronisation inside the container.
  • Log files are persisted to /data/logs/traefik for access logging and troubleshooting.

Networks

• Two networks are defined, and traefik connects to both:
  • ${CNT_NET1} and ${CNT_NET2} are custom networks, allowing traefik service to interact with other containers and services on these networks.

The networks section defines external networks (ext-net and all-net), using the docker bridge driver for communication.

With all of that out of the way, we are now ready to start the traefik container. Please note that, for ACME to function properly and request certificates for services that define it in the label section, you need to have proper DNS resolution, meaning that the requested FQDN needs to exist in your DNS. In my case, I needed to add the traefik-01.demo.networktechguy.com entry to my DNS. Since I'm using internal DNS for name resolution, that is enough for traefik to use ACME to request the certificate.

I usually run my docker containers with the following commands:

docker compose down && docker compose pull && docker compose up -d

This way I'm sure that it always pulls the newest image before deploying it.

Finally, we are ready to check how things worked out! Go to your workstation and make sure that you've added the root certificate to your Authorities list (click on Import... button if you haven't already and locate your root certificate and make it trusted for both options - to identify websites and to identify mail users) in Firefox. Other browsers may use system store to recognise the root certificates, but Firefox does not, so you have to add it manually.

In my case, it looks like this:

Finally, we are ready to see the traefik's dashboard - in my case, the URL is https://traefik-01.demo.networktechguy.com, and here's how it looks like on a computer that trusts the root CA:

We can examine the details a little bit closer:

As you can see, traefik requested certificate is functioning properly and Firefox has no issues with trusting it since we are trusting the root CA itself. Even though the signing CA is the intermediate CA, it is still working properly because all certificates below the root CA are also trusted! You will also notice that the certificate itself is valid for a year, which directly ties to the configuration we did previously for ACME provider inside of step-ca where we defined the default validity for all ACME certificates to be one year!

And that's it! Now you can use your new traefik instance to link your services with your internal PKI and avoid making a public knowledge of all the services you may be running on your network - something like this:

Of course, on computers that don't trust the root CA, you will still get this (or similar) message:

If you are using Windows workstations, just install the root CA by double clicking on it and choose Computer store if you want to enable it for all users and make sure it's installed under trusted Root CA's. If you run an AD domain, it can be done via Group Policy, if I'm not mistaken.

And that's all there's to it. From this point on, you can easily use the step certificate resolver for all your traefik-enabled services instead of using Let's Encrypt or ZeroSSL. Of course, if you are hosting some publicly available services, this will not work unless you allow them to first download the root certificate, but that's a different story. I usually use Let's Encrypt for all services that are on my network that are publicly available, and step for those that are meant to be accessed only internally. In one of the next blog posts we will discuss how to enable mTLS (mutual TLS) on your websites - in short, how to verify not only the server, but for server to also authenticate the user via user certificate!