Let'sEncrypt with Dynamic Updates to DNS - RFC2136

I've been using Cloudflare for a while now and I'm quite happy with their free plan. However, one thing that I'm missing there is the fact that you can't create subdomains - at least I don't think there is one. Since I have a subscription to Azure, I would simply create new DNS zones there and point from Cloudflare to Azure for all of my subdomains - and it has been working quite well, actually. One thing that I'm not happy with though is the fact that it's quite a complex solution, especially for someone who's not really very well versed in cloud technologies. This also creates significant security risks if you allow access into your Azure account with improperly configured account for accessing DNS Zones. I did follow some tutorials on how to do that, but I still don't feel very good about the whole deal - I'm not a cloud engineer after all.

So why did I even go through the hassle of getting things set up in Azure for my subdomains? Most of my subdomains have A records for services that are not externally accessible in my internal DNS server, some of them actually have zero accessible services from outside, but I still like to use LetsEncrypt/ZeroSSL certificates with them so that my browsers are not complaining about self-signed certificates. Since probably all of my services are now behind traefik, HTTP verification for my services is also not an option, at least I don't think it is - that would also mean that those services would have to have publicly resolvable DNS names, and I really, really don't want to do that for those that I only use internally.

So, how to resolve this conundrum? Well, here's where Dynamic Updates (RFC2136) comes in play. This system allows dynamic updates to DNS, which is really very handy with _acme-challenge.servicename.sub.domain.xyz type of records. If you have a setup similar to mine, you will be happy to know that traefik natively supports RFC2136 as a certificate resolver provider and can create TXT records for your services on the fly. What about safety, you may ask? Well, there are two ways of doing this - one way is to have a DNS server with split horizon configured, which means that it will respond to queries differently based on the configuration - one set of replies when queries are coming from outside of your network, the other set of replies when queries are coming from an external network. This is a pretty good solution when you work in production environments and want to avoid complexities of multiple DNS server - however, we are talking about homelabs here, so why not make it a little bit simpler when it comes to configuration.

Also, I was recently hunting for a new DNS server - I've been using Windows DNS server for a while now, but I'm really not happy with it - not that it doesn't work, it's just that administering it is pretty cumbersome in my opinion and even though there is WinAdmin that allows basic DNS server administration, it lacks any kind of advanced features in it. You can probably do stuff via PowerShell, but I'm not a Windows guy and PS is really not my forte. Also, from what I know, it doesn't support RFC2136.

To my rescue - Technitium DNS Server! I'm really not sure how I've never heard about this one before, to be honest. It has absolutely everything I need - a DNS/DHCP server (if it had IPAM included in it, it would be the best open source product in DDI category - if not the only one that I know of) with a functional web interface that completely works for me. It supports advanced DNS protocols as well, so it was a no-brainer for me. I really can't say enough good words about this product.

So, how does this work for subdomains? First of all, let's take a look at the regular DNS query from a user on the Internet for an IP address of srvc.sub.domain.xyz. Since this is an internally hosted service that will not be externally accessible, there will be no entry for it in the public DNS:

If you are running an internal DNS (as you should if you have multiple internal services), that looks something like this:

Why is it important to understand the difference between these queries and how they are working? Well, in most cases, you do not want to expose your internal DNS server to outside - mostly, if not exclusively, for security reasons. But if you run your DNS server internally only, how will LetsEncrypt or ZeroSSL confirm ownership of your (sub)domain? Well, this is why we need to have an external DNS server for our subdomain as well. This is where the simpler approach comes into play, but requires double the work when setting things initially.

Please keep in mind that DNS challenges that will come from LetsEncrypt/ZeroSSL will not actually look for specific A records - they will look for TXT records that your reverse proxy will dynamically create and remove once it's done. Hence, the query will be a little bit different in that case:

To start with this setup, you will need to have a publicly accessible server - any cloud provider will do - and install DNS server software on it. I really recommend going with Technitium, but you can choose whatever you want, as long as it supports RFC21316.

In my case, I'm using, as mentioned before, traefik as my reverse proxy, so this will explain how to use its native support for RFC2136 to dynamically update DNS entries in my external DNS server.

This is how the complete setup will work:

Before we start, a few warnings:

• patch your external DNS servers with all applicable security patches whenever they become available
• never ever create DNS entries for services/hosts that you don't want to have publicly accessible
• limit access to DNS server via through DNS requests only from Internet
• do not expose administrative access to the DNS server to the Internet

With this in mind, I would definitely recommend to limit access to the DNS server's administrative interface from your IP only or, better yet, set a VPN tunnel to it and access it from your own environment/homelab through the tunnel only.

In my environment, I actually have a server with Hetzner where all of my VMs are behind a firewall, so I created a VM within a danger zone that has no access to the other zones, but is accessible from Internet via NAT rule on the firewall. So even if it gets compromised, it will not be able to get to any other zone hosted behind the firewall.

With that behind us, how do we configure things?

First thing to do, after we have our DNS server configured, is to create a TSIG Key. To do that in Technitium DNS, choose Settings, then TSIG and click on Add:

Give a name to the key and for algorithm choose HMAC-SHA256 if it's not selected already. Leave the shared secret field blank as it will populate automatically once you save settings.

Next step is to go under Zones and click on Add Zone if you don't have one already:

Give it a name and set the type to Primary Zone. I also like to use Zone Serial option, but it is - optional.

Once you've selected your options, click on Add and you will be taken to the zone configuration. Click on Options and then click on Zone Options:

Under Zone Options, click on Dynamic Updates (RFC2136) and select Allow. You can select a different option if you want (or if you have a static IP address), but that is how I set it up.

Scroll down and under Security Policy click on Add to select the previously created TSIG Key. Under the Domain Name, I tried multiple different options, but the only one that actually worked for me was to allow wildcard for names under the subdomain. You may want to look into that further if you want to play it safe. Under Allowed Record Types, I entered TXT as I only want to allow TXT records through Dynamic Updates.

Finally, click on Save and that should be it when it comes to allowing Dynamic Updates to your zone (subdomain). Now, as long as your reverse proxy can reach the external DNS server, it should be able to write TXT records for ACME challenges when requested to do so by LetsEncrypt/ZeroSSL.

These are my settings in traefik to enable RFC2136 in my certificate resolvers section.

      - --certificatesresolvers.le.acme.dnschallenge=true
      - --certificatesresolvers.le.acme.dnschallenge.provider=rfc2136
      - [email protected]
      - --certificatesresolvers.le.acme.storage=acme.json
      - --certificatesresolvers.le.acme.dnschallenge.delayBeforeCheck=60
      - --certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53
      - --certificatesresolvers.le.acme.dnschallenge.disablepropagationcheck=true

Additionally, you need to set environment variables:

      - RFC2136_TSIG_KEY=dns-challenge
      - RFC2136_TSIG_SECRET=**THIS-WILL-BE-YOUR-KEY**
      - RFC2136_TSIG_ALGORITHM=HMAC-SHA256
      - RFC2136_NAMESERVER=**x.y.z.d**

These are pretty much self-explanatory variables - in my case, I named the TSIG Key dns-challenge so I've entered that as a value for RFC2136_TSIG_KEY, RFC2136_TSIG_SECRET will be whatever Technitium DNS server generated for you, RFC2136_TSIG_ALGORITHM should be set to HMAC-SHA256 if you selected that as an option when creating the key and, finally, RFC2136_NAMESERVER should be the IP address of your external DNS server that traefik can reach.

Restart your traefik container and now, when creating new services, instruct them to use le as certificate resolver (since that is the name of the resolver set in the configuration, you can actually name it whatever you want).

If you changed your existing le certificate resolver, future services will use RFC2136 as the provider for DNS challenges, and all existing services that already use le will use RFC2136 when renewing their certificates.

To test this out, create a new service under traefik and observe in the Zone configuration on the DNS server creation of _acme-challenge.xyz TXT records while verifying ownership of the domain. Don't worry, traefik will remove these records once the certificate has been issued.

And that's it, this is how you can set up your subdomain for DNS challenges by using RFC2136 - this also works with domains, but you would then have to host your main domain on your DNS server as well and I prefer to have those in Cloudflare for a bunch of reasons (stability, DDoS protection, 'anonimity`, etc.).

One last time - please make sure that only services that you want to expose to the Internet have entries in your external DNS server!