July 1, 2018

To the Cloud: DNS Edition

Replacing my personal DNS infrastructure with a managed service.

Background

For many years I was running my own DNS server infrastructure where I hosted personal domains just because I considered it a fun way to play with things. Lately that enjoyment had faded and the feeling of "I wonder what cool thing I can do today" had been mostly replaced with "I wonder if this is the day I spend on OS upgrades and general upkeep".

When this train of thought had been going on for long enough I realized it was a good time to offload my DNS hosting to someone else so I could spend time on other exciting stuff.

The old stack

At it's peak I was utilizing a total of 7 machines to host my DNS:

  • ns-signer1: Running OpenDNSSEC which took care of signing my zones.
  • ns-master1: The hidden master running NSD receiving signed zones from ns-signer1 (as well as from a friends BIND server which also published its zones through my infrastructure) and fanning them out to the slave servers.
  • ns1-ns4: The slaves published in NS records, also running NSD, actually handling requests.
  • ns-lab1: Not really part of the core service, running a custom built DNS server daemon for supporting dynamic responses: 
    $ dig time.sigterm.se TXT +short
"2018-07-01T10:18:05Z"


$ dig @8.8.8.8 whoami.sigterm.se TXT +short
"74.125.74.6:36291"

For the most part the machines were virtual amd64 OpenBSD machines running on two different VPS providers for redundancy. For some time I also mixed in more exotic architectures, running ns3 on sparc64 and ns4 on powerpc.

Fun fact: that ns4 machine was the basis for this misc@ post which after a short off-list discussion with deraadt@ himself got what I would argue to be an official stamp of "Shitty Machine" from the OpenBSD project, that being the conclusion from him after I verified the results on -current. You can imagine what kind of rock solid DNS infrastructure you missed out on!

Handling these machines are also part of the reason I got involved in Ansible development, adding support for managing services and packages on OpenBSD.

Finding a good home for my zones

Since I have traditionally been running all my personal stuff on unmanaged VPS machines, and my workplace operates most stuff on-premise, I have not been that much exposed to more managed and cloudy services. For this reason I felt this would be an interesting path to new learning experiences.

It turns out there are quite a lot of providers to choose from, and with projects such as octoDNS there are even opportunities for orchestrating multiple providers at the same time which of course is great from a redundancy perspective.

Quickly I noticed that it is pretty common to charge on a per-query basis (Dyn, Google Cloud DNS, Amazon Route 53) and this made me feel uneasy. The idea that having some buggy application of my own creation querying the crap out of my zone, malicious intent from someone else, or straight up interesting blog content (right...) causing unexpected bills triggered my paranoid side.

Looking around some more I noticed what Cloudflare had to offer, and I felt I had struck virtual gold. Using their Basic Plan I could host my zones for free, with unmetered queries and bandwidth, as reinforced in this support page.

From a management perspective the availability of an API also made me feel that it would support any future automation shenanigans I could throw at it. Being able to activate DNSSEC with the click of a button (except for the DS record replacement dance I had to do at my registrar) was a welcome feature as that meant I could keep my zones signed after the move.

Migrating zones was fairly easy since you can import BIND formatted zone files directly in the interface. I made sure to have "DNS only" mode for all records so I was not using any of their the CDN proxy magic (not that it would incur any extra cost, just that I wanted to use them purely for DNS and nothing else at this point).

Conclusion

With only ns-lab1 remaining I have now replaced 6 of my 7 DNS machines with a free Cloudflare service. Events such as problematic leap seconds and Cloudbleed shows that even such a big provider is not immune to problems with infrastructure and in that regard I have put myself at the mercy of a single point of failure.

Even so my tiny setup couldn't even begin to compare with the global presence offered by Cloudflare when it comes to latency, reliability or resiliency. At the amazing cost of "free" I am really happy with this setup.

In case I do decide to expand to more DNS providers in the future octoDNS already has a provider for Cloudflare, so that would be an interesting way forward in that case.