SystemD has some bizarre concepts on DNS

Image Credit (I think): 

So for a little background on this one, this mostly came up because I was setting a up a little Splunk Collector container for a malware lab and the resulting troubleshooting is how I went down the SystemD DNS confusion hole. 

There I am, sitting at my desk, fiddling with a docker container for Splunk behind a corporate firewall (and thus an HTTPS intercept). As such I had to do some funny things to get the whole kit and caboodle to connect to the internet in the first place. So when my dig request for fairly innocuous websites was failing I wasn't insufferably concerned; however, what did catch my eye is that the CentOS 7 box was resolving the address against a 127.0.0.0/8 address (specifically 127.0.0.53). 

Alright, that's bizarre, is this little booger running its own DNS server somehow? A test from another VM to make sure that it didn't respond to DNS queries at least lightened my fears, but it didn't help explain what was going on. So you go googling to figure out what in the world it's up to. The first thing I had to learn is that CentOS had, like the debian world, had moved to SystemD. 

So in order to understand what was happening, I needed to understand not necessarily how CentOS (or REHL derivatives) handled DNS but rather how SystemD did. The archwiki has a fairly good explanation of how SystemD's DNS mechanics work: https://wiki.archlinux.org/index.php/Systemd-resolved 

Some key things to note here /etc/resolv.conf is little more then a symbolic link to /etc/systemd/resolved.conf, so you bump into quite a few post on stackoverflow and the rest will talk about how even after changing resolv.conf there would be no actual changes, even manually setting the DNS servers (rather then having the host grab them from DHCP). 

The thing is that in order to make any sort of changes you'll need to restart systemd-resolved before it even bothers to recheck the configuration file. This is great and all, but there's something even more important to know, SystemD will check that stub resolver before it goes out to check the real DNS servers (or at least according to the documentation). 

The prevailing wisdom appears to be that in order to get around this you just use resolveconf which is a package you can get and it will tweak some settings (namely permissions on resolv.conf apparently). 

But the question is why does it do this in the first place? Why does dig google.com fail and resolve locally anyway? Well this is where we get to learn some things

First, for single host name DNS request (ie "computer1", "dc1") etc. without a TLD and the likes will NOT be resolved by a unicast DNS request (eg the standard one you're thinking of over port 53 to your DNS server). Instead SystemD will only attempt to resolve them via a protocol called LLMNR (Or Link-Local Multicast Name resolution) which as it's name implies is a method for using the local network and then multicasting a request and hoping the other host on the network also use this protocol and hand out a response... which is fantastic up until you know, you're on a corporate network that uses routing and such at which point your hopes of multi-casting routing being done properly are... well nill. [1]

But wait, there's more! If resolveD does a lookup against the first DNS server in your resolv.conf and for whatever reason it's not fast enough to respond or doesn't respond to that specific request then resolveD will "remember" this and just not ask it for any DNS. This means that rather then the expected behavior of crawling down the DNS server list for each request, rresolveD will instead just go with whatever server bothered to respond to it last time [2]

There's a problem with this, because in a LOT of the corporate world the different DNS servers are intended to act as zone boundaries. IE when the lookup against say 192.168.1.1 (Local DNS or what have you) fails, then it falls over to a public DNS server (Your 8.8.8.8 / 1.1.1.1's of the world). 

Look around the internet long enough and you'll see the vast majority of folks just tell you to remove resolveD entirely as it gets in the way of things more then it helps (apparently for a time there were some DNSSEC issues and the folks over at Ubuntu were just like "yeah turn it off"). 

Anyway, all of this to say, SystemD-resolved is weird