DNS_BAD_CONFIG after enabling magic DNS

Tailscale Version: 1.1.540-gce66dfebd
OS: Ubuntu 20.04 amd64

On my home network, I run a k3s cluster which has an instance of bitwarden running I use for my password manager. I’d like to be able to access it wherever I go, so I have each node in my cluster hooked up to tailscale as a machine. On my laptop I have my /etc/hosts set up to redirect *.homelab to my manager node, and the ingress is set up to send bitwarden.homelab requests to the pod serving bitwarden.I’d like to use magic DNS here, so that on my phone (where I can’t modify /etc/hosts ) I can access my bitwarden instance from anywhere. I figured I can enable magic DNS, then set up a second ingress in k3s to handle the host that’s generated for me by tailscale and route to bitwarden. However, when I enable magic DNS and set the DNS servers to 8.8.8.8 or 8.8.4.4, it seems to just break all my devices’ connections to the internet. I get DNS_PROBE_FINISHED_BAD_CONFIG in my browser when trying to go anywhere (a tailscale machine or any public website). I’m also running pihole in my cluster, and am using it for DNS on my laptop and phone, but even when I disable this and try again it still seems to break.

I already spoke with @danderson via slack trying to debug some of this, but was asked to raise it here instead.

We determined that the DNS resolution was working as expected, however both ping and trying to browse to my magic DNS name would just hang.

1 Like

Hmm, interesting. Some things to clarify:

  1. How did you determine that DNS resolution is working?

  2. Did you try forcing the DNS servers in the tailscale admin panel, but not enabling the MagicDNS checkbox?

  3. If you do #2, but manually query 100.100.100.100 using the ‘host’ command, what happens?

  4. If you do #2, but then manually edit resolv.conf to point at 100.100.100.100 instead of your regular DNS servers, what happens?

1 Like

Hi @apenwarr, thanks for replying

  1. We determined DNS resolution is working using the host command whilst I had magic DNS enabled. When I ran ping for the DNS name it also resolved, displaying the correct tailscale IP in the output. However, it hung indefinitely.

  2. I just tried this out, I set the DNS servers to 8.8.8.8 and 8.8.4.4 without enabling magic DNS, and found that chrome gives a DNS_BAD_CONFIG_ERROR.

  3. Here’s the command & output.

host 100.100.100.100                                                                   
Host 100.100.100.100.in-addr.arpa. not found: 3(NXDOMAIN)
  1. I have the same problem, any website that isn’t cached in my browser gives me that error.

Just trying some extra things out, when I enable DNS servers via the admin panel for each node in my k3s cluster homelab-0 to homelab-3. I have the exact same issue:

Here’s me trying to ping homelab-1 from homelab-0:

ping homelab-1
PING homelab-1.davidsbond93.gmail.com.beta.tailscale.net (<TAILSCALE IP>) 56(84) bytes of data.

--- homelab-1.davidsbond93.gmail.com.beta.tailscale.net ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9208ms

When disabled:

ping homelab-1
PING homelab-1 (<LAN IP>) 56(84) bytes of data.
64 bytes from homelab-1 (<LAN IP>): icmp_seq=1 ttl=64 time=2.56 ms

The issue of things hanging indefinitely seems reproducable on all machines on my network. The issue happens on both the stable and unstable versions

I’ve also found without enabling magic DNS or adding my own DNS servers I cannot ping any machine via its tailscale IP besides the local one. Not sure if that is intended behaviour?

Using tailscale ping I get a pong back from my machines.

1 Like

In order to test more ping things, I added hello.ipn.dev as an external machine and can ping it fine, the problem seems isolated to my local machines.

This has led me to believe the problem is down to my router. It’s a proprietary one provided by my ISP with a lot of functionality locked down. I may just need a proper 3rd party router.

1 Like

Hi David, and welcome to Tailscale forum!
Let us know how the new router works out and if it fixes your issue.

Hi David,

Just to clarify, are you saying that given an node address like, say, 100.1.1.1, that “ping 100.1.1.1” won’t work, but you can “ping x” if you set up “x” in a local DNS server or if you enable MagicDNS, it does work? That’s very strange; DNS settings should have nothing to do with pinging an IP address directly.

Does ping -n work any differently?

Hey @apenwarr, the behaviour I’m seeing is like so:

  • Pinging any of my machines via the tailscale ip will hang, no packets come back
  • Pinging the local ip address of any of my machines directly works fine, so 192.168.1.<whatever>
  • Adding hello.ipn.dev as an external machine and pinging it works fine, using both hello.ipn.dev and its tailscale ip.
  • Using the tailscale ping command gives me back a pong regardless of which machine I use.

From what I can see, using -n works no differently. This seems to be happening independently of magic DNS being on or off.

I’m still not quite clear what’s happening. Are you saying that changing the DNS settings in tailscale affects whether you can ping tailscale nodes using their IP address?

So:

  1. If you have Tailscale’s DNS settings blank, nodes are unpingable, by name nor by IP
  2. If you have Tailscale’s DNS settings set to 8.8.8.8, nodes are pingable, by name and by IP

Right? If so, that’s super weird and will need some investigation.

If I set the tailscale DNS settings, I get a DNS_BAD_CONFIG error on my devices trying to browse the internet. I’ve just replicated the behaviour on a fresh ubuntu install as well. I also cannot ping my devices by their tailscale IP if the tailscale DNS is set to 8.8.8.8 or blank.

1 Like

@apenwarr I just fixed my problem with the hanging pings. I had removed all my ACL settings before and blocked myself out of everything. So now I can ping things via their tailscale IP again.

I’m still having the issue where changing DNS settings in tailscale causes the DNS_BAD_CONFIG error.

Sorry, this thread fell off my inbox. I’m glad you fixed at least part of the problem by opening up your ACLs. If you’re still having trouble, I’d suggest a) install tailscale 1.2.x on all your devices since our new stable release is out now, with MagicDNS support; and b) before setting DNS entries in the admin panel, try probing your DNS server using the host command (with an explicit DNS server argument) to see if that works at least. If it does, after enabling DNS and MagicDNS, try the host command again to see if it still works, before checking your browser. This will help narrow down the problem you might be having.

Hey @apenwarr, no problem. I’ve actually managed to get the things working that I wanted to. The only outstanding issue seems to be how Chrome reacts to enabling custom nameservers.

I can reproduce on both my phone and laptop when adding a custom nameserver that all name resolution fails with this DNS_BAD_CONFIG error. I can still reach everything via the terminal and in alternative browsers. It’s a weird one, and likely not an issue of Tailscale’s at this point by the looks of it.

Huh, that definitely is weird. Have you tried installing the Wireguard
client instead? You could use a nonsense config file, but plug in a
DNS server, to see if you get the same behaviour when the wireguard
'link" is enabled. If you do, that’s a sign it’s not tailscale
specific.

I also have this issue on Chromium in Ubuntu 20.04 after enabling Magic DNS.

Firefox, Brave, Microsoft Edge and all other software in the box have no DNS issues,.

Version detail:
Ubuntu 20.04.1 LTS
Chromium 87.0.4280.88 (Official Build) snap (64-bit)

The error:
image

Just wanted to add that I’m seeing this issue as well. If I enter 8.8.8.8 or 1.1.1.1 into the nameservers field I just can’t really use my network any more.

@davidsbond, @itorres or anyone else running into this – what is the output from grep nameserver /etc/resolv.conf when Chrome is complaining about DNS_BAD_CONFIG?

Just another Tailscale user, but I’m chasing DNS problems as well and noticed a misconfigured /etc/resolv.conf. I’m curious if that’s what is happening here.

On your ubuntu system, are you using systemd-resolved?

@33b5e5 nameserver 100.100.100.100

@within I’m on a pretty vanilla Ubuntu install, but checking my running services I’m using resolvconf. Any way I can double check if that’s what I’m using or not?