Custom DERP Server Not Being Used, Latency Check Fails

1

I’m attempting to run my own DERP server as outlined here: Custom DERP Servers · Tailscale Docs

I’ve set up a small Ubuntu 20.04 VPS, pointed my domain to it, opened ports 443/TCP and 3478/UDP, installed the latest Go and DERPER, launched the binary, and added the server to my JSON policy.

However, none of my devices use it and running tailscale netcheck shows this:

  • Nearest DERP: New York City
    • DERP latency:
      • nyc: 93.8ms (New York City)
      • dfw: 111.4ms (Dallas)
      • ord: 111.6ms (Chicago)
      • sfo: 143.7ms (San Francisco)
      • sea: 147.1ms (Seattle)
      • lhr: 166.1ms (London)
      • fra: 169.7ms (Frankfurt)
      • sao: 201.4ms (São Paulo)
      • tok: 269.4ms (Tokyo)
      • sin: 310.2ms (Singapore)
      • syd: 314.8ms (Sydney)
      • blr: (Bangalore)
      • custom-derp: ()

As shown, it’s not even detecting the latency. I’ve seen a few other folks having similar issues, but with no posted solution. I’m more than happy to provide any additional details to get this resolved.

2

Got these errors in the terminal as well:

2021/12/15 18:30:36 http: TLS handshake error from REDACTED-IP:13944: EOF
2021/12/15 18:36:20 http: TLS handshake error from REDACTED-IP:46136: acme/autocert: missing server name

3

Can you send your tailscale ip address to support@tailscale.com? I can look at our telemetry to see what is happening.

1 Like
4

That means the UDP isn’t working yet. I imagine something else is blocking the UDP still: OS firewall, cloud firewall, ISP? You check all those?

5

Yes, everything I’m able to check seems fine. Netcat on my machine shows that the UDP port is open.

6

That’s one of the three places I mentioned :slight_smile:

Is this on a cloud? Does it have a firewall?

Does your ISP restrict inbound traffic?

Are you behind a home NAT device?

7

I was able to test and connect to your DERP server, and got a ~32ms time. It was able to relay for me.

8

As mentioned, it’s running on a VPS and ports 443/TCP and 3478/UDP have been opened on the firewall. There is no other firewall system running on the VPS that I’m aware of, but I can reach out to my provider to double check.

My home ISP does have a strict CGNAT, but that shouldn’t matter since that’s the whole point of DERP servers, right? I’m able to connect to the default Tailscale DERP servers, of course.

I can access the server confirmation page through my web browser and can send data back and forth over Netcat on 3478/UDP. My device has no problem reaching the server, but Tailscale still refuses to work with it.

9

That’s interesting. Despite being able to reach it through my web browser and with Netcat, Tailscale still refuses to work with it for me. If I set OmitDefaultRegions to true, my devices are no longer able to get a Tailscale IP. This is what tailscale netcheck then shows:

Report:

  • UDP: false
  • IPv4: (no addr found)
  • IPv6: no
  • MappingVariesByDestIP:
  • HairPinning:
  • PortMapping:
  • Nearest DERP: unknown (no response to latency probes)
10

Here’s the relevant portion of my ACL, if that helps.

// Custom DERP
  "derpMap": {
    "OmitDefaultRegions": true,
    "Regions": { "900": {
      "RegionID": 900,
      "RegionCode": "custom-derp",
      "Nodes": [{
          "Name": "1",
          "RegionID": 900,
          "HostName": "derp.REDACTED.TLD"
      }]
    }}
  },
11

Any ideas why it would work for other people and not for me?

12

By chance: are you in China, and/or is the DERP server in China?

The Great Firewall has at various points interfered with DERP connections, and its behavior has changed over time.

13

No, the server and myself are in the USA.

14

I’ve tried this across multiple network connections and devices. Tailscale simply refuses to associate with or even show the latency on my DERP server. What gives?

15

I’m not immediately sure, though security policies in hosting providers can be involved in blocking connectivity to custom DERP servers.

Could I ask though: why do you want to run a DERP? A custom DERP is available for special circumstances but isn’t something which people should need to routinely do. There are DERP servers scattered around the planet to provide this service.

16

Ok, I’ll try another VPS provider as a last ditch.

The reason I want a custom DERP server is because I’m using exit nodes and often connecting over relays due to my nodes being behind a strict NAT. The default DERP servers are sometimes sluggish and I’d just rather be routing through a server that’s only being used by me.

17

I would (strongly) recommend we try to fix direct connections to the exit node rather than set up a custom DERP to work around not being able to establish direct connections.

A direct connection is possible if either end of the tunnel can be reached on a UDP destination port which the other node can predict. Only one end of the connection needs this. If you are behind a strict NAT box with no UPnP/NAT-PMP/PCP, then perhaps the other end of the connection can be configured.

If the exit node is running in a cloud provider, almost all of them have a way to allow UDP port 41641 to ingress. There are examples of doing so in the Knowledge Base:

Regarding the custom DERP server: you’re welcome to try a different VPS if you want, but the problem a custom DERP is intended to solve is not the problem you’re facing. Latency and throughput will be best using direct connections, not any kind of relay.

The public DERPs are clusters of machines meshed together in a facility with high bandwidth connections available. A Custom DERP server might do better than this when dedicated for just one tailnet, but it isn’t assured.

18

Unfortunately, the network where my exit node and home sever are located does not allow any sort of port forwarding or public access. Very strict NAT and there’s no way around it.

When I need to access things remotely, it’s usually over cellular with CGNAT or a WISP connection with a moderate NAT. I can do some more digging, but I’m afraid I’m stuck with having to use DERP relays.