No luck getting a direct connection on vultr managed kubernetes and out of ideas

SUPPORT QUESTIONS
1

Prefacing this that I used tailscale on kubernetes clusters a lot with great success and never had issues

I recently set up a managed cluster on vultr and have no luck ever getting a direct connection with tailscale. I’m using it in the sidecar configuration like I always do, that is working completely fine on my other non-vultr k8s clusters. So far here’s what I tried

  • Disabled UFW (ufw disable) on the host
  • Updated iptables to accept everything and not have any restrictions
  • Removed any network firewall on vultr
  • Added a network firewall group on vultr that explicitly allows everything
  • Enabled IPv6 support on the entire kubernetes cluster with the hope that a direct GUA to the pod will make connection easier
  • RandomizeClientPort
  • Added -port setting to tailscaled, and created a NodePort service for the same port linked to the pod
  • Created a calico GlobalNetworkPolicy + NetworkPolicy that allows everything on all ports explicitly

Tailscale netcheck:

    * UDP: true
    * IPv4: yes, xxx:30747
    * IPv6: yes, [xxx]:3763
    * MappingVariesByDestIP: true
    * HairPinning: false
    * PortMapping:
    * CaptivePortal: true

vultr is using calico cni for networking, but as I mentioned above, I already tried adding explicit policies

At this point I’m out of ideas. I tried everything I could think of, and I still can’t get a direct connection going. Neither to my machines at home, nor to any other servers in my tailnet. My inkling is that the host is making issues but no idea what to try outside of iptbales + ufw disable

I also opened a ticket with vultr support to see if they can help

Anyone got an idea what to try here?

2

Noone an idea? Anything to try?

3

I had a bit more time for debugging

Installed a tailscale node onto the host directly and was able to get a direct connection with my other devices. I was also able to get a direct connection from host into ts-sidecar within the k8s cluster

This tells me that something in the cluster is blocking the connection. Looking at threads like this: Tailscale proxy in k8s with cilium works with pod not with svc? - #5 by farcaller

I am pretty sure that it’s calico messing with the direct connection. I’m not used to managing calico because my previous clusters were using cilium, but if anyone has an idea…