Tailscale version = 1.30.2
Your operating system & version = Ubuntu 22.04
I have a central tailscale router that allows my non-tailscale LAN devices to get to my remote tailscale devices. Everything works fine for a few days, but inevitably I will be unable to get to one of my remote boxes and restarting tailscaled on that box doesn’t fix it. I have to restart tailscaled on the tailscale router.
Note, while I am unable to get to one remote node, I can still get to other remote nodes just fine.
When the issue happens, I start to see open-conn-track timeouts to tcp 22, but udp 161 continues to be functional.
Here is an the series of log events that occur right when connectivity is lost and never regained, until I restart tailscaled on the Ubuntu router:
Oct 3 17:53:48 tailscale1 tailscaled[1243]: health("overall"): ok
Oct 3 17:53:57 tailscale1 tailscaled[1243]: Accept: UDP{10.5.22.197:63593 > 100.116.207.99:161} 88 ok out
Oct 3 17:54:01 tailscale1 CRON[1074636]: (root) CMD (cd /etc/puppet/code && /usr/bin/git pull && /usr/bin/puppet apply /etc/puppet/code/manifests/site.pp)
Oct 3 17:54:07 tailscale1 tailscaled[1243]: Accept: UDP{10.5.22.197:63607 > 100.116.207.99:161} 88 ok out
Oct 3 17:54:11 tailscale1 tailscaled[1243]: magicsock: closing connection to derp-4 (idle), age 1m0s
Oct 3 17:54:11 tailscale1 tailscaled[1243]: magicsock: 1 active derp conns: derp-16=cr3h7m0s,wr3h7m0s
Oct 3 17:54:11 tailscale1 tailscaled[1243]: magicsock: adding connection to derp-4 for [izYTS]
Oct 3 17:54:11 tailscale1 tailscaled[1243]: magicsock: 2 active derp conns: derp-4=cr0s,wr0s derp-16=cr3h7m0s,wr3h7m0s
Oct 3 17:54:11 tailscale1 tailscaled[1243]: derphttp.Client.Send: connecting to derp-4 (fra)
Oct 3 17:54:12 tailscale1 tailscaled[1243]: magicsock: derp-4 connected; connGen=1
Oct 3 17:54:15 tailscale1 tailscaled[1243]: Accept: TCP{10.5.22.197:64746 > 100.116.40.40:22} 52 ok out
Oct 3 17:54:19 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64746 => 100.116.40.40:22) to node [izYTS]; online=yes, lastRecv=40s
Oct 3 17:54:22 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64746 => 100.116.40.40:22) to node [izYTS]; online=yes, lastRecv=43s
Oct 3 17:54:27 tailscale1 tailscaled[1243]: Accept: UDP{10.5.22.197:63876 > 100.116.207.99:161} 88 ok out
Oct 3 17:54:34 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64746 => 100.116.40.40:22) to node [izYTS]; online=yes, lastRecv=10s
Oct 3 17:54:35 tailscale1 tailscaled[1243]: Accept: TCP{10.5.22.197:64754 > 100.116.207.99:22} 52 ok out
Oct 3 17:54:37 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64754 => 100.116.207.99:22) to node [9IlVV]; online=yes, lastRecv=5s
Oct 3 17:54:40 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64754 => 100.116.207.99:22) to node [9IlVV]; online=yes, lastRecv=8s
Oct 3 17:54:42 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64758 => 100.116.207.99:22) to node [9IlVV]; online=yes, lastRecv=10s
Oct 3 17:54:44 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64760 => 100.116.40.40:22) to node [izYTS]; online=yes, lastRecv=5s
Oct 3 17:54:45 tailscale1 tailscaled[1243]: open-conn-track: timeout opening (TCP 10.5.22.197:64758 => 100.116.207.99:22) to node [9IlVV]; online=yes, lastRecv=3s