OPNSense with subnet router - can't find other nodes

Tailscale version: 1.20.4
Your operating system & version: OPNsense 22.1.5-amd64
FreeBSD 13.0-STABLE

I’m trying to set my OPNSense firewall up so that devices on my WLAN can access tailscale devices. I set up Tailscale on OPNSense and ran it with --advertise-routes to add my internal subnets (192.168.88.0/24, 192.168.1.0/24), and I see an interface on the OPNSense device, it has an IP and I set up default rules to allow all traffic in and out on that interface. From a Chromebook on 192.168.1.0/24, I can ssh, https, ping to the tailscale ip of the firewall, but not any of the other nodes I have on tailscale.

I do have NAT-PMP enabled; I have “WAN” as the external interface and “LAN, TAILSCALE” as internal interfaces.

To clarify, the chromebook is on a different 192.168.1.0/24 network? Or are you trying to route from that network, through tailscale, and back to that network?

I have my chromebook on 192.168.88.0/24, I have servers on a different vlan that’s on 192.168.1.0/24. I’m expecting OPNSense to route for me between Tailscale and both of my 192.168 subnets. So, I have a Tailscale node on Amazon AWS that has a 100. address, I cannot ping, ssh or https to it from the chromebook. I can ping, ssh, https to the 100. address of the OPNSense firewall itself from the chromebook, so I think there’s a route or rule missing.

Unfortunately I think this isn’t going to work using FreeBSD. What you’re trying to set up is a site-to-site VPN. On Linux, tailscale supports a --snat-subnet-routes=false option to make the 100.x.y.z addresses visible to nodes on locally attached subnets. The work to implement --snat-subnet-routes=false for FreeBSD hasn’t been done.

So:

  1. The Chromebook does not have Tailscale installed (some Chromebooks can run Android apps, including the Tailscale app)
  2. The Chromebook can ping the 100.x.y.z address of the OPNsense router
  3. The Chromebook can not ping the 100.q.r.s address of the AWS node, and it isn’t clear why

Is that the remaining mystery to solve?

If so, I think it is that FreeBSD will answer an IP address for any of its interfaces. That is, if it has:

  • eth0 with an IP address of 192.168.88.1
  • tailscale0 with an IP address of 100.q.r.s

If the Chromebook sends a ping packet to destination IP address 100.q.r.s, and the FreeBSD system receives it from eth0, the FreeBSD system will send a ping response back out the Ethernet.

understood. So, maybe I need to swap opnsense for openwrt? Is there a timeline to get --snat-subnet-route=false on freebsd?

There is not a schedule for --snat-subnet-route=false on FreeBSD. You’re welcome to try OpenWRT if you’d like. It is not a supported platform but people report success using the upstream package.

The most straightforward way to get a site-to-site VPN is with a regular Linux system, such as a Raspberry Pi.

I’m still confused about this; I have a Linux device on my network. Can I use that as my subnet router for my wlan?

I’ve put that Linux device (ubu 18.04 with an ARM cpu) on tailscale, I ran sudo tailscale up --advertise-routes=192.168.1.0/24,192.168.88.0/24 --snat-subnet-routes=false, then I went into the web ui for tailscale and authorized that node for subnets. I also removed and re-added my router, to remove the advertise-routes I had added there. I wasn’t sure if I would stumble over the free account’s limit of 1 subnet router per network, which is why I did that; I think that’s someplace that the UX could be improved, btw.

I tested by using my phone, without tailscale installed. If I’m on my WLAN, I can ping 100.100.100.100. If I turn off Wifi and use 5G, I can’t ping 100.100.100.100. However, in neither case am I able to ping the 100.x IP addresses of any of my devices on Tailscale. Prior to making these changes, I was at least able to ping the 100.x IP address of my OPNSense router, but now that doesn’t work either.

Do I need to add routes on OPNSense manually? What I would expect to happen is, my OPNSense router would see the routes being advertised by my Linux device, and send packets that way; though I’d be concerned that because I’m using --advertise-routes=192.168.1.0/24,192.168.88.0/24 that my OPNsense router would route local traffic via tailscale.

The ultimate goal I’m trying to get to is, I have a machine running in the cloud (an AWS EC2 instance) which I have on Tailscale, and I want to be able to connect to it from an Oculus Quest on my WLAN, which afaik can’t run Tailscale. I think I can handle the MDNS reflection, but the routing isn’t there yet. I guess I could try sideloading tailscale onto the quest…

Hello.

Yes. Any linux node (and many others) on your network can act as a subnet router

In order to use the routing from any device which doesn’t have tailscale installed, then you will need to add a static route. If you want bidirectional routing between the subnet and the tailnet, you will also need to use --snat-subnet-routes=false in addition to the static route to turn off source NAT on the subnet router - allowing the LAN devices to see the tailscale ip addresses of the nodes which are starting the connection.

Would I set a static route on my router to send 100.X through the subnet router’s 100.X ip address as a next hop?

That should be handled by tailscale.
The LAN devices don’t know where 100.64.0.0/10 is, so you use the LAN IP. The subnet router already has an internal route to 100.64.0.0/10 so if tailscale is set to route packets from the LAN, then it should just work.