Oh, that’s interesting. I’ve seen these sorts of peering problems before. For example, when I was living in NYC, one of my ISPs decided to have a dispute with Netflix and had mysteriously bad problems viewing Netflix content, but if I VPNed into a nearby datacenter, the problems went away.
One thing you could try would be to set up a tailscale node that advertises 10.8.0.0/16 (with --accept-routes=false) and routes to that subnet via openvpn.
Or, as you suggested, you can try to play tricks with ‘ip rule’. If you can get openvpn to tag its packets with the special fwmark that tailscale uses, you can make it not send its traffic over tailscale. Or you should be able to add an extra ‘ip rule’ table above tailscale’s table that lets you do whatever you want. I’m not 100% sure, but I don’t think iptables will be the problem so much as the routes.