Can't get direct connection when 2 internet is connected

I have 2 internet connection from 2 different routers connected to my laptop using different adapter (Windows). Whenever I connect both internet source (which I do whenever I’m home), Tailscale seems to not be able to connect to my server directly.

Brief description of my network
Ext. VPN => Always on with either network connected
Internet 1 => 192.168.1.10 | LTE, Metric 25
Internet 2 => 192.168.100.10 | Metric 75
Tailscale => Metric 100 (set using powershell)

My other tailscale client is directly connected to internet 2 (192.168.100.5). If only internet 2 is connected, tailscale can directly connect to my server

C:\Users\bagusnl_reg>tailscale ping 100.83.61.125
pong from server (100.83.61.125) via 192.168.100.5:41641 in 2ms

Though when the internet 1 is also connected, tailscale fails to connect directly

C:\Users\bagusnl_reg>tailscale ping 100.83.61.125
pong from server (100.83.61.125) via DERP(sin) in 78ms
pong from server (100.83.61.125) via DERP(sin) in 71ms
pong from server (100.83.61.125) via DERP(sin) in 75ms
pong from server (100.83.61.125) via DERP(sin) in 73ms
pong from server (100.83.61.125) via DERP(sin) in 63ms
pong from server (100.83.61.125) via DERP(sin) in 117ms
pong from server (100.83.61.125) via DERP(sin) in 277ms
pong from server (100.83.61.125) via DERP(sin) in 64ms
pong from server (100.83.61.125) via DERP(sin) in 57ms
pong from server (100.83.61.125) via DERP(sin) in 59ms
direct connection not established

This happens when either VPN is on or off