Dynamic public ip. Tailscale switches to derp and doesn't falls back to direct until I restart

Tailscale version: 1.42.0
Your operating system & version: DietPi with linux kernel 5.10.110

My setup has a subnet router (at site A) which connect all device in some subnets.
at site B, a plain Tailscale installation with forwarding enabled on the DietPi OS. I have a router which routes subnets of Site A to this Tailscale installation. This helps me connect to all services at site A without exposing site B.

When I start my tailscale at site B, i see direct connections (since I have added port forwarding for 41641).
I can see good ping times (<80ms) and good speed also, 30- 40 Mbps.
After some time it starts using relay.

I am just guessing that changing public ip is the reason. Can you point me out how can I know for certain? Checking logs for certain items and all?

Btw, using tailscale status and tailscale ping to figure out if using relay.

What can i do to optimize this?