Greetings,
TLDR;
a pika consumer app connecting to a rabbitmq server via tailscale addresses has a connection reset exception which seems to have a co-occurrence with a system log entry of
tailscaled[]: magicsock: closing connection to derp-10 (idle), age 2m0s. The consumer app is implemented using threads so that the rabbitmq heartbeats should be sent even while the submission is being processed.
-
Where my I find details to debug what tailscale is seeing? In part because, if the syslog message is accurate, the socket is being reported as idle. However I believe that the rabbitmq heartbeats are being sent.
-
Is it possible to tweak the idle time period, which appears to be set at 2m0s?
Description
I have a web server, which also a rabbitmq server, we’ll call WRH (web-rabbitmq-host). This is a simple prototype, so no redundancy, etc.
A user web submission results in a message published to the rabbitmq service on WRH.
A separate machine hosts the service which will process the submission, we’ll call SPH (submission-processing-host). The SPH service connects to the rabbitmq service on WRH, via tailscale IP addressing.
The rabbitmq publish and consumer apps are implemented in python using the pika library. The consumer app, on SPH, has a blocking connection to the rabbitmq server on WRH.
The consumer app will be triggered to fetch the message by the pika rabbitmq library. When triggered, the consumer app processes the message in a secondary thread so that the main thread may continue to occasionally respond the the rabbitmq messages, in particular, the heartbeats.
Originally the consumer app simply set a long timeout for not receiving heartbeats, this did not prevent the connection reset exception. Then the app was re-worked to use threads so that the heartbeats would be processed and not simply accepting a long timeout. This too has resulted in connection reset exceptions.
There was a realization that there was one more component that I was not considering - that being tailscale.
Looking at logs, there is some correlation of connection reset exceptions and tailscaled[]: magicsock: closing connection to derp-10 (idle), age 2m0s entries.
My worst case approach to debugging is to bypass tailscale, having to setup direct public IP address access, in order to partition the problem into whether or not tailscale is involved in the connection reset exception. I really don’t want to have to do this.
Any references to others using rabbitmq through tailscale and what they have used to configure would likewise be useful.
Best regards,
…Otto