SOCKS proxy support struggles

hey y’all, I’m trying to do something kind of fancy –

I’ve got two Tailscale nodes successfully on the VPN:

  1. A client container inside a k8s cluster
  2. A VM in a cloud service provider, running a postgres database.

The goal is to connect from the container in the k8s cluster to the postgres database.

I’m more or less following the userspace networking guides that create a socks proxy, as described in the articles on Heroku, Cloud Run, etc.

I’ve set ALL_PROXY appropriately and verified I can ping successfully between the nodes, and I was able to curl a webserver on my laptop while on the VPN, and it respected ALL_PROXY, so I didn’t even have to pass it any extra flags, it just worked!

However, it looks like not everything plays nice with ALL_PROXY, for example, psycopg, a Python library that uses libpq to connect to postgres databases. When I attempt to make a connection to the ipv6 address with Python code that uses psycopg, I get a “cannot assign requested address” error.

Is there a nifty workaround for this, some sort of fancy ssh port forwarding I can do maybe? I tried a few ssh -L commands but either I’m not getting the syntax right (the manpage is really sparse here), or I might be headed in the right direction.

Would love some thoughts!

Hmm, from what I can tell, it looks like libpq is unable to use SOCKS proxies? That’s a surprising limitation after decades of development on their part.

A workaround could be to run a simple bouncer program that listens on a localhost socket and does the SOCKS proxying for you. For example, redsocks can do this: https://github.com/darkk/redsocks

Hi, I’m running into this same question, but struggling a bit with the proposed solution. The main way to use redsocks appears to be with iptables, which aiui isn’t an option because I don’t have NET_ADMIN. If your suggestion was to use it non-transparently, that sounds great, but redsocks isn’t very well documented and I haven’t been able to figure out a way to set it up that way. I did also try using socat, which looked like a great fit, but it only has SOCKS4 support and it looks like the Tailscale SOCKS implementation is SOCKS5 only. The other option on my list is proxychains-ng, but I’d prefer avoiding modifying the main container (aside from what address it connects to) if possible. Any further clarity you could give would be much appreciated, I’m not a networking person so I’m a bit out of my depth. Cheers!

EDIT: It looks like ncat should work, but I’m still wrestling with it a bit. libpq is throwing “server closed the connection unexpectedly”. I can see from ncat that it’s receiving the connections and doesn’t appear to be throwing any errors. I’m running tailscaled on max verbosity but am not seeing any messages about the proxied requests, so I’m having a hard time pinning down where the problem is.

You might find it more productive to build tailscaled from source, to be able to add debugging: GitHub - tailscale/tailscale: The easiest, most secure way to use WireGuard and 2FA.

This is an example Dockerfile which downloads and builds tailscale from a branch named dgentry/dns-copy. You could fork the tailscale repo, push debugging code to your fork, and have the Dockerfile build from source.

The SOCKS server code is in net/socks5/socks5.go

In Go one can add printf like so:
fmt.Printf("debug code: %d\n", d)
https://pkg.go.dev/fmt#hdr-Printing

1 Like

Got it working!

Solution for anyone who ends up here from googling libpq and SOCKS: use ncat, but make sure not to use versions 7.90 or 7.91, as they have a very tricky bug that breaks everything :sweat_smile: 7.92 works fine.

Thanks for the help.

Would you mind sharing an example of connecting with psql over a socks5 connection using nc?