Unable to SSH with - but all other services work, can netcat to ssh port

Linux
1

Very confused. I have a number of Ubuntu and Pi computers. I can ssh between them via “older” methods, i.e port forwarding on routers, public key, etc

But I can’t ssh between most of them, using tailscale - port is open, it just hangs. All ACL’s are in their default state - never been touched.

All other services work, I can RDP/VNC, or use a netcat server, and ping. nmap scan shows all correct ports are open. I can netcat (nc server 22) and manually connect to the SSHD just fine, it’s listening and responding - I just can’t ssh to it, always just hangs. Doesn’t seem to make a difference which direction I try and connect. ssh -vvv

debug1: Connecting to 100.108.42.127 [100.108.42.127] port 22.
debug1: connect to address 100.108.42.127 port 22: Connection timed out

Now, here is where it gets even weirder…in one case, I can ssh FROM a raspberry pi, but not FROM Ubuntu (both cases trying to connect to the same Ubuntu machine)

I am currently on my home network (Internet via LTE), I have two devices on my local wifi - my standard Ubuntu laptop, and a Raspberry Pi. I am trying to connect to a Ubuntu Server back in the city a few hundred km away. I’ll call it city_ubuntu

I can not ssh from my laptop to city_ubuntu, but I can ssh from my raspberry pi to city_ubuntu. This is despite my laptop and RaPi both right next to me on the same network. I am very confused by that. On my laptop, I can ping the server, I can RDP to it, I can netcat to the ssh port - I can do everything except establish an ssh connection.

I have a number of other linux and pi devices in a third location too - generally speaking SSH doesn’t work between any of them, except for the occasional exception like above.

tailscale status shows everything as up. Ubuntu file wall off. /etc/hosts.allow is empty

Any pointers? Anything else to try? more info here

2

Ok, a solution has been found - possible bug/incompatability with tailscale and Openssh 8.2

I manually compilied an earlier version Openssh 7.9 at it works!

Server is v8.4

So, connecting via openssh from 8.2 to 8.4 does not work for me, if using tailscale.

3

Isn’t is equally likely that there is a problem with the combination of settings being used on the Ubuntu system with OpenSSH 8.2, and the problem doesn’t involve Tailscale at all?

4

Well, I can ssh from the Ubuntu client (v20.04 / v8.2) to the Ubuntu server (v21.04 / v8.4) just fine when not using tailscale.

But as soon as I use the Tailscale IP - it fails. But if I downgrade to v7.9 then it works.

So, does that exclude it being an issue with the Ubuntu system? It’s only toggling Tailscale that causes the issue.

Also, if it was a machine issue, then would I would expect it to not connect at all, or get some ssh -vvv error message - but it just opens the connection, and the timesout.

I just ran another test, this time, for the ssh server I am connecting to , I swapped out the Ubuntu 21 machine, for a RaspberryPi4, and repeated the tests.

This Pi is running Openssh v7.9

Behavior is the same: I can not ssh to this machine from Ubuntu 20 / v8.2, but as soon as I use openssh v7.9, then I can successfully connect. (with tailscale).

When not using tailscale, I can connect just fine, with both Openssh v7.9 or v8.2.

Any further throughts? Anything you want me to try?

5

Also of note, I can ssh from ubuntu_city to ubuntu_home, but not the other way around

ubuntu_city: (v21) ssh -V
OpenSSH_8.4p1 Ubuntu-5ubuntu1.2, OpenSSL 1.1.1j 16 Feb 2021

ubuntu_home: (v20) ‘ssh -V’
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020

But, on ubuntu_home, if I manually compile OpenSSH v7.9, then I am able to connect, via tailscale, to ubuntu_city.

Note: I removed .ssh/config during testing

6

Note: if I manually compile Openssh v8.2, then it ssh-over-tailscale works.

If I use the system binary, also v8.2, then ssh-over-tailscale does not work.

If I use the system binary, for regular ssh-not-over-tailscale, then it works. Just doesnt work over tailscale.

Further info here

7

I am not able to reproduce this.

Fresh install of Ubuntu can ssh to Ubuntu and Raspberry Pi over tailscale using the system binary for OpenSSH 8.2

$: ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
8

I’m getting very similar results connecting to a Kali Linux box (based on Debian). All wireguard looks good, i can come from Kali to my laptop fine, but i can’t connect via SSH to Kali. I see the port open, works fine every other way, just not through tailscale. On Kali the ssh version is OpenSSH 8.7p1. I believe it must be something in latter versions of openssh?

@Jay regarding SSH working outside of tailscale. Yes, that is actually my only access into this box is via ssh to a public IP. For additional testing, i updated the sshd_config to uncomment to allow all interfaces. For grins i also uncommented allowTcpForwarding and PermitTunnel and none of those seemed to help. thanks!

(Sorry as a new user it looks like i’m limited to 3 replies total)

9

Can you capture the output of both ssh -G <host> and ssh -vv <host> for both a working and a non-working case? SSH doesn’t generally do anything fancy IP wise, so it’s quite odd to observe this effect. It is however quite possible that there are things like hostname canonicalization or other config settings at play.

10

I just did a clean install of Kali and tested incoming and outgoing ssh with no issues. Did you make any changes to your setup?

11

Thanks for doing that! The only ‘different’ things about this scenario is:

  • Latest Kali running on m5-2xlarge instance at AWS. Only inbound rule is for SSH from any, outbound allows all.
  • To connect via SSH i do use a ssh key to connect (-i option)
  • Tailscale was installed while sudo to root, i ssh as user kali

That’s all i can think of that might be different. I just installed from Kali’s ami.

thanks!

12

And you’re able to ssh into Kali from a non-tailscale device?

15

Thanks. I see you scrubbed some IPs, which is perfectly fine - but it’s noteworthy that the address started 10.x. The CGNAT range that Tailscale uses starts 100.x.

Are you trying to use a subnet route setup?

Even if you are, we can diagnose most simply to begin with by using the Tailscale IP of the target. You can get your local Tailscale address with tailscale ip -4 and you can get the Tailscale address of other hosts by name that way too, like tailscale ip -4 somehostname.

16

@raggi @Jay

Well, i’m today’s biggest idiot. Raggi, that was it…i literally was using 10.x out of ingrained habit and not 100.x. Irritating when you stare right at it over and over and never see that. Thanks for the extra eyes! thanks for troubleshooting and spending time on this. Virtual beer on me, thank you!

1 Like
17

Happy to be a rubber duck, and glad you got it working.

18

Happens to all of us - happy Friday/Weekend!

1 Like