Permission denied (tailscale)

SUPPORT QUESTIONS
1

Hi,

I am trying to use tailscale ssh. But whenever i try to connect to the server I get a Permission denied (tailscale). Does this error message mean that the ACL is misconfigured? Because when I check the service out on the tailscale website it tells me that I can connect to it.

1 Like
2

I’m also having this issue. Windows 11 machine, no other network issues.

3

Permission denied (tailscale)

this means the ACL does not allow you to access the endpoint
Check the src and/or dst is correctly set. Most likely the source is disallowed to access the tagged machine as a destination

4

I’m also having this issue but for the life of me, I can’t figure out why. Here is my entire ACL file, with some substitution. I have a personal account, so I am the only user, and I’ve not shared anything with anyone.

// Example/default ACLs for unrestricted connections.
{
	// Declare static groups of users beyond those in the identity service.
	"groups": {
		"group:admins": ["myself@gmail.com"],
	},
	// Declare convenient hostname aliases to use in place of IP addresses.
	"hosts": {
		"host1": "100.87.40.8",
		"host2": "100.69.15.21",
	},
	"tagOwners": {
		"tag:server":   ["myself@gmail.com"],
		"tag:digocean": ["myself@gmail.com"],
	},
	// Access control lists.
	"acls": [
		// Match absolutely everything. Comment out this section if you want
		// to define specific ACL restrictions.
		{
			"action": "accept",
			"src":    ["*"],
			"dst":    ["*:*"],
		},
	],
	"ssh": [
		{
			"action": "accept",
			"src":    ["myself@gmail.com"],
			"dst":    ["myself@gmail.com"],
			"users":  ["autogroup:nonroot", "root", "myself"],
		},
	],
}

When I attempt to SSH to either host1 or host2 (both of which show as online in the admin console), I get the Permission denied (tailscale).

For whatever it is worth, on both hosts, when I run sudo tailscale up --ssh --advertise-tags=tag:server,tag:digocean I get an error message:

Tailscale SSH enabled, but access controls don’t allow anyone to access this device. Ask your admin to update your tailnet’s ACLs to allow access.

It seems like my ACLs are configured to allow this.

I suspect that UFW may be getting in the way with one of the hosts, but the other one doesn’t have a firewall running (yet), and it is performing exactly the same on both.

1 Like
5

Here are some interesting logs from the machine running --ssh

Sep 04 15:23:11 hostname-01 tailscaled[9676]: Accept: TCP{100.78.63.4:53454 > 100.69.15.21:22} 64 tcp ok
Sep 04 15:23:11 hostname-01 tailscaled[9676]: Accept: TCP{100.78.63.4:53454 > 100.69.15.21:22} 52 tcp non-syn
Sep 04 15:23:11 hostname-01 tailscaled[9676]: Accept: TCP{100.78.63.4:53454 > 100.69.15.21:22} 73 tcp non-syn
Sep 04 15:23:12 hostname-01 tailscaled[9676]: ssh-conn-20220904T152311-98f4864d46: handling conn: 100.78.63.4:53454->myself@100.69.15.21:22```

This looks to me like the traffic is being allowed. The odd thing is that the logs show that this is happening over the course of 2 seconds, but on the client side, it exits immediately (0.005 seconds).
6

I wonder if I should start my own thread? Don’t want to hijack this one if I can on a completely different path.

Here’s what I see when I crank up the verbosity on the client side:

myself@myselfs-mac ~ $ ssh -vvv 100.69.15.21
OpenSSH_8.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug2: resolve_canonicalize: hostname 100.69.15.21 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/myself/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/myself/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: ssh_connect_direct: entering
debug1: Connecting to 100.69.15.21 [100.69.15.21] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /Users/myself/.ssh/id_rsa type -1
debug1: identity file /Users/myself/.ssh/id_rsa-cert type -1
debug1: identity file /Users/myself/.ssh/id_dsa type -1
debug1: identity file /Users/myself/.ssh/id_dsa-cert type -1
debug1: identity file /Users/myself/.ssh/id_ecdsa type -1
debug1: identity file /Users/myself/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/myself/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/myself/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/myself/.ssh/id_ed25519 type 3
debug1: identity file /Users/myself/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/myself/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/myself/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/myself/.ssh/id_xmss type -1
debug1: identity file /Users/myself/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version Tailscale
debug1: compat_banner: no match: Tailscale
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 100.69.15.21:22 as 'myself'
debug3: record_hostkey: found key type ED25519 in file /Users/myself/.ssh/known_hosts:54
debug3: load_hostkeys_file: loaded 1 keys from 100.69.15.21
debug1: load_hostkeys: fopen /Users/myself/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-256,rsa-sha2-512,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:DjrdLsd16cSWDgKOQFrZVxL3jfONS74Jz7N8qohH8so
debug3: record_hostkey: found key type ED25519 in file /Users/myself/.ssh/known_hosts:54
debug3: load_hostkeys_file: loaded 1 keys from 100.69.15.21
debug1: load_hostkeys: fopen /Users/myself/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '100.69.15.21' is known and matches the ED25519 host key.
debug1: Found key in /Users/myself/.ssh/known_hosts:54
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/myself/.ssh/id_ed25519 ED25519 SHA256:ljKRTzRntahkHaIhmDcG1D1NCiCRJKg9+gC0q+Wlgfk agent
debug1: Will attempt key: /Users/myself/.ssh/id_rsa 
debug1: Will attempt key: /Users/myself/.ssh/id_dsa 
debug1: Will attempt key: /Users/myself/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/myself/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /Users/myself/.ssh/id_ed25519_sk 
debug1: Will attempt key: /Users/myself/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: tailscale
debug3: start over, passed a different list tailscale
debug3: preferred publickey,keyboard-interactive,password
debug1: No more authentication methods to try.
myself@100.69.15.21: Permission denied (tailscale).

Until the last 4 or 5 lines, this looks like a normal ssh connection negotiation using keys. But then it seems like the server is demanding an authentication type that the client doesn’t know about? Or the client is requesting an authentication type that the server doesn’t know about?

I get the same results when I pass -oPreferredAuthentications=tailscale to the ssh client.

7

I had a similar issue. Finally realized that, when you tag a machine, the owner gets removed (surprising)! Therefore, if you have tagged any hosts (as I see you have done), you also need to name the tags.

"tagOwners" doesn’t seem to affect anything. (also surprising)

Since Tailscale doesn’t seem to allow wildcards like "tag:*", as far as I can tell, you need to name every tag in every dst. (further surprise)

		{
			"action": "accept",
			"src":    ["autogroup:members"],
			"dst":    ["autogroup:self", "tag:production", "tag:staging", "tag:ci", "tag:wildcat"],
			"users":  ["autogroup:nonroot", "root"],
		},

Brief aside: I’m finding the SSH section unnecessarily complex and laborious to use… which is the first time I’ve said that about Tailscale!

But, now that I’ve named all my tags in the "dst" section, I can SSH successfully.

1 Like