Please help me figure out why my k8s proxy isn't working

Yep, I know, it’s a bad title. I’ll try to elaborate.

I’m following (roughly) the “Sample Proxy” section of the Kubernetes docs. Rather than check out the files and use the makefile, I copied the proxy YAML file and the RBAC files (what even are those called?) and manually substituted in the right values, after creating an nginx service as well:

❯ kubectl create deployment nginx --image nginx
deployment.apps/nginx created
❯ kubectl expose deployment nginx --port 80
service/nginx exposed
❯ kubectl get svc nginx -o=jsonpath='{.spec.clusterIP}'
10.43.72.233

I set that IP in the proxy.yaml file, as well as setting the TS_KUBE_SECRET to tailscale-auth-key as well as setting the serviceAccountName to my tailscale acct.

❯ kubectl get secrets
NAME                 TYPE     DATA   AGE
tailscale-auth-key   Opaque   5      20h

I got that running:

❯ kubectl apply -f k8s/proxy.yaml
pod/proxy created

And now I can see it in tailscale:

❯ tailscale status | grep proxy
100.90.90.35    proxy                offby1@      linux   -
❯ tailscale ping proxy
pong from proxy (100.90.90.35) via DERP(sea) in 11ms
pong from proxy (100.90.90.35) via DERP(sea) in 26ms
pong from proxy (100.90.90.35) via DERP(sea) in 13ms
pong from proxy (100.90.90.35) via 192.168.5.167:59653 in 6ms

However, when I try to curl it, it simply hangs (yes, I have magicDNS):

❯ curl http://proxy/
# ... wait forever here

If I look at the proxy logs, I see this as soon as I run curl (100.113.157.9 is my laptop’s tailscape IP):

2022/12/13 02:39:39 wgengine: idle peer [NwM5W] now active, reconfiguring WireGuard
2022/12/13 02:39:39 wgengine: Reconfig: configuring userspace WireGuard config (with 1/14 peers)
2022/12/13 02:39:39 magicsock: disco: node [NwM5W] d:7af7c3ee54fbc194 now using 192.168.2.12:41641
2022/12/13 02:39:39 Accept: TCP{100.113.157.9:60069 > 100.90.90.35:80} 64 tcp ok
2022/12/13 02:39:40 Accept: TCP{100.113.157.9:60069 > 100.90.90.35:80} 64 tcp ok
2022/12/13 02:39:41 Accept: TCP{100.113.157.9:60069 > 100.90.90.35:80} 64 tcp ok
2022/12/13 02:39:50 Accept: TCP{100.113.157.9:60069 > 100.90.90.35:80} 64 tcp ok

And I don’t see anything in the nginx container.

I’ve confirmed that I can connect to the nginx container from the proxy:

❯ kubectl exec --stdin --tty proxy -- /bin/sh
Defaulted container "tailscale" out of: tailscale, sysctler (init)
/ # nc 10.43.72.233 80
GET /
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

So… why is the proxy not working? I’ve clearly missed something.

Probably not very helpful, but I am having the exact same issue. Somehow in Kubernetes the proxy is only receiving inbound traffic (Accept: TCP) but isn’t sending anything back out. I don’t have a solution yet.

However, there does seem to be some progress on an official Kubernetes operator which could maybe give some insight on how to solve this? See k8s-operator feature request and docker hub.

having the same issue as well