Please help me figure out why my k8s proxy isn't working

Yep, I know, it’s a bad title. I’ll try to elaborate.

I’m following (roughly) the “Sample Proxy” section of the Kubernetes docs. Rather than check out the files and use the makefile, I copied the proxy YAML file and the RBAC files (what even are those called?) and manually substituted in the right values, after creating an nginx service as well:

❯ kubectl create deployment nginx --image nginx
deployment.apps/nginx created
❯ kubectl expose deployment nginx --port 80
service/nginx exposed
❯ kubectl get svc nginx -o=jsonpath='{.spec.clusterIP}'

I set that IP in the proxy.yaml file, as well as setting the TS_KUBE_SECRET to tailscale-auth-key as well as setting the serviceAccountName to my tailscale acct.

❯ kubectl get secrets
NAME                 TYPE     DATA   AGE
tailscale-auth-key   Opaque   5      20h

I got that running:

❯ kubectl apply -f k8s/proxy.yaml
pod/proxy created

And now I can see it in tailscale:

❯ tailscale status | grep proxy    proxy                offby1@      linux   -
❯ tailscale ping proxy
pong from proxy ( via DERP(sea) in 11ms
pong from proxy ( via DERP(sea) in 26ms
pong from proxy ( via DERP(sea) in 13ms
pong from proxy ( via in 6ms

However, when I try to curl it, it simply hangs (yes, I have magicDNS):

❯ curl http://proxy/
# ... wait forever here

If I look at the proxy logs, I see this as soon as I run curl ( is my laptop’s tailscape IP):

2022/12/13 02:39:39 wgengine: idle peer [NwM5W] now active, reconfiguring WireGuard
2022/12/13 02:39:39 wgengine: Reconfig: configuring userspace WireGuard config (with 1/14 peers)
2022/12/13 02:39:39 magicsock: disco: node [NwM5W] d:7af7c3ee54fbc194 now using
2022/12/13 02:39:39 Accept: TCP{ >} 64 tcp ok
2022/12/13 02:39:40 Accept: TCP{ >} 64 tcp ok
2022/12/13 02:39:41 Accept: TCP{ >} 64 tcp ok
2022/12/13 02:39:50 Accept: TCP{ >} 64 tcp ok

And I don’t see anything in the nginx container.

I’ve confirmed that I can connect to the nginx container from the proxy:

❯ kubectl exec --stdin --tty proxy -- /bin/sh
Defaulted container "tailscale" out of: tailscale, sysctler (init)
/ # nc 80
<!DOCTYPE html>
<title>Welcome to nginx!</title>

So… why is the proxy not working? I’ve clearly missed something.

Probably not very helpful, but I am having the exact same issue. Somehow in Kubernetes the proxy is only receiving inbound traffic (Accept: TCP) but isn’t sending anything back out. I don’t have a solution yet.

However, there does seem to be some progress on an official Kubernetes operator which could maybe give some insight on how to solve this? See k8s-operator feature request and docker hub.

having the same issue as well