Debug Distroless and Scratch Containers Live with kubectl debug

What you'll build

You'll use kubectl debug to inject an ephemeral busybox (and later netshoot) container into a running scratch-based pod, share the target's PID namespace, and dig into its processes and filesystem using only tools from the debug image. The original container keeps running untouched.

Prerequisites

• Kubernetes 1.25+ (ephemeral containers are GA; GKE, EKS, and AKS enable them by default from that version)
• kubectl within one minor version of your cluster: run kubectl version and check both client and server
• update permission on pods/ephemeralcontainers in your target namespace
• Cluster nodes with outbound access to pull images from Docker Hub (or a mirror)

Step 1: Deploy a minimal target pod

We need a container with no shell and no package manager. The Kubernetes pause image is a static binary in a scratch-based image and fits perfectly as a stand-in for any distroless production app.

kubectl run minimal-app \
  --image=registry.k8s.io/pause:3.9 \
  --restart=Never

kubectl get pod minimal-app

It'll reach Running in seconds.

Step 2: Confirm the problem

kubectl exec -it minimal-app -- /bin/sh

OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown
error: command terminated with exit code 126

Same result with /bin/bash or any other shell. Nothing is there to run. This is the distroless debugging wall.

Step 3: Inject an ephemeral debug container

kubectl debug adds an ephemeral container to the live pod spec. The --target flag is what makes it useful: it tells the container runtime to run your debug container inside the same PID namespace as the named container, so you can see its processes.

kubectl debug -it minimal-app \
  --image=busybox:1.36 \
  --target=minimal-app \
  -- sh

Wait a moment for the image pull, then you'll drop into a shell. Without --target, your debug container gets its own isolated PID namespace and the target's processes are invisible. Don't skip it.

kubectl run names the container the same as the pod (minimal-app), so --target=minimal-app is correct here. For other pods, verify with:

kubectl get pod minimal-app -o jsonpath='{.spec.containers[*].name}'

Step 4: Inspect processes and filesystem

Inside the busybox shell:

ps aux

PID   USER     TIME  COMMAND
    1 root      0:00 /pause
   13 root      0:00 sh
   19 root      0:00 ps aux

PID 1 is the target container's process. The mount namespace is not shared via --target, only the PID namespace. Access the target's filesystem through /proc:

ls /proc/1/root/

For the pause image this is nearly empty, but for a real distroless Go or Python app you'd see your binary, config files, certificates, everything.

# Read the target process's environment variables
cat /proc/1/environ | tr '\0' '\n'

# Check open file descriptors
ls -la /proc/1/fd/

# Resolve the running binary path
readlink /proc/1/exe

The network namespace is shared automatically because all containers in a pod share the pod's network stack. No extra flags needed for that.

Step 5: Richer network debugging with netshoot

For real network diagnosis, swap in nicolaka/netshoot. It ships ss, tcpdump, dig, curl, nmap, and more. Exit the current session first, then:

kubectl debug -it minimal-app \
  --image=nicolaka/netshoot:latest \
  --target=minimal-app \
  -- bash

# Listening ports and active connections
ss -tlnp

# DNS resolution against the in-cluster resolver
dig kubernetes.default.svc.cluster.local

# Live packet capture on the pod interface
tcpdump -i eth0 -n port 8080

Pin to a specific netshoot version tag in CI or production tooling rather than :latest.

Verify it works

From a second terminal while the session is active:

kubectl describe pod minimal-app

Look for the Ephemeral Containers section near the bottom:

Ephemeral Containers:
  debugger-xxxxx:
    Image:    busybox:1.36
    State:    Running
    ...

Two behaviors to internalize: ephemeral containers are appended permanently to the pod spec and can't be removed without deleting the pod. They also can't restart if they exit. If you close the shell and need another session, just run kubectl debug again; a new ephemeral container gets appended.

Troubleshooting

"ephemeral containers are disabled for this cluster" You're on Kubernetes older than 1.25, or --feature-gates=EphemeralContainers=false was explicitly set on the API server. Upgrade the cluster or remove that flag from your API server config.

RBAC error: cannot update resource pods/ephemeralcontainers This subresource needs its own explicit grant, separate from basic pod permissions. Check your access:

kubectl auth can-i update pods/ephemeralcontainers

If it returns no, ask your cluster admin to add update on pods/ephemeralcontainers to your role.

ps aux shows only your own shell, not the target process You either omitted --target or the container name is wrong. Verify the container name and re-run with the correct --target value.

Debug image fails to pull on the node Cluster nodes in restricted environments may not reach Docker Hub. Mirror busybox:1.36 or nicolaka/netshoot to your internal registry (ECR, Artifact Registry, Harbor) and reference that URL instead.

Next steps

• Copy mode for crashlooping pods: kubectl debug -it <pod> --copy-to=debug-copy --set-image=app=<new-image> clones the pod with a different image. Useful when the original container is in CrashLoopBackOff and there's nothing live to attach to.
• --profile=sysadmin (kubectl 1.27+): Grants the ephemeral container elevated privileges. With it, you can run nsenter -t 1 -m -- sh inside the debug container to fully enter the target's mount namespace, or attach strace to the target PID.
• Scoped RBAC: Create a debug-role ClusterRole granting only update pods/ephemeralcontainers, bound to a short-lived service account. Keeps audit logs clean and limits blast radius.
• The Kubernetes ephemeral containers reference covers the full spec fields if you need to automate injection via the API directly rather than through kubectl debug.