Skip to main content
Back to Library
mastering ckad certified kubernetes application developer

Debugging and Logging Solutions

8 min read Chapter 63 of 87
Summary

Step-by-step solutions for Exercise 3 (attach an ephemeral...

Step-by-step solutions for Exercise 3 (attach an ephemeral debug container to a running nginx Pod) and Exercise 4 (retrieve logs from a crashed container using --previous). Includes all commands, expected output, and explanations.

Debugging and Logging Solutions

Exercise 3: Attach an Ephemeral Debug Container

Step 1: Create the Target Pod

Save the following manifest as debug-target.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: debug-target
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80

Apply it and wait for the Pod to be Running:

kubectl apply -f debug-target.yaml
kubectl wait --for=condition=Ready pod/debug-target --timeout=60s

Verify:

kubectl get pods debug-target
NAME           READY   STATUS    RESTARTS   AGE
debug-target   1/1     Running   0          10s

Step 2: Attempt Direct Debugging (Why Ephemeral Containers Exist)

Before using kubectl debug, consider what happens if you try kubectl exec:

kubectl exec -it debug-target -- wget -qO- http://localhost:80

This works for nginx because the nginx image includes basic utilities. But many production images are distroless — they do not include a shell, and kubectl exec fails:

# This would fail on a distroless image:
# kubectl exec -it distroless-pod -- /bin/sh
# error: Internal error occurred: error executing command in container:
# failed to exec in container: exec failed: unable to start container process:
# exec: "/bin/sh": stat /bin/sh: no such file or directory

Ephemeral containers solve this by adding a new container to the Pod with whatever debugging tools you need.

Step 3: Attach the Ephemeral Container

kubectl debug -it debug-target --image=busybox:1.36 --target=nginx

Breaking down the flags:

  • -it: Interactive terminal. -i passes stdin to the container, -t allocates a TTY.
  • --image=busybox:1.36: The image for the ephemeral container. BusyBox includes basic utilities: sh, wget, ps, ls, cat, netstat.
  • --target=nginx: Shares the process namespace with the nginx container. This lets you see nginx’s processes with ps. Without --target, the ephemeral container has its own isolated process namespace.

Expected output:

Defaulting debug container name to debugger-xxxxx.
If you don't see a command prompt, try pressing enter.
/ #

You now have a shell inside the ephemeral container, which shares the Pod’s network namespace and (via --target) the nginx container’s process namespace.

Step 4: Verify nginx Is Responding

From inside the ephemeral container:

wget -qO- http://localhost:80

Expected output:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
</html>

The ephemeral container shares the Pod’s network namespace, so localhost:80 reaches the nginx container’s port. This is the same behavior as any sidecar container — all containers in a Pod share the same network stack.

Step 5: Inspect Running Processes

ps aux

Expected output (with --target=nginx):

PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
   29 nginx     0:00 nginx: worker process
   30 nginx     0:00 nginx: worker process
   38 root      0:00 sh
   44 root      0:00 ps aux

The process list shows both the nginx processes (from the target container) and the current shell and ps command (from the ephemeral container). PID 1 is the nginx master process. This confirms the --target flag is working — without it, ps aux would show only the ephemeral container’s own processes.

Step 6: Additional Diagnostics

Check DNS resolution:

nslookup kubernetes.default.svc.cluster.local
Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

Check environment variables of the nginx process:

cat /proc/1/environ | tr '\0' '\n'
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=debug-target
NGINX_VERSION=1.25.5
...
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443

These diagnostics use the shared process namespace to read environment variables directly from nginx’s /proc/1/environ, which is accessible because both containers share the PID namespace.

Step 7: Exit and Verify

Exit the ephemeral container:

exit

The ephemeral container stops, but the Pod continues running:

kubectl get pods debug-target
NAME           READY   STATUS    RESTARTS   AGE
debug-target   1/1     Running   0          2m

The ephemeral container remains in the Pod spec. You can see it with:

kubectl describe pod debug-target

Under the Ephemeral Containers section:

Ephemeral Containers:
  debugger-xxxxx:
    Container ID:  containerd://abc123...
    Image:         busybox:1.36
    State:         Terminated
      Reason:      Completed
      Exit Code:   0
    Ready:          False

The ephemeral container cannot be removed — it stays in the Pod spec until the Pod is deleted. This is a known limitation.

Cleanup

kubectl delete pod debug-target

Exercise 4: Retrieve Logs from a Previous Container Crash

Step 1: Create the Crashing Pod

Save the following manifest as crash-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: crash-pod
spec:
  containers:
    - name: app
      image: busybox:1.36
      command:
        - sh
        - -c
        - "echo 'Starting up...' && echo 'FATAL: config not found' >&2 && exit 1"

This container prints a startup message to stdout, writes an error to stderr, and exits with code 1. It will immediately enter CrashLoopBackOff.

Apply it:

kubectl apply -f crash-pod.yaml

Step 2: Observe the CrashLoopBackOff

Wait a few seconds, then check the status:

kubectl get pods crash-pod

Expected output:

NAME        READY   STATUS             RESTARTS      AGE
crash-pod   0/1     CrashLoopBackOff   3 (10s ago)   25s

The container starts, runs the command, exits with code 1, and Kubernetes restarts it. The back-off delay increases with each restart.

Step 3: Attempt Normal Log Retrieval

kubectl logs crash-pod

This might show the current container instance’s output — which could be either the startup message from the latest attempt, or an empty result if the container is in back-off (waiting to restart). The output may look like:

Starting up...
FATAL: config not found

Or it may be empty if the container hasn’t restarted yet during the back-off period.

Step 4: Retrieve Previous Container Logs

The --previous flag retrieves logs from the container instance that ran before the current one:

kubectl logs crash-pod --previous

Expected output:

Starting up...
FATAL: config not found

Both stdout and stderr are captured by kubectl logs. The output contains:

  • Starting up... — written to stdout by echo 'Starting up...'
  • FATAL: config not found — written to stderr by echo 'FATAL: config not found' >&2

Kubernetes captures both streams. The --previous flag ensures you see the output from the container that crashed, not the one currently in back-off or just starting.

Step 5: Confirm the Exit Code

kubectl describe pod crash-pod

Look for the container state information:

Containers:
  app:
    Container ID:  containerd://abc123...
    Image:         busybox:1.36
    Command:
      sh
      -c
      echo 'Starting up...' && echo 'FATAL: config not found' >&2 && exit 1
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 01 Mar 2026 10:00:05 +0000
      Finished:     Sat, 01 Mar 2026 10:00:05 +0000
    Ready:          False
    Restart Count:  4

Key fields:

  • State: Waiting / CrashLoopBackOff — The container is in back-off, waiting for the next restart attempt.
  • Last State: Terminated / Error / Exit Code: 1 — The previous container exited with code 1. The Started and Finished timestamps are nearly identical, confirming the container ran for less than a second.
  • Restart Count: 4 — The container has been restarted 4 times. Each restart runs the same failing command.

The Events section confirms the pattern:

Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Normal   Scheduled  60s                default-scheduler  Successfully assigned default/crash-pod to ...
  Normal   Pulled     5s (x5 over 60s)   kubelet  Container image "busybox:1.36" already present on machine
  Normal   Created    5s (x5 over 60s)   kubelet  Created container app
  Normal   Started    5s (x5 over 60s)   kubelet  Started container app
  Warning  BackOff    3s (x8 over 55s)   kubelet  Back-off restarting failed container app in pod crash-pod_default(...)

The BackOff warning with x8 over 55s shows the exponential back-off pattern. The kubelet keeps restarting the container but increases the delay between attempts.

Step 6: Understanding the Back-Off Pattern

The CrashLoopBackOff delay follows an exponential pattern:

RestartDelay
1st10 seconds
2nd20 seconds
3rd40 seconds
4th80 seconds
5th160 seconds
6th+300 seconds (5-minute cap)

After the 5th restart, the delay caps at 5 minutes. This means debugging a CrashLoopBackOff becomes slower over time — the container runs less frequently, giving you fewer windows to capture its logs. Using --previous is essential because it preserves the most recent crash output regardless of the back-off state.

Cleanup

kubectl delete pod crash-pod

Key Takeaway

The --previous flag is the single most important debugging tool for CrashLoopBackOff Pods. Without it, you’re looking at the current container instance — which may be starting up, in back-off, or showing only a partial startup log. With --previous, you see the complete output from the container instance that failed. Combine it with kubectl describe pod to get the exit code, which narrows the diagnosis:

Exit CodeMeaning
0Normal exit (Pod’s restartPolicy triggers restart)
1Application error (check logs for details)
126Command not executable
127Command not found (wrong command in spec)
137SIGKILL (OOM kill or external termination)
143SIGTERM (graceful shutdown)