Logging, Events, and Debug Containers

When a Pod misbehaves, probes tell Kubernetes that something is wrong. Logs and events tell you what went wrong. Probes trigger automated responses — restarts, endpoint removal. Logs and events enable human diagnosis.

This section covers the full debugging toolkit: how to retrieve container output, how to read cluster events, how to systematically diagnose common failure states, and how to attach debug containers to running Pods when the original image lacks diagnostic tools.

Container Logs with kubectl logs

Every container in Kubernetes writes to stdout and stderr. The kubelet captures this output and stores it on the node. kubectl logs retrieves it.

Basic Usage

kubectl logs my-pod

This prints the current log output from the only container in the Pod, or the first container if there are multiple. For single-container Pods, this is the most common invocation.

Multi-Container Pods

When a Pod has more than one container, you must specify which container’s logs to retrieve:

kubectl logs my-pod -c sidecar

Without -c, kubectl returns an error listing the available container names. If the Pod has an init container whose logs you need:

kubectl logs my-pod -c init-db

Init container logs are available after the init container has completed (or failed). They are essential for debugging Pods stuck in Init:Error or Init:CrashLoopBackOff.

Previous Container Logs

When a container crashes and restarts, the current logs belong to the new instance. The previous instance’s logs — the ones that contain the crash — are available with --previous:

kubectl logs my-pod --previous

For multi-container Pods:

kubectl logs my-pod -c app --previous

This flag is indispensable for CrashLoopBackOff debugging. The current container may have only startup messages. The previous container’s logs contain the error that caused the crash. Kubernetes retains the previous container’s logs until the next restart or until the kubelet’s log retention policy evicts them.

Following Logs in Real Time

kubectl logs my-pod -f

The -f flag streams log output, similar to tail -f. This is useful for watching a running container’s behavior in real time — watching requests come in, watching for error patterns, observing startup sequences. Press Ctrl+C to stop following.

Combine with --since to avoid fetching the entire log history:

kubectl logs my-pod -f --since=5m

This starts streaming from 5 minutes ago, skipping older entries.

Filtering by Label

To stream logs from all Pods matching a label selector:

kubectl logs -l app=web

This retrieves logs from every Pod with the label app=web. Add --all-containers to include all containers in multi-container Pods:

kubectl logs -l app=web --all-containers

And --prefix to prepend the Pod and container name to each line:

kubectl logs -l app=web --all-containers --prefix

Output:

[pod/web-abc12/app] GET /api/users 200 12ms
[pod/web-def34/app] GET /api/users 200 15ms
[pod/web-abc12/sidecar] syncing config...

This is invaluable when debugging a Deployment with multiple replicas — you can see all replicas’ output interleaved with clear identification.

Tail and Timestamps

Limit the number of lines returned:

kubectl logs my-pod --tail=50

Add timestamps to each log line:

kubectl logs my-pod --timestamps

Output:

2026-03-01T10:15:32.123456789Z Starting application...
2026-03-01T10:15:33.456789012Z Listening on port 8080

Cluster Events

Events are Kubernetes objects that record state transitions and significant occurrences. They are stored in etcd and have a default retention of one hour (configurable by the cluster administrator). Events provide context that logs cannot — they describe what Kubernetes itself did, not what the application did.

Viewing Events via kubectl describe

kubectl describe pod my-pod

The Events section at the bottom of the output shows events related to this specific Pod:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned default/my-pod to node-1
  Normal   Pulling    2m                 kubelet            Pulling image "nginx:1.25"
  Normal   Pulled     1m50s              kubelet            Successfully pulled image "nginx:1.25"
  Normal   Created    1m50s              kubelet            Created container nginx
  Normal   Started    1m49s              kubelet            Started container nginx

This is the most common way to investigate a specific Pod. The events tell a story: image was pulled, container was created, container was started. When something goes wrong, the story breaks:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  5m                 default-scheduler  Successfully assigned default/my-pod to node-1
  Normal   Pulling    5m                 kubelet            Pulling image "nginx:latest"
  Warning  Failed     4m (x3 over 5m)    kubelet            Error: ImagePullBackOff
  Warning  Failed     4m                 kubelet            Failed to pull image "nginx:latest": rpc error: code = Unknown desc = Error response from daemon: manifest not found

Event Types

Kubernetes events have two types:

Normal: Expected operations — scheduling, image pulling, container starting, probe passing.
Warning: Unexpected or problematic operations — failed image pulls, probe failures, insufficient resources, back-off restarts, OOM kills.

Warning events are your primary signal that something needs attention.

Viewing All Events

To see all events in the current namespace:

kubectl get events

Sort by creation timestamp to see the most recent events last:

kubectl get events --sort-by='.lastTimestamp'

Filter to Warning events:

kubectl get events --field-selector type=Warning

Watch events in real time:

kubectl get events --watch

For events across all namespaces:

kubectl get events --all-namespaces --sort-by='.lastTimestamp'

Events disappear after their retention period. If you need historical events, you need a log aggregation pipeline — but that is outside the CKAD scope.

Systematic Debugging by Pod State

Every Pod failure manifests as a specific status visible in kubectl get pods. Each status points to a different category of problem. A systematic approach saves time in both production incidents and exam scenarios.

Pending

Symptoms: Pod stays in Pending state indefinitely.

Meaning: The scheduler cannot place the Pod on any node.

Diagnosis:

kubectl describe pod pending-pod

Look at the Events section for scheduling messages:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  30s   default-scheduler  0/3 nodes are available:
           3 Insufficient cpu.

Common causes:

Cause	Event Message	Fix
Insufficient CPU/memory	”Insufficient cpu” or “Insufficient memory”	Reduce resource requests, add nodes, or free resources
No matching node selector	”didn’t match Pod’s node affinity/selector”	Correct the `nodeSelector` or add labels to nodes
Taint not tolerated	”had taint {key=value:NoSchedule}“	Add a toleration to the Pod spec
PVC not bound	”persistentvolumeclaim is not bound”	Create a matching PV or fix the StorageClass

Key point: Pending Pods have never started a container. There are no container logs to inspect. Events and kubectl describe are your only tools.

CrashLoopBackOff

Symptoms: Pod repeatedly starts and crashes. Status alternates between Running and CrashLoopBackOff. Restart count climbs.

Meaning: The container starts, runs for a few seconds (or milliseconds), and exits with a non-zero exit code. Kubernetes restarts it, it crashes again, and the back-off delay increases exponentially (10s, 20s, 40s, up to 5 minutes).

Diagnosis:

# Check the exit code
kubectl describe pod crashing-pod | grep -A5 "Last State"

# Read the PREVIOUS container's logs (the crash)
kubectl logs crashing-pod --previous

Common causes:

Cause	What You See	Fix
Application error	Stack trace in `logs --previous`	Fix the application code or configuration
Missing config/secret	”file not found” or “env var not set” in logs	Mount the correct ConfigMap or Secret
Wrong command	`Exit Code: 127` (command not found)	Fix the `command` or `args` in the container spec
Liveness probe too aggressive	”Liveness probe failed” in events	Increase `initialDelaySeconds` or add a startup probe
OOM Kill	`Exit Code: 137`, OOMKilled reason	Increase memory limits or fix the memory leak

Key point: Always use --previous to see the logs from the container that actually crashed. The current container’s logs show a fresh start, not the failure.

ImagePullBackOff

Symptoms: Pod stays in ImagePullBackOff or ErrImagePull state.

Meaning: The kubelet cannot pull the container image.

Diagnosis:

kubectl describe pod image-pod

The Events section reveals the specific error:

Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Warning  Failed     10s   kubelet  Failed to pull image "myrepo/app:v2":
           rpc error: code = NotFound desc = failed to pull and unpack image:
           manifest unknown

Common causes:

Cause	Event Message	Fix
Image does not exist	”manifest unknown”	Verify the image name and tag
Tag does not exist	”not found”	Check available tags in the registry
Private registry, no credentials	”unauthorized” or “access denied”	Create an `imagePullSecret` and reference it in the Pod spec
Registry unreachable	”dial tcp: lookup registry.example.com: no such host”	Check network connectivity and DNS

Running but No Traffic

Symptoms: Pod shows Running status and 1/1 ready, but clients cannot reach it.

Meaning: The Pod is running, but traffic is not reaching it through the Service.

Diagnosis:

# Check if the Service selects the Pod
kubectl get endpoints my-service

# Check if readiness probe is passing
kubectl describe pod my-pod | grep -A3 Readiness

# Check Pod labels match Service selector
kubectl get pod my-pod --show-labels
kubectl get service my-service -o jsonpath='{.spec.selector}'

Common causes:

Cause	What You See	Fix
Readiness probe failing	`0/1` in READY column, empty endpoints	Fix the readiness probe or application
Label mismatch	Pod labels don’t match Service selector	Correct Pod labels or Service selector
Wrong port	Service `targetPort` doesn’t match container port	Align the port numbers
NetworkPolicy blocking	Endpoints present but connections refused	Check NetworkPolicy rules

Ephemeral Debug Containers

Minimal container images — distroless, scratch-based, Alpine without a shell — lack diagnostic tools. You cannot kubectl exec into a container that has no shell. You cannot run curl in a container that does not have it. Ephemeral containers solve this problem.

An ephemeral container is a temporary container added to a running Pod for debugging purposes. It shares the Pod’s network namespace (and optionally the process namespace of another container), giving it access to localhost ports, network interfaces, and process listings. It is never restarted and does not have ports, probes, or resource limits applied.

Attaching a Debug Container

kubectl debug -it my-pod --image=busybox:1.36 --target=app

This command:

Adds an ephemeral container using the busybox:1.36 image to the running Pod my-pod.
The --target=app flag shares the process namespace with the app container, letting you see its processes with ps aux.
The -it flags attach an interactive terminal.

Inside the debug container, you can run diagnostic commands:

# Check network connectivity from the Pod's perspective
wget -qO- http://localhost:8080/healthz

# List processes in the target container
ps aux

# Check DNS resolution
nslookup my-service.default.svc.cluster.local

# Inspect environment variables of the target process
cat /proc/1/environ | tr '\0' '\n'

# Check open ports
netstat -tlnp

Using a Full Debug Image

For more advanced debugging, use an image with a richer toolkit:

kubectl debug -it my-pod --image=nicolaka/netshoot --target=app

The netshoot image includes curl, dig, tcpdump, strace, iperf, and other network debugging tools.

Ephemeral Container Limitations

Ephemeral containers cannot be removed from a Pod once added. They remain in the Pod spec until the Pod is deleted.
They do not support probes, ports, or resource limits.
They require the EphemeralContainers feature gate (enabled by default since Kubernetes 1.25).
They are visible in kubectl describe pod under the Ephemeral Containers section.

Node Debugging

Sometimes the problem is not the Pod but the node. Disk pressure, network misconfiguration, kubelet issues, or container runtime problems require node-level investigation. kubectl debug node creates a privileged Pod on the specified node with the node’s root filesystem mounted.

kubectl debug node/my-node -it --image=ubuntu

Inside the debug Pod:

# Access the node's filesystem
chroot /host

# Check kubelet logs
journalctl -u kubelet --no-pager --since "10 minutes ago"

# Check disk space
df -h

# Check running containers
crictl ps

# Check node network
ip addr show
iptables -L -n

The /host directory is the node’s root filesystem. By running chroot /host, you operate as if you were SSH’d into the node. This is a powerful capability — use it responsibly and only when node-level diagnosis is needed.

CKAD Exam Tips

Know kubectl logs --previous. The exam regularly places a CrashLoopBackOff scenario that requires retrieving the previous container’s logs. Practice this until it is instinctive.
Events first, logs second. For Pending and ImagePullBackOff states, there are no container logs. kubectl describe pod events are the only diagnostic.
Familiarize yourself with kubectl debug. Ephemeral containers are testable exam material. Know the command syntax and the --target flag.
Sort events by timestamp. kubectl get events --sort-by='.lastTimestamp' puts the most recent — and most relevant — events at the bottom.

Exercises

Exercise 1: Diagnose and Fix a CrashLoopBackOff from a Broken Liveness Probe

Create a Pod with a liveness probe that checks a nonexistent HTTP path. Observe the Pod entering CrashLoopBackOff. Identify the problem using events and logs, then fix the probe to point to a valid path.

Pod spec requirements:

Image: nginx:1.25
Liveness probe: httpGet on path /does-not-exist, port 80
periodSeconds: 2, failureThreshold: 3

Steps:

Create the Pod and wait for CrashLoopBackOff.
Use kubectl describe pod to identify the probe failure in events.
Fix the liveness probe path to / (nginx returns 200 on the root path).
Verify the Pod becomes healthy and stays Running.

Exercise 2: Diagnose a Pod Stuck in Pending

Create a Pod that requests an unreasonable amount of CPU, causing it to stay in Pending indefinitely. Diagnose the issue from events.

Pod spec requirements:

Image: nginx:1.25
Resource request: cpu: 100 (100 whole CPUs — no node can satisfy this)

Steps:

Create the Pod and observe it staying in Pending.
Use kubectl describe pod and kubectl get events to identify the scheduling failure.
Note the specific event message explaining why no node can accommodate the Pod.
Delete the Pod.

Exercise 3: Attach an Ephemeral Debug Container

Create a running nginx Pod, then attach an ephemeral debug container to it. Use the debug container to verify the nginx process is running and the web server is responding.

Steps:

Create a Pod running nginx:1.25.
Use kubectl debug to attach a busybox:1.36 ephemeral container with --target=nginx.
Inside the debug container, run wget -qO- http://localhost:80 to verify nginx is serving.
Run ps aux to see the nginx processes (visible because of shared process namespace).
Exit the debug container.

Exercise 4: Retrieve Logs from a Previous Container Crash

Create a Pod that crashes immediately on startup, then retrieve the logs from the crashed container.

Pod spec requirements:

Image: busybox:1.36
Command: sh -c "echo 'Starting up...' && echo 'FATAL: config not found' >&2 && exit 1"

Steps:

Create the Pod. It will crash and enter CrashLoopBackOff.
Use kubectl logs --previous to retrieve the previous container’s output.
Identify the error message (FATAL: config not found) in the logs.
Use kubectl describe pod to confirm the exit code is non-zero.
Delete the Pod.