Reclaim Policy and emptyDir Solutions

Solution: Exercise 3 — Reclaim Policy Observation

This exercise has two parts that demonstrate the behavioral difference between the Delete and Retain reclaim policies. You will create PVCs, delete them, and observe what happens to the underlying PVs.

Part A: Delete Reclaim Policy

The standard StorageClass in Kind has a Delete reclaim policy. When a PVC using this class is deleted, the dynamically provisioned PV is also deleted along with its data.

Step 1: Create the PVC

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: delete-test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
EOF

The PVC will show Pending because Kind’s standard StorageClass uses WaitForFirstConsumer. Create a temporary Pod to trigger provisioning:

kubectl run pv-trigger --image=busybox:1.36 --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "busybox",
        "image": "busybox:1.36",
        "command": ["sleep", "60"],
        "volumeMounts": [{"name": "vol", "mountPath": "/mnt"}]
      }],
      "volumes": [{"name": "vol", "persistentVolumeClaim": {"claimName": "delete-test"}}]
    }
  }'

Wait for the Pod to start:

kubectl get pod pv-trigger -w

Step 2: Record the PV Name

Once the PVC is bound, record the dynamically created PV name:

PV_NAME=$(kubectl get pvc delete-test -o jsonpath='{.spec.volumeName}')
echo "PV name: $PV_NAME"

Output:

PV name: pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890

Verify the PV exists:

kubectl get pv "$PV_NAME"

Expected output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   AGE
pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890   1Gi        RWO            Delete           Bound    default/delete-test  standard       20s

Note the RECLAIM POLICY column shows Delete.

Step 3: Delete the Pod and PVC

First delete the Pod so the PVC is no longer in use:

kubectl delete pod pv-trigger

Then delete the PVC:

kubectl delete pvc delete-test

Step 4: Check the PV

kubectl get pv "$PV_NAME" 2>/dev/null || echo "PV not found — it was deleted"

Output:

PV not found — it was deleted

What happened: The Delete reclaim policy instructs Kubernetes to remove the PV object and its backing storage when the PVC is deleted. The local-path-provisioner cleaned up the directory on the node. This is the default behavior for dynamically provisioned volumes in most StorageClasses, including Kind’s standard class.

This is the expected production behavior for disposable storage — build caches, temporary computation results, test data. You do not want orphaned disks accumulating costs with no PVC referencing them.

---

Part B: Retain Reclaim Policy

With the Retain policy, deleting the PVC leaves the PV and its data intact. The PV transitions to Released but cannot be rebound to a new PVC without administrator intervention.

Step 1: Create the PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: retain-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/retain-data
    type: DirectoryOrCreate

Save as retain-pv.yaml and apply:

kubectl apply -f retain-pv.yaml

Verify the PV is available:

kubectl get pv retain-pv

Expected output:

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   AGE
retain-pv   1Gi        RWO            Retain           Available           manual          5s

The PV shows Available — no PVC has claimed it yet.

Step 2: Create the PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: retain-test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: manual

Save as retain-test-pvc.yaml and apply:

kubectl apply -f retain-test-pvc.yaml

Verify binding:

kubectl get pvc retain-test

Expected output:

NAME          STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
retain-test   Bound    retain-pv   1Gi        RWO            manual         5s

The PVC has bound to retain-pv because the capacity (1Gi), access mode (RWO), and StorageClass (manual) all match. Unlike the dynamic provisioning in Part A, this is static provisioning — the PV was created ahead of time and Kubernetes found it when the PVC was submitted.

Also verify from the PV side:

kubectl get pv retain-pv

Expected output:

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   AGE
retain-pv   1Gi        RWO            Retain           Bound    default/retain-test   manual          30s

The CLAIM column now shows default/retain-test, confirming the binding.

Step 3: Delete the PVC

kubectl delete pvc retain-test

Step 4: Check the PV Status

kubectl get pv retain-pv

Expected output:

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                 STORAGECLASS   AGE
retain-pv   1Gi        RWO            Retain           Released   default/retain-test   manual          1m

The PV is now Released, not Available. The CLAIM column still shows default/retain-test even though that PVC no longer exists.

Why the PV Is Not Available

A Released PV retains a reference to its former PVC in the spec.claimRef field. Kubernetes will not bind a new PVC to this PV because the old claim reference still exists. This is a safety mechanism — it prevents a new PVC from accidentally accessing another application’s data.

Inspect the claim reference:

kubectl get pv retain-pv -o jsonpath='{.spec.claimRef}' | python3 -m json.tool

Output:

{
    "apiVersion": "v1",
    "kind": "PersistentVolumeClaim",
    "name": "retain-test",
    "namespace": "default",
    "uid": "abc12345-def6-7890-ghij-klmn12345678"
}

How an Administrator Reuses a Retained PV

To make the PV available for a new PVC, an administrator must remove the claimRef:

kubectl patch pv retain-pv --type=json -p='[{"op": "remove", "path": "/spec/claimRef"}]'

Verify the PV is back to Available:

kubectl get pv retain-pv

Expected output:

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   AGE
retain-pv   1Gi        RWO            Retain           Available           manual          2m

Now the PV can be bound to a new PVC. The data from the previous usage is still on the hostPath directory (/mnt/retain-data). The administrator can choose to clean it up or leave it for the next consumer.

Summary: Delete vs Retain

Aspect	Delete	Retain
PV after PVC deletion	Deleted automatically	Moves to Released
Backing storage	Removed by provisioner	Left intact
New PVC can bind?	N/A (PV gone)	Not until claimRef is removed
Use case	Disposable, ephemeral data	Databases, critical data
Default on Kind?	Yes (standard class)	No (must set explicitly)

Common Mistakes

Forgetting to delete the Pod before the PVC: If a Pod is still mounting a PVC, the PVC enters Terminating state but is not actually deleted until the Pod is removed. Delete the Pod first, then the PVC.

Expecting a Released PV to auto-rebind: With Retain policy, the PV stays Released indefinitely. You must manually patch out the claimRef or delete and recreate the PV.

Wrong StorageClass matching: In Part B, the PVC’s storageClassName: manual must match the PV’s storageClassName: manual. A mismatch causes the PVC to stay Pending because no PV in the manual class satisfies the request, and no provisioner exists for the manual class to create one dynamically.

Confusing Recycle with Retain: The Recycle policy (which wipes data and makes the PV available again) is deprecated. If you see it in older documentation, ignore it. The CKAD tests Retain and Delete only.

Cleanup

kubectl delete pv retain-pv --ignore-not-found