Kubernetes Resource Conflicts: How VPA and Scheduler Mismatches Cause Production Outages

⚔️ Kubernetes Civil War: When VPA Fights the Scheduler (And Your Pods Pay the Price)

The Kubernetes Vertical Pod Autoscaler (VPA) can render pods permanently unschedulable by recommending resource requests that exceed the physical capacity of any single node in the cluster. While the scheduler makes a one-time placement promise, the VPA operates on continuous revision, often evicting pods without regard for topology or image locality.

Why This Matters

Technical automation models often fail when they ignore the physical constraints of the underlying infrastructure. In Kubernetes, the VPA and the scheduler represent two systems speaking different languages about the same resources; without explicit constraints like maxAllowed or PodDisruptionBudgets, this lack of coordination results in unplanned restarts, cold-start penalties, and SLO-burning 503 errors.

Key Insights

• The VPA Recommender builds an exponential decay histogram targeting the 90th percentile for CPU and memory to ensure workloads are spiky-traffic-aware.
• Concurrent use of HPA and VPA on the same CPU/Memory metrics creates a ‘Feedback Loop From Hell’ where competing controllers destabilize the cluster.
• The VPA Updater component proactively evicts pods using SIGTERM, ignoring graceful drains unless a PodDisruptionBudget (PDB) is explicitly defined.
• The Kubernetes scheduler has no memory of why it placed a pod, meaning every VPA-triggered reschedule loses carefully tuned topology and image locality context.

Working Examples

The non-negotiable fix to prevent VPA from recommending unschedulable resource sizes.

spec:
  resourcePolicy:
    containerPolicies:
    - containerName: api
      maxAllowed:
        cpu: "4"
        memory: 8Gi
      minAllowed:
        cpu: 100m
        memory: 128Mi

A safe HPA configuration using custom metrics to avoid feedback loops with VPA.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  metrics:
  - type: Pods
    pods:
      metric:
        name: requests_per_second
      target:
        type: AverageValue
        averageValue: 1000m

A PodDisruptionBudget acting as a ‘seatbelt’ to prevent VPA from evicting all replicas at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: "80%"
  selector:
    matchLabels:
      app: api-server

Practical Applications

• Use Case: Deploying stateless applications with updateMode: Auto while enforcing a PDB to maintain 80% availability during tuning. Pitfall: Running Auto mode without a PDB can lead to simultaneous eviction of all replicas.
• Use Case: Gradual VPA adoption following the SRE Graduation Ladder: starting with ‘Off’ for 2-4 weeks, then ‘Initial’, then ‘Auto’. Pitfall: Applying VPA to StatefulSets or batch jobs which cannot safely restart mid-operation.
• Use Case: Integrating KEDA for HPA scaling based on Kafka lag or SQS length while letting VPA handle resource right-sizing. Pitfall: Allowing both VPA and HPA to target CPU metrics, leading to oscillating pod counts and resource instability.

References:

• https://dev.to/npayyappilly/kubernetes-civil-war-when-vpa-fights-the-scheduler-and-your-pods-pay-the-price-3omo