The Runbook Is Already Lying to You: Solving Documentation Rot with AI Agents

The Runbook Is Already Lying to you.

Iyanu David highlights the terminal rot of static runbooks in high-velocity deployment environments. Datadog’s Bits agent reportedly achieved a 95% reduction in MTTR by automating routine triage tasks.

Why This Matters

The technical reality is that infrastructure changes faster than documentation can be updated, leading to knowledge entropy where SREs must treat runbooks as hypotheses rather than maps. While AI agents promise to bridge this gap via Retrieval-Augmented Generation, they introduce risks including garbage-in-garbage-out retrieval from uncurated indices and potential blast radius issues under automated execution.

Key Insights

• Datadog Bits agent compressed manual telemetry correlation across multiple surfaces into automated sequences, citing a 95% MTTR reduction.
• Retrieval-Augmented Generation (RAG) systems in incident response use vector indices to surface architecture docs and logs for LLM reasoning.
• Knowledge quality degrades in vector indices because semantic similarity does not equal epistemic quality, often surfacing outdated 2-year-old runbooks.
• PagerDuty’s tiered model categorizes incidents into Tier-1 fully automated, Tier-2 agent-assisted, and Tier-3 human-led responses.
• On-call fatigue shifts from high-volume interruptions to high-cognition triage of complex, novel failure modes that AI patterns cannot solve.

Practical Applications

• Use Case: Automating Tier-1 incidents like auto-scaling or dead-letter queue flushing where remediation is idempotent. Pitfall: Executing automated actions without situational awareness, such as scaling during a migration, can cause catastrophic state changes.
• Use Case: Enhancing vector retrieval by adding YAML frontmatter to runbooks with last-validated dates and alert types. Pitfall: Treating agent recommendations as authoritative without an advisory mode calibration period, leading to unverified execution.

References:

• https://dev.to/iyanu_david/the-runbook-is-already-lying-to-you-557o