Preventing AI Agent Configuration Drift with Agent Contract Testing

I built a tiny CI tool to keep AI agent configs from drifting in my repo

Developer Ramanpreet Singh built a lightweight Python validator to enforce tool permissions for AI agents within the CI pipeline. The tool utilizes a YAML-based registry to define agent Access Control Lists (ACLs) and call graphs to prevent unauthorized tool execution.

Why This Matters

In modern development, AI agent permissions are often scattered across READMEs and prompts, creating a gap where CI pipelines fail to detect when actual behavior drifts from documentation. This tool addresses the risk of agents executing unauthorized or sensitive actions, such as direct email sending or production deletion, by shifting permission enforcement from external documentation to repo-level contract tests.

Key Insights

• The tool uses a YAML registry to define tool-ACLs, preventing agents from calling unauthorized tools like ‘production_delete’ or ‘direct_email_send’.
• A Python-based validator script fails CI builds if an agent declares a tool not granted in the ACL or attempts an invalid call graph jump.
• A ~100-line Python module ‘agent_ops_guard’ provides runtime enforcement, raising ‘PolicyDenied’ before sensitive tools are executed in the runner.
• The system is designed as a narrow contract-test layer, specifically excluding orchestration, sandboxing, or LLM evaluation scopes covered by tools like LangGraph or Promptfoo.
• The ‘agent_ops_validate.py’ script provides a —strict mode to validate the repo against defined policy files without external dependencies.

Working Examples

YAML configuration defining tool permissions and blocked actions for specific agents.

# .agent-ops/registry/tool-acl.yaml
backend-builder:
  tools:
  - repo_read
  - repo_write_backend
  - run_backend_tests
security-reviewer:
  tools:
  - repo_read
  - dependency_scan
blocked_tools:
  - direct_email_send
  - production_delete

Python runtime enforcement module to block actions before tool execution.

from agent_ops_guard import AgentOpsGuard
guard = AgentOpsGuard(".")
guard.assert_tool_allowed("backend-builder", "repo_read")
guard.assert_call_allowed("orchestrator", "backend-builder")

Practical Applications

• Use Case: CI validation for AI agents where a ‘backend-builder’ is restricted to ‘repo_read’ and ‘run_backend_tests’. Pitfall: Relying on scattered Notion docs leads to silent permission drift and unauthorized tool access.
• Use Case: Runtime enforcement using ‘AgentOpsGuard’ to block unauthorized agent-to-agent calls in the orchestrator layer. Pitfall: Treating agent rules as process isolation rather than repo-level contracts can lead to architecture bypass.

References:

• https://dev.to/ramanpreet_singh_701cac53/i-built-a-tiny-ci-tool-to-keep-ai-agent-configs-from-drifting-in-my-repo-4lod
• https://github.com/RPSingh1990/agent-contract-tests