Securing AI Agents at the Tool Layer with agent-probe v0.5.0

You Can Security-Test Any AI Agent in 3 Lines of Python

Developer Jackson released agent-probe v0.5.0 to address the critical vulnerability gap where AI agents fail at the tool and memory layers rather than just the LLM. The tool enables deterministic security probing of any Python-based agent framework with just three lines of code.

Why This Matters

Technical security for AI agents has traditionally focused on prompt-level red-teaming, yet real-world failures occur when bad delegation turns an agent into an attacker’s proxy. While tools like PyRIT and Garak test model outputs, they often miss confused deputy attacks and parameter injection in multi-step workflows. By targeting the function layer directly, engineers can prevent privilege escalation and data exfiltration before deployment, avoiding the high cost of resource abuse or system prompt leakage in production environments.

Key Insights

• agent-probe v0.5.0 introduces FunctionTarget to wrap any callable agent, eliminating the need for HTTP-only testing bottlenecks (Jackson, 2026)
• The tool executes 20 probes across 7 categories, specifically targeting ASI-07 system prompt extraction and memory poisoning attacks
• SARIF 2.1.0 output support allows for seamless integration with GitHub Security and CodeQL, providing structured remediation data
• A zero-dependency architecture ensures the tool remains lightweight and secure, utilizing only the Python standard library
• Deterministic pattern-based probing removes the need for expensive LLM API keys during the security testing phase

Working Examples

Wrapping a standard Python function as a security probe target

from agent_probe import FunctionTarget, run_probes, format_text_report

def my_agent(message: str) -> str:
    # ... your agent logic ...
    return response

target = FunctionTarget(my_agent, name="my-agent")
results = run_probes(target)
print(format_text_report(results))

Integrating agent-probe with the LangChain framework

from langchain.agents import AgentExecutor

executor = AgentExecutor(agent=agent, tools=tools)
target = FunctionTarget(
    lambda msg: executor.invoke({"input": msg})["output"],
    name="langchain-agent",
)

GitHub Actions workflow for automated agent security gating

- name: Run agent security probes
  run: |
    python -c "
    from agent_probe import FunctionTarget, run_probes, format_sarif
    from my_app.agent import chat
    target = FunctionTarget(chat, name='my-agent')
    results = run_probes(target)
    with open('agent-probe.sarif', 'w') as f:
        f.write(format_sarif(results))
    if results.overall_score < 70:
        raise SystemExit(f'Score {results.overall_score}/100 below threshold')
    "

Practical Applications

• Use Case: Implementing FunctionTarget within unit test suites to detect parameter injection in tool calls during local development. Pitfall: Relying on stateless LLM testers that fail to catch multi-step memory poisoning attacks.
• Use Case: Exporting security findings to SARIF format for centralized vulnerability management in platforms like Defect Dojo. Pitfall: Treating AI agent security as a separate silo rather than integrating it into standard CI/CD security gates.
• Use Case: Protecting against A2A (Agent-to-Agent) privilege escalation by analyzing structured tool call responses for unsafe patterns. Pitfall: Assuming that model-level safety filters will prevent tool-layer abuse in complex autonomous workflows.

References:

• https://dev.to/claude-go/you-can-security-test-any-ai-agent-in-3-lines-of-python-4gmb
• https://github.com/claude-go/agent-probe.git