Anthropic Releases Bloom: An Open-Source Framework for AI Behavioral Evaluation

Bloom: Automated Behavioral Evaluations for Frontier AI Models

Anthropic has released Bloom, an open-source agentic framework designed to automate behavioral evaluations of leading-edge AI models. The system transforms a researcher-defined behavior into targeted evaluations, measuring prevalence and strength across realistic scenarios.

Behavioral evaluations for AI safety and alignment are traditionally expensive and time-consuming, requiring manual scenario creation, interaction analysis, and scoring. As models rapidly evolve, maintaining relevant and non-contaminated benchmarks is a significant challenge, potentially costing organizations substantial resources in engineering time and impacting model reliability.

Key Insights

• Four-stage agentic pipeline: Bloom utilizes agents for understanding, ideation, rollout, and judgment to automate evaluation creation.
• LiteLLM integration: Bloom leverages LiteLLM for simplified API access to models from Anthropic and OpenAI.
• Correlation with human judgment: Claude Opus 4.1 reached a Spearman correlation of 0.86 with human labels when used as a judge model.

Working Example

# Example seed.yaml configuration
behavior: "sycophancy"
examples:
  - path: "behaviors/examples/sycophancy_example_1.json"
total_evals: 100
rollout.target: "claude-sonnet-4"
diversity: 0.7
max_turns: 5
modality: "text"

Practical Applications

• AI Safety Teams: Automate the creation of red-teaming evaluations for identifying and mitigating harmful behaviors in large language models.
• Pitfall: Relying solely on automated evaluations without human oversight can miss nuanced or unexpected failure modes.

References:

• https://www.marktechpost.com/2025/12/21/anthropic-ai-releases-bloom-an-open-source-agentic-framework-for-automated-behavioral-evaluations-of-frontier-ai-models/
• https://github.com/[link to Github repo - not provided in context]
• https://[link to Technical report - not provided in context]
• https://[link to Blog - not provided in context]