Qwen-Powered Agent Automates Data Investigations at $0.004 Per Query – With Zero False Positives

I Built an Agent That Refuses to Act — Here’s Why Qwen Made It Possible

Demba Fofana developed prove-or-abstain, an AI agent that replicates a full data team investigation in two seconds for $0.004 per query. The agent achieved perfect benchmark accuracy across three different Qwen model tiers without any false assertions or false abstentions.

Why This Matters

In practice, SaaS teams spend over $150K annually chasing metric anomalies because manual investigations are slow and error-prone—each drop triggers a four-hour process costing ~$900 with no guarantee of correct diagnosis. Ideal models assume perfect information and unbiased reasoning; reality introduces confirmation bias where analysts chase ghost causes based on gut feelings rather than rigorous statistical gates.

Key Insights

• The four mathematical gates – materiality (>2% drop), localization (≥55% concentrated), significance (two-proportion z-test at p ≤ 0.01), clean separation (rate vs mix effect disentangled) – prevent any action unless all conditions pass perfectly.
• A self‑verified column mapping system uses two‑pass verification via Qween LLM catching misclassifications like treating Temperature as dimension instead covariate then labeling self_verified=false transparently auditable.
• When tested against ten real public datasets including Titanic passenger manifests university majors employment stats airline counts penguin populations car fuel economy Gapminder seven out ten produced ABSTAIN because most real-world shifts are genuinely systemic not actionable locally saving users from costly wrong diagnoses.
• The determinism guard tests every dimension skipped during initial LLM-driven exploration preventing lazy hallucinations where model might claim nothing worth examining while actually missing something important ensuring traceable decision path back pure math.

Working Examples

Clone repository run mock mode locally requiring zero API keys

git clone https://github.com/Demba09/prove-or-abstain
cd prove-or-abstain && pip install -r requirements.txt
QWEN_MOCK=1 uvicorn api.app:app
# Open http://localhost:8000 — no API key needed

Practical Applications

• Data engineering team investigating conversion drops – pitfall spending hours segmenting only find systemic shift whereas this agent reaches same conclusion instantly without human bias
• Product manager triggering rollback based incomplete analysis – pitfall acting entangled rate/mix effects wastes developer time reverts features unnecessarily due lack statistical rigor prevented automatic ABSTAIN escalation naming exact failing condition

References:

• From internal analysis