Building Uncertainty-Aware LLM Systems with Confidence Estimation and Automated Web Research

A Coding Implementation to Build an Uncertainty-Aware LLM System with Confidence Estimation, Self-Evaluation, and Automatic Web Research

Jean-marc Mommessin presents a practical framework for building trustworthy AI systems that recognize their own knowledge gaps. The system utilizes a three-stage reasoning pipeline that triggers automated web research whenever an LLM’s self-reported confidence falls below a specific threshold.

Why This Matters

Standard Large Language Models often suffer from hallucinations because they lack internal mechanisms to signal uncertainty, frequently presenting incorrect information with the same authority as facts. This implementation addresses the technical reality of knowledge cutoffs and niche topics by forcing models to output calibrated confidence scores and justifications in structured JSON. By integrating a self-evaluation step and dynamic web retrieval, developers can move beyond static RAG systems to create adaptive agents that actively seek external evidence only when their internal weights are insufficient, reducing both errors and unnecessary computation costs.

Key Insights

• Calibrated Confidence Scaling: The system implements a granular 0.0-1.0 scale where 0.90-1.00 represents well-established facts and 0.00-0.29 indicates minimal reliable knowledge.
• Three-Stage Pipeline: The workflow consists of initial generation with confidence reporting, a self-criticism phase for logical consistency, and an optional research synthesis phase.
• Threshold-Triggered Research: Automated web research via DuckDuckGo (DDGS) is dynamically activated when the model’s revised confidence score is low (e.g., below 0.55).
• Structured JSON Interoperability: All stages utilize OpenAI’s json_object response format to ensure that reasoning, answers, and metadata can be programmatically parsed and stored.
• Evidence Synthesis: A dedicated ‘Research Synthesizer’ role combines preliminary answers with live web snippets to produce updated, evidence-grounded final responses.

Working Examples

The initial stage of the pipeline defining the LLMResponse structure and the query function for calibrated confidence estimation.

from dataclasses import dataclass, field
from openai import OpenAI

@dataclass
class LLMResponse:
    question: str
    answer: str
    confidence: float
    reasoning: str
    sources: list[str] = field(default_factory=list)
    researched: bool = False
    raw_json: dict = field(default_factory=dict)

SYSTEM_UNCERTAINTY = """
You are an expert AI assistant that is HONEST about what it knows and doesn't know.
For every question you MUST respond with valid JSON only:
{
"answer": "<your best answer>",
"confidence": <float 0.0-1.0>,
"reasoning": "<explain knowledge gaps>"
}
""".strip()

def query_llm_with_confidence(question: str) -> LLMResponse:
    completion = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0.2,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": SYSTEM_UNCERTAINTY},
            {"role": "user", "content": question},
        ],
    )
    raw = json.loads(completion.choices[0].message.content)
    return LLMResponse(
        question=question,
        answer=raw.get("answer", ""),
        confidence=float(raw.get("confidence", 0.5)),
        reasoning=raw.get("reasoning", ""),
        raw_json=raw,
    )

Practical Applications

• Decision Support Systems: Implementing ‘Self-Critic’ agents in financial analysis to flag when market data is outdated or logically inconsistent. Pitfall: Setting the confidence threshold too high, which may cause the system to ignore valid internal knowledge and increase latency through unnecessary research.
• Dynamic Knowledge Retrieval: Using the auto-research trigger for real-time events like population statistics or software versions. Pitfall: Relying on raw web snippets without a synthesis stage can lead to conflicting or noisy final answers if search results are contradictory.

References:

• https://www.marktechpost.com/2026/03/21/a-coding-implementation-to-build-an-uncertainty-aware-llm-system-with-confidence-estimation-self-evaluation-and-automatic-web-research/