Taming LLM Output Chaos: A 3-Tier Normalisation Pattern

The Problem: LLMs Don’t Follow Instructions

Large Language Models (LLMs) struggle to consistently adhere to specific output formats, even with detailed prompting. For example, requesting “DRAINS” relationships from an LLM can yield eleven variations like “drains,” “depletes,” and “exhausts,” hindering application reliability.

Why This Matters

Ideal LLM applications assume perfect instruction following, but reality introduces significant output variation. This inconsistency breaks downstream processes expecting specific data structures, leading to failures and requiring manual intervention; the author’s initial collision detection rate was only 15-20% before implementing this pattern.

Key Insights

• Prompting Consistency: Prompting alone achieves approximately 70% consistency in LLM outputs.
• Semantic vs. Syntax: LLMs understand concepts but not necessarily the strict syntax required by applications.
• RapidFuzz: A fast, C++-based fuzzy string matching library that balances flexibility and precision, used by projects like Cognee.

Working Example

RELATION_TYPE_MAP = {
"drains": "DRAINS",
"depletes": "DRAINS",
"exhausts": "DRAINS",
}

SEMANTIC_KEYWORDS = {
"DRAINS": [
"drain",
"exhaust",
"deplet",
],
}

def _keyword_match_relation(relation_type: str) -> str | None:
    normalized = relation_type.lower()
    for canonical_type, keywords in SEMANTIC_KEYWORDS.items():
        for keyword in keywords:
            if keyword in normalized:
                return canonical_type
    return None

Practical Applications

• Sentinel (CLI tool): Uses this pattern to reliably detect energy conflicts in personal schedules by normalizing LLM-generated relationship types.
• Pitfall: Relying solely on prompt engineering for structured LLM output will lead to unpredictable behavior and application failures.

References:

• https://dev.to/armelhbobdad/taming-llm-output-chaos-a-3-tier-normalisation-pattern-41jm