Scaling AI: Solving the Infrastructure Fragmentation of LLM Reasoning

Why LLM Reasoning Is Breaking AI Infrastructure (And How to Fix It)

Jonathan Murray reports that while “thinking” improves model accuracy, it creates critical bottlenecks in production infrastructure. Developers are currently managing inconsistent reasoning schemas across OpenAI, Anthropic, and Google AI. This fragmentation forces teams to build complex middleware instead of core product features.

Why This Matters

The technical reality of LLM reasoning is a fragmented landscape where providers use different effort levels, token budgets, and output schemas, such as OpenAI’s effort levels versus Anthropic’s token budgets. This lack of abstraction means that simple API routing becomes a maintenance-heavy middleware layer, leading to unpredictable token usage and billing inconsistencies that prevent effective scaling and cost forecasting.

Key Insights

• OpenAI uses varying reasoning effort levels (low, medium, high) while Anthropic requires explicit reasoning token budgets as of 2026.
• Output fragmentation exists because some models return separate reasoning blocks while others mix reasoning directly into standard responses.
• The absence of a shared schema across providers like Google AI and OpenAI prevents standardized multi-model AI system interfaces.
• Billing models are inconsistent, with some providers exposing reasoning tokens explicitly and others bundling them into total usage metrics.
• Multi-model switching introduces system instability due to changes in input formats and reasoning structures even within a single provider’s endpoints.

Practical Applications

• Use case: Tuning reasoning budgets across multiple providers. Pitfall: Abandoning portability due to fragile adapter layers that break when output schemas change.
• Use case: Implementing cost translation layers for budget control. Pitfall: Over-reasoning on trivial queries which wastes tokens and inflates operational expenses.
• Use case: Maintaining persistent context across different model versions. Pitfall: Token explosion resulting from a lack of reasoning continuity and state management.

References:

• https://dev.to/backboardio/why-llm-reasoning-is-breaking-ai-infrastructure-and-how-to-fix-it-2aik