LLM Evals on Real Traffic — Not Just Test Suites

The eval gap

Grepture has introduced automated LLM-as-a-judge scoring directly within its AI gateway. This system evaluates production traffic in real-time to bridge the gap between static test suites and messy real-world user prompts.

Why This Matters

Traditional evaluation relies on “golden examples” in CI pipelines that often fail to represent the complexity of live production data. Real user prompts are longer and more unpredictable, leading to edge cases that static test fixtures cannot anticipate. By running evals on the gateway, teams can identify distribution shifts and model regressions that occur after deployment without managing separate batch jobs or data export pipelines. This approach transforms production logs from passive data into a continuous quality signal that follows prompt versions automatically.

Key Insights

• LLM-as-a-judge scoring provides 0-to-1 metrics with written reasoning based on real logs (Grepture, 2026)
• Standard templates include Relevance, Helpfulness, Toxicity, Conciseness, Instruction following, and Hallucination metrics
• Sampling rates allow teams to score a fraction of traffic, such as 1-10%, to maintain statistical significance while reducing judge token costs
• Filters enable targeted evaluation of specific models, providers, or prompt IDs to isolate performance issues
• Production traffic evaluation catches long-tail failures and model regressions that pre-deploy testing misses

Practical Applications

• Use case: Continuous quality monitoring of customer-facing models using Grepture’s sampling rate to manage judge token costs
• Pitfall: Relying solely on pre-deploy test suites, which can lead to undetected 40% hallucination rates on specific user query classes that weren’t in the original fixtures
• Use case: Tracking prompt drift by using filters to evaluate specific prompt versions managed separately from code
• Pitfall: Exporting logs to separate evaluation pipelines, creating friction that often prevents teams from implementing production evals

References:

• https://dev.to/grepture/llm-evals-on-real-traffic-not-just-test-suites-3k4c