{"post":{"slug":"online-evals-for-llm-apps-2026","title":"Online Evals: Measuring an LLM App in Production","excerpt":"Offline evals tell you whether a change broke your test set. Online evals tell you whether it broke real users. Most teams only build the first and wonder why quality drifts.","category":"ai","date":"August 18, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">Y</span>our offline suite is green. Every example passes, the scores are up from last month, and the deploy went out clean. Three weeks later support tells you the assistant has been giving people the wrong refund policy since roughly the fourteenth.\n\nNothing failed. The test set never contained a refund question phrased the way real customers phrase it, the retrieved document changed upstream, and no offline eval was ever going to catch either. This is the gap online evaluation exists to close: measuring the system on the traffic it actually receives, rather than on the traffic you imagined when you wrote the tests.\n\n## What changed in 2026\n\n- **Evaluation stopped being a pre-deploy gate.** Teams that treated evals purely as CI kept shipping green and regressing in production. Continuous measurement became the expected shape rather than an advanced practice.\n- **Sampling strategies got serious.** Scoring every request is affordable in a demo and ruinous at volume. Stratified sampling — deliberately over-sampling rare and high-risk request types — became the default approach.\n- **Implicit signals overtook explicit feedback.** Thumbs ratings are sparse and biased toward the angry. What users *do* after a response turned out to be a much better estimator of whether it worked.\n- **Drift detection moved from research to operations.** Watching the distribution of refusal rates, response lengths, tool-call frequencies and retrieval hit rates caught real degradation that per-response scoring missed entirely.\n\n## Offline vs online\n\n| | Offline evals | Online evals |\n| --- | --- | --- |\n| Runs | Before deploy, in CI | Continuously, on live traffic |\n| Input | A fixed curated set | Real user requests |\n| Ground truth | Known in advance | Usually absent |\n| Catches | Regressions against known cases | Drift, novel inputs, upstream changes |\n| Misses | Anything not in the set | Anything that fails silently and invisibly |\n| Cost | Bounded and predictable | Proportional to traffic and sample rate |\n| Blocks a deploy | Yes | No — it tells you after |\n\nNeither replaces the other. Offline evals give you a fast, deterministic signal against cases you have decided matter — which is the only thing that can gate a deploy. Online evals tell you about the enormous space of inputs you never anticipated, which is where production systems actually break. A team with only offline evals ships confidently into surprises; a team with only online evals detects problems it has no way to reproduce.\n\n## The signals worth collecting\n\nThe best production quality signals are usually already in your logs, unlabelled.\n\n**Retries and rephrasings.** A user who immediately asks the same thing differently is telling you the first answer failed. This is the highest-signal, lowest-effort metric most teams are not capturing.\n\n**Edits before use.** In any tool where the output is a draft, how much the user changes before sending it is a direct quality measure with no annotation cost.\n\n**Abandonment.** Requests where the user leaves without acting on the response. Noisy individually, meaningful in aggregate.\n\n**Copy and export events.** The positive counterpart — output the user found good enough to take somewhere else.\n\n**Tool-call and retrieval failures.** Not a quality judgement, but a strong leading indicator. Retrieval hit rates that fall precede answer quality that falls.\n\n**Refusal rate.** A number that moves when a prompt change makes the model more cautious, which is a real regression that no accuracy metric will report.\n\nExplicit feedback still has a place — it is unambiguous when you get it — but treat it as a sparse supplement rather than the main instrument. Response times and error rates come along for free from [LLM observability](/blog/llm-observability-2026) tooling you likely already have.\n\n## Scoring without doubling your bill\n\nThe obvious implementation — run a judge model over every response — costs roughly as much as serving the traffic. Three adjustments make it affordable.\n\n**Sample, and sample unevenly.** A few percent of ordinary traffic is enough to track a distribution. Then deliberately over-sample the categories where a failure is expensive: anything touching money, anything where the model refused, anything where a tool call errored, anything unusually long or short. Weight the aggregate back so your numbers stay representative.\n\n**Use cheap models for cheap checks.** Not every question needs a frontier judge. Format compliance, length limits, presence of a citation, and forbidden-phrase detection are string and schema problems. Reserve model-based judgement for the questions that genuinely require it — and validate the judge itself against human labels, per [LLM-as-judge](/blog/llm-as-judge-explained-2026).\n\n**Score asynchronously.** Nothing about production evaluation belongs in the request path. Queue the samples, score them out of band, and keep user latency untouched.\n\n## Common mistakes\n\n- **Alerting on individual bad outputs.** Every LLM system produces some. Alert on rates and distributions, not instances, or the alert becomes noise and gets muted.\n- **Judging with the model under test.** Self-evaluation inherits the same blind spots that produced the error.\n- **No sampling strategy.** Uniform sampling over skewed traffic means your rare, high-risk cases are almost never scored.\n- **Collecting signals with no baseline.** A refusal rate of 4% means nothing until you know last month's was 1%.\n- **Ignoring segments.** An aggregate that holds steady while one customer's experience collapses is a metric doing active harm.\n- **Treating online scores as ground truth.** They are estimates from an imperfect judge on a sample. Trends are reliable; absolute values are not.\n\n## FAQ\n\n**What sample rate is right?**\nEnough that your metric moves visibly before a human would notice the problem. For most products a low single-digit percentage of general traffic plus much heavier sampling of risky categories is a reasonable starting point, tuned from there.\n\n**Can I run my offline eval set against production traffic?**\nNot directly — production requests have no expected outputs. What works well is the reverse: promote interesting production samples into your offline set once a human has labelled them. That pipeline is what keeps [golden datasets](/blog/llm-golden-datasets-2026) current instead of frozen at launch.\n\n**How do I know the judge is right?**\nPeriodically have a human label a sample the judge already scored and measure agreement. If agreement drifts, the judge needs attention before its output means anything.\n\n**Is this the same as observability?**\nOverlapping but distinct. Observability tells you what happened — latency, errors, token counts, traces. Online evals tell you whether what happened was any good. You need the first to debug and the second to know there is something to debug.\n\n## Where to go next\n\nBuild the offline foundation first with [the AI evals guide](/blog/ai-evals-guide-2026) and [what is an AI eval harness](/blog/what-is-an-ai-eval-harness-2026). For the traces that make a bad production sample reproducible, [AI agent observability](/blog/ai-agent-observability-2026) covers what to emit, and [LLM regression testing](/blog/llm-regression-testing-2026) covers stopping the same failure twice.","hue":203,"metaTitle":"Online Evals: Measuring LLM Apps in Production","metaDescription":"How online evaluation differs from offline testing, which production signals actually predict quality, and how to sample and alert without doubling your bill.","keywords":["online evals","production llm evaluation","online vs offline evals","llm quality monitoring","production ai metrics","implicit user feedback llm","llm drift detection","ai observability evals 2026"],"tldr":["**Offline and online evals answer different questions** — one asks \"did this change break what I tested\", the other asks \"is production actually working right now\".","**Sample, do not score everything** — judging every request doubles your inference bill to measure something a few percent would tell you.","**Implicit signals beat explicit feedback** — retries, edits, abandonment and copy events are abundant and honest; thumbs-up buttons are neither.","**Alert on distribution shifts, not individual bad outputs** — a single poor answer is noise, a moved refusal rate is a signal.","**Skip:** online evals before you have offline ones. Production scoring on a system with no regression suite tells you something is wrong without helping you find out what."],"readTime":"5 min read","wordCount":1137},"related":[{"slug":"what-is-model-monitoring-2026","title":"What Is Model Monitoring? The 2026 MLOps Guide","excerpt":"Model monitoring tracks whether a deployed machine learning model is still performing well after launch. Here is what to actually measure and how in 2026.","category":"ai","date":"July 23, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":320},{"slug":"ai-observability-explained-2026","title":"AI observability explained for 2026","excerpt":"Traditional monitoring was not built for LLM apps that fail in silent, non-crashing ways. Here is what AI observability actually covers and why it needs different signals.","category":"ai","date":"July 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200},{"slug":"ai-observability-tools-2026","title":"AI observability tools in 2026: tracing, cost, and drift that actually scale","excerpt":"AI apps fail in ways traditional APM misses. Here are the observability tools built for LLMs in 2026, what they cover, and where each one fits.","category":"ai","date":"February 5, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":205}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"nua-stock-explained-2026","title":"Net Unrealized Appreciation: Employer Stock in a 401k","excerpt":"Rolling employer stock into an IRA can forfeit favourable capital gains treatment. Here is how the NUA election works and when it beats a normal rollover.","category":"finance","date":"August 18, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":288},"older":{"slug":"pcie-6-explained-2026","title":"PCIe 6.0 Explained: Doubling Again, With New Signalling","excerpt":"PCIe 6.0 doubles bandwidth using a new modulation scheme that required adding error correction. Here is what that means for latency, cost, and who needs it.","category":"tech","date":"August 18, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":251}}