Your offline suite is green. Every example passes, the scores are up from last month, and the deploy went out clean. Three weeks later support tells you the assistant has been giving people the wrong refund policy since roughly the fourteenth.
Nothing failed. The test set never contained a refund question phrased the way real customers phrase it, the retrieved document changed upstream, and no offline eval was ever going to catch either. This is the gap online evaluation exists to close: measuring the system on the traffic it actually receives, rather than on the traffic you imagined when you wrote the tests.
What changed in 2026
- Evaluation stopped being a pre-deploy gate. Teams that treated evals purely as CI kept shipping green and regressing in production. Continuous measurement became the expected shape rather than an advanced practice.
- Sampling strategies got serious. Scoring every request is affordable in a demo and ruinous at volume. Stratified sampling — deliberately over-sampling rare and high-risk request types — became the default approach.
- Implicit signals overtook explicit feedback. Thumbs ratings are sparse and biased toward the angry. What users do after a response turned out to be a much better estimator of whether it worked.
- Drift detection moved from research to operations. Watching the distribution of refusal rates, response lengths, tool-call frequencies and retrieval hit rates caught real degradation that per-response scoring missed entirely.
Offline vs online
|
Offline evals |
Online evals |
| Runs |
Before deploy, in CI |
Continuously, on live traffic |
| Input |
A fixed curated set |
Real user requests |
| Ground truth |
Known in advance |
Usually absent |
| Catches |
Regressions against known cases |
Drift, novel inputs, upstream changes |
| Misses |
Anything not in the set |
Anything that fails silently and invisibly |
| Cost |
Bounded and predictable |
Proportional to traffic and sample rate |
| Blocks a deploy |
Yes |
No — it tells you after |
Neither replaces the other. Offline evals give you a fast, deterministic signal against cases you have decided matter — which is the only thing that can gate a deploy. Online evals tell you about the enormous space of inputs you never anticipated, which is where production systems actually break. A team with only offline evals ships confidently into surprises; a team with only online evals detects problems it has no way to reproduce.
The signals worth collecting
The best production quality signals are usually already in your logs, unlabelled.
Retries and rephrasings. A user who immediately asks the same thing differently is telling you the first answer failed. This is the highest-signal, lowest-effort metric most teams are not capturing.
Edits before use. In any tool where the output is a draft, how much the user changes before sending it is a direct quality measure with no annotation cost.
Abandonment. Requests where the user leaves without acting on the response. Noisy individually, meaningful in aggregate.
Copy and export events. The positive counterpart — output the user found good enough to take somewhere else.
Tool-call and retrieval failures. Not a quality judgement, but a strong leading indicator. Retrieval hit rates that fall precede answer quality that falls.
Refusal rate. A number that moves when a prompt change makes the model more cautious, which is a real regression that no accuracy metric will report.
Explicit feedback still has a place — it is unambiguous when you get it — but treat it as a sparse supplement rather than the main instrument. Response times and error rates come along for free from LLM observability tooling you likely already have.
Scoring without doubling your bill
The obvious implementation — run a judge model over every response — costs roughly as much as serving the traffic. Three adjustments make it affordable.
Sample, and sample unevenly. A few percent of ordinary traffic is enough to track a distribution. Then deliberately over-sample the categories where a failure is expensive: anything touching money, anything where the model refused, anything where a tool call errored, anything unusually long or short. Weight the aggregate back so your numbers stay representative.
Use cheap models for cheap checks. Not every question needs a frontier judge. Format compliance, length limits, presence of a citation, and forbidden-phrase detection are string and schema problems. Reserve model-based judgement for the questions that genuinely require it — and validate the judge itself against human labels, per LLM-as-judge.
Score asynchronously. Nothing about production evaluation belongs in the request path. Queue the samples, score them out of band, and keep user latency untouched.
Common mistakes
- Alerting on individual bad outputs. Every LLM system produces some. Alert on rates and distributions, not instances, or the alert becomes noise and gets muted.
- Judging with the model under test. Self-evaluation inherits the same blind spots that produced the error.
- No sampling strategy. Uniform sampling over skewed traffic means your rare, high-risk cases are almost never scored.
- Collecting signals with no baseline. A refusal rate of 4% means nothing until you know last month's was 1%.
- Ignoring segments. An aggregate that holds steady while one customer's experience collapses is a metric doing active harm.
- Treating online scores as ground truth. They are estimates from an imperfect judge on a sample. Trends are reliable; absolute values are not.
FAQ
What sample rate is right?
Enough that your metric moves visibly before a human would notice the problem. For most products a low single-digit percentage of general traffic plus much heavier sampling of risky categories is a reasonable starting point, tuned from there.
Can I run my offline eval set against production traffic?
Not directly — production requests have no expected outputs. What works well is the reverse: promote interesting production samples into your offline set once a human has labelled them. That pipeline is what keeps golden datasets current instead of frozen at launch.
How do I know the judge is right?
Periodically have a human label a sample the judge already scored and measure agreement. If agreement drifts, the judge needs attention before its output means anything.
Is this the same as observability?
Overlapping but distinct. Observability tells you what happened — latency, errors, token counts, traces. Online evals tell you whether what happened was any good. You need the first to debug and the second to know there is something to debug.
Where to go next
Build the offline foundation first with the AI evals guide and what is an AI eval harness. For the traces that make a bad production sample reproducible, AI agent observability covers what to emit, and LLM regression testing covers stopping the same failure twice.