A failing web service throws errors. A failing agent finishes successfully and hands back a confident, wrong answer. That single difference is why standard application monitoring is close to useless for agents: your error rate is near zero, your latency is fine, your dashboards are green, and your users are unhappy.
Agent observability is the practice of instrumenting for the failures that do not raise exceptions.
What changed in 2026
- The run became the unit of analysis. Tracing tools reorganized around whole agent runs — the full tree of model calls, tool invocations, and handoffs — rather than individual API requests.
- Semantic conventions stabilized. Open telemetry standards for generative AI spans matured enough that switching observability vendors stopped requiring re-instrumentation.
- Cost joined the trace. Token spend per span became a first-class field rather than something reconciled from a separate billing export weeks later.
- Online evaluation moved into production. Rather than only scoring offline test sets, teams began running lightweight judges against sampled live traffic and alerting on score drift.
What to actually capture
| Signal |
Why it matters |
Alert on |
| Steps per run |
Rises sharply when the agent is confused |
p95 above your normal ceiling |
| Repeated identical tool calls |
The clearest sign of a non-converging loop |
Any call repeated three times in a run |
| Tool error rate |
Silent tool failures cause silent bad answers |
Any sustained increase |
| Tokens per successful run |
The real unit economics |
Week-over-week drift |
| Handoff depth |
Deep chains lose context |
Runs exceeding your designed cap |
| Context fill at each step |
Predicts quality decay before it shows |
Sustained fill above 60 percent |
| Sampled judge score |
Direct quality signal |
Rolling average drop |
The two cheapest and most predictive are step count and repeated tool calls. Neither requires a model to compute, both are available from ordinary tracing, and both move before output quality visibly degrades. If you instrument nothing else, instrument these.
Context fill is the underrated one. When runs start pushing into the unreliable range described in context rot, answers get worse with no other visible symptom. Tracking fill per step gives you a warning that a summarization or compaction step is overdue.
Traces, not logs
Ordinary logs are the wrong shape here. A log line per model call gives you forty disconnected entries and no way to see that call twenty-nine repeated call seventeen. You need a tree: a root span for the run, child spans for each model call and tool invocation, with inputs, outputs, token counts, and timing attached to each.
That structure is what makes debugging tractable. When a user reports a bad answer, you open the run, read the tree, and find the step where it went wrong — usually a tool that returned an empty result the model then interpreted as a negative finding, or a handoff that dropped a constraint.
For the general practice, AI observability explained covers the foundations and AI observability tools surveys the vendor landscape. The agent-specific addition is that your spans need to record tool arguments and results, not just model calls, because most real failures happen at that boundary.
Common mistakes
- Storing full prompt and completion text in your general log pipeline. Those payloads contain user data. Sample them, redact them, and keep them in a store with an appropriate retention policy.
- Measuring cost per call. The meaningful denominator is a successful outcome. Cheap failing runs are not cheap.
- No sampling strategy for quality. You cannot judge every run. Sample a fixed percentage plus every run that trips a leading indicator.
- Ignoring the tool layer. A tool returning an empty array instead of an error is the single most common silent failure, and it is invisible unless you trace tool results.
- Alerting only on aggregates. Aggregate quality can look stable while one request category degrades badly. Segment by task type.
FAQ
Do I need a dedicated LLM observability vendor?
Not necessarily. If you already run distributed tracing, the generative AI semantic conventions let you extend it. Dedicated tools mainly add prompt-aware UIs and built-in evaluation, which is worth money once you are running online judges.
How much sampling is enough for quality scoring?
For most teams, one to five percent of runs plus all flagged runs gives a usable signal without a large judging bill. Increase it after a release, decrease it once stable.
What is a reasonable step-count ceiling?
Whatever your normal p95 is, plus headroom. The absolute number matters less than the deviation — an agent that normally takes six steps and suddenly takes twenty is telling you something regardless of the cap.
Does tracing add meaningful latency?
Span creation is negligible next to model calls. The cost is storage and, if you run online judges, the judging calls themselves.
Where to go next
For scoring agent behavior systematically, read AI agent evaluation frameworks and LLM as judge explained. If your traces reveal context loss between agents, AI agent handoff patterns covers the fix.