Observability for LLM applications is not APM with different labels. A service that returns HTTP 200 with a hallucinated answer is failing — your uptime dashboard will never show it. In 2026, the teams that build reliable AI products instrument for what actually fails: quality degradation, cost explosions, silent refusals, and retrieval failures that only appear as vague user frustration.
What changed in 2026
- OpenTelemetry for LLMs standardized. The OpenTelemetry Semantic Conventions for Gen AI (v1.27+) define standard span attributes for LLM calls — model name, token counts, cost, prompt/completion hashes. Vendor lock-in dropped significantly.
- LLM observability platforms matured. Langfuse (open-source), Braintrust, and Arize Phoenix all offer full-featured platforms with evals, human annotation, and cost dashboards. Picking one and staying is now viable.
- Continuous eval became standard. Shipping a production quality eval that runs on sampled live traffic — not just at deploy — is a recognized engineering practice, not a research experiment.
- Cost attribution got granular. Modern platforms attribute token costs to individual features, users, and workflows, enabling cost-per-task analysis that drives product decisions.
The LLM observability stack
Layer 1 — Tracing: Every LLM call is a span. Capture: model ID, input tokens, output tokens, latency (TTFT + total), cost estimate, prompt hash, output hash, session/user ID, feature name.
Layer 2 — Logging: Store the full prompt + completion for a configurable sample rate (10–100% depending on sensitivity). This is the raw material for debugging and eval datasets.
Layer 3 — Metrics: Aggregate traces into metrics: p50/p95/p99 latency, token usage by feature, cost per task, error rate, refusal rate, cache hit rate.
Layer 4 — Evals: Continuously run LLM-as-judge on sampled production outputs. Score quality dimensions relevant to your use case (accuracy, helpfulness, tone).
Layer 5 — Alerts: Trigger on metric anomalies: cost spike (>2× baseline), quality drop (eval score drops >10 points), error rate (>5%), latency p95 (>3s).
What to instrument
| Signal |
Why it matters |
How to capture |
| Token counts (input/output) |
Cost driver, billing |
Provider usage API |
| Time to first token (TTFT) |
Perceived latency |
Streaming timing |
| Total latency |
End-to-end performance |
Span start/end |
| Cache hit/miss |
Cost efficiency |
Provider cache field |
| Retrieval context |
RAG quality debugging |
Log chunks passed to LLM |
| Tool calls |
Agent behavior |
Log every tool invocation |
| Model version |
Regression source |
Log exact model ID |
| User session |
User-level debugging |
Session/user ID header |
The metrics that actually matter
Cost per resolved task — group cost by workflow completion (did the user get what they needed?), not raw API spend. This is the unit that drives product ROI decisions.
Quality score (sampled eval) — daily LLM-as-judge score on 5–10% of production traffic. A slow drift from 4.2 to 3.8 over two weeks is a real signal, invisible to error logs.
Retrieval precision — for RAG systems, what percentage of retrieved chunks were actually used by the LLM in its answer? High retrieval, low usage = retrieval quality problem.
Refusal rate — how often does the model refuse to answer? Spikes indicate prompt changes, adversarial users, or model policy updates.
P95 latency by feature — latency varies dramatically by feature; a search feature degrading from 800ms to 2s is invisible in aggregate.
How to pick your observability tool
| Requirement |
Tool |
| Open-source, self-hosted |
Langfuse |
| Best eval workflow |
Braintrust |
| ML + LLM unified |
Arize Phoenix |
| Existing DataDog/Grafana stack |
OTel exporter → your existing stack |
| Lightweight, just tracing |
Helicone |
For most teams: Langfuse (self-hosted or cloud) covers 80% of needs, integrates with LangChain/LlamaIndex, and is free to self-host.
Common mistakes
Logging only errors. A 200 OK with a useless answer is a failure. Log everything and evaluate the content, not just the status code.
Sampling out your tail. 1% sampling looks fine in dashboards but completely misses rare-but-critical failure modes. Sample 100% of failures, 10–20% of successes.
No cost attribution. "Total monthly spend is $4,000" is useless. "Feature X costs $2,300/month, of which 40% is a 3,000-token system prompt that could be cached" drives decisions.
Forgetting multi-step traces. In agentic workflows, a single user action triggers 3–15 LLM calls. Trace them all under a single root span so you can see the full cost and latency of one user task.
Privacy non-compliance. Storing full prompts/completions may capture PII. Implement hash-only logging for PII-sensitive fields, with full content stored in a separate access-controlled store.
What to skip
- Building your own LLM logging library — the OTel semantic conventions and existing platforms are good enough; reinventing the data model costs months.
- Alerting on input token count in isolation — it fluctuates with usage patterns; alert on cost-per-task regression instead.
- Manual log inspection at scale — without structured queries and eval scoring, log inspection doesn't scale past a few hundred calls/day.
FAQ
How do I correlate LLM calls with user outcomes?
Pass a session_id and user_id through every LLM call span. When a user rates an answer or completes a workflow, record that outcome against the session ID and join in your analytics.
What's a healthy cost-per-task target?
Depends entirely on your use case. For a customer support bot, <$0.10 per resolved ticket is typically viable. Build your own baseline, then work to improve it.
Should I log full prompts in production?
Log full prompts at a sampling rate (10–20%), with PII scrubbing applied. Full logging is critical for debugging but has storage and compliance costs.
How do I detect prompt injection in logs?
Flag outputs that contain instruction-like phrases ("ignore previous instructions", "you are now"), large behavioral shifts from baseline, or tool calls that don't match the user's stated intent.
Where to go next