AI apps fail differently than web apps. The status code is 200, the latency is fine, and the response is wrong. Traditional APM (Datadog, New Relic, Sentry) instruments the wrong layer — it sees the call to OpenAI but not the prompt, the tools, or the agent's reasoning. LLM-native observability fills that gap, and in 2026 the category has consolidated around four serious players plus the existing APM vendors getting better.
What changed in 2026
- OpenTelemetry GenAI semantic conventions are stable. You can now instrument once and route to multiple backends.
- Trace + eval converged. Pulling a production trace into an eval set is one click in every major tool.
- Cost observability got serious. Per-tenant, per-feature, per-model unit-cost dashboards are now the default — bad prompts caught before invoice.
The four LLM-native tools
LangSmith (LangChain). Deepest agent tracing. Best for stacks already on LangChain/LangGraph. Self-host option exists. Eval and trace are tightly integrated.
Langfuse. Open-source, self-host first. Excellent OSS UX, OTel-native, generous free tier. Best for teams that want LangSmith-class capability without proprietary lock-in.
Helicone. Proxy-based — sits between your app and the model API. Lowest integration friction (one URL change). Best for teams that want cost dashboards and basic traces fast, before they invest in deeper instrumentation.
Arize Phoenix. Production observability with full ML drift detection (embedding space, prediction drift). Best for teams that already do ML observability or need that depth alongside LLM tracing.
Plus the APM giants
Datadog LLM Observability and New Relic AI Monitoring added LLM-specific dashboards in 2025–2026. The trade-off: deep integration with your existing infra signals (DB, queue, latency) but shallower than the natives on prompt/eval workflows.
| Tool |
Best at |
Hosted |
Free tier |
| LangSmith |
Agent tracing, eval integration |
Yes / self-host |
Yes |
| Langfuse |
OSS-first, OTel-native |
Self-host / cloud |
Yes |
| Helicone |
Lowest-friction proxy + cost |
Cloud |
Yes |
| Arize Phoenix |
Drift + ML observability depth |
Self-host / Arize AX |
Yes (OSS) |
| Datadog LLM |
APM integration |
Cloud |
No |
What to instrument from day one
You don't need everything at once. Start here:
- Trace every model call with input, output, latency, token count, model name, cost.
- Tag traces by feature (chat, search, summary) and tenant. Without these, dashboards are useless.
- Alert on unit-cost drift. If
tokens_per_user / DAU jumps 30%, you have a regression — possibly a prompt change, possibly a user-input attack.
- Sample a percentage to evals. The first month of running this catches 80% of "I think it's getting worse" anecdotes with data.
By month three, add: tool-call success rate, escalation rate, judge-score drift, embedding-space drift on retrieved context.
What to ignore at first
Don't wire up everything OTel emits. The volume kills costs and signal-to-noise. Start with model calls and tool calls; add intermediate steps when an incident demands it.
FAQ
Can I just use Datadog if I'm already on it?
For basic visibility yes. Switch to or supplement with a native tool when prompt iteration speed matters or you start running evals.
Self-host or SaaS?
SaaS unless data residency or cost dictates otherwise. Self-host adds 1–2 weeks of ops work that rarely pays off.
What does this typically cost?
$0 (OSS) to a few thousand a month at moderate scale. Most teams under-budget by 5×; budget for it.
Do I need both tracing and evals?
Yes. Tracing tells you what happened in production; evals tell you whether a change is better. They're complementary, not redundant.
Where to go next
For related material see AI evals frameworks in 2026, Vector database comparison in 2026, and RAG vs fine-tuning in 2026.