{"post":{"slug":"multi-turn-evals-2026","title":"Multi-Turn Evals: Testing Conversations, Not Single Answers","excerpt":"A model that answers every question correctly in isolation can still lose the thread by turn four. Single-turn evaluation cannot see that, and most evaluation is single-turn.","category":"ai","date":"February 10, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">Y</span>our evaluation set has three hundred questions and the model answers them well. In production, users report that it \"forgets things\" and \"goes in circles\". Both observations are correct, and your evaluation cannot detect either, because every question is asked in isolation.\n\nConversations fail differently from single questions. The characteristic failures — losing a constraint stated three turns ago, misresolving \"the second one\", accepting a correction and then reverting to the original error — only appear across turns.\n\n## What changed in 2026\n\n- **Chat became the dominant interface.** Most LLM products are conversational, which made single-turn-only evaluation an obvious gap.\n- **Simulated users matured.** Scripted and model-driven user simulators became a practical way to generate reproducible multi-turn tests.\n- **Trajectory scoring spread.** Assessing the path rather than only the final answer became standard for agentic and conversational evaluation.\n- **Agent evaluation converged with it.** An agent run is a multi-turn interaction, so the same techniques applied.\n\n## What breaks across turns\n\n| Property | Failure it produces |\n| --- | --- |\n| Reference resolution | \"The second one\" resolves to the wrong thing |\n| Constraint retention | A limit stated at turn two is ignored at turn six |\n| Correction handling | User corrects an error, model reverts later |\n| Topic tracking | Model answers a previous question, not the current one |\n| Consistency | Model contradicts what it said earlier |\n| Appropriate clarification | Guesses rather than asking when genuinely ambiguous |\n\nConstraint retention is the one users complain about most. Someone says \"keep responses under fifty words\" at the start; by turn eight the responses are three paragraphs. The instruction is still in the context and its influence has faded relative to more recent content.\n\nCorrection handling is the most damaging. A user corrects a factual error, the model acknowledges it, and two turns later reasons from the original wrong value — which reads as not listening.\n\n## Scripting the user\n\nReproducibility is the design problem. Real transcripts are realistic and unusable for comparison, since you cannot replay a human.\n\n**Fixed scripts** are the reliable default: predetermined user turns, delivered regardless of what the model says. Every run is identical, so results are comparable across model and prompt versions. The limitation is that if the model responds unexpectedly, turn four may not follow sensibly — which is acceptable for a test suite and would be strange in a demo.\n\n**Simulated users** — a model playing the user role with a goal — produce more natural conversations and reintroduce variability, since the simulator responds differently to different model outputs. Useful for exploration, weaker for regression detection.\n\nMost suites want fixed scripts for the regression tier and simulated users for broader exploration. The scripts should encode the specific properties above: state a constraint early and check it later, use a pronoun referring back several turns, correct an error and verify it stays corrected.\n\n## Score the path\n\nThe final answer alone is insufficient, for the same reason it is insufficient in agent evaluation: a correct answer reached through a wrong path is not reliable.\n\nAssert on the trajectory. Did the model retain the constraint at each turn where it applied? Did it resolve the reference correctly at the turn where it mattered? Did it ask for clarification where the input was genuinely ambiguous?\n\nPer-turn assertions localise the failure, which is the practical benefit — knowing the conversation broke at turn four is far more useful than knowing the final answer was wrong. This is the same reasoning that makes per-step spans valuable in [agent observability](/blog/ai-agent-observability-2026).\n\nProperty assertions transfer well here. Response length, format compliance, presence of a required disclaimer, and absence of a forbidden claim are all checkable per turn without a judge model, per [LLM regression testing](/blog/llm-regression-testing-2026).\n\n## Common mistakes\n\n- **Single-turn evaluation for a chat product.** Misses the entire class of conversational failure.\n- **Real transcripts as regression tests.** Not reproducible.\n- **Final-answer-only scoring.** Hides where it broke.\n- **No constraint-retention cases.** The most common user complaint, untested.\n- **Conversations too short.** Failures appear at depth; three turns may not reach it.\n- **Simulated users in the regression tier.** Variability defeats comparison.\n\n## FAQ\n\n**How long should test conversations be?**\nLong enough to reach where failures occur, which is usually deeper than expected. If your users have twenty-turn conversations, a five-turn test does not cover the behaviour they experience.\n\n**Should I test with compaction enabled?**\nYes, if production uses it — compaction is precisely where constraints get lost. Testing without it means testing a different system, per [context compaction](/blog/context-compaction-explained-2026).\n\n**How does this differ from agent evaluation?**\nVery little structurally. An agent run is a multi-turn interaction where the other party is a set of tools rather than a person, and trajectory scoring applies identically.\n\n**Can I automate the scoring?**\nProperty assertions, yes and cheaply. Judgements about whether a response was contextually appropriate need a judge model, with the usual validation — see [LLM-as-judge](/blog/llm-as-judge-explained-2026).\n\n## Where to go next\n\nFor the single-turn foundation, read [golden datasets](/blog/llm-golden-datasets-2026) and [LLM regression testing](/blog/llm-regression-testing-2026). For the drift that degrades any suite over time, [eval drift](/blog/eval-drift-explained-2026).","hue":214,"metaTitle":"Multi-Turn Evals: Testing Conversations, Not Answers","metaDescription":"Why single-turn evaluation misses conversational failures, how to script reproducible multi-turn tests, and which properties to assert on.","keywords":["multi turn evals","conversation evaluation llm","dialogue testing","context retention testing","trajectory scoring","chatbot quality","simulated user testing","ai evaluation 2026"],"tldr":["**Failures compound across turns** — a small early error becomes the premise for everything after it.","**A scripted user beats a real transcript for reproducibility** — fixed inputs make runs comparable.","**Test the properties conversations break** — reference resolution, constraint retention, correction handling.","**Score the trajectory, not just the last message** — a right answer reached through a wrong path will not repeat.","**Skip:** multi-turn evaluation for single-shot tasks. Extraction and classification have no conversation to break."],"readTime":"4 min read","wordCount":861},"related":[{"slug":"rag-evaluation-metrics-2026","title":"RAG Evaluation Metrics: Measuring Retrieval and Generation","excerpt":"A wrong RAG answer has two possible causes and one combined score cannot tell them apart. Here is how to measure each stage separately and act on the result.","category":"ai","date":"July 24, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":206},{"slug":"online-evals-for-llm-apps-2026","title":"Online Evals: Measuring an LLM App in Production","excerpt":"Offline evals tell you whether a change broke your test set. Online evals tell you whether it broke real users. Most teams only build the first and wonder why quality drifts.","category":"ai","date":"August 18, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":203},{"slug":"ai-evals-guide-2026","title":"AI evals guide in 2026: how to measure whether your LLM actually works","excerpt":"Evals are the engineering discipline that separates AI projects that ship from ones that stall. This is the practical 2026 guide to building, running, and acting on LLM evaluations.","category":"ai","date":"March 20, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":215}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"iphone-17-pro-rumors-2026","title":"iPhone 17 Pro rumors in 2026: what is plausible vs noise","excerpt":"A clear-eyed look at the credible iPhone 17 Pro leaks for September 2026 — what's likely real, what's analyst-noise, and what to actually budget for.","category":"tech","date":"February 10, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":220},"older":{"slug":"notebooklm-vs-chatgpt-2026","title":"NotebookLM vs ChatGPT in 2026: when each one wins","excerpt":"NotebookLM and ChatGPT solve different problems. Here is when NotebookLM beats general chat — and when ChatGPT is still the right answer.","category":"ai","date":"February 10, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":195}}