{"post":{"slug":"cross-encoder-reranking-2026","title":"Cross-Encoder Reranking: The Cheapest Big Quality Win","excerpt":"Reranking a shortlist with a model that reads query and document together is usually the highest-return change in a retrieval pipeline. Here is how to do it.","category":"ai","date":"August 11, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span>n embedding-based retriever encodes the query and each document separately, then compares the resulting vectors. That independence is what makes it fast — documents are encoded once, ahead of time — and it is also what limits its accuracy, because the model never sees the query and document together.\n\nA cross-encoder does see them together. It reads the pair and produces a relevance score with full attention across both, which is substantially more accurate and far too slow to run over an entire corpus.\n\n## What changed in 2026\n\n- **Reranking became standard rather than an optimization.** Retrieve-then-rerank moved into default RAG architecture guidance.\n- **Small rerankers got good.** Compact models delivered most of the quality of larger ones at much lower latency, which made reranking affordable in interactive systems.\n- **Rerank-as-a-service appeared.** Hosted reranking endpoints removed the need to deploy and scale a separate model.\n- **Candidate set sizing got attention.** Recognition spread that reranking a small candidate set wastes the technique, since the reranker can only reorder what retrieval surfaced.\n\n## Why it works\n\n| | Bi-encoder retrieval | Cross-encoder reranking |\n| --- | --- | --- |\n| Query and document processed | Separately | Together |\n| Documents encoded | Once, in advance | Per query |\n| Attention across the pair | None | Full |\n| Accuracy | Good | Substantially better |\n| Speed over a whole corpus | Fast | Impractical |\n| Speed over a shortlist | N/A | Acceptable |\n\nThe two-stage structure resolves the tension. Retrieval casts a wide net cheaply, reranking applies expensive accuracy to a small set. You get most of the cross-encoder's quality at a fraction of its cost.\n\n## Getting the candidate set right\n\nThis is where most reranking implementations underperform. If retrieval returns ten candidates and you rerank them to pick five, the reranker can only reorder those ten — and if the best document ranked fifteenth, it is not there and reranking cannot recover it.\n\nRetrieve generously. Pulling a substantially larger candidate set than you intend to keep is what gives the reranker room to work. The cost is a linear increase in reranking latency, which is the tradeoff to tune.\n\nThe pattern that works: retrieve a broad set with hybrid search, rerank all of them, keep the top few for the model's context. Recall comes from the wide retrieval, precision comes from the reranker, and the final context stays small — which also helps with the degradation described in [context rot explained](/blog/context-rot-explained-2026).\n\nAnd check the order of operations. Reranking cannot fix poor recall. If your labelled set shows the right document frequently is not retrieved at all, that is a retrieval problem, and hybrid search or better chunking is the fix. Reranking helps when the right document is retrieved but ranked below the cutoff.\n\n## Common mistakes\n\n- **Reranking a candidate set that is too small.** Nothing to find, nothing to gain.\n- **Reranking when recall is the bottleneck.** Reordering bad candidates does not help.\n- **Using an oversized reranker.** Small models capture most of the benefit at much lower latency.\n- **Reranking in the critical path without measuring latency.** A forward pass per candidate adds up.\n- **Passing all reranked results to the model.** Rerank widely, keep few.\n\n## FAQ\n\n**How many candidates should I rerank?**\nEnough that the right document is reliably in the set. Check where the correct answer typically ranks in your retrieval results and set the candidate count comfortably above it.\n\n**How much latency does it add?**\nA model forward pass per candidate, batched. With a small reranker and a moderate candidate set it is typically tens of milliseconds, which is small next to generation.\n\n**Do I need to host the reranker?**\nNot necessarily. Hosted reranking endpoints exist and are convenient, though they add a network round trip.\n\n**Is reranking better than late interaction?**\nDifferent tradeoffs — reranking keeps storage low and adds query latency; late interaction pays in storage. See [late interaction retrieval](/blog/late-interaction-retrieval-2026).\n\n## Where to go next\n\nFor the retrieval stage feeding it, read [hybrid search with BM25](/blog/hybrid-search-bm25-2026). For the alternative architecture, [late interaction retrieval](/blog/late-interaction-retrieval-2026), and for measuring the improvement, [RAG evaluation metrics](/blog/rag-evaluation-metrics-2026).","hue":205,"metaTitle":"Cross-Encoder Reranking Guide for 2026","metaDescription":"How cross-encoder reranking improves retrieval quality, why candidate set size matters, latency tradeoffs, and when reranking will not help.","keywords":["cross encoder reranking","reranker model retrieval","two stage retrieval pipeline","candidate set size","retrieval precision improvement","bi encoder vs cross encoder","rag quality improvement","search reranking latency","retrieval architecture 2026"],"tldr":["**A cross-encoder scores query and document together** — far more accurate than comparing two independent embeddings.","**It only runs on a shortlist** — retrieving broadly then reranking the top candidates is what makes it affordable.","**Retrieve more than you think before reranking** — the reranker can only reorder what retrieval found.","**Latency is the cost** — a forward pass per candidate, which adds up.","**Skip:** reranking when your retrieval recall is poor. Fix recall first; reordering bad candidates does not help."],"readTime":"3 min read","wordCount":700},"related":[{"slug":"rerankers-explained-2026","title":"Rerankers: Fixing the Order of What You Retrieved","excerpt":"Vector search returns roughly relevant results in roughly the right order. A reranker reads each candidate against the query properly and reorders them, which is a different and better operation.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":207},{"slug":"reranking-in-rag-2026","title":"Reranking in RAG explained in 2026: better answers, cheaply","excerpt":"Reranking is a second-stage step that reorders retrieved documents by true relevance before the model reads them. Here is the 2026 guide to how it works, why it lifts RAG quality, and what it costs.","category":"ai","date":"June 30, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":260},{"slug":"late-interaction-retrieval-2026","title":"Late Interaction Retrieval: Between Dense and Cross-Encoder","excerpt":"Late interaction keeps per-token embeddings and matches them individually. Here is what that buys over single-vector search and what it costs in storage.","category":"ai","date":"July 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":218}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"continuous-batching-explained-2026","title":"Continuous Batching Explained: Why LLM Servers Got Fast","excerpt":"Static batching makes every request in a batch wait for the slowest one. Continuous batching lets finished sequences leave and new ones join mid-flight. The throughput difference is large.","category":"code","date":"August 11, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":305},"older":{"slug":"dc-fast-charging-explained-2026","title":"DC Fast Charging Explained: Why Your Speed Varies So Much","excerpt":"Two cars at the same charger get very different speeds, and the same car varies by session. Here is what actually controls charging rate and how to plan around it.","category":"tech","date":"August 11, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":231}}