An embedding-based retriever encodes the query and each document separately, then compares the resulting vectors. That independence is what makes it fast — documents are encoded once, ahead of time — and it is also what limits its accuracy, because the model never sees the query and document together.
A cross-encoder does see them together. It reads the pair and produces a relevance score with full attention across both, which is substantially more accurate and far too slow to run over an entire corpus.
What changed in 2026
- Reranking became standard rather than an optimization. Retrieve-then-rerank moved into default RAG architecture guidance.
- Small rerankers got good. Compact models delivered most of the quality of larger ones at much lower latency, which made reranking affordable in interactive systems.
- Rerank-as-a-service appeared. Hosted reranking endpoints removed the need to deploy and scale a separate model.
- Candidate set sizing got attention. Recognition spread that reranking a small candidate set wastes the technique, since the reranker can only reorder what retrieval surfaced.
Why it works
|
Bi-encoder retrieval |
Cross-encoder reranking |
| Query and document processed |
Separately |
Together |
| Documents encoded |
Once, in advance |
Per query |
| Attention across the pair |
None |
Full |
| Accuracy |
Good |
Substantially better |
| Speed over a whole corpus |
Fast |
Impractical |
| Speed over a shortlist |
N/A |
Acceptable |
The two-stage structure resolves the tension. Retrieval casts a wide net cheaply, reranking applies expensive accuracy to a small set. You get most of the cross-encoder's quality at a fraction of its cost.
Getting the candidate set right
This is where most reranking implementations underperform. If retrieval returns ten candidates and you rerank them to pick five, the reranker can only reorder those ten — and if the best document ranked fifteenth, it is not there and reranking cannot recover it.
Retrieve generously. Pulling a substantially larger candidate set than you intend to keep is what gives the reranker room to work. The cost is a linear increase in reranking latency, which is the tradeoff to tune.
The pattern that works: retrieve a broad set with hybrid search, rerank all of them, keep the top few for the model's context. Recall comes from the wide retrieval, precision comes from the reranker, and the final context stays small — which also helps with the degradation described in context rot explained.
And check the order of operations. Reranking cannot fix poor recall. If your labelled set shows the right document frequently is not retrieved at all, that is a retrieval problem, and hybrid search or better chunking is the fix. Reranking helps when the right document is retrieved but ranked below the cutoff.
Common mistakes
- Reranking a candidate set that is too small. Nothing to find, nothing to gain.
- Reranking when recall is the bottleneck. Reordering bad candidates does not help.
- Using an oversized reranker. Small models capture most of the benefit at much lower latency.
- Reranking in the critical path without measuring latency. A forward pass per candidate adds up.
- Passing all reranked results to the model. Rerank widely, keep few.
FAQ
How many candidates should I rerank?
Enough that the right document is reliably in the set. Check where the correct answer typically ranks in your retrieval results and set the candidate count comfortably above it.
How much latency does it add?
A model forward pass per candidate, batched. With a small reranker and a moderate candidate set it is typically tens of milliseconds, which is small next to generation.
Do I need to host the reranker?
Not necessarily. Hosted reranking endpoints exist and are convenient, though they add a network round trip.
Is reranking better than late interaction?
Different tradeoffs — reranking keeps storage low and adds query latency; late interaction pays in storage. See late interaction retrieval.
Where to go next
For the retrieval stage feeding it, read hybrid search with BM25. For the alternative architecture, late interaction retrieval, and for measuring the improvement, RAG evaluation metrics.