Embedding search has a structural limitation that has nothing to do with model quality. The query and the documents are encoded separately — each becomes a vector without ever seeing the other — and then compared by distance.
That is what makes it fast: document vectors are computed once at indexing time and reused for every query. It is also what makes it approximate. The encoder had to compress a document into a fixed vector without knowing what would be asked of it.
What changed in 2026
- Reranking became a default pipeline stage. Retrieve-wide-then-rerank settled as standard rather than an advanced addition.
- Small rerankers got good. Compact cross-encoders reached quality that made the latency cost acceptable for interactive use.
- Rank fusion became a companion. Combining several retrieval signals before reranking improved the candidate pool — see rank fusion.
- Measurement improved. Teams started distinguishing recall problems from ordering problems rather than treating "bad results" as one issue.
Bi-encoder versus cross-encoder
|
Embedding search |
Reranker |
| Encoding |
Query and document separately |
Together, as a pair |
| When computed |
Documents at index time |
Every candidate, at query time |
| Speed |
Very fast over millions |
Slow per candidate |
| Quality |
Approximate |
Substantially better |
| Scales to |
Whole corpus |
Tens of candidates |
A reranker takes the query and one document as a single input and produces a relevance score. Because it sees both at once, it can attend to how specific query terms relate to specific document passages — the interaction that separate encoding necessarily discards.
The cost is that nothing can be precomputed. Every query-document pair is a fresh forward pass, which is why rerankers cannot search a corpus and embeddings can.
Retrieve wide, rerank narrow
The pipeline that follows is straightforward and the sizing is where judgement enters.
Retrieve a generous candidate set with cheap vector or hybrid search — several dozen. Rerank those. Keep the top handful for the prompt.
The wide first stage exists because retrieval optimises recall and reranking optimises precision. You want the correct document somewhere in the candidates; the reranker's job is to move it to the top. Retrieving only five candidates leaves the reranker nothing to work with.
The narrow second stage exists because context is expensive and models attend unevenly across long inputs. Five well-ordered chunks beat twenty poorly-ordered ones.
Sizing the first stage is a measurement question: increase candidate count until recall stops improving, then stop. Every additional candidate costs a reranker call and adds latency.
It cannot fix recall
The most important limitation, and the most common misapplication.
A reranker reorders what it is given. If the document containing the answer was never retrieved, no amount of reranking produces it. Reranking a candidate set that does not contain the answer produces a confidently-ordered list of wrong documents.
Which is why diagnosis order matters. Measure hit rate first — did the correct chunk appear in the candidate set at all? If it did not, the problem is chunking, embeddings, or query formulation, and a reranker is the wrong investment. If it appeared at position 30, reranking is exactly the fix. Retrieval metrics covers making that distinction.
Common mistakes
- Adding a reranker to fix low recall. It reorders; it does not retrieve.
- Too few candidates. Nothing to reorder.
- Too many candidates. Latency and cost with diminishing recall gains.
- Not measuring before and after. Reranking should visibly improve ordering metrics; if it does not, something is wrong.
- Reranking on the wrong text. Score the same chunk text you will put in the prompt, not a truncated version.
- Ignoring latency. A reranker call per candidate is real time on an interactive path.
FAQ
How much does it improve results?
Substantially where ordering is the problem, and not at all where recall is. That is why measuring which you have first is the whole recommendation.
Does it replace hybrid search?
No — they compose well. Hybrid retrieval builds a better candidate pool; reranking orders it. Using both is common and sensible.
How many candidates should I rerank?
Enough that recall plateaus, typically some tens. Measure it rather than picking a number.
Can a general LLM do the reranking?
It can, and it is usually slower and more expensive than a purpose-trained reranker for the same job. A general model is a reasonable way to test whether reranking helps before investing in a dedicated one.
Where to go next
For diagnosing whether you have a recall or ordering problem, read retrieval metrics. For building a better candidate pool, rank fusion, and for the indexing-side improvement, contextual retrieval.