A new embedding model publishes benchmark results comfortably ahead of the one you deployed eighteen months ago. Switching looks like a configuration change — swap the model name and carry on.
It is not. Vectors produced by different models occupy entirely different spaces, and distances between them are meaningless. Not approximate, not degraded: meaningless. A search comparing a new-model query vector against old-model document vectors returns results, ranked confidently, with no error anywhere, and they are nonsense.
Which makes an embedding change a full re-index of your corpus, and that is a project.
What changed in 2026
- Model turnover stayed fast. Meaningfully better embedding models kept arriving, which means most teams face this decision periodically rather than once.
- Variable dimensionality spread. Models supporting truncatable embeddings let you trade accuracy for storage after the fact, which changes the sizing conversation.
- Multimodal shifted requirements. Teams wanting to index images alongside text often need a different model family entirely, forcing a migration for capability rather than quality.
- Benchmark scepticism grew. Enough teams found leaderboard rankings not reproducing on their own corpora that domain benchmarking became the expected first step.
Benchmark on your corpus, not a leaderboard
Public benchmarks measure general-purpose performance across broad datasets. Your corpus is not general-purpose, and rankings frequently reorder on domain-specific content — legal text, code, medical notes, and internal jargon all behave differently from web prose.
Before committing to a migration, build a small evaluation from your own material: a few dozen real queries with known-correct chunks, exactly as described in retrieval metrics. Embed a sample of your corpus with the candidate model, run the queries, and compare hit rate against your current model on the same set.
This takes an afternoon and regularly overturns the assumption that motivated the migration. A model several places higher on a leaderboard can perform worse on a specialised corpus, and finding that out before re-embedding ten million chunks is worth the afternoon.
Compare on cost and latency too. A larger, better model that doubles your indexing bill and adds latency to every query may not be worth a modest accuracy gain.
Dimensionality has consequences
| Factor |
Larger vectors |
Smaller vectors |
| Retrieval accuracy |
Generally better |
Generally worse |
| Storage and memory |
Proportionally more |
Less |
| Search latency |
Slower |
Faster |
| Index build time |
Longer |
Shorter |
| Cost at scale |
Higher |
Lower |
Moving from a smaller to a larger dimensionality multiplies your vector storage directly. On a large corpus that can mean substantially more memory, which for an in-memory index is a real infrastructure change rather than a config tweak.
Models supporting truncation are useful here: they let you keep a shorter prefix of the vector with graceful degradation, so you can tune the accuracy-cost trade after the fact rather than committing up front. Where available, benchmark at several lengths — the shortest acceptable one is frequently much shorter than the default.
Running the migration
Dual-write, then cut over is the standard shape and the one that avoids downtime.
- Stand up a second index with the new model, alongside the existing one.
- Backfill the corpus into the new index. This is bulk, offline, latency-insensitive work — exactly what batch APIs are for, at roughly half the cost.
- Dual-write new documents to both indexes during the backfill, so the new index does not fall behind.
- Verify by running your evaluation queries against the new index and comparing to the baseline you measured earlier.
- Cut over reads once you are satisfied. Keep the old index briefly.
- Decommission the old index after a period with no need to roll back.
The step people skip is the second index. Migrating in place — deleting and re-embedding as you go — means a period where the index contains vectors from both models, which is the incomparability problem in production. Every query during that window returns partly-garbage results, silently.
Rolling back is why the old index stays. If the new model underperforms in ways your evaluation missed, flipping reads back is a configuration change rather than another full re-index.
Common mistakes
- Migrating in place. Mixed vectors, silent nonsense, no error.
- Trusting a leaderboard. Rankings often do not survive a specialised corpus.
- Not measuring a baseline first. Without a before, you cannot tell if the after is better.
- Ignoring dimensionality cost. Larger vectors change your infrastructure bill.
- Backfilling synchronously. Pays full price for work with no latency requirement.
- Deleting the old index immediately. Removes the cheap rollback.
- Forgetting the query path. Queries must be embedded with the same model as the documents — an obvious statement that is a common bug during cutover.
FAQ
How do I know a migration is worth it?
Measured improvement on your own evaluation set, weighed against the re-indexing cost and any change in storage and latency. A hit rate improvement of a few points on a large corpus may not justify the project; a substantial one usually does.
Can I keep both models running permanently?
You can run two separate indexes and query both, but merging their results is awkward since the scores are not comparable. Rank fusion works around it. Generally it is complexity worth avoiding unless the two models serve genuinely different content types.
What about the chunks themselves?
A re-index is a natural moment to revisit chunking, and changing both at once means you cannot attribute the difference. If you want to improve chunking too, do it as a separate measured change — see RAG chunking strategies.
Does this apply to reranking models?
Much less painfully. A reranker scores query-document pairs at query time rather than producing stored vectors, so swapping one is a deployment rather than a re-index.
Where to go next
For measuring the before and after properly, read retrieval metrics. For the cheap way to run the backfill, batch APIs for LLMs, and for the storage layer this all lands in, best vector databases for RAG.