Vector embeddings are the numerical backbone of modern AI applications. Every semantic search, every RAG pipeline, every recommendation engine that "understands" what you mean rather than what you typed is built on them. Despite this, most engineers learn embeddings just enough to paste the API call and move on — which explains why retrieval quality is often the first thing to disappoint in production.
What changed in 2026
- Embedding models got significantly better. The gap between OpenAI's
text-embedding-3-large, Cohere's Embed v4, and the best open-source models (Nomic Embed, E5-Mistral) narrowed substantially — and multimodal embeddings (text + image in the same space) became production-ready.
- pgvector hit maturity. Version 0.8+ added HNSW indexing with competitive ANN performance, making Postgres the default choice for teams already running it.
- Matryoshka embeddings went mainstream. Models trained with Matryoshka Representation Learning let you truncate embedding dimensions at inference time (e.g., 1536 → 256) with graceful quality degradation — cutting storage and search cost significantly.
- Rerankers became standard. Embedding-based first-stage retrieval + a cross-encoder reranker is now the default RAG retrieval stack, not a premium add-on.
How embeddings work
An embedding model maps text (or an image, or code) to a dense vector — typically 768 to 3072 floating-point numbers. The training objective forces semantically similar inputs to produce vectors that are close together by cosine similarity or dot product.
At query time: embed the query → find the K nearest stored vectors → return those documents. The math is a nearest-neighbor search in a high-dimensional space.
What "close together" means is determined by the training data and objective. A model trained on code will have fn sort_by_key close to sorted(list, key=lambda x: x.key). A general model may not.
Embedding model comparison (2026)
| Model |
Dimensions |
Best for |
Self-hostable |
| OpenAI text-embedding-3-large |
256–3072 (Matryoshka) |
General English text |
No |
| Cohere Embed v4 |
1024 |
Multilingual, long docs |
No |
| Nomic Embed 1.5 |
64–768 (Matryoshka) |
English, self-host, cost |
Yes |
| E5-Mistral-7B |
4096 |
High-quality open-source |
Yes (GPU) |
| Voyage Code 3 |
1024 |
Code search |
No |
| CLIP / SigLIP |
512–1024 |
Image + text |
Yes |
For most English-only RAG apps, text-embedding-3-small (1536-dim, truncatable to 256) is the cost-quality sweet spot. Go to 3-large only if retrieval quality is measurably worse.
Chunking: the part that actually determines quality
Bad chunking hurts more than a mediocre embedding model. Rules that work:
- Target 200–500 tokens per chunk with ~50-token overlap for most prose.
- Never split mid-sentence. Use sentence-aware splitting (spaCy, NLTK, or LangChain's
RecursiveCharacterTextSplitter).
- Chunk by semantic unit for structured docs. One heading section = one chunk for docs/wikis.
- Keep metadata in the chunk. Title, section header, and source URL embedded in the chunk text dramatically improve relevance.
- For code, chunk by function/class, not by line count.
See RAG chunking strategies in 2026 for a deeper treatment.
How to pick your vector storage
| Scale |
Recommendation |
| < 100K vectors, already on Postgres |
pgvector with HNSW index |
| < 5M vectors, need filtering + metadata |
Qdrant (self-host or cloud) |
| > 5M vectors or multi-tenant SaaS |
Pinecone, Weaviate, or Qdrant cloud |
| Serverless / no infra |
Pinecone serverless or Turbopuffer |
| Existing Elasticsearch/OpenSearch |
Use their vector search (kNN) |
Don't add a new vector database if you already have Postgres and you're under a few million vectors. pgvector's HNSW index is fast enough for most applications.
Common mistakes
No reranker. Embedding similarity is approximate. A cross-encoder reranker (Cohere Rerank, BGE reranker) on the top-20 embedding results improves precision significantly at low added cost.
Same chunk size for all content types. Dense technical docs need smaller chunks than narrative prose. Tune per content type.
Ignoring embedding drift. If your base model changes, re-embed your entire corpus — mixed embeddings from different models in the same index produce garbage results.
Cosine vs. dot product confusion. Most modern embedding models expect dot product (inner product) similarity, not cosine, when using normalized vectors. Check your index configuration.
Storing embeddings in application memory. Even at 100K vectors, you need an indexed store; in-memory numpy arrays don't scale past a few queries/second.
What to skip
- Fine-tuning embedding models before validating the retrieval pipeline end-to-end — fix chunking and reranking first, they move the needle more.
- Very high-dimensional embeddings (3072+) when you can truncate to 256 with Matryoshka and get 95% of the quality at 1/10 the storage and search cost.
- Storing raw float32 vectors at scale — quantize to int8 or use product quantization to reduce storage by 4–8× with marginal quality loss.
FAQ
How do I evaluate retrieval quality?
Measure hit rate (is the relevant chunk in the top-K?), MRR, and nDCG on a labeled eval set. At minimum, track how often the final answer is grounded in retrieved context.
Can I embed images and text together?
Yes, with multimodal models like CLIP or SigLIP. Both map to the same vector space so you can search "sunset over mountains" and retrieve both text descriptions and matching photos.
How often should I re-embed my data?
When content changes significantly, when you upgrade the embedding model, or when retrieval metrics degrade. Schedule periodic re-embedding checks.
What's the difference between bi-encoder and cross-encoder?
Bi-encoders (embedding models) encode query and document separately — fast, scalable, used for first-stage retrieval. Cross-encoders process the pair together — slow, precise, used for reranking.
Where to go next