Type "person holding a red umbrella in the rain" into a photo library and get back photos matching that description — none of which were tagged with any of those words. Nothing performed a keyword match. The sentence and the images were converted into vectors in the same space, and the system returned the pictures whose coordinates sat closest to the sentence's.
That shared space is the entire idea behind multimodal embeddings, and understanding its geometry is what separates a search feature that works from one that returns confidently irrelevant results.
What changed in 2026
- Multimodal retrieval became a standard RAG component. Document pipelines stopped discarding images. Charts, diagrams, and screenshots are now indexed alongside the text rather than dropped at ingestion.
- Video entered the same space. Embedding video segments for retrieval by description moved from research demo to shipping feature, mostly by treating video as sampled frames plus a transcript.
- Unified models displaced separate encoders. Rather than one model per modality plus an alignment layer, single models handling several modalities natively became the practical default.
- The threshold problem got acknowledged. Enough teams shipped cross-modal search with a single similarity cutoff and got bad results that per-modality calibration is now treated as a required step.
How the shared space is built
Two encoders, one target. An image encoder turns a picture into a vector; a text encoder turns a sentence into a vector of the same dimensionality. During training, matched pairs — a photo and its caption — are pushed toward each other, and mismatched pairs are pushed apart.
Do that across a very large number of pairs and something useful emerges: the region of the space representing "dog on a beach" ends up close to both photos of dogs on beaches and sentences describing them. No shared vocabulary, no tagging, no explicit mapping. Just a geometry where semantically related things across modalities end up near each other.
The consequences are worth being precise about. You can search images with text, search text with images, and cluster mixed content in one operation. What you cannot do is assume the space is uniform.
Where the space gets lumpy
This is the practical part, and it is where most implementations go wrong.
The modality gap. Text embeddings and image embeddings do not fully overlap even after alignment — they occupy somewhat separate regions of the space. Text-to-text similarities run higher than text-to-image similarities for genuinely equivalent matches. A cosine similarity of 0.8 might be an excellent cross-modal hit and an unremarkable text-to-text one.
The consequence: a single relevance threshold across a mixed corpus systematically favours one modality. If your search returns mostly text when images would serve better, this is usually why. Calibrate per modality pair — sample real matched and unmatched examples and find the cutoff for each combination separately.
Detail versus gist. Image embeddings capture overall content well and fine detail poorly. "A street scene with cars" retrieves reliably; "the license plate reads ABC-123" does not. The embedding is a summary, and specifics are what summaries lose.
Text inside images is invisible. A screenshot full of words embeds as "a screenshot", not as its contents. If your corpus has slides, scanned documents, or UI captures, run OCR and index the extracted text separately. This single oversight accounts for a large share of disappointing multimodal search deployments.
Vectors from different models are incomparable. Not less accurate — meaningless. Distances only have significance within one model's space, so changing embedding models means re-embedding the entire corpus. Budget for that before you pick one. How vector embeddings work covers the fundamentals this rests on.
Making it work in practice
Hybrid retrieval outperforms pure vector search consistently enough to be the default recommendation. Combine embedding similarity with whatever structured signals you have — filename, capture date, folder, tags, extracted OCR text — and merge the rankings. Vectors handle the fuzzy semantic part; metadata handles the precise part; neither is good at the other's job.
Chunking matters differently here. A long document splits into text chunks, but an image is atomic — you cannot half-embed a photograph. For documents containing both, keeping an image with its caption and surrounding paragraph as a unit preserves context that embedding them separately destroys. The reasoning in RAG chunking strategies transfers, with the caveat that images set their own boundaries.
Storage grows faster than people expect. Multimodal vectors tend toward higher dimensionality, and an image corpus produces a lot of them. Check that your vector store handles the volume and dimensionality before committing — best vector databases for RAG covers the options.
Common mistakes
- One similarity threshold across modalities. The modality gap makes a single cutoff wrong for at least one direction.
- Mixing embeddings from different models or versions. Produces plausible-looking nonsense rather than an error.
- Skipping OCR. Text in images is invisible to image embeddings, and this is the most common cause of "the search cannot find the slide I know exists".
- Expecting fine-grained detail. Embeddings capture gist. Small text, exact counts, and precise attributes need another mechanism.
- Discarding metadata. Date, location, and filename are precise signals that vectors approximate badly.
- Not re-embedding after a model change. The old vectors are not degraded, they are incompatible.
FAQ
Do I need a special vector database?
Not usually — most handle multimodal vectors fine since they are just vectors. Check dimensionality limits and how well the store handles your volume, and confirm it supports filtering by metadata alongside vector search, since hybrid retrieval depends on that.
Can I search video?
Yes, typically by sampling frames and embedding them plus the transcript. Retrieval granularity becomes the design question: a whole video, a scene, or a moment. Finer granularity means better precision and considerably more vectors.
How does this fit into RAG?
It extends retrieval to non-text content. The rest of the pipeline is unchanged — retrieve relevant chunks, put them in context, generate. The wrinkle is that a retrieved image needs a model that can actually see it in the generation step.
Is a caption plus text embedding good enough instead?
Sometimes, and it is a legitimate cheaper path: caption every image with a vision model, then embed the captions as ordinary text. You lose visual nuance the caption omitted, and you gain a searchable, inspectable, debuggable text corpus. For many products that is the better trade.
Where to go next
Start with how vector embeddings work if the single-modality case is still new. For storing and querying at scale, best vector databases for RAG, and for the retrieval pipeline these plug into, RAG chunking strategies.