A multimodal model does not process pixels directly in its language layers. An image passes through a vision encoder that converts it into a sequence of vectors, and those enter the model's context as tokens alongside the text.
That conversion is where most practical behaviour comes from — the cost, the resolution limits, and the reason a model can describe a chart accurately and misread a number on it.
What changed in 2026
- Resolution handling improved. Dynamic approaches adapting to image size and aspect ratio replaced fixed square resizing.
- Token costs became visible. Providers exposed image token counts, which made the cost of high-resolution inputs concrete.
- Document understanding matured. Multimodal models handling page layout directly reduced reliance on separate parsing.
- OCR retained a role. For precise text extraction, dedicated OCR remained more reliable than asking a vision model to read — see OCR pipelines.
Images cost tokens
The practical consequence people encounter first: an image consumes context and costs money in proportion to its resolution.
A modest image might cost a few hundred tokens. A high-resolution one, tiled into many pieces, can cost thousands — more than several pages of text. Sending several high-resolution images in one request can dominate the entire prompt budget.
| Image handling |
Token cost |
Detail preserved |
| Low resolution, single pass |
Low |
Overall scene only |
| Standard resolution |
Moderate |
Most visible content |
| High resolution with tiling |
High |
Small text, fine detail |
| Multiple high-res images |
Very high |
Everything, expensively |
Which makes resolution a cost lever you control. If the task is "what is in this photograph", low resolution is sufficient and much cheaper. If it is "what does the small print say", you need resolution and should expect to pay for it.
Tiling, and what it costs
Large images exceed what an encoder processes in one pass, so they are typically split into tiles, each encoded separately, often alongside a downscaled view of the whole.
That preserves detail — small text in one region survives because that tile was encoded at full resolution — and it multiplies the token count, since each tile contributes its own tokens.
It also introduces a subtle limitation: relationships spanning tile boundaries can be weaker than relationships within a tile. A table split across tiles may be read less reliably than one contained in a single tile, which is one reason document layout affects extraction accuracy in ways that look arbitrary.
The encoder is a bottleneck
The framing that explains most failures: the encoder compresses an image into a fixed-size representation. It is a summary, and summaries lose specifics.
What survives compression well: overall scene, objects present, general layout, colour, obvious relationships.
What survives poorly: small text, precise numeric values, exact positions, fine visual distinctions, and long strings of characters like identifiers or serial numbers.
This is why a model can accurately describe a chart's shape and trend while misreading its axis labels. The gist compressed well; the digits did not.
The practical response is to match the tool to the requirement. For extracting exact text — an invoice number, a total, a date — dedicated OCR is more reliable than asking a vision model to read, and the two compose well: OCR extracts the text, the model reasons about it. For understanding layout, structure, and meaning, the vision model is the right tool.
Common mistakes
- Sending maximum resolution by default. Expensive, frequently unnecessary.
- Expecting exact text extraction from a vision model. Use OCR for precision.
- Ignoring image token cost in budgets. Can exceed the text entirely.
- Not checking what resolution the provider actually uses. Your upload may be downscaled before encoding.
- Sending many images per request. Multiplies cost quickly.
- Assuming detail is preserved. The encoder summarises.
- Poor source quality. A blurry photograph does not improve at high resolution.
FAQ
How do I reduce image costs?
Send the lowest resolution that supports the task, crop to the relevant region rather than sending the whole page, and avoid sending several images where one would do.
Can a vision model read a document reliably?
It reads well for understanding and less reliably for exact transcription. For fields you will act on — amounts, identifiers, dates — validate against OCR or against the source, per LLM output validation.
Does image order matter?
Position in context affects attention, so images in the middle of a long prompt may receive less weight — the same effect as with text, per context extension.
What about video?
Typically handled as sampled frames plus a transcript, which means the cost is frames times per-image cost. Sampling rate becomes the main cost and fidelity lever.
Where to go next
For precise text extraction, read OCR pipelines. For searching images by description, multimodal embeddings, and for validating extracted values, LLM output validation.