Almost every retrieval system in production uses a chunk size and overlap copied from whatever tutorial the author read first. Those defaults are reasonable starting points and they are not tuned for your documents, and the difference between a default and a tuned setting on a specific corpus can be substantial.
Understanding what each parameter does makes the tuning quick.
What changed in 2026
- Structure-aware splitting became the default recommendation. Splitting on document structure — headings, sections, paragraphs — before falling back to size limits clearly outperformed pure character counting.
- Semantic chunking matured. Methods that split where topic shifts occur, rather than at fixed lengths, became practical enough for production use.
- Late chunking gained attention. Embedding a long document first and then pooling into chunk representations preserved surrounding context that independent chunk embedding discards.
- Overlap fell out of favour as a primary lever. With better splitting strategies, large overlaps became less necessary and their storage cost more apparent.
What each parameter does
| Parameter |
Increasing it |
Decreasing it |
| Chunk size |
More context per chunk, less precise matching |
Precise matching, may lose surrounding meaning |
| Overlap |
Less risk of splitting a relevant passage |
Less storage, fewer near-duplicate results |
| Number retrieved |
Higher recall, more distractors in context |
Cleaner context, risk of missing the answer |
Overlap has exactly one purpose: preventing a passage that answers a question from being cut in half at a chunk boundary, leaving neither half sufficient. A modest overlap handles that. A large overlap mostly produces chunks that are substantially the same text, which inflates your index and means a retrieval returning several chunks may return the same content repeatedly — consuming context that could have held something new.
Chunk size is the more consequential parameter. Small chunks embed precisely, so similarity search matches them well, and they may lack the surrounding context needed to actually answer. Large chunks carry context and embed diffusely, so a specific question matches them less reliably. The right size depends on how your documents are written: dense reference material suits smaller chunks, narrative or explanatory content suits larger.
Tuning it properly
Split on structure first. If your documents have headings, sections, or clear paragraph boundaries, use them. A chunk that corresponds to a real section is more coherent than one that ends mid-sentence at character six hundred, and structure-aware splitting reduces how much overlap you need because you are no longer cutting arbitrarily.
Then measure recall at a few chunk sizes on a labelled set. This is a small experiment — three or four configurations, one evaluation run each — and it settles the question empirically rather than by intuition. The measurement setup is in RAG evaluation metrics.
Keep overlap modest once structure-aware splitting is in place. Its job is largely done by splitting sensibly.
Consider adding context to each chunk rather than increasing its size. Prepending the document title and section heading to a chunk before embedding gives it locating context at negligible storage cost, and frequently improves retrieval more than a size change does.
Common mistakes
- Fixed character splitting on structured documents. Cuts through headings, tables, and sentences.
- Large overlaps by default. Storage cost and duplicate retrieval results.
- Tuning chunking before checking whether retrieval method is the problem. Hybrid search often helps more.
- One chunk strategy for a mixed corpus. Reference documents and long-form prose want different treatment.
- Never measuring. Defaults persist for years because nobody ran the experiment.
FAQ
What is a good starting chunk size?
Whatever corresponds to a coherent unit in your documents — a section, a few paragraphs. Then measure two sizes either side of it.
Do I need overlap with structure-aware splitting?
Less. A small overlap still helps at section boundaries where a topic continues across them.
Should chunks be the same size?
Not necessarily. Structure-aware splitting produces variable sizes, and that is fine — coherence matters more than uniformity.
What about tables and code?
Both break badly under naive splitting. Handle them as units where possible, and consider the visual approach in multimodal RAG explained for table-heavy documents.
Where to go next
For the broader strategy, read RAG chunking strategies. For measuring the effect, RAG evaluation metrics, and for retrieval methods that reduce chunking sensitivity, hybrid search with BM25.