Chunking is the unglamorous part of RAG that determines whether your retrieval is accurate or mediocre — and most teams pick a default and never revisit it. The chunk size, splitting method, and overlap strategy you choose will affect your recall, your context quality, and ultimately your answer quality more than almost any other parameter. Here is what the 2026 state of the art actually looks like.
What changed in 2026
- Semantic chunking went from research to default. LlamaIndex and LangChain both ship semantic splitters as first-class options. The quality gain over fixed-size splitting is well-documented across public benchmarks.
- Embedding models got document-structure-aware. Models like
text-embedding-3-large and Cohere's embed-v4 are trained on structured documents, making structure-aware splitting (by heading, section, paragraph) more effective.
- Multi-vector retrieval became standard. Storing both a summary embedding and chunk-level embeddings for the same content lets you retrieve at the right granularity. pgvector, Qdrant, and Weaviate all support this natively.
- Late chunking / context-aware embedding (Jina's technique) propagates document-level context into each chunk embedding, reducing the context blindness problem of independent chunk encoding.
Chunking methods compared
| Method |
How it splits |
Retrieval precision |
Context quality |
Compute cost |
| Fixed-size (token count) |
Every N tokens |
Medium |
Medium |
Very low |
| Fixed-size + overlap |
Every N tokens, slide M |
Medium–high |
Medium |
Very low |
| Sentence splitter |
Sentence boundaries |
High |
Low (tiny chunks) |
Low |
| Recursive character splitter |
Hierarchy of delimiters |
Medium–high |
Medium |
Low |
| Semantic splitter |
Embedding cosine shift |
High |
High |
Medium |
| Hierarchical (parent-child) |
Two-level: parent + child |
Very high |
High |
Medium |
| Document-structure aware |
Headings, sections, tables |
High (for structured docs) |
Very high |
Low–medium |
Fixed-size with overlap
The simplest approach: split on token count N, slide forward by N - overlap tokens. A 512-token chunk with 10% overlap (51-token overlap) is the classic starting point. Easy to implement, consistent size for vector indexing, but blind to semantic boundaries — a sentence about a contract deadline might span two chunks, losing meaning in both.
When to use: quick prototypes, very homogeneous documents (uniform prose, no tables or headers).
Semantic chunking
Calculate the embedding for each sentence, then find points where the cosine similarity between adjacent sentences drops sharply — that's a topical boundary. Split there. The result is variable-length chunks that correspond to coherent topics rather than arbitrary token counts.
Implementation with LlamaIndex:
from llama_index.core.node_parser import SemanticSplitterNodeParser
splitter = SemanticSplitterNodeParser(
buffer_size=1,
breakpoint_percentile_threshold=95,
embed_model=embed_model,
)
Trade-off: requires embedding every sentence during indexing (~2–5× more embedding calls), but retrieval quality improvement is typically 10–20% on recall@5.
Hierarchical (parent-child) chunking
Index small child chunks (128–256 tokens) for precise retrieval, but when a child chunk is retrieved, return its parent chunk (512–2048 tokens) to the LLM for generation. This combines the precision of small chunks with the context richness of large ones.
Parent: full section (1024 tokens) ← sent to LLM
└── Child chunk 1 (256 tokens) ← retrieved by vector search
└── Child chunk 2 (256 tokens)
└── Child chunk 3 (256 tokens)
LlamaIndex's HierarchicalNodeParser and LangChain's ParentDocumentRetriever implement this pattern. This is the recommended default for production RAG over structured documents.
Document-structure-aware splitting
For PDFs, HTML, and Markdown with explicit structure, split on headings and sections first, then apply semantic or fixed-size splitting within sections. Tables should stay together as a unit (never split mid-table). Code blocks should be isolated as atomic chunks.
Tools: LlamaParse for PDFs (see AI PDF extraction in 2026), Unstructured.io for mixed document types, custom Markdown splitters for wikis.
How to pick
- Homogeneous prose (articles, reports)? → Semantic chunking with 10–15% overlap.
- Structured documents (PDFs with headings, wikis, docs)? → Structure-aware splitting + hierarchical retrieval.
- Mixed content with tables and code? → Keep tables/code atomic, apply semantic splitting to prose sections.
- Need fast indexing at scale? → Fixed-size with overlap as a baseline, then benchmark semantic splitting on a sample.
- Always run retrieval evals (recall@5, MRR) before choosing — the right answer depends on your data.
Common mistakes
Choosing chunk size by instinct. Benchmark it. A 256-token chunk might outperform 1024 on your dataset even if conventional wisdom says otherwise.
No overlap. A sentence about your most important fact split across a boundary is invisible to both chunks. Always use overlap.
Splitting tables. A table split mid-row is worse than useless — it produces misleading partial data. Tables must be atomic chunks.
Ignoring metadata. Every chunk should carry source, section heading, page number, and document ID. Without metadata, you can't filter, you can't cite, and you can't debug retrieval failures.
Re-indexing without an eval regression test. Changing chunking strategy changes which queries work. Always run your eval set before and after.
What to skip
- Sentence-level chunks for generation context — they're too small to give the LLM useful context. Retrieve small, return larger.
- 512-token fixed chunks as a permanent default — it's a starting point, not a destination.
- Manual chunk size tuning without evals — you need a recall metric to know if you're improving.
FAQ
What chunk size is best in 2026?
For most use cases: 512–1024 tokens for parent chunks, 128–256 for child chunks in hierarchical retrieval. Benchmark against your specific data before committing.
How does chunking interact with embedding model choice?
Embedding models have a token limit (usually 512–8192). Chunks longer than the model's limit are truncated. Match your chunk size to your model's effective encoding window.
Should I chunk code differently from prose?
Yes. Code should be chunked by function or class, not by token count. A 50-line function is one semantic unit — splitting it destroys meaning.
How do I evaluate chunking quality?
Build a question-answer eval set from your documents. Measure recall@5 (does the correct chunk appear in the top 5 results?) before and after changing chunking strategy.
Where to go next