The early assumption was that more data is better and filtering is a refinement. Repeated results pointed the other way: careful curation of a smaller corpus consistently outperforms scaling up a noisy one, and the gap is not marginal.
That reframes data work from collection to selection. What you exclude matters as much as what you gather.
What changed in 2026
- Curation became the differentiator. With architectures converging, data quality emerged as a primary source of capability differences.
- Deduplication became standard and thorough. Near-duplicate detection, not just exact matching, became the expected baseline.
- Synthetic data matured with caveats. Generated data proved genuinely useful and required its own filtering and contamination checks.
- Mixture proportions got studied properly. How much code, how much of each language, and how much of each domain became deliberate decisions rather than accidents of collection.
Deduplication first
The highest-return single step, and the one most often done incompletely.
Web-scale corpora contain enormous redundancy — the same article syndicated across sites, boilerplate repeated on every page, near-identical documents differing in a header. Training on that spends compute learning the same thing repeatedly and pushes the model toward memorisation of duplicated passages.
Exact deduplication catches identical documents and misses the common case. Near-duplicate detection — documents differing only in small ways — removes considerably more and is where most of the benefit is.
Deduplication should also run across dataset splits. Near-duplicates spanning your training and evaluation sets inflate scores in exactly the way eval contamination describes.
Quality filtering
Beyond duplicates, a large share of scraped content is unusable: navigation boilerplate, machine-generated spam, truncated pages, and text so malformed it teaches nothing.
| Filter |
Removes |
Cost |
| Length bounds |
Fragments and boilerplate |
Trivial |
| Character distribution |
Encoding errors, gibberish |
Trivial |
| Repetition within document |
Spam, generated filler |
Cheap |
| Language identification |
Content in unintended languages |
Cheap |
| Perplexity against a reference model |
Low-quality text |
Moderate |
| Model-based quality classifier |
Nuanced low quality |
Expensive |
Cheap heuristics do most of the work. Length bounds, character distribution checks, and within-document repetition detection remove a substantial fraction of unusable content for almost no compute.
Model-based classifiers catch what heuristics miss and carry a risk worth naming: a classifier trained to recognise "high quality" tends to encode a particular style, and filtering hard on it narrows the distribution in ways that can hurt generality.
Mixture matters
Composition shapes capability as directly as volume does. The proportion of code determines coding ability. The proportion of each language determines multilingual performance — and interacts with tokenizer training, since under-represented languages suffer twice.
Two findings worth knowing. Adding code improves reasoning on non-code tasks, so code proportion is not purely about coding ability. And repeating high-quality data for a small number of epochs generally beats adding a larger quantity of lower-quality data — quality-weighted repetition is a real technique rather than a compromise.
Getting proportions right is more consequential than most incremental data collection, and it is cheaper to adjust.
Common mistakes
- Collecting before filtering. The gain from filtering usually exceeds the gain from more volume.
- Exact deduplication only. Near-duplicates are the larger problem.
- Not deduplicating across splits. Silent evaluation inflation.
- Aggressive model-based filtering. Narrows the distribution.
- Ignoring mixture proportions. Shapes capability more than total size.
- Unfiltered synthetic data. Can carry the generator's biases and memorisation.
- No decontamination against evaluations. Scores that mean nothing.
FAQ
How much data do I need?
For pretraining, an enormous amount — and quality-filtered beats larger-and-noisier at essentially every scale. For fine-tuning, far less than people expect; a few thousand excellent examples outperform tens of thousands of mediocre ones.
Is synthetic data useful?
Genuinely, particularly for covering cases real data lacks. It needs the same filtering as any data, plus a check that it does not reproduce content from evaluation sets.
How do I filter for a specific domain?
Classify and select rather than collecting narrowly. Domain data alone tends to produce a narrow model; a domain-weighted mixture over a broad base works better.
Does this apply to fine-tuning datasets?
More strongly, because the datasets are small enough that a few bad examples represent a meaningful fraction. Every example in a small fine-tuning set should be one you would defend — see instruction tuning.
Where to go next
For the supervised stage that consumes curated data, read instruction tuning. For the contamination risk curation must address, eval contamination, and for how tokenisation interacts with data mixture, tokenizer training.