Most discussion of AI security concerns attacking a deployed model — jailbreaks, prompt injection, extraction. Data poisoning works earlier in the lifecycle. The attacker contributes corrupted examples to the training data, the model learns from them, and the resulting behaviour ships as a property of the weights.
There is no runtime filter that catches this, because from the model's perspective nothing is going wrong. It is doing exactly what it was trained to do.
What changed in 2026
- Research clarified the scale required. Findings suggested the number of poisoned samples needed can be closer to a small fixed count than a fixed proportion of the dataset, which makes poisoning large training corpora more feasible than previously assumed.
- Fine-tuning datasets drew attention. As adaptation became routine, the practice of fine-tuning on third-party instruction datasets emerged as the realistic exposure for most organizations.
- Provenance tooling matured. Tracking where training data came from and what changed became a documented practice rather than an afterthought.
- Model supply chain scrutiny increased. Downloading weights from public hubs came under the same kind of examination as downloading software dependencies.
Attack types
| Attack |
Goal |
Detection difficulty |
| Availability poisoning |
Degrade overall model quality |
Easier; general performance drops |
| Targeted poisoning |
Wrong behaviour on specific inputs |
Hard; general metrics look fine |
| Backdoor with trigger |
Normal until a trigger appears, then attacker-chosen behaviour |
Very hard; requires knowing the trigger |
| Bias injection |
Skew outputs on a topic |
Hard; may look like ordinary bias |
| Data extraction setup |
Make memorization of specific content more likely |
Hard |
Backdoors are the concerning category because ordinary evaluation cannot find them. The model performs normally on every test you run, because your tests do not contain the trigger. It behaves as intended until someone supplies the specific pattern, at which point the planted behaviour activates.
What actually defends
For the vast majority of teams, the practical exposure is not poisoning a foundation model's pretraining corpus — that is a nation-state-scale concern. It is fine-tuning on a dataset you did not create.
Inspect fine-tuning data. If you are adapting a model on an instruction dataset downloaded from a public hub, you are trusting whoever assembled it. Sample it, read it, and run automated checks for anomalous patterns. The dataset-quality discipline in LoRA fine-tuning guide doubles as a security control.
Track provenance. Know where every training example came from and what transformations were applied. This is the same reasoning behind SBOM and supply chain security, applied to data rather than code.
Be careful with user-contributed data. Any pipeline that feeds user input back into training is a poisoning vector by design. Rate-limit contribution, require review, and never train on unfiltered user submissions.
Verify model weights. Downloading weights from a public hub carries the same trust question as any dependency. Prefer signed artifacts from accountable publishers, and check that what you downloaded matches what was published.
Evaluate broadly. Targeted poisoning is invisible on narrow test sets. A diverse evaluation set covering many input categories is more likely to surface anomalies, even though it cannot find a hidden trigger.
Common mistakes
- Fine-tuning on unexamined third-party datasets. The most realistic exposure and the easiest to reduce.
- Training on unfiltered user input. A poisoning channel you built yourself.
- Assuming general benchmarks catch it. Targeted attacks leave aggregate metrics intact.
- No provenance records. Impossible to audit after the fact.
- Unverified weights from public hubs. Same trust problem as an unsigned package.
FAQ
Could someone poison a major foundation model?
It is a real research concern for pretraining corpora built from web data. For an individual organization it is not the practical threat; your fine-tuning data is.
Can I detect a backdoor after training?
Not reliably without knowing the trigger. Research on detection exists and is not something to depend on operationally. Prevention through data control is the workable path.
Does retrieval avoid this risk?
It shifts it. Retrieval avoids baking bad data into weights, and a poisoned document in your corpus becomes an injection problem instead — see prompt injection defense.
Is synthetic training data safer?
It removes the third-party dataset risk and introduces its own quality concerns. It also depends on the model generating it, which has its own provenance.
Where to go next
For runtime attacks, read prompt injection defense and membership inference attacks. For supply chain practice, SBOM and supply chain security.