After instruction tuning, a model produces reasonable responses. The next problem is that "reasonable" covers a wide range, and some reasonable responses are clearly better than others — more helpful, better calibrated, appropriately cautious.
Teaching that requires a different kind of signal. You cannot demonstrate the single correct answer, because there is not one. What you can do is show pairs and say which is preferred.
What changed in 2026
- DPO and its variants became the default starting point. Simpler training with competitive results made direct methods the pragmatic choice for most teams.
- RLHF stayed relevant at the frontier. Where maximum control over the objective matters, the reward-model pipeline continued to justify its complexity.
- Preference data remained the bottleneck. Both methods need the same comparisons, and collecting good ones stayed the expensive part.
- Automated preference labelling spread. Using a strong model to generate comparisons became common, with the usual caveats about inheriting its biases.
The two pipelines
RLHF runs in stages. Collect preference comparisons. Train a reward model to predict which response a human would prefer. Then use reinforcement learning to update the policy so it produces responses the reward model scores highly, with a penalty for drifting too far from the starting model.
DPO collapses that. It derives a loss function that optimises the same underlying objective directly from the preference pairs, with no separate reward model and no sampling loop. Training looks much more like ordinary supervised learning.
|
RLHF |
DPO |
| Reward model |
Required |
None |
| Training loop |
Sampling and RL |
Standard supervised |
| Compute cost |
High |
Moderate |
| Implementation complexity |
High |
Low |
| Control over objective |
Fine-grained |
Less direct |
| Reward hacking risk |
Real |
Not applicable |
| Typical use |
Frontier labs |
Most practical work |
Reward hacking, and why DPO sidesteps it
The characteristic RLHF failure is worth understanding because it explains much of the appeal of direct methods.
The reward model is an approximation of human preference, trained on finite data. The policy is then optimised to maximise its score. Any systematic error in that approximation becomes something the optimiser will find and exploit.
In practice that shows up as responses that score well and are worse: excessive hedging, unnecessary length, formulaic structure that the reward model rewarded, or confident agreement with the user. The policy is doing exactly what it was told; the target was wrong.
Mitigations exist — the divergence penalty that keeps the policy near its starting point is precisely this — and tuning that penalty is a real and finicky part of running RLHF.
DPO has no reward model to hack, which removes this failure mode. It has its own: it can overfit the preference dataset, and it is more sensitive to the quality and coverage of those pairs since there is no intermediate model smoothing them.
The data is the hard part
Both methods consume the same input, and it is where the cost and the quality ceiling live.
Preference pairs need to be comparable — two responses to the same prompt, differing in ways that matter. Pairs where one response is obviously broken teach little, because the distinction is trivial. The useful signal comes from pairs that are both plausible and genuinely different in quality.
Annotator agreement is the practical constraint. If two people disagree about which response is better, the label is noise, and enough noise makes the training signal meaningless. Measuring agreement before scaling collection is worth the delay.
Automated labelling with a strong model is fast and inherits that model's preferences, including its biases toward length, formatting, and confident phrasing. A human-labelled subset to validate the automated labels is the usual compromise — the same discipline as LLM-as-judge.
Common mistakes
- Running either before instruction tuning. Preference methods refine; they do not create instruction-following.
- Ignoring annotator agreement. Noisy labels produce a model optimised toward noise.
- Obvious-versus-broken pairs. No useful signal in a trivial comparison.
- Untuned divergence penalty in RLHF. Too weak and the policy degenerates; too strong and nothing changes.
- Trusting automated preferences wholesale. They encode another model's biases.
- No evaluation independent of the preference data. Improvement on the training objective is not improvement.
FAQ
Which should I use?
DPO or a variant, unless you have a specific reason for the reward-model pipeline. It is simpler to run, cheaper, and competitive on typical tasks. Reach for RLHF when you need fine control over a composite objective.
Can I use both?
Yes — training a reward model for evaluation while using DPO for optimisation is a reasonable combination. See reward models.
How much preference data is needed?
Considerably less than instruction data, and quality-sensitive. Thousands of well-constructed pairs can move behaviour meaningfully; tens of thousands of noisy ones may not.
Does this reduce hallucination?
Somewhat and indirectly, by preferring appropriately-hedged responses over confident wrong ones — provided your preference data actually reflects that. It is not a substitute for grounding.
Where to go next
For the stage that must come first, read instruction tuning. For the reward model component, reward models, and for the safety behaviour these methods shape, refusal training.