On-policy versus off-policy is one of the oldest distinctions in reinforcement learning, and it turned out to matter enormously for how large language models get post-trained. The short version: on-policy methods insist on learning from data the current version of the model actually generated, while off-policy methods can learn from data generated by some other policy entirely — including a fixed, pre-collected dataset. That difference drives real tradeoffs in cost, stability, and infrastructure complexity that shaped the shift toward methods like DPO in LLM training.
What changed in 2026
- Off-policy and offline preference methods became the default starting point for many teams post-training LLMs, largely because they avoid the cost and complexity of live sampling during training that on-policy methods like PPO require.
- On-policy methods stayed relevant where distribution shift matters most. For tasks where the model current behavior needs to directly shape what it learns from — such as more recent variants used in reasoning-focused training — on-policy or hybrid approaches remained in active use.
- Hybrid and iterative approaches grew, where teams periodically regenerate fresh on-policy samples and re-run offline preference optimization on them, capturing some of the accuracy benefit of on-policy learning without running a full live RL loop continuously.
- GRPO and related group-relative methods gained adoption as an alternative on-policy approach that removes the need for a separate learned reward model in some setups, simplifying part of the classic RLHF pipeline described in our RLHF guide.
The core distinction
- On-policy: the data used to update the model must come from the current version of that same model's own behavior. If the model changes, you need fresh data from the new version to keep training correctly. PPO, the algorithm behind classic RLHF, is on-policy.
- Off-policy: the data can come from a different policy — an older version of the model, a different model entirely, or a static human-labeled preference dataset. The training algorithm has to account mathematically for the fact that the data was not generated by the policy currently being updated.
- Offline: a stricter version of off-policy where no new data is generated at all during training — you train entirely on a fixed, pre-collected dataset. DPO training is the most widely known example in LLM post-training.
On-policy vs off-policy methods compared
| Method |
Type |
Needs live sampling during training |
Needs a separate reward model |
| PPO |
On-policy |
Yes |
Yes |
| GRPO |
On-policy |
Yes |
Often not, depending on setup |
| DPO |
Offline / off-policy |
No |
No |
| KTO |
Offline / off-policy |
No |
No |
| IPO |
Offline / off-policy |
No |
No |
The practical upshot: on-policy methods generally track the model actual current behavior more faithfully, which can matter for correcting subtle distribution drift, but they cost meaningfully more compute and engineering effort because you must keep generating and scoring fresh samples throughout training.
Why the shift toward offline methods happened
Running an on-policy RL loop for LLM training means standing up live generation infrastructure, a reward model or scoring mechanism, and careful monitoring for instability — reward hacking, mode collapse, and KL-divergence blowups are all real operational risks. Offline methods like DPO reformulate the same underlying preference-learning objective as a simpler supervised-style loss computed directly on a fixed dataset of preferred and rejected responses. That simplicity is why so many teams adopted offline methods as a default, even though the underlying mathematics involves real approximations relative to a true on-policy solution.
Common pitfalls
- Treating offline methods as a strictly inferior shortcut. For many practical preference-alignment goals, the accuracy gap versus a well-tuned on-policy pipeline is small enough that the operational simplicity is worth it.
- Ignoring distribution drift with purely offline training. If your static preference dataset was generated by a meaningfully different model version, training against it can push the current model in directions that do not match its actual current failure modes.
- Assuming any one method is universally correct. Reasoning-heavy tasks, agentic tasks, and simple preference-tuning tasks have all favored different points on the on-policy to offline spectrum in practice; test on your own task rather than defaulting blindly.
FAQ
Is DPO a type of reinforcement learning?
It is derived from the same theoretical objective as RLHF's reinforcement learning step, but it is typically described as offline preference optimization rather than reinforcement learning in the classic sense, since it does not use an explicit reward model or on-policy sampling loop.
Why is PPO still used if offline methods are simpler?
Because on-policy sampling can better capture the current model actual behavior, which matters for tasks sensitive to distribution shift, and because more recent on-policy variants like GRPO have addressed some of PPO's historical complexity.
Can you combine on-policy and off-policy training?
Yes — iterative approaches that periodically regenerate on-policy samples and then run offline optimization on them are a common middle ground, balancing cost against accuracy.
Does the on-policy vs off-policy choice affect final model quality?
It can, but the size of the effect is task-dependent. For many preference-alignment goals the difference is modest; for tasks highly sensitive to distribution shift, on-policy or hybrid approaches tend to hold an edge.
Where to go next