You want a number that says how good a response is. Human judgement provides it, slowly and expensively, at a rate that cannot keep up with a training loop generating thousands of samples.
A reward model is the substitute: a model trained to predict what a human would have said. It is the component that makes reinforcement learning from human feedback possible, and its limitations propagate into everything trained against it.
What changed in 2026
- Reward models found a second life as evaluators. Even where direct preference methods replaced RLHF for training, reward models remained useful for scoring at scale.
- Length and formatting bias got measured properly. Explicit debiasing became standard rather than an afterthought.
- Ensembles gained ground. Several reward models disagreeing became a signal that a response is ambiguous, which is more informative than a single confident score.
- Distribution shift stayed the core problem. No amount of tuning solved a reward model going stale as the policy it scores improves past its training data.
Trained on comparisons, not ratings
The design choice that makes this work: humans are asked which of two responses is better, not to score each on a scale.
Absolute ratings are notoriously inconsistent. One annotator's 7 is another's 4, the same person drifts over a session, and the scale means different things for different prompts. Pairwise comparisons are far more stable — people are reliably better at "which of these two" than at "how good is this".
The model is then trained so that the preferred response receives a higher score than the rejected one. The absolute values are arbitrary; only the ordering carries meaning, which is worth remembering when someone quotes a reward score as though it were a measurement.
What it actually learns
The critical framing: it predicts what a human would prefer, which is not the same as what is correct.
Humans preferring an answer is correlated with that answer being good, and the correlation is imperfect in specific, exploitable directions:
Length. Longer responses are systematically preferred in preference data, largely because thoroughness reads as quality. A reward model learns this, and anything optimised against it becomes verbose. This is the single most documented artefact, and controlling for it explicitly — normalising by length, or balancing the training pairs — is now standard practice.
Confidence. A confident wrong answer frequently beats a correctly hedged one. Annotators reward decisiveness, and the model learns to reward it too.
Formatting. Bullet points, headers, and structure score well independent of content quality.
Agreement. Responses that agree with the premise of the question are preferred over ones that correct it, which trains sycophancy.
| Artefact |
Symptom in the trained policy |
| Length bias |
Padded, verbose responses |
| Confidence bias |
Fewer appropriate hedges |
| Format bias |
Bullet points everywhere |
| Agreement bias |
Sycophancy, less pushback |
These are not bugs in the reward model. They are accurate reflections of preference data, which makes them harder to remove than a straightforward error.
Going stale
The subtler operational problem: a reward model is trained on responses from a particular policy. As training improves that policy, its outputs move outside the distribution the reward model saw.
The reward model becomes progressively less reliable exactly as the policy gets better, which is the worst possible timing. Scores keep rising while actual quality plateaus or degrades — a divergence that only human evaluation reveals.
Mitigations: refresh the reward model periodically with new comparisons on current policy outputs, hold out human evaluation as a check that the reward score is still tracking something real, and watch for the score climbing while independent measures do not move. That last signal is the clearest indication that you are optimising an approximation rather than the thing itself — a specific instance of the general problem in eval drift.
Common mistakes
- Treating the score as a quality measurement. It is a learned prediction with known biases.
- No length control. Guarantees verbosity in anything optimised against it.
- Never refreshing it. It goes stale precisely as the policy improves.
- No independent human check. Removes your only signal that the proxy still tracks reality.
- Comparing scores across reward models. The scale is arbitrary and model-specific.
- Training on low-agreement comparisons. Noise in, noise out.
FAQ
Do I need one if I use DPO?
Not for training — that is the point of direct methods. A reward model remains useful for evaluating candidates at scale, which is a separate job. See DPO vs RLHF.
Can I use a general LLM as a reward model instead?
That is essentially what LLM-as-judge does, and it works with similar caveats — biases toward length and confidence appear there too. A purpose-trained reward model is typically cheaper to run at scale; a general model is easier to start with. See LLM-as-judge.
How do I know if mine is any good?
Held-out agreement with human preferences on comparisons it never saw. Track it over time, because it will decline as your policy drifts.
What is an ensemble worth?
Disagreement between several reward models flags responses where the preference signal is genuinely ambiguous, which is useful information a single score hides.
Where to go next
For the training methods that consume reward models, read DPO vs RLHF. For the same reliability problem in evaluation, LLM-as-judge and eval drift.