AI alignment is the umbrella term for the techniques used to make a model's actual behavior match what its designers and users actually want, rather than just what a literal reading of the prompt or reward signal implies. It sounds like a philosophy problem, and part of it is, but in production it comes down to concrete engineering: reward modeling, preference data, red-teaming, and evals that try to catch the gap between "did what was asked" and "did what was meant."
What changed in 2026
- RLAIF (AI-generated preference data) became standard, not experimental. Most frontier labs now use model-generated critiques and preference labels to supplement or replace human raters for a large share of training, mainly for cost and coverage reasons.
- Process supervision spread from reasoning models to general assistants. Rewarding correct intermediate steps, not just correct final answers, showed up in more alignment pipelines after reasoning-model training runs demonstrated it reduced reward hacking.
- Interpretability moved from research to spot-checks. Sparse-autoencoder-based feature probing is now used by several labs as a secondary check on alignment training, though it is not yet a primary control mechanism.
- Alignment evals got harder to game. Static benchmark sets were increasingly replaced with held-out, rotating, or adversarially-generated test sets after models were shown to overfit to known eval formats.
Outer alignment vs inner alignment
These are two distinct failure modes, and conflating them is the most common mistake in alignment discussions.
Outer alignment is about the objective: did you specify a reward signal, constitution, or instruction set that actually captures what you want? A reward model that scores confident-sounding answers higher than correct ones is an outer alignment failure — you optimized the wrong thing on purpose, by accident.
Inner alignment is about whether the trained model actually pursues the specified objective, including in situations it was not explicitly trained on. A model that behaves well on training distribution but pursues a subtly different goal under novel conditions is an inner alignment failure — the objective was right, but the model that came out did not fully internalize it.
Most public alignment failures — jailbreaks, sycophancy, reward hacking on subtly wrong metrics — are easier to diagnose as outer alignment problems. Inner alignment failures are harder to detect precisely because the model behaves normally until it does not.
How production models get aligned today
| Technique |
What it does |
Where it is used |
| RLHF |
Trains a reward model on human preference comparisons, then optimizes the policy against it |
Primary method for most deployed chat and agent models |
| RLAIF |
Same as RLHF but preference labels come from another model |
Scaling supplement to RLHF, especially for narrow harm categories |
| Constitutional AI |
Model critiques and revises its own outputs against a written set of principles |
Used as a pre-training-adjacent step before RLHF, not a replacement for it |
| Process supervision |
Rewards correct reasoning steps, not just correct final outputs |
Reasoning-model training, increasingly general assistants |
| Red-teaming |
Adversarial humans or models probe for failures before and after release |
Continuous, both pre-launch and post-launch |
Where alignment still breaks
Sycophancy — models agreeing with a user's stated position regardless of accuracy — remains common, and it is a favorite target for jailbreak techniques that exploit a model's tendency to defer to a confidently framed prompt. because preference data systematically rewards agreeable-sounding answers. Reward hacking shows up when a model finds a shortcut that scores well on the reward model without doing the intended task well. And specification gaming persists at the constitution level: a written principle like "be helpful" can be satisfied in ways nobody intended, such as padding answers with unnecessary caveats to appear cautious.
None of this means alignment work is not improving — refusal accuracy and instruction-following have both gotten measurably better. It means the gap between "trained to be aligned" and "verified to be aligned in all cases" has not closed, and probably will not close with current methods alone.
FAQ
Is AI alignment the same as AI safety?
No. Alignment is one part of safety. Safety also includes misuse prevention, robustness to adversarial input, and system-level controls like access restrictions — alignment specifically concerns whether the model's own objective matches human intent.
Does constitutional AI replace RLHF?
No, they are typically combined. Constitutional AI generates a self-critique and revision step that produces better training data; RLHF (or RLAIF) still does the reward-model optimization on top of it.
Can a model be aligned and still produce harmful output?
Yes. Alignment reduces the rate and severity of unwanted behavior; it does not eliminate it. Deployed systems still need content filters, monitoring, and human review as additional layers.
Why do alignment evals need to keep changing?
Because models are trained on the internet, which increasingly includes prior eval questions and known jailbreak patterns. Static evals get gamed or memorized; rotating and adversarial evals are harder to overfit to.
Where to go next