A standard transformer processes every token through every layer. The token for a comma receives exactly the same computation as the token carrying the meaning of a sentence.
That uniformity is simple to implement and obviously wasteful. Most tokens in most text are easy — predictable function words, punctuation, common continuations — and do not need the model's full depth.
What changed in 2026
- Conditional computation moved from research toward practice. Architectures varying compute per token became a serious efficiency direction rather than a curiosity.
- The distinction from mixture of experts sharpened. Both are conditional computation and they route differently, and the terms stopped being conflated.
- Capacity-capped routing became the workable form. Fixed per-layer capacity gave predictable compute, which is what makes it deployable.
- Serving implications got studied. Variable per-token compute interacts with batching in ways that required work to exploit.
How routing works
At each layer, a small router scores every token in the sequence. Tokens scoring highest are processed by the layer normally; the rest bypass it via a residual connection, arriving at the next layer unchanged by this one.
The critical design detail is the capacity cap: each layer processes a fixed fraction of tokens, not "however many the router thinks need it". That makes total compute deterministic and known in advance, which is what allows efficient batching and predictable latency.
Without a cap, compute per sequence would vary with content, batching would become awkward, and latency would be unpredictable — which is precisely why simpler early-exit schemes proved difficult to deploy despite being conceptually appealing.
The router is trained alongside the model, learning which tokens benefit from processing at which depth. What it learns tends to be intuitive: content words and tokens at points of genuine uncertainty get more depth; function words and highly predictable continuations get less.
Not mixture of experts
The two get conflated because both route tokens conditionally. They route along different axes.
|
Mixture of depths |
Mixture of experts |
| Routes |
Whether to process at this layer |
Which expert processes this token |
| Saves |
Compute, by skipping |
Compute, by activating a subset |
| Parameter count |
Similar to dense |
Much larger than active |
| Memory |
Similar to dense |
Larger — all experts stored |
| Axis |
Depth |
Width |
Mixture of experts keeps every token going through every layer and replaces each layer with several parallel branches, activating a few. Parameters grow enormously while active compute stays moderate — so the model is memory-hungry and compute-efficient.
Mixture of depths keeps the layer structure and lets tokens skip layers. Parameter count stays comparable to a dense model; compute falls.
They address different constraints, and they can be combined. If memory is your limit, mixture of experts makes it worse. If compute is your limit, either helps.
What it means in practice
For anyone not training models, this is an architectural property of the model you selected rather than something you configure. There is no serving-side switch.
What it affects is the relationship between model size and inference cost. A model using conditional computation may have a parameter count suggesting one cost and an actual inference cost meaningfully lower — which makes parameter count an even weaker proxy for cost than it already was.
The practical implication is the same as for mixture of experts: compare models on measured cost and latency for your workload rather than on architectural specifications. The relationship between size and cost is no longer straightforward, and headline parameter counts mislead in both directions.
Common mistakes
- Confusing it with mixture of experts. Different axis, different constraint relieved.
- Expecting a serving-side control. It is architectural.
- Inferring cost from parameter count. Conditional computation breaks that relationship.
- Assuming uniform quality across token types. Tokens routed through fewer layers may be handled less well in unusual contexts.
- Comparing architectures on paper. Measure on your workload.
FAQ
Does it reduce quality?
The premise is that skipped computation was not contributing much for those tokens, and results generally support that at moderate skip rates. Aggressive skipping degrades quality, and the capacity cap is what keeps it in the useful range.
Can it combine with mixture of experts?
Yes — they operate on different axes and compose. Combined architectures exist and add routing complexity.
Does it help with memory?
No. It reduces compute while parameter storage is unchanged. If your constraint is memory, this does not help — quantisation does.
How do I know if a model uses it?
Model documentation, where disclosed. From outside, the signal is inference cost lower than parameter count would suggest.
Where to go next
For the width-axis counterpart, read mixture of experts. For the serving-side efficiency levers you do control, continuous batching and quantization explained.