{"post":{"slug":"mixture-of-depths-2026","title":"Mixture of Depths: Not Every Token Needs Every Layer","excerpt":"A standard transformer spends identical compute on every token, whether it is a comma or the crux of an argument. Routing tokens through fewer layers reclaims that waste.","category":"ai","date":"February 4, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span> standard transformer processes every token through every layer. The token for a comma receives exactly the same computation as the token carrying the meaning of a sentence.\n\nThat uniformity is simple to implement and obviously wasteful. Most tokens in most text are easy — predictable function words, punctuation, common continuations — and do not need the model's full depth.\n\n## What changed in 2026\n\n- **Conditional computation moved from research toward practice.** Architectures varying compute per token became a serious efficiency direction rather than a curiosity.\n- **The distinction from mixture of experts sharpened.** Both are conditional computation and they route differently, and the terms stopped being conflated.\n- **Capacity-capped routing became the workable form.** Fixed per-layer capacity gave predictable compute, which is what makes it deployable.\n- **Serving implications got studied.** Variable per-token compute interacts with batching in ways that required work to exploit.\n\n## How routing works\n\nAt each layer, a small router scores every token in the sequence. Tokens scoring highest are processed by the layer normally; the rest bypass it via a residual connection, arriving at the next layer unchanged by this one.\n\nThe critical design detail is the **capacity cap**: each layer processes a fixed fraction of tokens, not \"however many the router thinks need it\". That makes total compute deterministic and known in advance, which is what allows efficient batching and predictable latency.\n\nWithout a cap, compute per sequence would vary with content, batching would become awkward, and latency would be unpredictable — which is precisely why simpler early-exit schemes proved difficult to deploy despite being conceptually appealing.\n\nThe router is trained alongside the model, learning which tokens benefit from processing at which depth. What it learns tends to be intuitive: content words and tokens at points of genuine uncertainty get more depth; function words and highly predictable continuations get less.\n\n## Not mixture of experts\n\nThe two get conflated because both route tokens conditionally. They route along different axes.\n\n| | Mixture of depths | Mixture of experts |\n| --- | --- | --- |\n| Routes | Whether to process at this layer | Which expert processes this token |\n| Saves | Compute, by skipping | Compute, by activating a subset |\n| Parameter count | Similar to dense | Much larger than active |\n| Memory | Similar to dense | Larger — all experts stored |\n| Axis | Depth | Width |\n\nMixture of experts keeps every token going through every layer and replaces each layer with several parallel branches, activating a few. Parameters grow enormously while active compute stays moderate — so the model is memory-hungry and compute-efficient.\n\nMixture of depths keeps the layer structure and lets tokens skip layers. Parameter count stays comparable to a dense model; compute falls.\n\nThey address different constraints, and they can be combined. If memory is your limit, mixture of experts makes it worse. If compute is your limit, either helps.\n\n## What it means in practice\n\nFor anyone not training models, this is an architectural property of the model you selected rather than something you configure. There is no serving-side switch.\n\nWhat it affects is the relationship between model size and inference cost. A model using conditional computation may have a parameter count suggesting one cost and an actual inference cost meaningfully lower — which makes parameter count an even weaker proxy for cost than it already was.\n\nThe practical implication is the same as for [mixture of experts](/blog/mixture-of-experts-explained-2026): compare models on measured cost and latency for your workload rather than on architectural specifications. The relationship between size and cost is no longer straightforward, and headline parameter counts mislead in both directions.\n\n## Common mistakes\n\n- **Confusing it with mixture of experts.** Different axis, different constraint relieved.\n- **Expecting a serving-side control.** It is architectural.\n- **Inferring cost from parameter count.** Conditional computation breaks that relationship.\n- **Assuming uniform quality across token types.** Tokens routed through fewer layers may be handled less well in unusual contexts.\n- **Comparing architectures on paper.** Measure on your workload.\n\n## FAQ\n\n**Does it reduce quality?**\nThe premise is that skipped computation was not contributing much for those tokens, and results generally support that at moderate skip rates. Aggressive skipping degrades quality, and the capacity cap is what keeps it in the useful range.\n\n**Can it combine with mixture of experts?**\nYes — they operate on different axes and compose. Combined architectures exist and add routing complexity.\n\n**Does it help with memory?**\nNo. It reduces compute while parameter storage is unchanged. If your constraint is memory, this does not help — quantisation does.\n\n**How do I know if a model uses it?**\nModel documentation, where disclosed. From outside, the signal is inference cost lower than parameter count would suggest.\n\n## Where to go next\n\nFor the width-axis counterpart, read [mixture of experts](/blog/mixture-of-experts-explained-2026). For the serving-side efficiency levers you do control, [continuous batching](/blog/continuous-batching-explained-2026) and [quantization explained](/blog/quantization-explained-2026).","hue":196,"metaTitle":"Mixture of Depths: Not Every Token Needs Every Layer","metaDescription":"How mixture-of-depths routing skips layers for easy tokens, how it differs from mixture of experts, and why capacity caps keep compute predictable.","keywords":["mixture of depths","conditional computation","token routing","adaptive compute","transformer efficiency","mixture of experts comparison","early exit","model architecture 2026"],"tldr":["**Uniform compute per token is wasteful** — trivial tokens receive the same processing as difficult ones.","**A router decides which tokens skip which layers** — capacity per layer is capped, so cost stays predictable.","**It differs from mixture of experts** — one skips layers, the other selects among parallel branches.","**The capacity cap keeps compute deterministic** — unlike dynamic early exit, which does not.","**Skip:** treating this as a serving-side lever. It is an architectural choice made at training time."],"readTime":"4 min read","wordCount":823},"related":[{"slug":"mixture-of-experts-explained-2026","title":"Mixture of experts explained in 2026: how MoE models work","excerpt":"Mixture of experts models activate only part of the network per token, giving huge capacity at a fraction of the compute. Here is how MoE works, why it dominates 2026 frontier models, and its tradeoffs.","category":"ai","date":"June 29, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":320},{"slug":"moe-routing-explained-2026","title":"MoE Routing Explained: How Experts Get Chosen","excerpt":"A mixture-of-experts model activates a fraction of its parameters per token. Here is how the router decides, why load balancing is hard, and what it means for serving.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":218},{"slug":"vocabulary-size-explained-2026","title":"Vocabulary Size: Trading Memory for Token Efficiency","excerpt":"A larger vocabulary means fewer tokens per document and a much larger embedding layer. The tradeoff has moved as models and context windows have grown.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":198}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"matter-smart-home-guide-2026","title":"Matter smart home in 2026: why it finally works and how to set one up","excerpt":"After three years of broken promises, Matter actually works in 2026. Here's how to pick a hub, what devices play nicely, and the gotchas that remain.","category":"tech","date":"February 4, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":215},"older":{"slug":"ppf-vs-elss-vs-nps-2026","title":"PPF vs ELSS vs NPS 2026: which Indian tax-saver actually wins?","excerpt":"PPF, ELSS, and NPS compared on returns, lock-in, taxation, and liquidity in 2026 — and why the right choice depends on which tax regime you're under.","category":"finance","date":"February 4, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":290}}