{"post":{"slug":"moe-routing-explained-2026","title":"MoE Routing Explained: How Experts Get Chosen","excerpt":"A mixture-of-experts model activates a fraction of its parameters per token. Here is how the router decides, why load balancing is hard, and what it means for serving.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span> dense model runs every parameter for every token. A mixture-of-experts model does not: a small routing network looks at each token and selects a handful of expert subnetworks to process it, leaving the rest idle. The model can hold far more total parameters while doing far less arithmetic per token.\n\nThat trade is why mixture architectures became common at the frontier. It also creates a specific set of operational problems that dense models do not have.\n\n## What changed in 2026\n\n- **The architecture became mainstream.** Mixture designs moved from a frontier-lab curiosity to a common choice across both closed and open-weight releases.\n- **Expert granularity increased.** Designs shifted toward more, smaller experts with finer-grained routing rather than a few large ones.\n- **Serving infrastructure specialized.** Expert-parallel deployment, where different experts live on different devices, became a supported pattern in major serving stacks.\n- **Memory expectations got corrected.** Widespread confusion about active versus total parameters gave way to clearer guidance that memory tracks the total.\n\n## Active versus total parameters\n\n| | Dense model | Mixture-of-experts model |\n| --- | --- | --- |\n| Parameters used per token | All | A small fraction |\n| Parameters that must be in memory | All | All |\n| Arithmetic per token | Proportional to size | Proportional to active size |\n| Memory requirement | Proportional to size | Proportional to total size |\n| Training difficulty | Standard | Harder; routing must be balanced |\n| Serving simplicity | Simple | Expert placement matters |\n\nThis table is the whole practical story. A mixture model advertising a small active parameter count is telling you about compute, not memory. Every expert has to be loaded because any token might route to it, so self-hosting requires memory for the full parameter count. Teams who plan capacity from the active figure discover this at deployment.\n\n## The load balancing problem\n\nThe router is trained alongside the model, and left to itself it develops favourites — routing most tokens to a handful of experts while others receive almost nothing. That is bad twice over: the popular experts become a throughput bottleneck while the neglected ones remain undertrained and contribute little.\n\nTraining therefore includes an auxiliary objective pushing the router toward even utilization. Getting that balance right is the central difficulty in training these models, and an imbalanced router shows up as both worse quality and worse hardware utilization.\n\nAt serving time the same imbalance manifests as uneven load across devices when experts are distributed. If one device holds the experts that receive most of the traffic, it saturates while others idle — which is a scheduling problem of the kind covered in [AI workload scheduling](/blog/ai-workload-scheduling-2026).\n\nFor anyone consuming these models through an API, none of this is visible or actionable. It matters when you self-host, where it determines your hardware requirement and your achievable throughput. The broader architecture background is in [mixture of experts explained](/blog/mixture-of-experts-explained-2026).\n\n## Common mistakes\n\n- **Sizing memory from active parameters.** Memory tracks the total, always.\n- **Assuming mixture means cheap to run.** Cheap per token in compute, expensive in memory.\n- **Ignoring expert placement when distributing.** Uneven placement produces uneven load.\n- **Comparing models on total parameters alone.** Active count predicts speed; total predicts memory. Both matter.\n- **Expecting dense-equivalent quality at equal active size.** A mixture model with a given active count typically outperforms a dense model of that size, and comparisons should account for the memory it consumes.\n\n## FAQ\n\n**Why not just use a dense model?**\nFor a given compute budget per token, a mixture model can hold much more knowledge. The tradeoff is memory and training complexity.\n\n**How many experts activate per token?**\nDesign-dependent, and typically a small number out of many. Finer-grained designs with more, smaller experts have become common.\n\n**Does routing add latency?**\nThe routing computation itself is small. The latency risk is uneven load across devices in a distributed deployment.\n\n**Can I fine-tune a mixture model?**\nYes, with the usual parameter-efficient methods, and routing behavior can shift during adaptation in ways worth monitoring.\n\n## Where to go next\n\nFor the architecture, read [mixture of experts explained](/blog/mixture-of-experts-explained-2026). For memory planning, [GPU memory estimate for LLMs](/blog/gpu-memory-estimate-llm-2026), and for serving efficiency, [AI workload scheduling](/blog/ai-workload-scheduling-2026).","hue":218,"metaTitle":"MoE Routing Explained: Expert Selection 2026","metaDescription":"How mixture-of-experts routing selects experts per token, why load balancing matters, and why memory requirements track total rather than active parameters.","keywords":["moe routing explained","mixture of experts router","sparse activation models","expert load balancing","active vs total parameters","expert parallelism serving","top-k gating","moe inference memory","model architecture 2026"],"tldr":["**A router picks a small number of experts per token** — total parameters are large, active parameters are small.","**This decouples capacity from compute cost** — more knowledge without proportionally more arithmetic per token.","**Memory does not shrink** — every expert must be resident even though few run per token.","**Load balancing is the core engineering problem** — an unbalanced router leaves hardware idle and some experts undertrained.","**Skip:** assuming a mixture model is cheaper to self-host. Active parameters drive compute; total parameters drive memory."],"readTime":"3 min read","wordCount":713},"related":[{"slug":"mixture-of-experts-explained-2026","title":"Mixture of experts explained in 2026: how MoE models work","excerpt":"Mixture of experts models activate only part of the network per token, giving huge capacity at a fraction of the compute. Here is how MoE works, why it dominates 2026 frontier models, and its tradeoffs.","category":"ai","date":"June 29, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":320},{"slug":"what-is-a-parameter-count-2026","title":"What Is a Parameter Count in 2026? Reading the Number","excerpt":"The 7B, 70B, or 400B label on a model name is a parameter count — but it measures capacity, not quality, and it is easy to misread. Here is how to interpret it.","category":"ai","date":"July 20, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150},{"slug":"best-ai-api-gateways-2026","title":"Best AI API Gateways in 2026: Multi-Model Routing Tools","excerpt":"The best AI API gateways in 2026 sit between your app and every model provider it calls, handling fallback, caching, and cost tracking so you don't hardcode a single vendor's SDK into your codebase.","category":"ai","date":"July 29, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":210}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"lora-fine-tuning-guide-2026","title":"LoRA Fine-Tuning: Adapting a Model Without Retraining It","excerpt":"LoRA trains a small set of extra weights instead of the whole model. Here is when adaptation beats prompting, what data you actually need, and how to avoid overfitting.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":213},"older":{"slug":"parking-ticket-appeal-2026","title":"Appealing a Parking Ticket: The Grounds That Work","excerpt":"Parking penalties are frequently issued in error and most go unchallenged. Here is what constitutes a valid ground and how the appeal process runs.","category":"guides","date":"June 26, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":40}}