Mixture of experts is the architecture behind a striking fact about 2026 frontier models: they can carry a trillion parameters yet cost no more per token than a model a tenth the size. The trick is that they do not use all their parameters at once. Instead of one dense network processing every token, an MoE model is split into many specialized subnetworks, and a router decides which few to activate for each token. Most of the model sits idle at any given step.
What changed in 2026
- MoE became the frontier default. Most of the largest, most capable models now use sparse mixture-of-experts designs rather than dense ones.
- Fine-grained routing matured. Architectures moved toward many smaller experts with more of them active per token, improving quality without inflating compute.
- Serving tooling caught up. Inference stacks got better at expert parallelism and memory management, making MoE models more practical to deploy outside hyperscale labs.
How MoE works
A standard transformer applies the same feed-forward layer to every token. An MoE layer replaces that single feed-forward block with many of them — the "experts" — plus a small router (or gating network):
- The router looks at each token and scores the experts.
- It selects the top-k experts (often two) for that token.
- Only those experts process the token; their outputs are combined, weighted by the router's scores.
- Every other expert does no work for that token.
Because only k of many experts run per token, the compute per token depends on the active experts, not the total. That is the whole point.
Active vs total parameters
This is the number people get wrong. An MoE model has two very different parameter counts:
| Metric |
Meaning |
Drives |
| Total parameters |
All experts combined |
Model capacity and memory footprint |
| Active parameters |
What runs per token |
Compute cost and latency |
| Experts per token (k) |
How many fire at once |
Quality vs compute balance |
| Number of experts |
Total specialists available |
Capacity and routing complexity |
A model might have a trillion total parameters but activate only tens of billions per token. It reasons with the compute budget of a small model while holding the knowledge capacity of a huge one — but you still need enough memory to hold every expert.
Why it saves compute but not memory
MoE reduces floating-point operations per token, which cuts inference cost and speeds up generation. It does not reduce memory: all experts must be resident because the router might call any of them on the next token. This is why MoE models are cheap to run per token yet demanding on VRAM, and why techniques like quantization matter so much for actually deploying them.
The tradeoffs
- Routing instability. If the router sends most tokens to a few favorite experts, the rest never train well. Load-balancing losses exist to spread traffic, and tuning them is delicate.
- Serving complexity. Distributing experts across devices (expert parallelism) adds engineering overhead that dense models avoid.
- Uneven latency. Routing decisions can create imbalance across a batch, complicating throughput planning.
- Fine-tuning friction. Adapting an MoE model is trickier than a dense one; approaches like LoRA vs full fine-tuning need extra care around which parts to adapt.
FAQ
Is a mixture of experts model smarter than a dense one?
Not inherently. MoE gives more capacity per unit of compute, so at a fixed serving budget it often reaches higher quality. But a well-trained dense model can match a poorly trained MoE. The advantage is efficiency, not magic.
Why does MoE need so much memory if it uses few parameters per token?
Because any expert might be selected for the next token, all of them must be loaded in memory. Only the compute is sparse; the storage is not.
What does the router do?
The router is a small learned network that scores experts for each token and selects the top few. Its quality determines whether experts specialize usefully or collapse onto a few overused ones.
How many experts activate per token?
Commonly two, though fine-grained designs activate more, smaller experts. The count trades quality against compute — more active experts generally means better results and higher cost.
Where to go next