The first question when self-hosting a model is whether it fits. The usual approach — parameter count times bytes per parameter — gives the weight memory and nothing else, and teams who size hardware on that number discover the gap when concurrency rises and the server runs out.
Weights are the floor. Several other things sit on top of them.
What changed in 2026
- Quantization became the default for serving. Reduced-precision inference moved from an optimization to the normal way open models are deployed, since the quality cost is small at moderate precision.
- Cache memory got respected in planning. Wider understanding of KV cache scaling meant capacity plans started including it rather than discovering it.
- Paged allocation reduced waste. Serving stacks with block-based cache allocation substantially cut the fragmentation overhead that used to inflate real requirements.
- Unified memory devices changed the picture. Machines sharing memory between processor and accelerator made capacity, rather than a fixed accelerator memory size, the constraint.
Inference memory components
| Component |
Scales with |
Notes |
| Model weights |
Parameters times bytes per parameter |
The floor; precision is the lever |
| KV cache |
Context length times batch size times layers |
Can exceed weights at long context |
| Activations |
Batch size and hidden size |
Modest for inference |
| Framework overhead |
Fixed-ish |
Allow real headroom |
| Fragmentation |
Allocation strategy |
Much reduced with paged allocation |
The practical method is: compute weight memory from parameters and precision, add your expected cache based on the context length and concurrency you actually need, then add meaningful headroom for overhead. A server sized to exactly the weights will fail the moment two users hold long conversations simultaneously.
For a mixture-of-experts model, remember that all experts must be resident even though few activate per token, so weight memory tracks total rather than active parameters — the point made in MoE routing explained.
Training is a different order of magnitude
| Component |
Relative to weights |
| Weights |
Baseline |
| Gradients |
Roughly the same again |
| Optimizer state |
Commonly several times the weights |
| Activations for backpropagation |
Scales with batch and sequence length |
| Total for full fine-tuning |
Several times inference |
This multiplier is exactly why parameter-efficient methods exist. Freezing the base model eliminates gradients and optimizer state for the frozen parameters, which removes most of the training footprint — the mechanism described in LoRA fine-tuning guide. Combining that with a quantized frozen base is what brings large-model adaptation onto a single accelerator.
Activation memory during training is the term people forget. It scales with both batch size and sequence length, and gradient checkpointing trades compute for a substantial reduction in it when memory is the binding constraint.
Common mistakes
- Sizing on weights alone. Works until real concurrency arrives.
- Ignoring context length in capacity planning. Cache scales with it directly.
- Assuming active parameters set memory for mixture models. They set compute; total sets memory.
- No headroom. Fragmentation and framework overhead are real.
- Planning training memory from inference numbers. Off by a large multiple.
- Aggressive quantization without measuring. Quality degrades first on hard reasoning and long context.
FAQ
How much does quantization save?
Roughly proportional to the precision reduction on weights. Cache can often be quantized separately for further savings.
Does quantization hurt quality?
At moderate levels, minimally for most tasks. Degradation appears first on complex reasoning and long context, so test on your hardest cases rather than assuming.
Can I split a model across accelerators?
Yes, and it introduces communication between devices that becomes a throughput factor — which is why interconnect matters, per AI compute leasing explained.
What about running on a laptop?
Unified memory machines can hold surprisingly large models, with bandwidth rather than capacity often becoming the limit. See AI PC TOPS explained.
Where to go next
For the cache detail, read KV cache explained. For training efficiently, LoRA fine-tuning guide and PEFT methods explained.