Parameters are the numbers inside a neural network that get adjusted during training and then stay fixed during use. Each one is either a weight, which scales the strength of a connection between two artificial neurons, or a bias, which shifts the output of a neuron up or down. A model described as having 70 billion parameters has, quite literally, 70 billion such numbers, most of them arranged into large matrices that get multiplied against input data as it flows through the network.
What changed in 2026
- Active parameters became the number that matters for cost. As mixture-of-experts architectures went mainstream, vendors started reporting both total and active parameter counts, because inference cost tracks the latter, not the former.
- Parameter count lost ground as a marketing headline. After several years of raw scale as the main selling point, 2025 and 2026 releases increasingly emphasized benchmark performance per dollar over headline parameter size.
- Quantization decoupled parameter count from memory footprint. A 70 billion parameter model can now run in well under half its original memory size with 4-bit quantization, which changes what "big" even means for deployment purposes.
How parameters get set
At the start of training, parameters are initialized to small random values — the network knows nothing. During training, the model makes a prediction, compares it to the correct answer using a loss function, and then adjusts every parameter slightly in the direction that would have reduced the error, a process called backpropagation combined with gradient descent. Repeated over trillions of tokens, this nudging process is what turns random numbers into a model that can complete a sentence, answer a question, or write code.
Nothing about a parameter is symbolic or interpretable on its own — you cannot point at one weight and say what concept it represents. Understanding emerges from the pattern across billions of them, which is part of why transformer architecture design (see what transformer architecture actually is) matters as much as raw count.
Parameter count: what it does and does not tell you
| Signal |
What it reflects |
Limitation |
| Total parameter count |
Overall model capacity |
Says nothing about training data quality |
| Active parameter count |
Compute cost per inference step (MoE models) |
Ignored by casual size comparisons |
| Parameters per training token |
Roughly, how well-trained the model is relative to its size |
Not always disclosed by vendors |
| Benchmark scores |
Actual task performance |
Can be gamed or overfit to public test sets |
A well-trained 8 billion parameter model can beat a poorly-trained 30 billion parameter one on real tasks. Parameter count sets an upper bound on capacity, not a guarantee of quality.
Dense models vs mixture-of-experts
In a dense model, every parameter is used for every token processed — a 70B dense model does a 70B-parameter amount of work on every single token. In a mixture-of-experts model, the network is split into many specialized sub-networks ("experts"), and a router activates only a handful of them per token. A model might have 400 billion total parameters but only activate 30 billion per token, giving it much of the capacity of a huge model at closer to the inference cost of a smaller one. Our deeper explainer on mixture of experts covers the routing mechanics in detail.
FAQ
Is a bigger parameter count always better?
No. It raises the ceiling on what the model could learn, but training data volume, data quality, and architecture choices determine how much of that ceiling gets used.
How is parameter count different from file size?
File size depends on precision — a model stored in 16-bit precision takes roughly twice the disk space of the same parameter count stored in 8-bit, and quantization shrinks it further without changing the parameter count itself.
What is the difference between parameters and hyperparameters?
Parameters are learned automatically during training. Hyperparameters — learning rate, batch size, number of layers — are chosen by engineers before training starts and are not learned from data.
Why do companies report active parameters separately now?
Because for mixture-of-experts models, active parameters predict inference cost and latency far more accurately than total parameter count does.
Where to go next