The same open-weight model served by two providers can differ by a large factor in tokens per second, by a meaningful margin in output quality, and by several multiples in price. That is counterintuitive — it is the same weights — and it is entirely explained by how each provider runs them.
Knowing what varies is the difference between an informed choice and picking the cheapest row in a comparison table.
What changed in 2026
- The market segmented. Providers differentiated into speed-optimized, price-optimized, and enterprise-compliance tiers rather than competing on a single axis.
- Quantization disclosure improved. Pressure from users produced clearer statements about the numeric precision at which models are served, after a period where cheap prices concealed reduced precision.
- Independent benchmarking matured. Third-party measurement of latency and throughput across providers became a normal input to procurement.
- Data terms became a differentiator. As enterprise adoption grew, contractual commitments about retention and training became as important as performance.
What actually differs
| Dimension |
Why it varies |
How to check |
| Tokens per second |
Hardware generation, batching strategy |
Measure with your prompt lengths |
| Time to first token |
Queue depth, prefill efficiency |
Measure at your traffic pattern |
| Numeric precision |
Provider chooses the quantization |
Ask directly; compare outputs |
| Price per token |
All of the above, plus margin |
Compare at equal precision |
| Context window offered |
May be capped below model maximum |
Read the model page |
| Rate limits |
Capacity allocation |
Check the tier you would buy |
| Data retention |
Policy, not technology |
Read the terms |
| Uptime and regions |
Infrastructure footprint |
Status history, not the status page today |
Quantization is the one that most often explains a price difference. Serving a model at reduced numeric precision uses less memory and produces more tokens per second per device, which is a legitimate engineering choice and a real quality tradeoff. It is not always disclosed prominently. If two providers quote very different prices for the same model, precision is the first thing to ask about.
Evaluating one properly
Benchmark with your own prompts. Published throughput numbers are measured with prompt and output lengths that may bear no relation to yours, and throughput varies substantially with both. A provider fast on short prompts may not be fast on your ten-thousand-token documents.
Measure both latency metrics. Time to first token determines whether a streaming interface feels responsive; tokens per second determines how long a long generation takes. Optimizing the wrong one produces a system that feels slow for reasons your dashboard does not show.
Compare output quality directly. Run the same prompts against two providers serving the nominally identical model and diff the results. If one is quantized more aggressively, you will see it on harder inputs before you see it on easy ones — which is why an evaluation set matters more than spot checks. The method in eval-driven development for AI applies unchanged.
Read the data terms. For anything touching customer data, retention period, training use, and processing location are contractual questions that no amount of performance advantage compensates for.
Common mistakes
- Comparing prices without comparing precision. It is the most common apples-to-oranges error in this market.
- Trusting published benchmarks. They use their prompt shapes, not yours.
- Ignoring rate limits on the tier you would actually buy. Headline capacity is often not what a new account receives.
- Single-provider dependency without an abstraction. Keep the provider behind an interface so switching is a config change — the same discipline as AI model deprecation planning.
- Overlooking context window caps. Providers sometimes serve a model with a smaller window than the model supports.
FAQ
Is hosted inference cheaper than self-hosting?
Usually, until volume is very high and sustained. Self-hosting adds operational responsibility that hosted providers absorb; the crossover is a utilization question covered in AI compute leasing explained.
Does quantization always hurt quality?
Not always noticeably. Modest quantization is frequently indistinguishable on ordinary tasks and degrades first on hard reasoning and long context. Test on your hardest cases.
Should I use multiple providers?
For redundancy on critical paths, yes, if the abstraction cost is low. Quality differences between providers make it more involved than a simple failover.
How do I check licensing for the models they serve?
Provider hosting does not change the model's licence terms for your use. See open-weight model licensing.
Where to go next
For self-hosting economics, read AI compute leasing explained. For cutting spend before switching providers, batch inference cost savings and AI model routers explained.