Six services in your company call language models. Each has its own API key in its own secret store, its own retry logic written by a different person, its own idea of what to log. Nobody can answer what the company spends on inference, per feature, this month. Rotating a leaked key means finding every service that has it. Switching models means six pull requests in four languages.
An LLM gateway is a service every model call goes through. It holds the credentials, applies the policies, records what happened, and forwards the request. Each of those problems becomes one place instead of six.
What changed in 2026
- Gateways became standard at organisational scale. Enough companies hit the scattered-integration problem that a central proxy stopped being unusual architecture.
- Cost attribution became the leading driver. The question "which feature is spending this" turned out to be the one executives ask, and it is very hard to answer without a chokepoint.
- Semantic caching moved into the gateway. Caching responses to similar requests at the proxy layer became a common feature rather than something each service built.
- The abstraction tax got clearer. As provider-specific capabilities grew, flattening every provider to a common interface visibly cost teams the features they wanted.
What it centralises
| Concern |
Without a gateway |
With one |
| API keys |
In every service |
One place, rotated once |
| Retry and fallback |
Reimplemented per service |
Uniform policy |
| Rate limiting |
Per service, uncoordinated |
Global budget enforcement |
| Cost attribution |
Reconstructed from bills |
Tagged at the chokepoint |
| Model switching |
A change per service |
Configuration |
| Audit logging |
Inconsistent |
Complete by construction |
| Caching |
Per service, if at all |
Shared across all callers |
The key management row is the strongest single argument and the one most often underweighted. Credentials in a dozen repositories are a security posture nobody chose deliberately. A gateway makes rotation an operation rather than a project.
The cost row is the one that gets it funded. Attribution is genuinely hard to retrofit, and a chokepoint every call passes through is the natural place to tag spend by team, feature, and customer — see agent cost attribution.
The rate limiting row matters more than it looks. Without coordination, six services each retrying independently against a shared provider quota produce exactly the pile-on that makes an incident worse. One place enforcing a global budget prevents that.
What it costs you
It is now in the request path. Every AI feature depends on it. That means real availability work — redundancy, health checks, a clear failure mode — and a latency budget for the extra hop. A gateway that adds meaningful latency to every call is a tax on every feature.
Abstraction hides capability. This is the substantive tradeoff. Providers differ in what they offer beyond basic completion, and a gateway presenting one uniform interface either exposes provider-specific parameters as passthrough — leaking the abstraction — or drops them, costing you features. Teams that adopted a gateway for portability and then needed a provider-specific capability discover this quickly.
It becomes a bottleneck organisationally. If adding a model or a parameter requires a change to a shared service owned by another team, the gateway is now in everyone's critical path for shipping.
The mitigation for the second is to design for passthrough from the start: a common interface for what genuinely is common, and an explicit escape hatch for provider-specific parameters. Pretending providers are interchangeable is what causes the pain.
When a library is enough
For a single service calling a single provider, a gateway is infrastructure with no coordination problem to solve. The value comes from having many callers.
A shared internal library is the middle path: one implementation of retries, logging, and key handling, imported rather than called over the network. No extra hop, no availability dependency, no abstraction layer. The cost is language lock-in and coordinated upgrades — the same tradeoff as the sidecar pattern at a different layer.
Reach for a gateway when you have several services, more than one language, a real need for centralised cost and policy control, or a security requirement that credentials not be distributed. Below that, a library is less to run.
Common mistakes
- Building one before you need it. Infrastructure ahead of the problem.
- No availability plan. A single-instance gateway is a single point of failure for every AI feature.
- Flattening away provider features. Design passthrough in from the start.
- Not tagging at the gateway. The main reason to have a chokepoint, frequently unimplemented.
- Adding meaningful latency. Measure the hop; it applies to every call forever.
- Making it a deployment bottleneck. If every model addition needs another team, teams route around it.
- Caching without care. Semantic caching can return a response to a similar question, which is not always acceptable.
FAQ
Build or buy?
Buying gets you features quickly and adds a vendor to your critical path. Building is straightforward for the basics and grows into a real project once you want caching, routing, and quotas. Start with whichever gets the chokepoint in place — the value is in having one, not in who wrote it.
Does it help with rate limits?
Substantially. A central view of consumption lets you queue, prioritise, and shed load coherently rather than having six services discover a shared limit independently — see thundering herd and cache stampede.
What about streaming?
It must pass through cleanly, and this is worth testing specifically. A gateway that buffers a streaming response and forwards it at the end has silently broken every streaming UI behind it.
Does it replace observability tooling?
It gives you a natural place to emit from and is not a substitute for tracing across your application — see LLM observability.
Where to go next
For the spend visibility a gateway makes possible, read agent cost attribution. For the failure handling it should centralise, LLM fallback strategies, and for the same tradeoff at the network layer, the sidecar pattern.