Model serving infrastructure is the system that takes a trained model and answers prediction requests against it at the latency, throughput, and scale a real application needs. Training produces a static artifact — a file of weights. Serving turns that artifact into a running service: loading it into memory, batching or streaming requests through it, scaling replicas with traffic, and rolling out new versions without breaking the requests already in flight. Most of the operational complexity in production machine learning lives here, not in training.
How it works
Serving splits into three patterns based on how fast an answer is actually needed. Batch serving runs predictions over a large dataset on a schedule and writes results somewhere to be read later — no request waits on the model in real time, so throughput matters far more than per-request latency. Online serving answers individual prediction requests synchronously, typically behind an API, with a latency budget measured in milliseconds; this is what powers a fraud check on checkout or a recommendation on page load. Streaming serving sits between the two, processing a continuous flow of events and emitting predictions as they occur, common for real-time scoring on event streams.
# Online serving: a request-response prediction endpoint
@app.post("/predict")
def predict(features: FeatureInput):
x = preprocess(features)
prediction = model.predict(x) # must fit the endpoint's latency budget
return {"score": float(prediction)}
Online serving is where most infrastructure investment goes, because it has to hold a latency budget under variable traffic, which batch serving does not need to worry about at all.
Why LLM serving needed its own engines
General-purpose model servers were built around classic ML models: fixed input shapes, single forward passes, predictable per-request cost. LLM inference breaks all three assumptions — output length varies per request, generation happens token-by-token, and a KV-cache has to be managed across many concurrent, variable-length requests without wasting GPU memory. That mismatch is why a separate category of inference engines emerged specifically for LLMs, built around continuous batching (packing new requests into GPU batches as older ones finish, instead of waiting for a fixed batch to complete) and efficient KV-cache handling.
Serving tool landscape
| Tool |
Best for |
Notable strength |
| Triton Inference Server |
Classic ML and deep learning models |
Multi-framework support, strong batching, mature |
| TorchServe |
PyTorch models |
Native PyTorch integration, straightforward for PyTorch-only shops |
| TensorFlow Serving |
TensorFlow models |
Mature, tightly integrated with the TF ecosystem |
| Ray Serve |
Mixed Python model pipelines |
Flexible composition of multiple models/steps in one service |
| vLLM |
LLM inference |
Continuous batching and PagedAttention KV-cache management, high throughput |
| KServe / Seldon Core |
Kubernetes-native model serving at scale |
Standardized deployment, autoscaling, canary rollout on K8s |
| Managed endpoints (SageMaker, Vertex AI) |
Teams wanting less operational ownership |
Autoscaling and infra managed, less control over serving internals |
Choosing between these is mostly about what you are serving (classic ML vs LLM) and how much serving infrastructure you want to operate yourselves versus hand to a managed platform.
Rolling out a new model version safely
- Shadow deployment. Send production traffic to the new version alongside the current one, but only return the current version's answer. Compare outputs offline before anyone depends on the new version.
- Canary rollout. Route a small percentage of real traffic to the new version and monitor latency and prediction quality before increasing that percentage.
- Automatic rollback triggers. Tie the rollout to concrete thresholds (latency regression, error rate, a quality metric dropping) so a bad version reverts without waiting for someone to notice manually.
- Full rollout. Only after the canary period holds steady across representative traffic, not just a quiet time window.
Common mistakes
- Serving every model behind one generic endpoint with no autoscaling. Traffic spikes and cold starts on an under-provisioned endpoint show up directly as latency the caller feels; size and autoscale serving infrastructure to the actual traffic shape.
- Applying classic serving patterns to LLM inference unchanged. A general-purpose model server without continuous batching or KV-cache management will underuse GPU capacity badly compared to a purpose-built LLM inference engine.
- Skipping shadow or canary rollout for "small" model updates. A retrained model with slightly different weights can regress in ways offline metrics did not catch; real traffic is the only place some failure modes show up.
- Ignoring cold start latency for infrequently-called models. A model that scales to zero saves cost but can add seconds of latency to the first request after idle time; decide deliberately whether that tradeoff is acceptable for the use case.
FAQ
Is model serving the same as deploying an API?
Related but not identical — serving infrastructure includes the API layer plus concerns an ordinary API does not have, like batching requests together for GPU efficiency and managing model memory footprint.
Why can't I just use a normal web server to serve an LLM?
You can, but a general-purpose server will not do continuous batching or KV-cache management, which is where most of the throughput gains in LLM inference actually come from — you would be leaving significant GPU efficiency on the table.
What is the difference between batch and online serving?
Batch serving runs predictions over a dataset on a schedule with no real-time caller waiting; online serving answers individual requests synchronously within a strict latency budget. The right choice depends entirely on whether anything is waiting on the answer in real time.
Do I need Kubernetes to serve models in production?
No, but tools like KServe and Seldon Core assume it, and it is a common choice at scale for standardized autoscaling and rollout. Smaller deployments often do fine with a managed endpoint or a simpler containerized service.
Where to go next
See feature store explained for how the inputs to these models stay consistent, LLMOps explained for the operational practices around LLM-specific serving, and SOAP vs REST for choosing the API style a serving endpoint exposes.