LLMOps is the set of practices for operating applications built on large language models in production — and it diverges from classic MLOps at almost every step after "call a model and get a prediction." A trained classifier has a fixed set of weights and a well-defined accuracy metric; an LLM application has a prompt that changes independently of any weights, output that is open-ended text rather than a single label, and a cost and latency profile driven by token counts rather than fixed inference time. LLMOps exists because none of the classic MLOps tooling — model registries, drift dashboards, accuracy-based CI gates — maps cleanly onto any of that.
The core idea
Classic MLOps assumes the artifact you version and monitor is the model itself: its weights, its training data, its accuracy against a labeled test set. LLMOps shifts that unit in two ways. First, for API-based or hosted LLMs, you rarely retrain anything — the artifact you actually change and need to version is the prompt (and, increasingly, the RAG pipeline feeding it context). Second, "correct" is no longer a single label to check against; it is a quality judgment about open-ended text, which needs an evaluation approach built for that, not a confusion matrix.
# A prompt registry entry, versioned like code
PROMPT_V3 = {
"version": "3",
"template": "Summarize the following ticket in 2 sentences...",
"model": "claude-sonnet-4-5",
"eval_score": 0.91, # against the golden dataset, not production traffic
}
That single change — versioning a prompt string and its associated eval score instead of a model artifact — is why LLMOps needed its own tooling rather than inheriting classic MLOps tools unchanged.
What LLMOps actually covers
Prompt versioning and registries. Every prompt change tracked like a code change, tied to the eval score it produced, so a regression can be traced to the exact prompt (or model, or retrieval config) that caused it, and rolled back deliberately rather than by guesswork.
Evaluation pipelines. A golden dataset of representative inputs, scored automatically — often by another LLM acting as a judge against a rubric, sometimes by simpler heuristics (does the output contain required fields, pass a regex, stay under a length limit) — run on every prompt or model change before it ships.
Cost and latency monitoring. Token spend per request, time-to-first-token, and total generation time become first-class production metrics, because unlike a classic model's near-constant inference cost, LLM cost and latency both scale with output length and can vary sharply between providers and models.
RAG-specific monitoring. For retrieval-augmented pipelines, the model can be working perfectly and still produce a bad answer because retrieval returned the wrong chunks. Monitoring retrieval quality (are the returned chunks relevant?) and groundedness (does the output actually match the retrieved context, or did the model hallucinate past it?) is a layer classic model monitoring has no equivalent for.
Guardrails and safety checks. Input and output filtering for prompt injection, unsafe content, and policy violations, running as a layer around the model call rather than something the model itself is trusted to enforce alone.
LLMOps vs classic MLOps
| Dimension |
Classic MLOps |
LLMOps |
| Primary artifact versioned |
Model weights |
Prompts (and RAG retrieval config) |
| Quality signal |
Accuracy, precision/recall against labels |
LLM-as-judge or rubric scoring against a golden set |
| Drift concern |
Statistical data/concept drift |
Prompt regressions, retrieval quality decay |
| Cost driver |
Roughly fixed per-inference compute |
Token count, varies per request and provider |
| Latency concern |
Fixed inference time |
Time-to-first-token, streaming, generation length |
| Safety layer |
Rarely a distinct concern |
Guardrails against injection and unsafe output |
The two disciplines share plenty — both need CI/CD, versioning discipline, and production monitoring — which is why most teams run them as adjacent, overlapping practices rather than entirely separate functions.
Common mistakes
- Reusing classic drift dashboards unchanged. A statistical drift metric tuned for numeric features says nothing about whether a prompt started producing worse summaries after a model provider's silent update.
- Evaluating only on vibes, not a golden dataset. Without a fixed, versioned set of test inputs and a repeatable scoring method, you cannot tell whether a prompt change actually helped or just felt better on the few examples someone tried.
- Ignoring cost until the bill arrives. Token spend can scale unexpectedly with longer conversations, larger retrieved contexts, or a provider price change; monitor cost per request in the same dashboard as quality, not as a separate finance concern.
- Treating the model as the only thing that can fail in a RAG pipeline. A retrieval step returning stale or irrelevant chunks produces a bad answer that no amount of prompt tuning on the generation side will fix.
FAQ
Is LLMOps just MLOps with a new name?
No. They overlap on CI/CD and monitoring discipline, but LLMOps centers on prompt versioning, open-ended output evaluation, and token cost — none of which classic MLOps tooling was built to handle.
Do I need LLMOps tooling if I fine-tune my own model instead of calling an API?
Partially — fine-tuning brings back some classic MLOps concerns (training data versioning, weight artifacts), but you still need prompt and evaluation tooling for how the fine-tuned model is actually invoked.
What is LLM-as-judge evaluation?
Using a separate LLM call, given a rubric, to score the quality of another LLM's output against a golden dataset — a practical substitute for human review at the volume automated evaluation pipelines need.
How is RAG monitoring different from model monitoring?
Model monitoring watches the model's output quality. RAG monitoring separately checks whether the retrieval step is returning relevant, current context, since a good model with bad retrieved context still produces a bad or hallucinated answer.
Where to go next
See model serving infrastructure for how these models actually get deployed, feature store explained for the equivalent consistency problem on the classic ML side, and WebSockets vs server-sent events for streaming LLM output to a client.