Someone asks why the assistant answered differently last Tuesday. Answering requires knowing which model version was serving, which prompt version it used, what effort and sampling settings applied, and whether an adapter was loaded.
In most deployments, at least one of those is unknown. The model was referenced by an alias that has since moved, the prompt was edited without a version, or the configuration lives in an environment variable nobody recorded.
What changed in 2026
- Model turnover made pinning essential. Frequent releases and deprecations meant unpinned references changed underneath running products.
- The composite unit got recognised. Teams stopped versioning models alone and started versioning the whole behaviour-determining bundle.
- Rollback speed became a requirement. As AI features became load-bearing, reverting in seconds rather than through a rebuild became necessary.
- Registries stayed optional for small deployments. The recognition that configuration in a repository covers most cases limited unnecessary tool adoption.
What actually determines behaviour
| Component |
Changes behaviour |
Commonly versioned |
| Model identifier and version |
Substantially |
Sometimes |
| System prompt |
Substantially |
Rarely |
| Tool definitions |
Substantially |
Almost never |
| Effort or reasoning settings |
Substantially |
Rarely |
| Sampling parameters |
Moderately |
Rarely |
| Adapter, if any |
Substantially |
Sometimes |
| Retrieval index version |
Substantially |
Rarely |
Look at the middle column against the right one. The things that determine what your product does are largely not tracked, which is why "why did it behave differently" is so often unanswerable.
Tool definitions deserve particular mention: they shape agent behaviour as much as the prompt does, and they are almost never treated as versioned artefacts — see tool schema design.
Never deploy against a moving alias
Provider aliases that point at "the latest version" are convenient and a genuine hazard. Your product's behaviour changes when the provider updates the alias, with no deploy on your side, no change in your repository, and no record.
Pin to a specific version. Upgrade deliberately, after running your evaluation against the new version — which is exactly what LLM regression testing is for.
The same applies to prompts. Production referencing "the current prompt" means an edit is an unreviewed deploy, per prompt registries.
Attach evaluation results to the version
A version with no scores is a version you cannot reason about. Attaching evaluation results at the point of creation turns "is the new one better" from an argument into a lookup, and it gives you a baseline to compare against when something regresses months later.
The results worth recording: your evaluation suite scores, cost per representative request, latency percentiles, and refusal rate. Those four cover the dimensions that actually differ between candidate configurations.
Rollback must be fast
The purpose of versioning is reverting. If reverting requires a code change, a build, and a deploy, it will take long enough that a bad configuration stays live through the incident.
Make the active version a runtime configuration value. Changing it should take effect within seconds without a rebuild. That converts a bad model or prompt from an outage into a brief blip.
The caveat is the one from prompt registries: runtime configuration makes changes easy, which means it needs a gate. Promotion to production should be a deliberate, recorded action rather than an edit.
Common mistakes
- Versioning the model but not the prompt. Incomplete; behaviour still unexplainable.
- Deploying against a moving alias. Silent behaviour changes.
- Rollback requiring a rebuild. Too slow to use during an incident.
- No evaluation results attached. Versions cannot be compared.
- Untracked tool definitions. A major behaviour determinant, invisible.
- Adopting a heavyweight registry for one model. Configuration in the repo is enough.
- No record of which version served a given request. Makes incident analysis guesswork.
FAQ
Do I need a dedicated registry tool?
Not for a small deployment — a versioned configuration file in your repository covers model, prompt, and settings adequately. Tools earn their place with many models, many prompts, non-engineer editing, or experimentation.
Should the retrieval index be versioned too?
Ideally yes. A changed index changes outputs as surely as a changed prompt, and re-indexing is a behaviour change that usually goes unrecorded — see embedding model migration.
How do I record which version served a request?
Emit it in your traces alongside the request. That single field makes incident analysis tractable, and it costs nothing — see LLM observability.
How long should I keep old versions?
Long enough to roll back to and to compare against — a few generations is usually sufficient. Keep the evaluation results indefinitely; they are small.
Where to go next
For the prompt half of the versioned unit, read prompt registries. For gating upgrades, LLM regression testing, and for comparing candidates before promotion, A/B testing LLM models.