At two in the morning during an incident, someone increases an instance size in the console and service is restored. Nobody updates the infrastructure code, because the incident is over and everyone goes back to sleep. Three weeks later a routine deploy reverts the change and the incident happens again.
That is drift, and the failure is not the console change — it was the right call at the time. The failure is that the loop never closed.
What changed in 2026
- Scheduled drift detection became standard. Running a comparison between code and reality on a schedule, rather than only at apply time, moved into normal pipeline practice.
- Break-glass workflows matured. Documented emergency access paths with automatic ticket creation reduced undocumented console changes.
- Detection granularity improved. Better filtering of expected differences reduced the noise that caused teams to ignore drift reports.
- Auto-remediation stayed controversial. Automatically reverting drift remained inadvisable for the same reason it always was.
Where drift comes from
| Source |
Typical case |
| Emergency console changes |
Incident mitigation, never reconciled |
| Autoscaling and managed adjustments |
Expected; should be excluded from detection |
| Provider-side defaults changing |
Cloud provider alters a default value |
| Manual experimentation |
Someone testing something in a shared environment |
| Another tool managing the same resource |
Two systems both think they own it |
| Partial apply failures |
Half a change applied, state and reality diverged |
The second row matters for signal quality. Autoscaling changes instance counts by design, and a detection system reporting that as drift trains people to ignore the report. Excluding fields that legitimately change outside the code is what keeps the signal meaningful.
The two-tools case is worth naming specifically, because it produces oscillation — each tool reverting the other's changes on every run — and it is frequently invisible until someone notices resources flapping.
Detecting and deciding
Run detection on a schedule, ideally daily, and alert on unexpected differences. The point is to find drift when someone can think about it calmly, not when they are trying to deploy something urgent.
When drift is found, the decision is not automatic. There are three reasonable outcomes.
Absorb it. The change was correct and should be permanent. Update the code to match reality, so the next apply preserves it. This is the right answer for most emergency changes, because they were made to fix a real problem.
Revert it. The change was accidental or inappropriate. Apply the code and restore the intended state.
Investigate. Something changed that nobody expected, which may indicate a compromised credential or another tool with unexpected access. This is rare and important.
Automatic reversion is tempting and dangerous. A system that silently reverts drift will, at some point, revert an incident mitigation and cause an outage. Detection should notify; a human should decide.
Reducing the cause
Detection manages drift; reducing console access prevents it. Read-only console access as the default, with a documented break-glass path for emergencies, removes most casual drift.
Make the break-glass path fast and logged. If emergency access is slow or requires waking someone, people will find another way around it. If it is a self-service elevation that creates a ticket automatically, the loop closes on its own — the ticket is the reminder to reconcile.
Make reconciliation part of incident follow-up. An incident whose mitigation was a console change has an action item to update the code, tracked like any other, per incident postmortem guide.
The time-bound elevation pattern in least privilege IAM is the mechanism that makes all of this workable.
Common mistakes
- Detecting only at apply time. Discovered at the worst moment.
- Automatic revert. Will eventually undo an incident mitigation.
- Not excluding expected changes. Noise trains people to ignore the report.
- No break-glass path. People find undocumented ways around the restriction.
- Standing console write access. The root cause of most drift.
- No reconciliation step after incidents. Where drift becomes permanent.
FAQ
How often should detection run?
Daily is a reasonable default. More frequently for high-change environments; the value is catching drift before it interacts with a deploy.
What about resources managed outside the code deliberately?
Exclude them explicitly rather than tolerating them as noise. Undeclared exclusions are indistinguishable from unnoticed drift.
Does drift mean someone did something wrong?
Usually it means someone fixed something under pressure. The process failure is the missing reconciliation, not the change.
Can I prevent drift entirely?
Not while humans have emergency access, which they need. The goal is fast detection and reliable reconciliation.
Where to go next
For state handling, read Terraform state management. For access control, least privilege IAM, and for closing the loop after incidents, incident postmortem guide.