Chaos engineering is the practice of deliberately injecting failure into a system — killing a server, adding network latency, exhausting memory — to verify that your assumptions about its resilience actually hold, instead of finding out during a real incident. It works because most outages are not caused by novel failures; they are caused by known failure modes that nobody tested combined with a system that behaved differently under real conditions than engineers assumed. Getting started does not require a Netflix-scale chaos platform: it requires a hypothesis, a small blast radius, and the discipline to run experiments before a failure finds you first.
How it works
Every chaos experiment follows the same shape. First, define a steady-state metric — something measurable that represents normal, healthy behavior, like request success rate or p99 latency. Second, form a hypothesis: "if the payment service's database connection drops, checkout latency will increase but stay under 2 seconds because the circuit breaker will fail over to cached pricing." Third, inject the failure in a controlled way with a defined blast radius — the smallest scope that could disprove the hypothesis. Fourth, observe whether steady state held. If it did, your resilience assumption is verified. If it did not, you found a real gap before a customer did.
Getting started, step by step
- Pick one dependency and one failure mode. Do not start with "test everything." Start with the dependency your team already suspects is fragile — a database, a third-party API, a specific service.
- Write the hypothesis down before running anything. "The system will continue serving reads from cache if the primary database becomes unreachable for 30 seconds" is testable. "See what happens" is not.
- Run it in staging first, with the smallest blast radius that can test the hypothesis. One pod, one instance, one percent of traffic — not the whole fleet.
- Automate the experiment and the abort condition. Every chaos tool needs a kill switch that reverts the injected failure immediately if steady-state metrics degrade past a safe threshold.
- Graduate to production only after staging passes repeatedly. Production chaos testing, often run as scheduled GameDays with the on-call team watching, is where most of the value is — staging rarely has production's real traffic patterns and data volume.
- Fix what breaks, then re-run the same experiment. A chaos practice that finds a gap and never re-tests it after the fix is not actually confirming the fix worked.
Chaos engineering tools compared
| Tool |
Approach |
Best for |
| Chaos Monkey (Netflix) |
Randomly terminates instances in production |
Simple instance-failure testing, the original tool that started the practice |
| Gremlin |
Managed SaaS with a large library of attack types and safety controls |
Teams wanting guardrails and a UI without building tooling in-house |
| Litmus Chaos |
Kubernetes-native, CNCF project, defined via CRDs |
Teams already running Kubernetes who want experiments as code |
| Chaos Mesh |
Kubernetes-native, similar scope to Litmus |
Alternative Kubernetes-native option with strong network fault injection |
| AWS Fault Injection Simulator |
Managed AWS service targeting EC2, ECS, EKS, RDS |
Teams fully on AWS wanting native integration with existing infrastructure |
Common mistakes
- Running the first experiment directly in production. Validate the hypothesis and the abort mechanism in staging first — the first time you use a new chaos tool is not the time to also be testing it against live customer traffic.
- No abort condition. Every experiment needs an automatic circuit breaker of its own: a defined threshold where the injected failure gets reverted immediately, not a manual process someone has to remember.
- Ignoring blast radius. Testing "what happens if the entire database cluster goes down" as your first experiment risks a real outage to answer a question a single-node failure could have answered more safely.
- Treating it as a one-time event. A single GameDay proves the system worked once, under one set of conditions. Chaos engineering earns its value as a recurring practice, especially after every major architecture change.
FAQ
Is chaos engineering only for large companies?
No. The scale changes, but the discipline — hypothesis, small blast radius, measured outcome — applies just as well to a five-service startup as it does to a company running thousands of services.
Do I need Kubernetes to do chaos engineering?
No, though tools like Litmus and Chaos Mesh assume it. Simpler experiments — killing a process, adding latency with tc on Linux, disabling a dependency's DNS entry — work on any infrastructure.
How is chaos engineering different from load testing?
Load testing verifies the system handles expected traffic volume. Chaos engineering verifies the system handles unexpected failure conditions. They test different assumptions and both belong in a mature testing practice.
What is a GameDay?
A scheduled, planned chaos experiment run with the responsible team present and watching, often simulating a specific incident scenario end to end, including whether alerts fire and runbooks work — not just whether the system technically survives.
Where to go next
Chaos experiments are how you validate that mechanisms like the circuit breaker pattern actually behave the way you designed them under real failure. The metric you protect during every experiment should tie back to your error budget, and if you are deciding who owns this practice organizationally, SRE vs DevOps explains where chaos engineering typically sits.