An A/B test sounds simple: show two versions, see which one wins. In practice, most A/B tests that teams run never reach a trustworthy answer — the sample is too small, the test gets cut short the moment it looks good, or the result gets read as a verdict on a hunch that was already decided before the test began. Running a test that actually tells you something takes a bit more discipline than clicking start on an experiment. Here is the practical version, without the statistics lecture.
What changed in 2026
- Sequential and Bayesian testing tools went mainstream. Most major platforms now offer always-valid inference, so you can check a test in progress without automatically inflating your false-positive rate the way classic fixed-horizon tests do — but you still need a stopping rule you set in advance.
- AI copilots draft hypotheses and variant copy faster, which is genuinely useful, but they cannot tell you whether your traffic supports the test you are about to run. That is still a math problem, not a prompting problem.
- Privacy changes squeezed sample sizes. Cookie deprecation and stricter tracking consent mean fewer teams have the clean, complete data they used to, so power calculations matter more than they did a few years ago, not less.
- More teams pair tests with qualitative signals — session recordings, support tickets, and user interviews — to sanity-check a surprising result before acting on it.
How to run an A/B test step by step
- Write the hypothesis first. If X happens, then Y will follow, because Z. If you cannot fill in the blanks, you are not ready to test.
- Pick one metric that decides the outcome. Track others for context, but decide in advance which number wins or loses the test.
- Calculate the sample size before launch, using your baseline conversion rate and the smallest lift worth detecting. This tells you how long the test needs to run.
- Change one variable at a time, unless you are deliberately running a multivariate test with the sample size to support it.
- Let it run the full planned duration, across full weekly cycles, so day-of-week effects do not skew the read.
- Analyze once, at the pre-agreed endpoint, and check for validity issues, such as sample ratio mismatch or novelty effects, before you trust the number.
Why most A/B tests fail before they start
The failure usually happens on day one, not in the analysis. Someone launches a test on a page that gets a few hundred visits a week, expecting a readable result in five days. The math does not support it — detecting a realistic 5 to 10 percent lift on low traffic can take weeks or months. Rather than run underpowered tests, teams are often better off testing bigger swings on smaller traffic, or picking a higher-traffic page to validate the methodology and applying the resulting discipline more broadly.
A/B testing methods compared
| Approach |
How it works |
Best for |
| Fixed-horizon A/B |
Set sample size and duration in advance, analyze once at the end |
Clean, classic tests with predictable traffic |
| Sequential / always-valid |
Check results continuously with a statistically valid stopping rule |
Teams that want to stop early on a clear win or loss |
| Multi-armed bandit |
Traffic shifts toward the winning variant automatically as data comes in |
Optimizing revenue during the test itself, not just after |
| Multivariate |
Tests combinations of several changes at once |
High-traffic pages where you need to test interactions between elements |
From result to business case
A statistically significant lift is a fact, not a decision. Before you propose the change more broadly, translate the number into what it is worth: revenue impact, implementation cost, and risk if the result does not hold up outside the test window. That translation is exactly what a business case is for, and it is worth building the habit of writing one even for changes that feel obviously good — the discipline catches results that look great in isolation but are not worth the engineering cost.
FAQ
How long should an A/B test run?
Long enough to hit your pre-calculated sample size, and at least one full week, ideally two to four, to average out day-of-week effects. Do not stop early just because the result looks good on day three.
What if the results are not statistically significant?
Treat it as inconclusive, not as a loss. Either the effect is smaller than you could detect, or there truly is no effect. Extending the test or trying a bigger change are both reasonable next steps.
Can I test more than one thing at once?
Yes, with a multivariate test or a bandit approach, but both need considerably more traffic than a single A/B comparison to produce a readable result. If in doubt, test one variable at a time.
Do I need a statistician to run a basic A/B test?
No — free sample size calculators and most testing platforms handle the math. You do need to resist the temptation to stop the test the moment it looks good, which is the single most common way non-statisticians break their own results.
Where to go next