Statistical Significance

Understanding Statistical Significance

In a two-variant test, a p-value of 0.05 means there is a 5% chance the observed lift happened by luck if the variants were actually identical. Below that threshold, the result is conventionally called "significant" — though the threshold is a convention, not a physical law, and the cost of false positives in commerce contexts should determine where you set it.

The most common testing mistakes are peeking and early stopping. A/B tests do not reach a stable lift on day one; the numbers wobble heavily in the first week while sample sizes build. Declaring a winner as soon as the dashboard flashes "significant" inflates the false-positive rate dramatically — a test peeked at every day will cross 95% significance by chance roughly 25% of the time even when there is no real effect.

Powering a test properly means calculating, before launch, the sample size needed to detect a minimum meaningful effect at the desired confidence level. A store with 1,000 orders per week, a 3% baseline conversion rate, and a desire to detect a 10% relative lift needs somewhere on the order of 50,000+ sessions per variant for a well-powered test. If the store cannot produce that volume in a reasonable window, it is more honest to test only larger hypothesized effects or use Bayesian methods that don't require a pre-set stopping rule.

Statistical significance is necessary but not sufficient. A significant result with a practically trivial effect size is noise dressed up in math. Reporting tests with effect size, confidence interval, and significance together gives decision-makers the full picture instead of a binary yes/no.