A/B test sample size, power and the peeking problem

The four inputs

Sample size depends on the baseline rate, the minimum detectable effect (MDE), the significance level (alpha, often 5%) and power (often 80%). Smaller effects need far more users: halving the MDE roughly quadruples the sample.

In an interview, state the inputs, give a rough number, then turn it into days of traffic. That last step is what makes it a business answer.

Why peeking breaks the math

A p-value assumes the sample size was fixed in advance. If you check daily and stop the first time it crosses 0.05, the real false positive rate is much higher than 5%.

Fixes: commit to a duration up front, or use a method designed for continuous monitoring such as sequential testing. Name one and why.

Run at least full weeks

Weekday and weekend users behave differently. Running whole weeks avoids a day-of-week mix shift being read as a treatment effect.

Common mistakes

• Quoting a sample size without stating the MDE it assumes.
• Stopping the test as soon as it looks significant.
• Mixing up relative and absolute lift when setting MDE.

Further reading

• Evan Miller: How Not To Run an A/B Test
• Evan Miller: Sample Size Calculator