Sample ratio mismatch: the check to run before reading any result

Experimentation · 6 min read

What a sample ratio mismatch is, how to test for it with a chi-squared check, and the usual root causes interviewers expect you to name.

What it is

A sample ratio mismatch (SRM) is when the observed split between control and treatment differs from the planned split by more than chance allows. A 50/50 test that lands 50.8/49.2 on a million users is almost certainly broken.

Treat SRM like a fever: it is a symptom. Do not read the metric results until you know the cause.

How to check

Run a chi-squared goodness-of-fit test of the observed counts against the expected ratio. Many teams use a strict threshold such as p < 0.001 because they check every test.

Common causes

Assignment bugs, redirects or slower page loads in one arm that drop users before logging, bot filtering applied unevenly, and triggering conditions that differ between arms.

Common mistakes

Reporting the lift anyway and adding a footnote.
Assuming a small percentage gap is fine on a large sample.
Fixing the split by reweighting instead of finding the cause.