Branch
Experimentation & A/B Testing interview questions
Design, power, guardrails, peeking and readouts for online experiments.
18 practice questions · 6 subtopics · hints and unlimited attempts
What this branch covers
Design & Power
Randomization units, MDE and sample size.
Design & Power questionsReadouts & Pitfalls
Peeking, multiple testing, novelty effects.
Readouts & Pitfalls questionsGuardrails & Metrics
Choosing primary, secondary and guardrail metrics.
Guardrails & Metrics questionsInterference & Marketplaces
Network effects, spillover and two-sided marketplace tests.
Interference & Marketplaces questionsMultiple Testing
Familywise error, corrections and disciplined metric selection.
Multiple Testing questionsData Quality
Sample-ratio mismatch, logging bugs and trustworthy experiment data.
Data Quality questions
All Experimentation & A/B Testing questions
Sample size for a 2% lift
Compute the per-arm sample size for a conversion test.
Design & Power · numeric · medium · 8 min
Why peeking inflates false positives
Explain the statistical cost of checking results daily.
Readouts & Pitfalls · multiple choice · easy · 5 min
Design metrics for a checkout redesign
ProChoose primary, secondary and guardrail metrics and defend them.
Guardrails & Metrics · scenario · medium · 20 min
Why user-level randomization breaks in a marketplace
ProYou want to test a new driver incentive at Uber.
Interference & Marketplaces · scenario · hard · 20 min
Big week-1 lift that fades to zero
A redesigned button shows +8% engagement in week 1, but by week 4 treatment and control are indistinguishable.
Readouts & Pitfalls · multiple choice · easy · 5 min
Sample size for a 1-point conversion lift
Baseline conversion is 10%.
Design & Power · numeric · medium · 8 min
False-positive risk with four variants
ProYou run one control against four independent treatment variants, testing each at α = 0.05.
Multiple Testing · numeric · medium · 6 min
Guardrails for a checkout change
ProYou're testing a one-click checkout that you expect to raise conversion.
Guardrails & Metrics · scenario · medium · 15 min
Wins in every segment, loses overall
ProYour treatment beats control in BOTH the mobile segment and the desktop segment, yet loses when you pool all users together.
Readouts & Pitfalls · multiple choice · medium · 6 min
A 50/50 split that came out 52/48
You designed a 50/50 experiment.
Data Quality · multiple choice · medium · 6 min
Choose the OEC for a homepage change
ProYou're testing a new homepage row on Netflix.
Guardrails & Metrics · scenario · medium · 15 min
What CUPED buys you
ProYour experiments are underpowered and you can't get more users.
Design & Power · multiple choice · hard · 7 min
A PM wants to ship the moment p < 0.05
A PM watches the experiment dashboard daily and wants to ship as soon as it crosses p < 0.05.
Readouts & Pitfalls · scenario · medium · 12 min
Measure surge pricing's effect on driver retention
ProUber is considering a surge-pricing policy that changes marketplace conditions for every driver and rider in a zone.
Design & Power · scenario · hard · 25 min
Triage a sample-ratio mismatch
ProA nominal 50/50 Meta messaging experiment reports 520,000 control users and 480,000 treatment users.
Readouts & Pitfalls · scenario · hard · 20 min
Test a new shoppable-ad ranking model
Meta has a new ranking model for shoppable-feed ads.
Design & Power · scenario · medium · 20 min
Spot a novelty effect in a VR interface test
A new VR interface lifts daily use by 18% in days 1–3, 7% in days 4–7, and 0% in days 8–21.
Readouts & Pitfalls · multiple choice · easy · 7 min
False-positive risk across four variants
You compare four independent treatment variants with one control.
Design & Power · numeric · medium · 10 min
Other branches
Start with Experimentation & A/B Testing
A free account keeps your attempt history and progress. Premium sets are included in Pro and Founder plans.