A/B Test Significance Calculator
Two-proportion z-test with p-value and 95 % verdict.
What it does
A/B tests compare two rates, but raw gaps mislead at small samples. A 15 % vs 12 % split means little on 200 visitors and a lot on 20,000. The z test turns visitors and conversions into one p-value.
The tool above takes both arms, pools the rate, and scores the gap with the two-proportion z and a two-sided normal p. It prints each rate, the p to four decimals, and a 95 % verdict. Counts stay whole; math handles the rest.
Penn State methods courses and the NIST statistics handbook teach this pooled form for comparing proportions, with the normal approximation once counts grow past single digits. The 0.05 line is convention, not physics, which is why the page shows p itself alongside the word.
Use it to stop arguments, not to bless peeking. Fix traffic splits and runtime first, read once, then confirm winners on fresh weeks before rebuilding the funnel around them.
How to use
- Enter visitors and conversions for A.
- Enter visitors and conversions for B.
- Read both rates and the two-sided p-value.
- Read the 95 % verdict.
- Check the uplift direction in the note.
- Fix sample size before the next run.
Formula
- nA, nB
- Visitors per arm, whole counts.
- cA, cB
- Conversions per arm, whole counts within visitors.
- p
- Pooled conversions over pooled visitors.
Rates are conversions over visitors per arm. The pooled rate assumes one shared truth under the null; the normal approximation needs decent counts per arm (rule of thumb: at least 10 conversions each). Verdict uses 0.05 two-sided.
Worked example
Example: 12 % vs 15 %
Landing page test: A 120/1000, B 150/1000.
- A: 120 conversions from 1000 visitors = 12.00 %.
- B: 150 from 1000 = 15.00 %.
- Pooled rate: 270 / 2000 = 13.5 %.
- z: 0.03 / 0.0153 = 1.96.
- p = 0.0496, under 0.05, so Significant at 95 %.
Reference table
| p | Read | Action |
|---|---|---|
| Under 0.01 | Strong signal | Ship and monitor |
| 0.01 to 0.05 | Signal | Ship, then confirm |
| 0.05 to 0.20 | Weak | Extend or rethink |
| Over 0.20 | Noise | Kill or redesign |
Limits
- Approximation needs counts; tiny arms mislead.
- Assumes random split and stable traffic.
- One metric at a time; comics of metrics inflate wins.
- Verdict is convention at 0.05, not proof.
Common mistakes
- Peeking daily and stopping on green.
- Testing five headlines and reporting one.
- Changing traffic mid-test.
- Counting bot or duplicate events.
- Reading rate gaps without visitors.
Try it yourself
Quick checks. Answers sit tucked under each question.
A 500/5000 vs B 560/5000?
Rates are 10.00 % and 11.20 %. p = 0.0513, so Not significant at 95 %.
How much traffic spots a 2 point lift near 10 %?
Roughly 3,800 per arm for 80 % power: 7.85 x 0.1956 / 0.0004 = 3,838. Small gaps are expensive; test bigger swings first.
Questions
The test is significant. Do we ship it?
Meaningful chance, not proof. At p 0.049 the winner could still be noise about 5 times in 100. Ship it, then watch the metric hold on fresh traffic.
One-sided or two-sided p?
Two-sided by default here: it flags change either way. One-sided suits rare directional bets but reads lower p values, so agree the side before starting.
Can small uplifts ever win?
Yes. Tiny edges need big samples: a 12.0 to 12.5 % move needs tens of thousands per arm. The tool shows p drifting down as traffic grows at fixed rates.
What bar should a store use?
Usually 95 % (p under 0.05). Money pages sometimes demand 99 %. Looser lines ship faster but cry wolf more; pick one line and hold it.
May I peek before it finishes?
Peeking inflates false wins fast. Fix the sample or the runtime up front, ignore the mid-flight p, and read once. Sequential methods exist for frequent checkers.
What breaks test math most?
Flicker (novelty), device mix shifts, weekday swings, and broken tracking. Segment by device and week, confirm the event fires, then trust the pooled math.