An A/B test is the only tool that gives a causal answer from observational-looking data, and it is easy to run in a way that produces a confident wrong one. This lesson is the arithmetic and the five failures that matter.
Why a difference is not a result
import numpy as np
rng = np.random.default_rng(5)
# Both variants are IDENTICAL. Same page, same rate, no difference.
TRUE_RATE = 0.10
N = 1000
differences = []
for _ in range(1000):
a = rng.binomial(N, TRUE_RATE) / N
b = rng.binomial(N, TRUE_RATE) / N
differences.append(b - a)
differences = np.array(differences)
print(f'two identical variants, n={N} each, 1000 repeats')
print(f'mean difference {differences.mean():+.4f}')
print(f'largest "win" {differences.max():+.2%}')
print(f'largest "loss" {differences.min():+.2%}')
print(f'apparent lift > 2 points: {(np.abs(differences) > 0.02).mean():.1%} of runs')
Open this example in the editor and change it. Nothing you type is saved.
Two identical pages, and in a meaningful fraction of runs one "beat" the other by more than two percentage points. Stop a test early, at a moment that looks good, and you can manufacture any result you like.
Decide the sample size first
The sample size you need depends on three things you must choose before starting: the baseline rate, the smallest lift worth detecting, and how often you are willing to be wrong in each direction.
import numpy as np
def sample_size(baseline: float, lift: float, power: float = 0.80,
alpha: float = 0.05) -> int:
"""Per-group sample size for a two-sided test of two proportions.
z values: 1.96 for alpha=0.05, 0.84 for power=0.80."""
z_alpha = 1.959964
z_beta = 0.841621 if power == 0.80 else 1.281552 # 0.80 or 0.90
p1 = baseline
p2 = baseline * (1 + lift)
p_bar = (p1 + p2) / 2
numerator = (z_alpha * np.sqrt(2 * p_bar * (1 - p_bar))
+ z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
return int(np.ceil(numerator / (p2 - p1) ** 2))
print(f'{"baseline":>9} {"lift":>6} {"per group":>12} {"total":>12}')
for baseline in [0.02, 0.10]:
for lift in [0.02, 0.05, 0.10, 0.20]:
n = sample_size(baseline, lift)
print(f'{baseline:>8.0%} {lift:>6.0%} {n:>12,} {2 * n:>12,}')
Open this example in the editor and change it. Nothing you type is saved.
Two facts fall out of that table and both change how tests get planned.
A smaller lift costs quadratically more. Detecting a 5% relative lift costs about four times as much traffic as a 10% one, and a 2% lift costs twenty-five times as much.
A lower baseline costs more. At a 2% conversion rate you need five times the traffic you would need at 10% for the same relative lift.
Which means: compute this before the test, and if the traffic is not available, do not run it. A test that cannot reach its sample size will end in an argument about an inconclusive result, having cost weeks.
import numpy as np
def sample_size(baseline, lift, power=0.80):
z_alpha, z_beta = 1.959964, 0.841621
p1, p2 = baseline, baseline * (1 + lift)
p_bar = (p1 + p2) / 2
return int(np.ceil(((z_alpha * np.sqrt(2 * p_bar * (1 - p_bar))
+ z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2)
/ (p2 - p1) ** 2))
WEEKLY_TRAFFIC = 24_000
for lift in [0.05, 0.10, 0.20]:
n = sample_size(0.08, lift)
weeks = 2 * n / WEEKLY_TRAFFIC
verdict = 'feasible' if weeks <= 4 else 'too slow - do not start'
print(f'{lift:>4.0%} lift: {2 * n:>9,} visitors, {weeks:>5.1f} weeks {verdict}')
Open this example in the editor and change it. Nothing you type is saved.
Confidence intervals say more than p-values
import numpy as np
def compare(successes_a, trials_a, successes_b, trials_b):
a, b = successes_a / trials_a, successes_b / trials_b
se = np.sqrt(a * (1 - a) / trials_a + b * (1 - b) / trials_b)
difference = b - a
low, high = difference - 1.96 * se, difference + 1.96 * se
# A z-test on the pooled proportion, for the p-value.
pooled = (successes_a + successes_b) / (trials_a + trials_b)
se_pooled = np.sqrt(pooled * (1 - pooled) * (1 / trials_a + 1 / trials_b))
z = difference / se_pooled if se_pooled else 0.0
return {'a': a, 'b': b, 'difference': difference,
'low': low, 'high': high, 'z': z}
for label, args in [
('clearly better', (800, 10_000, 920, 10_000)),
('probably better', (800, 10_000, 860, 10_000)),
('no idea', (80, 1_000, 92, 1_000)),
]:
r = compare(*args)
print(f'{label}')
print(f' A {r["a"]:.2%} B {r["b"]:.2%} difference {r["difference"]:+.2%}')
print(f' 95% CI [{r["low"]:+.2%}, {r["high"]:+.2%}] z = {r["z"]:+.2f}')
print(f' interval excludes zero: {not (r["low"] < 0 < r["high"])}')
print()
Open this example in the editor and change it. Nothing you type is saved.
The third case has the largest apparent lift and tells you the least: the interval runs from -1.5 points to +3.9 points, so the change could be a meaningful win or a meaningful loss.
Report the interval, not a verdict. "B is better" is a claim; "B is between 0.4 and 2.0 points better" is a result, and it lets the reader decide whether 0.4 points would be worth shipping.
The practical failures, in order of frequency
1. Peeking. Checking the result repeatedly and stopping when it looks good.
import numpy as np
rng = np.random.default_rng(23)
# Both variants identical. Peek every 500 visitors and stop at the
# first "significant" moment.
def run_with_peeking(checks, per_check=500, seed=0):
generator = np.random.default_rng(seed)
a_success = b_success = 0
for check in range(1, checks + 1):
a_success += generator.binomial(per_check, 0.10)
b_success += generator.binomial(per_check, 0.10)
n = check * per_check
pa, pb = a_success / n, b_success / n
pooled = (a_success + b_success) / (2 * n)
se = np.sqrt(pooled * (1 - pooled) * 2 / n)
if se and abs(pb - pa) / se > 1.96:
return True, check
return False, checks
false_positives = sum(run_with_peeking(20, seed=s)[0] for s in range(400))
print(f'20 peeks, 400 identical A/A tests')
print(f'"significant" at some point: {false_positives / 400:.1%}')
print('The advertised false-positive rate is 5%.')
Open this example in the editor and change it. Nothing you type is saved.
Peeking twenty times turns a 5% error rate into something several times larger. The fix is to fix the sample size in advance and look once, or to use a sequential method designed for continuous monitoring.
2. Unequal populations. The split must be random and must happen before any difference can occur.
import numpy as np
import pandas as pd
rng = np.random.default_rng(29)
n = 20_000
users = pd.DataFrame({
'user_id': np.arange(n),
'is_mobile': rng.random(n) < 0.6,
'is_returning': rng.random(n) < 0.35,
})
users['variant'] = rng.choice(['A', 'B'], n)
# The sanity check to run BEFORE looking at the outcome.
check = users.groupby('variant').agg(
users=('user_id', 'size'),
mobile=('is_mobile', 'mean'),
returning=('is_returning', 'mean'),
).round(4)
print(check.to_string())
print()
split = check['users'] / check['users'].sum()
print(f'split {split.iloc[0]:.1%} / {split.iloc[1]:.1%}')
print('If the covariates differ, the randomisation is broken and the')
print('result is uninterpretable - check this first, every time.')
Open this example in the editor and change it. Nothing you type is saved.
3. Stopping when the result arrives rather than when the sample does. Same failure as peeking, wearing a different hat.
4. Measuring the wrong thing. A button that gets more clicks and fewer purchases is a loss. Always have one primary metric, declared in advance, and a guardrail that would catch the obvious harm.
5. Testing many variants and reporting the winner. Twenty variants against one control, at a 5% error rate, gives you roughly one false winner by construction.
import numpy as np
rng = np.random.default_rng(37)
# Twenty variants, ALL identical to the control.
n = 5_000
control = rng.binomial(n, 0.10) / n
winners = 0
for _ in range(20):
variant = rng.binomial(n, 0.10) / n
se = np.sqrt(2 * 0.10 * 0.90 / n)
if (variant - control) / se > 1.96:
winners += 1
print(f'20 identical variants, {winners} "beat" the control')
print('With a 5% threshold, one false winner per twenty is the')
print('expected outcome, not bad luck.')
Open this example in the editor and change it. Nothing you type is saved.
The fix is a correction - divide your threshold by the number of comparisons - or, better, to test fewer things and have a reason for each.
A worked example
import numpy as np
import pandas as pd
rng = np.random.default_rng(seed=149)
# ---- 1. Plan, before any data exists. -------------------------------
BASELINE = 0.082
MIN_LIFT = 0.08 # 8% relative - the smallest worth shipping
WEEKLY_TRAFFIC = 26_000
def sample_size(baseline, lift, power=0.80):
z_alpha, z_beta = 1.959964, 0.841621
p1, p2 = baseline, baseline * (1 + lift)
p_bar = (p1 + p2) / 2
return int(np.ceil(((z_alpha * np.sqrt(2 * p_bar * (1 - p_bar))
+ z_beta * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2)
/ (p2 - p1) ** 2))
per_group = sample_size(BASELINE, MIN_LIFT)
weeks = 2 * per_group / WEEKLY_TRAFFIC
PLAN = {
'primary metric': 'checkout completion rate',
'guardrail': 'refund rate within 30 days must not rise',
'baseline': f'{BASELINE:.1%}',
'minimum lift worth shipping': f'{MIN_LIFT:.0%} relative',
'sample size per group': f'{per_group:,}',
'planned duration': f'{weeks:.1f} weeks',
'analysis': 'once, at the planned sample size - no peeking',
}
for key, value in PLAN.items():
print(f'{key:<28} {value}')
print()
# ---- 2. Run it. The TRUE lift here is 6%, below the threshold. ------
TRUE_A, TRUE_B = BASELINE, BASELINE * 1.06
n = per_group
assignments = pd.DataFrame({
'user_id': np.arange(2 * n),
'variant': np.repeat(['A', 'B'], n),
'is_mobile': rng.random(2 * n) < 0.61,
'is_returning': rng.random(2 * n) < 0.34,
})
rates = np.where(assignments['variant'] == 'A', TRUE_A, TRUE_B)
assignments['converted'] = rng.random(2 * n) < rates
assignments['refunded'] = assignments['converted'] & (rng.random(2 * n) < 0.031)
# ---- 3. Sanity checks, BEFORE the outcome. --------------------------
balance = assignments.groupby('variant').agg(
users=('user_id', 'size'),
mobile=('is_mobile', 'mean'),
returning=('is_returning', 'mean'),
).round(4)
print('randomisation check:')
print(balance.to_string())
split = balance['users'] / balance['users'].sum()
print(f'split {split.iloc[0]:.2%} / {split.iloc[1]:.2%} '
f'(expect 50/50 within a fraction of a point)')
print()
# ---- 4. The result, as an interval. ---------------------------------
result = assignments.groupby('variant').agg(
users=('user_id', 'size'),
conversions=('converted', 'sum'),
refunds=('refunded', 'sum'),
)
result['rate'] = result['conversions'] / result['users']
result['refund_rate'] = result['refunds'] / result['conversions']
a, b = result.loc['A'], result.loc['B']
difference = b['rate'] - a['rate']
se = np.sqrt(a['rate'] * (1 - a['rate']) / a['users']
+ b['rate'] * (1 - b['rate']) / b['users'])
low, high = difference - 1.96 * se, difference + 1.96 * se
relative = difference / a['rate']
print('primary metric: checkout completion')
print(f' A {a["rate"]:.3%} ({int(a["conversions"]):,} of {int(a["users"]):,})')
print(f' B {b["rate"]:.3%} ({int(b["conversions"]):,} of {int(b["users"]):,})')
print(f' difference {difference:+.3%} ({relative:+.1%} relative)')
print(f' 95% CI [{low:+.3%}, {high:+.3%}]')
print(f' excludes zero: {not (low < 0 < high)}')
print()
# ---- 5. The guardrail. ----------------------------------------------
gd = b['refund_rate'] - a['refund_rate']
gse = np.sqrt(a['refund_rate'] * (1 - a['refund_rate']) / max(a['conversions'], 1)
+ b['refund_rate'] * (1 - b['refund_rate']) / max(b['conversions'], 1))
print('guardrail: refund rate')
print(f' A {a["refund_rate"]:.2%} B {b["refund_rate"]:.2%} '
f'difference {gd:+.2%} (95% CI [{gd - 1.96 * gse:+.2%}, {gd + 1.96 * gse:+.2%}])')
print()
# ---- 6. The decision, against the threshold set in advance. ---------
minimum_absolute = BASELINE * MIN_LIFT
print(f'threshold set in advance: {minimum_absolute:+.3%} absolute')
print(f'observed: {difference:+.3%}')
print(f'the interval includes the threshold: {low <= minimum_absolute <= high}')
print()
if low > minimum_absolute:
print('SHIP: the whole interval is above the threshold.')
elif high < 0:
print('DO NOT SHIP: the whole interval is a loss.')
else:
print('INCONCLUSIVE at the planned sample size.')
print('The true lift is 6%, below the 8% we said was worth shipping,')
print('so the test was never powered to resolve it - which is the')
print('correct outcome, and the reason to set the threshold first.')
Open this example in the editor and change it. Nothing you type is saved.
The outcome there is the honest one, and the one teams find hardest: a real but small lift, below the threshold that was set in advance, with an interval that cannot rule out zero. Without the pre-registered threshold this would have become an argument. With it, the answer is a sentence.
When you cannot run a test
Sometimes randomisation is impossible - a pricing change, a legal requirement, a launch that affects everyone at once. The alternatives, in descending order of trustworthiness:
- A staged rollout by region or cohort, which is a randomised test with a coarser unit.
- Difference-in-differences: compare the change in the exposed group with the change in an unexposed one over the same period, which cancels out anything that affected both.
- An interrupted time series: fit the trend before the change and compare what followed with the extrapolation.
- A before-and-after comparison, which is the weakest and confounds the change with everything else that happened that week.
All four are weaker than a randomised test. Say which one you used, and say what it cannot rule out.
When it goes wrong
| Symptom | Cause |
|---|---|
| A clear winner that did not replicate | Stopped early, or peeked |
| An "inconclusive" result after weeks | Underpowered; compute n first |
| The variants differ before the change | Broken randomisation; check covariates |
| The winner lost money | The primary metric was the wrong one |
| One of twenty variants won | Expected by chance; correct for comparisons |
| The split is 53/47 | Assignment is not random; investigate |
| A big lift with a tiny sample | Noise; look at the interval |
| The result is reported as a verdict | Report the interval and the threshold |
A check you can run
Take the last A/B test your team ran and ask two questions: what sample size was agreed before it started, and how many times was the dashboard checked?
If the answer to the first is "we didn't" and the second is "daily", the result is not evidence, whatever it showed. Computing the sample size takes the ten lines above and five minutes, and doing it before the test is the difference between an experiment and an anecdote.