Every number computed from a sample is a guess with a range around it. This lesson is about how wide that range is, how to compute it without a statistics library, and the three places a dashboard pretends it does not exist.
Where the range comes from
If you measure a conversion rate on a thousand visitors, you did not measure the conversion rate. You measured one sample, and a different thousand visitors would have given a different answer. The range is how much that answer would wobble.
You can see the wobble directly by simulating it:
import numpy as np
rng = np.random.default_rng(3)
TRUE_RATE = 0.12
SAMPLE = 1000
# A thousand parallel universes, each with a thousand visitors.
samples = rng.binomial(SAMPLE, TRUE_RATE, size=1000) / SAMPLE
print(f'true rate {TRUE_RATE:.3f}')
print(f'mean of samples {samples.mean():.3f}')
print(f'spread (std) {samples.std():.4f}')
print(f'middle 95% {np.percentile(samples, 2.5):.3f} to '
f'{np.percentile(samples, 97.5):.3f}')
print(f'lowest seen {samples.min():.3f}')
print(f'highest seen {samples.max():.3f}')
Open this example in the editor and change it. Nothing you type is saved.
Nothing changed between those thousand universes except luck, and the measured rate ranged from roughly 9% to 15%. Any single one of them, reported on its own, would look like a fact.
Computing the interval without simulating
For a proportion there is a formula, and it is worth being able to write from memory:
import numpy as np
def proportion_interval(successes: int, trials: int, z: float = 1.96) -> tuple:
"""A 95% confidence interval for a proportion. z=1.96 for 95%,
2.58 for 99%, 1.64 for 90%."""
p = successes / trials
standard_error = np.sqrt(p * (1 - p) / trials)
return p - z * standard_error, p + z * standard_error
for trials in [100, 1_000, 10_000, 100_000]:
low, high = proportion_interval(int(trials * 0.12), trials)
print(f'n={trials:>7,} 12.0% [{low:.2%}, {high:.2%}] '
f'width {100 * (high - low):.2f} points')
Open this example in the editor and change it. Nothing you type is saved.
Two things to take from that table.
The width shrinks with the square root of n. To halve the interval you need four times the data; to make it ten times narrower you need a hundred times the data. That is why "collect more data" has a cost curve, and why a 2% lift needs a very large sample to detect.
At n=100 the interval is ±6 points. A rate reported as "12%" from a hundred observations is really "somewhere between 6% and 18%", which is almost no information at all.
For a mean rather than a proportion, the same shape:
import numpy as np
rng = np.random.default_rng(7)
values = rng.normal(240, 60, 400)
mean = values.mean()
standard_error = values.std(ddof=1) / np.sqrt(len(values))
low, high = mean - 1.96 * standard_error, mean + 1.96 * standard_error
print(f'mean {mean:.1f} se {standard_error:.2f} 95% CI [{low:.1f}, {high:.1f}]')
print(f'the interval is +/- {1.96 * standard_error / mean:.1%} of the mean')
Open this example in the editor and change it. Nothing you type is saved.
The bootstrap, for everything else
The formulas above assume a shape. The bootstrap assumes nothing: resample your own data, with replacement, and look at how much the answer moves.
import numpy as np
rng = np.random.default_rng(11)
# A heavily skewed quantity, where the formula for a mean is shaky.
order_value = rng.lognormal(3.6, 1.1, 800)
def bootstrap(values, statistic, draws=2000, seed=0):
generator = np.random.default_rng(seed)
n = len(values)
return np.array([
statistic(generator.choice(values, size=n, replace=True))
for _ in range(draws)
])
for name, statistic in [('mean', np.mean), ('median', np.median),
('p90', lambda v: np.percentile(v, 90))]:
draws = bootstrap(order_value, statistic)
point = statistic(order_value)
low, high = np.percentile(draws, [2.5, 97.5])
print(f'{name:<7} {point:7.2f} 95% CI [{low:7.2f}, {high:7.2f}] '
f'width {high - low:6.2f}')
Open this example in the editor and change it. Nothing you type is saved.
The median's interval is far narrower than the mean's, on exactly the same data, because the median is not dragged about by the tail. That is a second, quantitative argument for the median on skewed data.
The bootstrap works for any statistic you can compute - a ratio, a percentile, a difference between two groups, the output of a whole pipeline. It costs a few seconds and it removes the need to remember which formula applies.
The weekly dashboard problem
import numpy as np
import pandas as pd
rng = np.random.default_rng(13)
# 52 weeks with NO underlying change at all: a constant 8% rate.
TRUE = 0.08
VISITS = 1200
weekly = rng.binomial(VISITS, TRUE, 52) / VISITS
changes = np.diff(weekly) / weekly[:-1]
print(f'true rate {TRUE:.1%}, constant all year')
print(f'weekly rate ranged {weekly.min():.2%} to {weekly.max():.2%}')
print()
print(f'weeks "up" more than 10%: {(changes > 0.10).sum()}')
print(f'weeks "down" more than 10%: {(changes < -0.10).sum()}')
print(f'biggest weekly swing: {changes.max():+.1%}')
print(f'biggest weekly drop: {changes.min():+.1%}')
Open this example in the editor and change it. Nothing you type is saved.
Nothing changed, and the dashboard showed a double-digit move in a large fraction of weeks. Somebody will have been asked to explain each of them, and will have found an explanation, because there is always an explanation available after the fact.
The defence is to put the interval on the chart:
import numpy as np
rng = np.random.default_rng(13)
VISITS = 1200
weekly = rng.binomial(VISITS, 0.08, 8) / VISITS
half_width = 1.96 * np.sqrt(0.08 * 0.92 / VISITS)
for week, rate in enumerate(weekly, start=1):
low, high = rate - half_width, rate + half_width
bar = ' ' * int(low * 500) + '[' + '-' * max(int((high - low) * 500) - 1, 1) + ']'
print(f'week {week} {rate:.2%} {bar}')
print()
print(f'every interval is +/- {half_width:.2%}; they all overlap, so no week')
print('is distinguishable from any other')
Open this example in the editor and change it. Nothing you type is saved.
Once the bands are drawn, "week 5 was down" stops being a story, because week 5's band overlaps every other week's.
Small segments are mostly noise
import numpy as np
import pandas as pd
rng = np.random.default_rng(17)
TRUE = 0.10
segments = pd.DataFrame({
'segment': ['enterprise', 'mid-market', 'small', 'trial', 'edu', 'partner'],
'users': [40, 180, 2200, 9000, 25, 70],
})
segments['converted'] = [rng.binomial(n, TRUE) for n in segments['users']]
segments['rate'] = segments['converted'] / segments['users']
segments['half_width'] = 1.96 * np.sqrt(
segments['rate'] * (1 - segments['rate']) / segments['users'])
segments['low'] = (segments['rate'] - segments['half_width']).clip(0)
segments['high'] = segments['rate'] + segments['half_width']
view = segments.assign(
rate=lambda d: (d['rate'] * 100).round(1),
low=lambda d: (d['low'] * 100).round(1),
high=lambda d: (d['high'] * 100).round(1),
)[['segment', 'users', 'converted', 'rate', 'low', 'high']]
print(view.sort_values('rate', ascending=False).to_string(index=False))
print()
print(f'every segment has the SAME true rate of {TRUE:.0%}')
print('the "best" and "worst" segments differ only by sample size')
Open this example in the editor and change it. Nothing you type is saved.
Every segment was generated from the same 10% rate. The ranking is pure noise, and the small segments sit at the extremes because small samples vary more. A dashboard that ranks segments by rate will put the smallest ones at the top and the bottom, every time, forever.
The rule: do not rank a segment with fewer than a few hundred observations, and never without its interval.
Rounding and false precision
rate = 0.1237419
n = 1200
half_width = 0.0187
print(f'reported as {rate:.5%} - six significant figures from 1200 observations')
print(f'reported as {rate:.1%} - honest')
print(f'with its interval: {rate:.1%} (95% CI {rate - half_width:.1%} to {rate + half_width:.1%})')
print()
print('The interval is +/- 1.9 points. Every digit after the first')
print('decimal place is noise dressed as precision.')
Open this example in the editor and change it. Nothing you type is saved.
A good rule: report to one more significant figure than the interval justifies, and no more. If the interval is ±2 points, "12%" is right and "12.37%" is a lie about how much you know.
The question to ask of any difference
Given two numbers that differ, ask them in this order:
- How big is the difference, in the units people care about? Not "15% better" - "1.8 percentage points, which is about 340 more conversions a month".
- How big is the uncertainty? If the intervals overlap substantially, the difference may be nothing.
- Is it big enough to act on? A real difference that is too small to change a decision is not a finding.
import numpy as np
def compare(label, successes_a, trials_a, successes_b, trials_b):
a, b = successes_a / trials_a, successes_b / trials_b
se = np.sqrt(a * (1 - a) / trials_a + b * (1 - b) / trials_b)
difference = b - a
low, high = difference - 1.96 * se, difference + 1.96 * se
overlaps_zero = low < 0 < high
print(f'{label}')
print(f' A {a:.2%} (n={trials_a:,}) B {b:.2%} (n={trials_b:,})')
print(f' difference {difference:+.2%} 95% CI [{low:+.2%}, {high:+.2%}]')
print(f' could be zero: {overlaps_zero}')
print()
compare('tiny sample', 12, 100, 18, 100)
compare('same rates, big sample', 1200, 10_000, 1260, 10_000)
compare('clear difference', 1200, 10_000, 1450, 10_000)
Open this example in the editor and change it. Nothing you type is saved.
A worked example
import numpy as np
import pandas as pd
rng = np.random.default_rng(seed=131)
# A year of weekly conversion data for four segments. The TRUE rates
# are fixed and known here - which is what makes the noise visible.
TRUE_RATES = {'enterprise': 0.21, 'mid-market': 0.14, 'small': 0.11, 'trial': 0.11}
WEEKLY_TRAFFIC = {'enterprise': 35, 'mid-market': 260, 'small': 2600, 'trial': 7800}
weeks = pd.date_range('2026-01-05', periods=52, freq='W-MON')
rows = []
for segment, rate in TRUE_RATES.items():
traffic = WEEKLY_TRAFFIC[segment]
for week in weeks:
rows.append({
'week': week, 'segment': segment, 'visits': traffic,
'conversions': int(rng.binomial(traffic, rate)),
})
data = pd.DataFrame(rows)
print(f'{len(data)} segment-weeks, {data["visits"].sum():,} visits')
print()
# 1. The whole-year rate per segment, with an interval.
def interval(successes, trials, z=1.96):
p = successes / trials
se = np.sqrt(p * (1 - p) / trials)
return p, max(p - z * se, 0.0), p + z * se
annual = data.groupby('segment', as_index=False).agg(
visits=('visits', 'sum'), conversions=('conversions', 'sum'))
annual[['rate', 'low', 'high']] = annual.apply(
lambda r: pd.Series(interval(r['conversions'], r['visits'])), axis=1)
annual['true'] = annual['segment'].map(TRUE_RATES)
annual['covers_truth'] = (annual['low'] <= annual['true']) & (annual['true'] <= annual['high'])
print('annual rates, with 95% intervals:')
print(annual.assign(**{c: (annual[c] * 100).round(2) for c in ['rate', 'low', 'high', 'true']})
[['segment', 'visits', 'rate', 'low', 'high', 'true', 'covers_truth']]
.to_string(index=False))
print()
print('"small" and "trial" have the same true rate; the data agrees '
'because both intervals overlap.')
print()
# 2. The weekly view, which is where the noise lives.
weekly = data.copy()
weekly['rate'] = weekly['conversions'] / weekly['visits']
spread = weekly.groupby('segment')['rate'].agg(
worst='min', best='max', std='std')
spread['range_points'] = ((spread['best'] - spread['worst']) * 100).round(1)
spread['true'] = spread.index.map(TRUE_RATES)
print('weekly rates, by segment (true rate constant all year):')
print(spread.assign(**{c: (spread[c] * 100).round(1) for c in ['worst', 'best', 'true']})
[['true', 'worst', 'best', 'range_points']].to_string())
print()
print('The 35-visit segment swung across tens of percentage points')
print('with nothing changing. The 7,800-visit one barely moved.')
print()
# 3. How often a weekly chart would have shown a "significant" move.
alarms = {}
for segment, group in weekly.groupby('segment'):
changes = group['rate'].pct_change().dropna()
alarms[segment] = int((changes.abs() > 0.20).sum())
print('weeks that moved more than 20% from the previous week:')
for segment, count in alarms.items():
print(f' {segment:<12} {count:>2} of 51')
print('Every one of those would have been explained in a meeting.')
print()
# 4. Bootstrap the difference between two segments, with no formula.
def bootstrap_difference(a_rows, b_rows, draws=2000, seed=0):
generator = np.random.default_rng(seed)
out = np.empty(draws)
a_conv, a_vis = a_rows['conversions'].to_numpy(), a_rows['visits'].to_numpy()
b_conv, b_vis = b_rows['conversions'].to_numpy(), b_rows['visits'].to_numpy()
for i in range(draws):
ia = generator.integers(0, len(a_conv), len(a_conv))
ib = generator.integers(0, len(b_conv), len(b_conv))
out[i] = (b_conv[ib].sum() / b_vis[ib].sum()) - (a_conv[ia].sum() / a_vis[ia].sum())
return out
same = bootstrap_difference(data[data.segment == 'small'], data[data.segment == 'trial'])
different = bootstrap_difference(data[data.segment == 'small'],
data[data.segment == 'mid-market'])
for label, draws in [('small vs trial (truly equal)', same),
('small vs mid-market (truly different)', different)]:
low, high = np.percentile(draws, [2.5, 97.5])
print(f'{label}')
print(f' difference {draws.mean():+.3%} 95% CI [{low:+.3%}, {high:+.3%}] '
f'excludes zero: {not (low < 0 < high)}')
print()
# 5. How much data would be needed to see a one-point difference.
for n in [1_000, 5_000, 20_000, 100_000]:
se = np.sqrt(2 * 0.11 * 0.89 / n)
print(f'n={n:>7,} per group: 95% interval on a difference is '
f'+/-{1.96 * se * 100:.2f} points')
print()
print('A one-point difference needs roughly 20,000 per group to be')
print('distinguishable from nothing. That is the cost of precision.')
Open this example in the editor and change it. Nothing you type is saved.
Five things that example establishes with numbers rather than assertion. The annual intervals cover the true rates, which is the interval doing its job. Two segments with identical true rates produce overlapping intervals, correctly. The 35-visit segment swings wildly while the 7,800-visit one does not. The bootstrap separates a real difference from a fake one with no formula. And the last table prices precision: a one-point difference costs about twenty thousand observations per group to see.
When it goes wrong
| Symptom | Cause |
|---|---|
| A weekly metric swings and nobody knows why | Sampling noise; put the interval on the chart |
| The smallest segments are always the extremes | Small samples vary more; do not rank them |
| A rate is reported to four decimal places | False precision; round to the interval |
| Two overlapping intervals were called different | Overlap means it may be nothing |
| "We need more data" with no number | Width shrinks as the square root; price it |
| A skewed mean has a wide interval | It is dragged by the tail; use the median |
| A formula does not fit the statistic | Bootstrap it instead |
| A difference is real but tiny | Statistically detectable is not the same as useful |
A check you can run
Take any percentage on your dashboard, find its denominator, and compute 1.96 * sqrt(p * (1 - p) / n).
That is the half-width of its 95% interval. Compare it with the week-on-week movement everybody has been discussing. In my experience the movement is inside the interval more often than not, which means the discussion has been about noise - and the fix is one band drawn on one chart.