Better in Both, Worse Overall

  • Hard
  • Probability puzzles
  • Tests: Noticing that a pooled average hides its groups

The puzzle

Two treatments for kidney stones. On small stones A succeeds 81 times in 87 and B 234 in 270. On large stones A succeeds 192 in 263 and B 55 in 80.

A beats B on small stones and on large stones. Which is better overall - and how can the answer be B?

Stuck?

One at a time, smallest first. Open only as many as you need — the whole point of the puzzle is the part you work out yourself.

Hint 1

Work out the four percentages, then pool each treatment's totals.

Hint 2

Look at how many of each kind of case each treatment took on.

Show the answer Hide the answer

A wins both subgroups (93% v 87% and 73% v 69%) and loses overall: 78% against 83%. Both statements are true.

How to get there

Small stones: A 81/87 = 93.1%, B 234/270 = 86.7%. Large stones: A 192/263 = 73.0%, B 55/80 = 68.8%. A is ahead in both.

Pooled: A is (81 + 192)/350 = 78.0%, B is (234 + 55)/350 = 82.6%. B wins. Nothing has been miscounted - both treatments handled 350 cases.

The cause is which cases each took. Large stones are harder for everybody, and A took 263 of them against B's 80. A's overall rate is dragged down by the difficulty of its caseload, not by its performance. This is real data from a 1986 study, and it is why clinical results are reported by subgroup: pooling can reverse the direction of an effect.

The technique worth keeping

A weighted average is governed by the weights as much as the values. Before comparing two pooled rates, check whether the groups are the same size - if they are not, compare within groups and treat the pooled figure as describing the caseload rather than the treatment.