Accuracy is the wrong metric for most real problems, and the threshold that turns a probability into a decision is a business choice rather than a modelling one. Getting both right is usually worth more than any amount of tuning.
Accuracy lies when the classes are skewed
import numpy as np
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(3)
n = 10_000
# 2% fraud, which is a realistic rate.
X = rng.normal(size=(n, 5))
y = (rng.random(n) < 0.02).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
useless = DummyClassifier(strategy='most_frequent').fit(X_tr, y_tr)
print(f'fraud rate {y_te.mean():.2%}')
print(f'accuracy of "always say no fraud": {accuracy_score(y_te, useless.predict(X_te)):.3%}')
print()
print(pd.DataFrame(confusion_matrix(y_te, useless.predict(X_te)),
index=['actually legitimate', 'actually fraud'],
columns=['predicted legit', 'predicted fraud']).to_string())
print()
print('98% accurate, and it catches not one fraud. Accuracy on a')
print('skewed problem is a measure of the skew.')
Open this example in the editor and change it. Nothing you type is saved.
The confusion matrix, and the four numbers from it
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score
actual = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
predicted = np.array([0, 0, 0, 0, 1, 1, 0, 1, 1, 1])
matrix = confusion_matrix(actual, predicted)
tn, fp, fn, tp = matrix.ravel()
print(pd.DataFrame(matrix, index=['actual no', 'actual yes'],
columns=['predicted no', 'predicted yes']).to_string())
print()
print(f'true negatives {tn} false positives {fp}')
print(f'false negatives {fn} true positives {tp}')
print()
print(f'precision {tp / (tp + fp):.3f} - of those we flagged, how many were right')
print(f'recall {tp / (tp + fn):.3f} - of those that were positive, how many we caught')
print(f'F1 {f1_score(actual, predicted):.3f} - their harmonic mean')
print(f'specificity{tn / (tn + fp):>7.3f} - of the negatives, how many we left alone')
Open this example in the editor and change it. Nothing you type is saved.
The two words to keep straight:
- Precision answers "when we say yes, how often are we right?" It is about the cost of a false positive.
- Recall answers "of all the real cases, how many did we find?" It is about the cost of a false negative.
They trade off. Flagging everything gives perfect recall and terrible precision; flagging only the single most certain case gives the reverse.
Which to favour
import pandas as pd
print(pd.DataFrame([
{'problem': 'cancer screening',
'false negative costs': 'a missed diagnosis',
'false positive costs': 'an unnecessary follow-up test',
'favour': 'recall'},
{'problem': 'spam filter',
'false negative costs': 'a spam email in the inbox',
'false positive costs': 'a real email lost',
'favour': 'precision'},
{'problem': 'fraud blocking',
'false negative costs': 'a fraudulent transaction',
'false positive costs': 'a blocked legitimate customer',
'favour': 'depends on the amounts'},
{'problem': 'churn outreach',
'false negative costs': 'a customer leaves unprompted',
'false positive costs': 'a wasted phone call',
'favour': 'recall - calls are cheap'},
]).to_string(index=False))
Open this example in the editor and change it. Nothing you type is saved.
The question is never "which metric is best" but what does each kind of mistake cost. Write the two costs down in money or in hours, and the metric chooses itself.
The threshold is yours to pick
predict() applies a 0.5 threshold. There is nothing special about 0.5, and on a skewed problem it is almost always wrong.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(7)
n = 20_000
X = rng.normal(size=(n, 6))
score = 1.4 * X[:, 0] + 0.9 * X[:, 1] - 0.7 * X[:, 2] + rng.normal(0, 1.4, n)
y = (score > 3.0).astype(int) # about 2% positive
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
probabilities = model.predict_proba(X_te)[:, 1]
print(f'positive rate {y_te.mean():.2%}')
print()
rows = []
for threshold in [0.02, 0.05, 0.10, 0.25, 0.50, 0.75]:
flagged = (probabilities >= threshold).astype(int)
rows.append({
'threshold': threshold,
'flagged': int(flagged.sum()),
'precision': round(precision_score(y_te, flagged, zero_division=0), 3),
'recall': round(recall_score(y_te, flagged), 3),
'f1': round(f1_score(y_te, flagged, zero_division=0), 3),
})
print(pd.DataFrame(rows).to_string(index=False))
print()
print('One model, six different products. The threshold is the')
print('product decision and it does not require retraining.')
Open this example in the editor and change it. Nothing you type is saved.
Choosing the threshold by cost, which is the honest way:
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(11)
n = 20_000
X = rng.normal(size=(n, 6))
score = 1.4 * X[:, 0] + 0.9 * X[:, 1] - 0.7 * X[:, 2] + rng.normal(0, 1.4, n)
y = (score > 3.0).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
probabilities = model.predict_proba(X_te)[:, 1]
# The costs, in pounds, agreed with the business.
COST_FALSE_POSITIVE = 4 # the review costs four pounds of someone's time
COST_FALSE_NEGATIVE = 180 # an undetected case costs 180 pounds
rows = []
for threshold in np.arange(0.01, 0.80, 0.01):
flagged = probabilities >= threshold
false_positives = int((flagged & (y_te == 0)).sum())
false_negatives = int((~flagged & (y_te == 1)).sum())
rows.append({
'threshold': round(threshold, 2),
'reviewed': int(flagged.sum()),
'missed': false_negatives,
'cost': false_positives * COST_FALSE_POSITIVE
+ false_negatives * COST_FALSE_NEGATIVE,
})
table = pd.DataFrame(rows)
best = table.loc[table['cost'].idxmin()]
print(table.iloc[::8].to_string(index=False))
print()
print(f'cheapest threshold {best["threshold"]:.2f}: '
f'{int(best["reviewed"]):,} reviewed, {int(best["missed"])} missed, '
f'cost {best["cost"]:,.0f}')
print(f'at the default 0.50: cost '
f'{table.loc[(table["threshold"] - 0.5).abs().idxmin(), "cost"]:,.0f}')
Open this example in the editor and change it. Nothing you type is saved.
That difference is typically larger than anything a week of model tuning would buy, and it takes ten lines.
ROC AUC, and when it misleads
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score, average_precision_score
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(13)
n = 40_000
X = rng.normal(size=(n, 5))
latent = 1.3 * X[:, 0] + 0.7 * X[:, 1] + rng.normal(0, 1.5, n)
for rate_label, cut in [('balanced (50%)', 0.0), ('skewed (1%)', 3.6)]:
y = (latent > cut).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
probabilities = model.predict_proba(X_te)[:, 1]
print(f'{rate_label:<16} positive rate {y_te.mean():.2%} '
f'ROC AUC {roc_auc_score(y_te, probabilities):.3f} '
f'avg precision {average_precision_score(y_te, probabilities):.3f} '
f'(baseline {y_te.mean():.3f})')
print()
print('ROC AUC barely moves between the two. Average precision falls')
print('a long way, because it measures the thing that actually got')
print('harder: finding the rare positives.')
Open this example in the editor and change it. Nothing you type is saved.
ROC AUC is the probability that a random positive is scored above a random negative. It is threshold-free and comparable across datasets, which is why it is the default for model selection.
Its weakness is that it uses the true negative rate, and when negatives are overwhelming, a large number of false positives barely moves it. On a 1% problem, a model with an AUC of 0.95 can still be wrong four times out of five when it says yes.
Average precision (the area under the precision-recall curve) has the positive rate as its baseline and does not have this problem. For a skewed problem, report both.
For regression
import numpy as np
import pandas as pd
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
mean_absolute_percentage_error, r2_score)
rng = np.random.default_rng(17)
actual = rng.gamma(4, 300, 2000)
typical = actual + rng.normal(0, 150, 2000)
with_outliers = typical.copy()
with_outliers[rng.choice(2000, 20, replace=False)] += 9000
for name, predicted in [('typical errors', typical), ('plus 20 big misses', with_outliers)]:
print(f'{name}')
print(f' MAE {mean_absolute_error(actual, predicted):8.1f}')
print(f' RMSE {np.sqrt(mean_squared_error(actual, predicted)):8.1f}')
print(f' MAPE {mean_absolute_percentage_error(actual, predicted):8.1%}')
print(f' R2 {r2_score(actual, predicted):8.3f}')
print()
print('Twenty bad predictions out of two thousand barely move the MAE')
print('and move the RMSE a long way. Which you want depends on whether')
print('one big miss is worse than many small ones.')
Open this example in the editor and change it. Nothing you type is saved.
- MAE - the average error, in the units of the target. Robust, and the easiest to explain.
- RMSE - penalises large errors disproportionately. Use it when one big miss is much worse than several small ones.
- MAPE - percentage error. Useful across different scales, and undefined when the actual value is zero.
- R2 - the share of variance explained. Comparable within a dataset, meaningless across datasets.
A worked example
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import (average_precision_score, confusion_matrix,
precision_recall_curve, roc_auc_score)
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(seed=59)
n = 60_000
# Transactions. Fraud is rare, and the amounts vary enormously, so
# the cost of a miss is not constant.
amount = rng.lognormal(3.4, 1.1, n).round(2)
hour = rng.integers(0, 24, n)
age_days = rng.gamma(2, 180, n).round()
country_risk = rng.choice([0, 1, 2], n, p=[.85, .12, .03])
latent = (0.9 * np.log1p(amount) + 1.3 * country_risk
+ 0.8 * ((hour < 5) | (hour > 22)) - 0.004 * np.sqrt(age_days)
+ rng.normal(0, 1.1, n))
fraud = (latent > 7.0).astype(int) # about 1.8%, a realistic rate
X = pd.DataFrame({'amount': amount, 'hour': hour, 'age_days': age_days,
'country_risk': country_risk})
X_tr, X_te, y_tr, y_te, amount_tr, amount_te = train_test_split(
X, fraud, amount, test_size=0.3, random_state=0, stratify=fraud)
print(f'{len(X_tr):,} train, {len(X_te):,} test')
print(f'fraud rate {y_te.mean():.2%} ({int(y_te.sum())} cases)')
print(f'median fraudulent amount {np.median(amount_te[y_te == 1]):,.2f}')
print()
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
probabilities = model.predict_proba(X_te)[:, 1]
# ---- 1. The threshold-free numbers. --------------------------------
print(f'ROC AUC {roc_auc_score(y_te, probabilities):.3f} (baseline 0.500)')
print(f'average precision {average_precision_score(y_te, probabilities):.3f} '
f'(baseline {y_te.mean():.3f})')
print()
# ---- 2. The default threshold, which is a bad product. -------------
default = (probabilities >= 0.5).astype(int)
matrix = confusion_matrix(y_te, default)
tn, fp, fn, tp = matrix.ravel()
print('at the default threshold of 0.5:')
print(f' flagged {tp + fp:,} caught {tp} of {tp + fn} frauds '
f'({tp / (tp + fn):.0%}) false alarms {fp}')
print()
# ---- 3. Choose by cost, using the real amounts. --------------------
REVIEW_COST = 3.50 # a human review
CHARGEBACK_FEE = 25.00 # on top of losing the transaction amount
rows = []
for threshold in np.arange(0.005, 0.60, 0.005):
flagged = probabilities >= threshold
review_cost = int(flagged.sum()) * REVIEW_COST
missed = (~flagged) & (y_te == 1)
loss = float(amount_te[missed].sum()) + int(missed.sum()) * CHARGEBACK_FEE
rows.append({'threshold': round(threshold, 3),
'reviewed': int(flagged.sum()),
'caught': int((flagged & (y_te == 1)).sum()),
'missed': int(missed.sum()),
'review_cost': round(review_cost),
'fraud_loss': round(loss),
'total': round(review_cost + loss)})
table = pd.DataFrame(rows)
best = table.loc[table['total'].idxmin()]
default_row = table.loc[(table['threshold'] - 0.5).abs().idxmin()]
print(table.iloc[::15].to_string(index=False))
print()
print(f'cheapest threshold {best["threshold"]:.3f} total cost '
f'{best["total"]:,.0f} ({int(best["caught"])} caught, '
f'{int(best["missed"])} missed, {int(best["reviewed"]):,} reviewed)')
print(f'default threshold 0.500 total cost {default_row["total"]:,.0f}')
print(f'saving from moving the threshold alone: '
f'{default_row["total"] - best["total"]:,.0f}')
print()
# ---- 4. What a capacity constraint does instead. -------------------
CAPACITY = 400 # reviews the team can do per period
cutoff = float(np.sort(probabilities)[-CAPACITY])
flagged = probabilities >= cutoff
print(f'with capacity for {CAPACITY} reviews, the threshold is {cutoff:.3f}')
print(f' precision {float((flagged & (y_te == 1)).sum()) / flagged.sum():.1%} '
f'recall {float((flagged & (y_te == 1)).sum()) / y_te.sum():.1%}')
print(f' fraud value caught '
f'{float(amount_te[flagged & (y_te == 1)].sum()):,.0f} of '
f'{float(amount_te[y_te == 1].sum()):,.0f}')
print()
print('When the team has fixed capacity, the threshold is not chosen')
print('at all - it falls out of the capacity, and the metric to')
print('optimise is precision in the top N.')
Open this example in the editor and change it. Nothing you type is saved.
Three results there and the third is the one most often missed. The threshold-free numbers say the model is good. Moving the threshold from the default to the cost-minimising one saves a substantial sum with no retraining. And when the review team has fixed capacity, the threshold is determined by the capacity, which makes "precision at the top 400" the metric to improve rather than AUC.
When it goes wrong
| Symptom | Cause |
|---|---|
| 98% accurate, catches nothing | A skewed label; accuracy measures the skew |
| High AUC, useless in production | Few positives; look at average precision |
| The model never predicts the minority class | The 0.5 threshold; move it |
| Precision and recall both fell | The model got worse, not the threshold |
| F1 chosen without thinking | It weights the two errors equally; yours may not |
| RMSE dominated by a handful of rows | That is its purpose; use MAE if unwanted |
| MAPE is infinite | A zero actual value; use MAE |
| The metric does not match the decision | Write down the cost of each error first |
A check you can run
Take any classifier you have deployed and sweep its threshold from 0.01 to 0.9, computing at each step the number of each kind of error multiplied by its cost.
Plot or print the total. The minimum is almost never at 0.5, and the distance between it and where you are is usually worth more than the next model you were about to train.