A model is a function fitted to examples. You give it inputs and the answers, it finds a rule that reproduces the answers, and you hope the rule holds on inputs it has never seen. Everything else in this course is detail on that sentence.
The vocabulary, used precisely
import numpy as np
import pandas as pd
rng = np.random.default_rng(3)
n = 500
# Every row is an EXAMPLE. Every column except the last is a FEATURE.
# The last column is the LABEL - the thing we want to predict.
data = pd.DataFrame({
'tenure_months': rng.integers(1, 60, n),
'tickets_last_90d': rng.poisson(1.8, n),
'plan': rng.choice(['free', 'pro', 'team'], n, p=[.6, .3, .1]),
'monthly_spend': rng.gamma(3, 24, n).round(2),
})
risk = (0.55 - 0.012 * data['tenure_months'] + 0.09 * data['tickets_last_90d']
- 0.004 * data['monthly_spend'])
data['churned'] = rng.random(n) < risk.clip(0.02, 0.95)
print(f'{len(data)} examples, {data.shape[1] - 1} features, 1 label')
print(f'label balance: {data["churned"].mean():.1%} positive')
print(data.head(3).to_string(index=False))
Open this example in the editor and change it. Nothing you type is saved.
The words, each meaning exactly one thing:
- Example (or sample, instance, row): one thing you want a prediction about.
- Feature: an input column. Label (or target): the output column.
- Training: finding the rule. Inference (or prediction, scoring): applying it.
- Supervised: you have labels. Unsupervised: you do not, so there is nothing to predict and the task is structure-finding.
- Classification: the label is a category. Regression: the label is a number.
- Parameters: what the model learns. Hyperparameters: what you choose before training.
That last distinction matters more than it sounds. A model's parameters are fitted from the data; its hyperparameters are a choice you make and must therefore be chosen on data the final score is not computed from - which is the whole subject of the cross-validation lesson.
It learns association, not mechanism
import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
n = 2000
# Umbrella sales and traffic accidents are both caused by rain.
# Neither causes the other.
rain = rng.random(n) < 0.3
umbrellas = rng.poisson(np.where(rain, 40, 6))
accidents = rng.poisson(np.where(rain, 12, 4))
frame = pd.DataFrame({'umbrellas': umbrellas, 'accidents': accidents})
print(f'correlation {frame["umbrellas"].corr(frame["accidents"]):.3f}')
# A model predicts accidents from umbrella sales very well.
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X = frame[['umbrellas']].to_numpy()
y = frame['accidents'].to_numpy()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model = LinearRegression().fit(X_train, y_train)
print(f'R2 on held-out data: {r2_score(y_test, model.predict(X_test)):.3f}')
print()
print('The model is useful for PREDICTION and says nothing about cause.')
print('Banning umbrellas would not reduce accidents.')
Open this example in the editor and change it. Nothing you type is saved.
This is the single most important limitation to state out loud, because a model that predicts well is routinely presented as an explanation of what to change. A model answers "what is likely, given what I can see", not "what would happen if I changed something". The second question needs an experiment, or a causal method designed for it.
It cannot see what is not in the data
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
rng = np.random.default_rng(11)
n = 1200
# The true driver is a feature the model is NOT given.
quality = rng.normal(0, 1, n)
price = rng.normal(50, 10, n)
sales = 200 - 1.5 * price + 40 * quality + rng.normal(0, 8, n)
frame = pd.DataFrame({'price': price, 'sales': sales})
split = int(n * 0.7)
model = RandomForestRegressor(n_estimators=120, random_state=0).fit(
frame[['price']][:split], frame['sales'][:split])
predicted = model.predict(frame[['price']][split:])
print(f'MAE without the quality feature: '
f'{mean_absolute_error(frame["sales"][split:], predicted):.1f}')
frame['quality'] = quality
model_full = RandomForestRegressor(n_estimators=120, random_state=0).fit(
frame[['price', 'quality']][:split], frame['sales'][:split])
print(f'MAE with it: '
f'{mean_absolute_error(frame["sales"][split:], model_full.predict(frame[["price", "quality"]][split:])):.1f}')
print()
print('No algorithm recovers a variable it was never shown. "Try a')
print('better model" is the wrong response to a missing feature.')
Open this example in the editor and change it. Nothing you type is saved.
It assumes the future resembles the past
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
rng = np.random.default_rng(13)
# Train on a world where feature 0 matters.
X_train = rng.normal(0, 1, (2000, 2))
y_train = (X_train[:, 0] + rng.normal(0, 0.4, 2000) > 0).astype(int)
model = LogisticRegression().fit(X_train, y_train)
# The same world, later: nothing changed.
X_same = rng.normal(0, 1, (2000, 2))
y_same = (X_same[:, 0] + rng.normal(0, 0.4, 2000) > 0).astype(int)
print(f'accuracy, same world: {accuracy_score(y_same, model.predict(X_same)):.3f}')
# A changed world: the relationship flipped.
X_new = rng.normal(0, 1, (2000, 2))
y_new = (-X_new[:, 0] + rng.normal(0, 0.4, 2000) > 0).astype(int)
print(f'accuracy, changed world: {accuracy_score(y_new, model.predict(X_new)):.3f}')
print()
print('The model did not break. The world moved, and nothing in the')
print('model can tell you that it did - which is why monitoring exists.')
Open this example in the editor and change it. Nothing you type is saved.
Accuracy below 0.5 is worse than guessing. The model is confidently wrong, it reports no error, and only a monitoring system comparing predictions with outcomes will notice.
What it does not do
Four things people expect of a model that it does not provide:
It does not explain itself. Feature importance says which inputs the model used, not why the world works that way. A spuriously correlated feature gets high importance.
It does not know what it does not know. Asked about an input unlike anything in training, it answers confidently.
import numpy as np
from sklearn.ensemble import RandomForestRegressor
rng = np.random.default_rng(17)
X = rng.uniform(0, 10, (500, 1))
y = (2 * X[:, 0] + rng.normal(0, 0.5, 500))
model = RandomForestRegressor(n_estimators=100, random_state=0).fit(X, y)
for value in [5.0, 9.5, 50.0, 500.0]:
prediction = model.predict([[value]])[0]
truth = 2 * value
print(f'input {value:>6.1f} predicted {prediction:>7.2f} true {truth:>7.1f} '
f'{"in range" if value <= 10 else "EXTRAPOLATING"}')
print()
print('A tree model cannot extrapolate at all: it returns the nearest')
print('leaf it knows, forever. It does not say so.')
Open this example in the editor and change it. Nothing you type is saved.
It does not fix bad labels. If the label was recorded wrongly, the model learns the mistake faithfully.
It does not produce fairness. A model trained on past decisions reproduces the pattern of those decisions, including the parts nobody would defend.
When not to use it
import pandas as pd
print(pd.DataFrame([
{'situation': 'a rule would do',
'instead': 'write the rule - it is testable, explainable and free'},
{'situation': 'fewer than a few hundred labelled examples',
'instead': 'collect more, or use a rule'},
{'situation': 'the decision must be explainable in law',
'instead': 'a simple model you can write down, or a rule'},
{'situation': 'the cost of a wrong answer is catastrophic',
'instead': 'a model as an aid, with a human deciding'},
{'situation': 'you want to know what to CHANGE',
'instead': 'an experiment; a model answers a different question'},
{'situation': 'the labels are the thing you are trying to define',
'instead': 'define it first; a model cannot invent the target'},
]).to_string(index=False))
Open this example in the editor and change it. Nothing you type is saved.
The first row is the one most often ignored. A great deal of what gets built as a model is three conditions and a threshold, and the rule version is faster, cheaper, auditable, and does not need a retraining pipeline.
The honest framing
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix
rng = np.random.default_rng(seed=23)
n = 4000
# A churn model, framed honestly from the start.
tenure = rng.integers(1, 48, n)
tickets = rng.poisson(1.6, n)
spend = rng.gamma(3, 22, n)
risk = (0.62 - 0.013 * tenure + 0.10 * tickets - 0.006 * spend)
churned = (rng.random(n) < np.clip(risk, 0.02, 0.95)).astype(int)
X = np.column_stack([tenure, tickets, spend])
X_train, X_test, y_train, y_test = train_test_split(
X, churned, test_size=0.3, random_state=0, stratify=churned)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
predicted = model.predict(X_test)
accuracy = accuracy_score(y_test, predicted)
baseline = max(y_test.mean(), 1 - y_test.mean())
matrix = confusion_matrix(y_test, predicted)
print('THE CLAIM, written as it should be')
print()
print(f'Given tenure, ticket count and monthly spend, this model assigns')
print(f'each account a churn probability. On {len(y_test):,} accounts it did')
print(f'not see during training it is right {accuracy:.1%} of the time.')
print()
print(f'The baseline - always predicting the majority class - is '
f'{baseline:.1%},')
print(f'so the model adds {accuracy - baseline:+.1%}.')
print()
print('confusion matrix (rows: actual, columns: predicted):')
print(pd.DataFrame(matrix, index=['stayed', 'churned'],
columns=['predicted stay', 'predicted churn']).to_string())
print()
print('WHAT IT DOES NOT SAY')
for limitation in [
'that any of these features CAUSES churn',
'what would happen if we changed one of them',
'anything about accounts unlike those in the training data',
'anything about a world where the product or market changes',
]:
print(' -', limitation)
print()
print('HOW IT WOULD BE USED')
print(' - rank accounts by probability, work the top of the list')
print(' - a human reads the account before acting')
print(' - measure whether intervention changes the outcome, by experiment')
Open this example in the editor and change it. Nothing you type is saved.
Three parts, and the middle one is what separates a useful model from an overclaimed one. The score, with its baseline and the size of the test set. The list of things it does not say. How a decision will actually be made from it, including who looks at it.
That baseline line is the one most often missing. A model with 91% accuracy on a problem where 90% of cases are negative has learned almost nothing, and the accuracy figure conceals it completely.
When it goes wrong
| Symptom | Cause |
|---|---|
| High accuracy, no business value | A skewed label; compare with the baseline |
| The model is treated as an explanation | It learned association, not mechanism |
| "Try a better algorithm" does not help | A missing feature; no model recovers it |
| Accuracy fell below chance | The world changed; monitor outcomes |
| Confident answers on strange inputs | Models do not know their own limits |
| The model reproduced a past bias | It learned the labels it was given |
| A tree model would not extrapolate | By construction; it predicts the nearest leaf |
| A model where a rule would do | Three conditions is not machine learning |
A check you can run
Take any model in use and write down three sentences: what it predicts, what the baseline is, and what somebody does with the output.
If the second is missing, compute it - it takes one line and it is sometimes most of the reported score. If the third is vague, the model is not in production even if the code is running, and the first conversation to have is about the decision rather than the algorithm.