A model's score on data it was trained on is not evidence of anything. The split is what makes the number honest, and getting it wrong is the commonest way a project reports a result that does not survive contact with production.
Features, labels and the matrix
import numpy as np
import pandas as pd
rng = np.random.default_rng(3)
n = 1500
frame = pd.DataFrame({
'tenure_months': rng.integers(1, 60, n),
'tickets_90d': rng.poisson(1.7, n),
'plan': rng.choice(['free', 'pro', 'team'], n, p=[.6, .3, .1]),
'monthly_spend': rng.gamma(3, 25, n).round(2),
'signed_up': pd.Timestamp('2024-01-01') + pd.to_timedelta(rng.integers(0, 700, n), 'D'),
})
risk = 0.60 - 0.012 * frame['tenure_months'] + 0.09 * frame['tickets_90d']
frame['churned'] = (rng.random(n) < risk.clip(0.02, 0.95)).astype(int)
FEATURES = ['tenure_months', 'tickets_90d', 'plan', 'monthly_spend']
LABEL = 'churned'
X = frame[FEATURES]
y = frame[LABEL]
print(f'X {X.shape} y {y.shape} positive rate {y.mean():.1%}')
print(X.dtypes.to_dict())
Open this example in the editor and change it. Nothing you type is saved.
Two conventions worth following. X is the feature matrix and y the label vector; everyone uses those names and deviating from them makes code harder to read. And the feature list is a named constant, not an inline drop(columns=[...]), so that training and inference cannot drift apart.
The three-way split
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(5)
X = rng.normal(size=(2000, 4))
y = (X[:, 0] + rng.normal(0, 0.5, 2000) > 0).astype(int)
# First carve off the test set, and then do not touch it.
X_rest, X_test, y_rest, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y)
# Then split what remains into train and validation.
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=0.25, random_state=0, stratify=y_rest)
for name, part in [('train', y_train), ('validation', y_val), ('test', y_test)]:
print(f'{name:<11} {len(part):>5,} ({len(part) / len(y):.0%}) '
f'positive {part.mean():.1%}')
Open this example in the editor and change it. Nothing you type is saved.
Each set has one job and they are not interchangeable:
- Train - fit the parameters.
- Validation - choose hyperparameters, compare models, pick a threshold. Used many times.
- Test - the final estimate of performance. Used once.
stratify=y keeps the label proportions equal across the splits. Without it, a rare class can be unevenly distributed by chance and the scores become incomparable.
Random is not always right
A random split assumes the rows are independent. Three common situations where they are not, and a random split then leaks.
1. Time. If the model will predict the future, the test set must be the future.
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
rng = np.random.default_rng(11)
n = 3000
days = np.sort(rng.integers(0, 365, n))
# The relationship DRIFTS over the year.
strength = np.linspace(2.0, -0.5, 365)[days]
feature = rng.normal(0, 1, n)
label = (feature * strength + rng.normal(0, 1, n) > 0).astype(int)
frame = pd.DataFrame({'day': days, 'feature': feature, 'label': label})
# Random split: train and test are interleaved in time.
shuffled = frame.sample(frac=1.0, random_state=0)
split = int(n * 0.75)
model = LogisticRegression().fit(shuffled[['feature']][:split], shuffled['label'][:split])
random_score = accuracy_score(shuffled['label'][split:],
model.predict(shuffled[['feature']][split:]))
# Time split: train on the past, test on the future.
ordered = frame.sort_values('day')
model = LogisticRegression().fit(ordered[['feature']][:split], ordered['label'][:split])
time_score = accuracy_score(ordered['label'][split:],
model.predict(ordered[['feature']][split:]))
print(f'random split accuracy: {random_score:.3f}')
print(f'time split accuracy: {time_score:.3f}')
print()
print('The random split is optimistic because it lets the model train')
print('on days that come after days it is tested on.')
Open this example in the editor and change it. Nothing you type is saved.
2. Groups. If several rows belong to one entity - a customer, a patient, a device - all of that entity's rows must be on the same side.
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupShuffleSplit, train_test_split
from sklearn.metrics import accuracy_score
rng = np.random.default_rng(13)
n_customers, per_customer = 300, 8
customer = np.repeat(np.arange(n_customers), per_customer)
# A per-customer quirk the model can memorise.
customer_effect = rng.normal(0, 2, n_customers)[customer]
feature = rng.normal(0, 1, len(customer))
label = ((feature + customer_effect + rng.normal(0, 0.6, len(customer))) > 0).astype(int)
X = np.column_stack([feature, customer]) # customer id as a feature, deliberately
y = label
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
naive = accuracy_score(y_te, RandomForestClassifier(n_estimators=120, random_state=0)
.fit(X_tr, y_tr).predict(X_te))
splitter = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=0)
train_idx, test_idx = next(splitter.split(X, y, groups=customer))
grouped = accuracy_score(y[test_idx], RandomForestClassifier(n_estimators=120, random_state=0)
.fit(X[train_idx], y_tr := y[train_idx]).predict(X[test_idx]))
print(f'random split (same customer on both sides): {naive:.3f}')
print(f'grouped split (customers kept together): {grouped:.3f}')
print()
print('The gap is memorisation of individual customers, which is')
print('useless for a new customer - the case the model is for.')
Open this example in the editor and change it. Nothing you type is saved.
3. Near-duplicates. Rows that are copies of each other, or near copies - the same document scraped twice, the same transaction retried - put the same information on both sides.
import numpy as np
import pandas as pd
rng = np.random.default_rng(17)
base = pd.DataFrame({'a': rng.normal(size=400), 'b': rng.normal(size=400)})
# 15% of rows are duplicated, as happens in scraped or retried data.
duplicated = pd.concat([base, base.sample(60, random_state=0)], ignore_index=True)
exact = int(duplicated.duplicated().sum())
print(f'{len(duplicated)} rows, {exact} exact duplicates ({exact / len(duplicated):.0%})')
print('After a random split, roughly half of those pairs straddle the')
print('split - and the model has seen the answer.')
Open this example in the editor and change it. Nothing you type is saved.
Deduplicate before splitting, always. After splitting it is too late.
The test set is spent when you look at it
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
rng = np.random.default_rng(19)
# Pure noise: there is NOTHING to learn here.
X = rng.normal(size=(600, 30))
y = rng.integers(0, 2, 600)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
best = 0.0
for depth in range(1, 25):
score = accuracy_score(y_te, RandomForestClassifier(
max_depth=depth, n_estimators=60, random_state=0).fit(X_tr, y_tr).predict(X_te))
best = max(best, score)
print(f'true accuracy achievable: 0.500 (the labels are random)')
print(f'best of 24 settings, chosen ON THE TEST SET: {best:.3f}')
print()
print('Trying enough settings against the test set finds one that fits')
print('its noise. The reported number is then a selection artefact.')
Open this example in the editor and change it. Nothing you type is saved.
Every time you look at the test set and change something, it becomes a little more like a validation set. After a dozen such rounds the number it reports is optimistic by a few points, and nobody can say by how much.
The discipline: choose everything on the validation set or by cross-validation, and run the test set once, at the end, as the number you publish. If you must look again, say so, and treat the result as an estimate with a question mark.
The number worth watching
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
rng = np.random.default_rng(23)
n = 1200
X = rng.normal(size=(n, 8))
y = (X[:, 0] + 0.6 * X[:, 1] + rng.normal(0, 0.9, n) > 0).astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
rows = []
for depth in [2, 4, 6, 10, 16, None]:
model = RandomForestClassifier(max_depth=depth, n_estimators=150, random_state=0)
model.fit(X_tr, y_tr)
train_score = accuracy_score(y_tr, model.predict(X_tr))
test_score = accuracy_score(y_te, model.predict(X_te))
rows.append({'max_depth': str(depth), 'train': round(train_score, 3),
'test': round(test_score, 3), 'gap': round(train_score - test_score, 3)})
print(pd.DataFrame(rows).to_string(index=False))
print()
print('The gap between train and test is the overfitting measure.')
print('A gap near zero with a low score is underfitting; a large gap')
print('with a high train score is memorisation.')
Open this example in the editor and change it. Nothing you type is saved.
Always report both numbers. A test score alone cannot distinguish "this problem is hard" from "this model memorised the training set".
A worked example
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import GroupShuffleSplit, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
rng = np.random.default_rng(seed=31)
# Support tickets. Several per customer, spread over a year, and the
# relationship between features and outcome drifts slowly.
n_customers = 600
rows = []
for customer in range(n_customers):
quirk = rng.normal(0, 1.6) # a strong per-customer effect
plan = rng.choice(['free', 'pro', 'team'], p=[.5, .35, .15])
joined = int(rng.integers(0, 330)) # customers arrive over the year
for _ in range(int(rng.integers(3, 12))):
day = min(joined + int(rng.integers(0, 40)), 364)
drift = np.linspace(1.4, 0.4, 365)[day] # the world changes
complexity = rng.gamma(2, 1.2)
first_response = rng.gamma(2, 3)
score = (-1.1 + 0.42 * complexity * drift + 0.1 * first_response
+ quirk + rng.normal(0, 0.7))
rows.append({'customer': customer, 'day': day, 'plan': plan,
'complexity': complexity, 'first_response_h': first_response,
'escalated': int(score > 0)})
tickets = pd.DataFrame(rows)
# A feature computed over the WHOLE dataset: each customer's own
# escalation rate. It is the kind of thing that gets added without
# thinking, and it is exactly what a grouped split exists to expose.
tickets['customer_rate'] = tickets.groupby('customer')['escalated'].transform('mean')
HONEST_FEATURES = ['complexity', 'first_response_h', 'plan']
LEAKY_FEATURES = HONEST_FEATURES + ['customer_rate']
CATEGORICAL = ['plan']
LABEL = 'escalated'
print(f'{len(tickets):,} tickets from {tickets["customer"].nunique()} customers')
print(f'positive rate {tickets[LABEL].mean():.1%}')
print(f'duplicated rows: {int(tickets.duplicated().sum())}')
print()
def build(features, depth=8):
numeric = [f for f in features if f not in CATEGORICAL]
return Pipeline([
('prepare', ColumnTransformer([
('num', StandardScaler(), numeric),
('cat', OneHotEncoder(handle_unknown='ignore'), CATEGORICAL),
])),
('model', RandomForestClassifier(n_estimators=200, max_depth=depth, random_state=0)),
])
def evaluate(train, test, label, features):
model = build(features).fit(train[features], train[LABEL])
probabilities = model.predict_proba(test[features])[:, 1]
return {
'split': label,
'train_n': len(train), 'test_n': len(test),
'honest_auc' if features is HONEST_FEATURES else 'leaky_auc':
round(roc_auc_score(test[LABEL], probabilities), 3),
}
def both(train, test, label):
honest = evaluate(train, test, label, HONEST_FEATURES)
leaky = evaluate(train, test, label, LEAKY_FEATURES)
return {**honest, 'leaky_auc': leaky['leaky_auc']}
results = []
# 1. Random - wrong here on two counts at once.
train_idx, test_idx = train_test_split(
tickets.index, test_size=0.25, random_state=0, stratify=tickets[LABEL])
results.append(both(tickets.loc[train_idx], tickets.loc[test_idx], 'random (wrong)'))
# 2. Grouped by customer - fixes the leakage between a customer's rows.
splitter = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=0)
gi, gj = next(splitter.split(tickets, groups=tickets['customer']))
results.append(both(tickets.iloc[gi], tickets.iloc[gj], 'grouped by customer'))
# 3. Time-ordered - fixes training on the future.
ordered = tickets.sort_values('day')
cut = int(len(ordered) * 0.75)
results.append(both(ordered.iloc[:cut], ordered.iloc[cut:], 'time-ordered'))
# 4. Both at once, which is what production actually requires: a
# customer who has not been seen, in a month that has not happened.
cutoff_day = int(np.percentile(tickets['day'], 70))
early = tickets[tickets['day'] <= cutoff_day]
early_customers = set(early['customer'])
late = tickets[(tickets['day'] > cutoff_day) & (~tickets['customer'].isin(early_customers))]
print(f'time + unseen customers: {len(early):,} train, {len(late):,} test')
results.append(both(early, late, 'time + unseen customers'))
print()
print(pd.DataFrame(results).to_string(index=False))
print()
print('Two things in that table, and the second is the important one.')
print()
print('The honest column moves with the split, and the two')
print('time-respecting rows are the lowest - which is the drift')
print('in the data showing up as soon as the model is asked to')
print('predict a period it did not train on. Those are the rows')
print('that describe production.')
print()
print('The leaky column is high under EVERY split, including the')
print('strictest. customer_rate was computed over the whole dataset,')
print('so it carries the test labels into every fold - and no amount')
print('of splitting discipline removes a feature that was built from')
print('the answer. The split protects against one kind of leakage;')
print('this is the other kind, and it has its own lesson.')
print()
# 5. The test set, used ONCE, after everything is settled.
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=1)
rest_idx, test_idx = next(splitter.split(tickets, groups=tickets['customer']))
rest, held_out = tickets.iloc[rest_idx], tickets.iloc[test_idx]
inner = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=2)
tr_idx, val_idx = next(inner.split(rest, groups=rest['customer']))
train, validation = rest.iloc[tr_idx], rest.iloc[val_idx]
print(f'train {len(train):,} validation {len(validation):,} test {len(held_out):,}')
print(f'customer overlap train/test: '
f'{len(set(train["customer"]) & set(held_out["customer"]))}')
print()
# Tuning uses the HONEST features only - the leaky one is dropped.
best_depth, best_score = None, -1.0
for depth in [4, 6, 8, 12, None]:
pipeline = build(HONEST_FEATURES, depth).fit(train[HONEST_FEATURES], train[LABEL])
score = roc_auc_score(validation[LABEL],
pipeline.predict_proba(validation[HONEST_FEATURES])[:, 1])
print(f' max_depth {str(depth):<5} validation AUC {score:.3f}')
if score > best_score:
best_depth, best_score = depth, score
print(f'chosen on VALIDATION: max_depth={best_depth} (AUC {best_score:.3f})')
final = build(HONEST_FEATURES, best_depth).fit(rest[HONEST_FEATURES], rest[LABEL])
test_auc = roc_auc_score(held_out[LABEL],
final.predict_proba(held_out[HONEST_FEATURES])[:, 1])
print(f'TEST AUC (looked at once): {test_auc:.3f}')
print()
print('The validation number chose the model. The test number is the')
print('one that goes in the report, and it was computed once.')
Open this example in the editor and change it. Nothing you type is saved.
The table in step four carries two lessons. The honest column moves by several points across the four splits, and the two that respect time are the lowest - because the relationship in this data drifts, and a split that lets the model train on later days flatters it. Those two rows are the ones that describe production; reporting a looser number and deploying against a stricter reality is how a model that "worked in testing" disappoints.
The leaky column is the harder lesson: it stays high under every split, including the one with no shared customers and no shared time. customer_rate was computed across the whole dataset with groupby(...).transform('mean'), which is a line that appears in a great many feature-engineering notebooks and quietly carries the test labels into every fold. A disciplined split cannot save you from it. That is a different failure, and it gets its own lesson.
When it goes wrong
| Symptom | Cause |
|---|---|
| Great validation score, poor production | The split let information leak |
| The score fell when the split got stricter | It was leaking; the stricter number is real |
| Several rows per entity | Use a grouped split |
| The model predicts the future | Split in time, never randomly |
| The test score crept up over weeks | The test set was used for selection |
| A rare class is missing from a fold | Use stratify= |
| Duplicates straddle the split | Deduplicate before splitting |
| Only the test score is reported | Report the train score too; the gap is the signal |
A check you can run
Take your current split and ask two questions: could any entity appear on both sides, and does any test row come from a time before any training row?
If either is yes, re-split properly and re-measure. The number will fall. That fall is not a regression - it is the first honest estimate you have had, and it is far better to find it now than after deployment.