Features are where most of the gain in a model comes from, and where the most expensive mistake lives. Leakage - a feature that contains the answer - produces a result that looks excellent and is worth nothing, and it is the single most common reason a model fails after deployment.
Useful features
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
rng = np.random.default_rng(3)
n = 4000
frame = pd.DataFrame({
'income': rng.lognormal(10.3, 0.5, n).round(-2),
'loan_amount': rng.lognormal(9.0, 0.6, n).round(-2),
'applied_at': pd.Timestamp('2026-01-01') + pd.to_timedelta(rng.integers(0, 365 * 24, n), 'h'),
'employment_years': rng.gamma(2.2, 2.4, n).round(1),
})
risk = (-1.0 + 3.0 * (frame['loan_amount'] / frame['income'])
- 0.08 * frame['employment_years'] + rng.normal(0, 0.8, n))
frame['defaulted'] = (risk > 0).astype(int)
folds = StratifiedKFold(5, shuffle=True, random_state=0)
raw = ['income', 'loan_amount', 'employment_years']
base = cross_val_score(HistGradientBoostingClassifier(random_state=0),
frame[raw], frame['defaulted'], cv=folds, scoring='roc_auc').mean()
# A RATIO, which is the single most productive feature type.
frame['loan_to_income'] = frame['loan_amount'] / frame['income']
with_ratio = cross_val_score(HistGradientBoostingClassifier(random_state=0),
frame[raw + ['loan_to_income']], frame['defaulted'],
cv=folds, scoring='roc_auc').mean()
print(f'raw features only: AUC {base:.3f}')
print(f'plus the ratio: AUC {with_ratio:.3f}')
print()
print('The model could in principle learn the ratio from the two')
print('columns. Giving it directly saves it the trouble, and on a')
print('tree model it saves a great deal of depth.')
Open this example in the editor and change it. Nothing you type is saved.
The categories worth trying, roughly in order of how often they pay:
Ratios and differences. loan / income, price - cost, errors / requests. A tree can only split on one feature at a time, so a ratio it would need a dozen splits to approximate is nearly free as a column.
Dates broken into parts. A raw timestamp is useless to most models; its components are not.
import numpy as np
import pandas as pd
rng = np.random.default_rng(5)
stamps = pd.Series(pd.Timestamp('2026-01-01')
+ pd.to_timedelta(rng.integers(0, 365 * 24, 6), 'h'))
parts = pd.DataFrame({
'hour': stamps.dt.hour,
'day_of_week': stamps.dt.dayofweek,
'is_weekend': stamps.dt.dayofweek >= 5,
'month': stamps.dt.month,
'day_of_month': stamps.dt.day,
'is_month_end': stamps.dt.is_month_end,
})
# Cyclic encoding, so 23:00 and 00:00 are close rather than far apart.
parts['hour_sin'] = np.sin(2 * np.pi * stamps.dt.hour / 24).round(3)
parts['hour_cos'] = np.cos(2 * np.pi * stamps.dt.hour / 24).round(3)
print(parts.to_string(index=False))
Open this example in the editor and change it. Nothing you type is saved.
Counts and aggregates over a window. "Orders in the last 30 days", "mean session length last week". These are usually the strongest features in a behavioural model - and they are also where leakage enters, so the window must close before the prediction point.
Categorical encodings. One-hot for a handful of values; target encoding for many, computed inside the cross-validation fold.
Text length, token counts, flags. Cheap, and often surprisingly predictive.
Leakage
Leakage is any feature that would not be available, with that value, at the moment the prediction is actually made.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
rng = np.random.default_rng(7)
n = 3000
frame = pd.DataFrame({
'tenure': rng.integers(1, 60, n),
'tickets': rng.poisson(1.6, n),
'spend': rng.gamma(3, 25, n).round(2),
})
risk = -0.5 - 0.04 * frame['tenure'] + 0.4 * frame['tickets'] + rng.normal(0, 1, n)
frame['churned'] = (risk > 0).astype(int)
# A field filled in BECAUSE the customer churned.
frame['cancellation_reason_given'] = np.where(
frame['churned'] == 1, rng.random(n) < 0.85, rng.random(n) < 0.02).astype(int)
folds = StratifiedKFold(5, shuffle=True, random_state=0)
honest = ['tenure', 'tickets', 'spend']
print(f'honest features: AUC '
f'{cross_val_score(HistGradientBoostingClassifier(random_state=0), frame[honest], frame["churned"], cv=folds, scoring="roc_auc").mean():.3f}')
print(f'plus the leak: AUC '
f'{cross_val_score(HistGradientBoostingClassifier(random_state=0), frame[honest + ["cancellation_reason_given"]], frame["churned"], cv=folds, scoring="roc_auc").mean():.3f}')
print()
print('In production that field is empty for every account that has')
print('not churned yet. The model would score near chance.')
Open this example in the editor and change it. Nothing you type is saved.
Five shapes it takes, and all five appear in real projects:
1. A field recorded as a consequence. cancellation_reason, refund_amount, closed_date. Available in the training table, absent at prediction time.
2. An aggregate computed over the whole dataset.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold, GroupKFold
rng = np.random.default_rng(11)
n_customers, per_customer = 400, 6
customer = np.repeat(np.arange(n_customers), per_customer)
n = len(customer)
quirk = rng.normal(0, 1.5, n_customers)[customer]
feature = rng.normal(0, 1, n)
y = ((feature + quirk + rng.normal(0, 0.8, n)) > 0).astype(int)
frame = pd.DataFrame({'customer': customer, 'feature': feature, 'y': y})
# Computed over ALL rows, including the ones in the test fold.
frame['customer_rate'] = frame.groupby('customer')['y'].transform('mean')
folds = GroupKFold(5)
honest = cross_val_score(HistGradientBoostingClassifier(random_state=0),
frame[['feature']], frame['y'], cv=folds,
groups=frame['customer'], scoring='roc_auc').mean()
leaked = cross_val_score(HistGradientBoostingClassifier(random_state=0),
frame[['feature', 'customer_rate']], frame['y'], cv=folds,
groups=frame['customer'], scoring='roc_auc').mean()
print(f'feature only: AUC {honest:.3f}')
print(f'plus customer_rate: AUC {leaked:.3f}')
print()
print('A GROUPED split did not save it. The feature was built from the')
print('labels of every row, so it carries them into every fold.')
Open this example in the editor and change it. Nothing you type is saved.
3. Preprocessing fitted before the split. The scaler's mean, the imputer's median, the feature selector's choice - all learned from the test rows. The pipeline exists to prevent exactly this.
4. A future-dated value. A "last activity date" that is after the prediction date, a target encoding computed over all time.
5. Duplicate rows straddling the split. The same record on both sides, so the model has seen the answer.
How to find it
Three checks, in increasing order of effort.
1. The 0.99 rule. An AUC above about 0.99 on a messy real-world problem is leakage until proved otherwise.
import numpy as np
import pandas as pd
print(pd.DataFrame([
{'AUC': '0.60-0.75', 'reading': 'plausible for a hard behavioural problem'},
{'AUC': '0.75-0.90', 'reading': 'a good model, check it anyway'},
{'AUC': '0.90-0.97', 'reading': 'excellent, or a subtle leak'},
{'AUC': '0.97-0.999', 'reading': 'assume leakage; find it before celebrating'},
{'AUC': '1.000', 'reading': 'certainly leakage'},
]).to_string(index=False))
Open this example in the editor and change it. Nothing you type is saved.
2. Rank the features and interrogate the top one.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(13)
n = 3000
frame = pd.DataFrame({
'tenure': rng.integers(1, 60, n),
'tickets': rng.poisson(1.6, n),
'spend': rng.gamma(3, 25, n).round(2),
})
risk = -0.5 - 0.04 * frame['tenure'] + 0.4 * frame['tickets'] + rng.normal(0, 1, n)
frame['churned'] = (risk > 0).astype(int)
frame['exit_survey_sent'] = np.where(frame['churned'] == 1,
rng.random(n) < 0.9, rng.random(n) < 0.03).astype(int)
FEATURES = ['tenure', 'tickets', 'spend', 'exit_survey_sent']
X_tr, X_te, y_tr, y_te = train_test_split(frame[FEATURES], frame['churned'],
test_size=0.3, random_state=0, stratify=frame['churned'])
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
result = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
table = pd.DataFrame({'feature': FEATURES,
'importance': result.importances_mean.round(4)}
).sort_values('importance', ascending=False)
print(table.to_string(index=False))
print()
print('For the top feature, ask one question: WOULD I HAVE THIS VALUE,')
print('with this value, at the moment I need the prediction?')
print('For exit_survey_sent the answer is no, and the model is void.')
Open this example in the editor and change it. Nothing you type is saved.
3. Simulate the prediction moment. Build the feature table using only data with a timestamp before the prediction point, and re-score. Any feature that cannot be built that way is leakage by definition.
import numpy as np
import pandas as pd
rng = np.random.default_rng(17)
events = pd.DataFrame({
'customer': rng.integers(1, 40, 600),
'day': rng.integers(0, 180, 600),
'amount': rng.gamma(2, 30, 600).round(2),
})
PREDICTION_DAY = 120
# Features built ONLY from before the prediction day.
past = events[events['day'] < PREDICTION_DAY]
features = past.groupby('customer').agg(
orders_before=('amount', 'size'),
spend_before=('amount', 'sum'),
last_order_day=('day', 'max'),
).round(2)
features['days_since_last'] = PREDICTION_DAY - features['last_order_day']
print(f'{len(events)} events, {len(past)} before day {PREDICTION_DAY}')
print(features.head(4).to_string())
print()
print(f'max day used: {int(past["day"].max())} < {PREDICTION_DAY}')
print('Every feature is computable at the prediction moment. That is')
print('the only test that catches all five shapes of leakage.')
Open this example in the editor and change it. Nothing you type is saved.
Target encoding, done safely
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.preprocessing import TargetEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
rng = np.random.default_rng(19)
n = 4000
category = rng.integers(0, 120, n) # high cardinality
effect = rng.normal(0, 1.0, 120)[category]
y = ((effect + rng.normal(0, 1.0, n)) > 0).astype(int)
frame = pd.DataFrame({'category': category, 'noise': rng.normal(0, 1, n)})
folds = StratifiedKFold(5, shuffle=True, random_state=0)
# WRONG: encode using the whole dataset, then cross-validate.
leaky = frame.copy()
leaky['category_rate'] = pd.Series(y).groupby(frame['category']).transform('mean').to_numpy()
leaked_score = cross_val_score(HistGradientBoostingClassifier(random_state=0),
leaky[['category_rate', 'noise']], y,
cv=folds, scoring='roc_auc').mean()
# RIGHT: TargetEncoder inside the pipeline, fitted per fold with its
# own internal cross-fitting.
pipeline = Pipeline([
('encode', ColumnTransformer([('cat', TargetEncoder(), ['category'])],
remainder='passthrough')),
('model', HistGradientBoostingClassifier(random_state=0)),
])
honest_score = cross_val_score(pipeline, frame, y, cv=folds, scoring='roc_auc').mean()
print(f'encoded outside the pipeline: AUC {leaked_score:.3f}')
print(f'encoded inside the pipeline: AUC {honest_score:.3f}')
print()
print('The first number is the one that gets put in a slide. The')
print('second is the one that survives production.')
Open this example in the editor and change it. Nothing you type is saved.
A worked example
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import GroupKFold, cross_val_score, train_test_split
rng = np.random.default_rng(seed=89)
# Subscription accounts, with an event log. The question: will the
# account cancel in the 30 days after the prediction date?
n_accounts = 1200
PREDICTION_DAY = 240
HORIZON = 30
accounts = pd.DataFrame({
'account': np.arange(n_accounts),
'plan': rng.choice(['free', 'pro', 'team'], n_accounts, p=[.5, .35, .15]),
'signed_up_day': rng.integers(0, 200, n_accounts),
})
rows = []
for account, signed_up in zip(accounts['account'], accounts['signed_up_day']):
health = rng.normal(0, 1)
for day in range(int(signed_up), 300):
if rng.random() < 0.25 + 0.08 * health:
rows.append({'account': account, 'day': day,
'kind': rng.choice(['login', 'lesson', 'ticket'],
p=[.6, .3, .1]),
'value': rng.gamma(2, 10)})
accounts.loc[accounts['account'] == account, 'health'] = health
events = pd.DataFrame(rows)
# The label: cancelled in the window AFTER the prediction day.
cancel_risk = -1.4 - 0.9 * accounts['health'] + rng.normal(0, 0.6, n_accounts)
accounts['cancelled_in_window'] = (cancel_risk > 0).astype(int)
# A field that is only filled in once an account cancels - the leak.
accounts['cancellation_reason'] = np.where(
accounts['cancelled_in_window'] == 1,
rng.choice(['price', 'unused', 'competitor'], n_accounts), None)
print(f'{n_accounts:,} accounts, {len(events):,} events')
print(f'cancellation rate {accounts["cancelled_in_window"].mean():.1%}')
print()
# ---- 1. Build features from BEFORE the prediction day only. --------
past = events[events['day'] < PREDICTION_DAY]
recent = past[past['day'] >= PREDICTION_DAY - 30]
features = past.groupby('account').agg(
events_total=('day', 'size'),
active_days=('day', 'nunique'),
last_day=('day', 'max'),
mean_value=('value', 'mean'),
).reset_index()
features['days_since_last'] = PREDICTION_DAY - features['last_day']
recent_counts = recent.groupby(['account', 'kind']).size().unstack(fill_value=0)
recent_counts.columns = [f'recent_{c}' for c in recent_counts.columns]
features = features.merge(recent_counts.reset_index(), on='account', how='left').fillna(0)
# A ratio: recent activity against the account's own history.
features['recent_share'] = (
features[[c for c in features.columns if c.startswith('recent_')]].sum(axis=1)
/ features['events_total'].clip(lower=1)).round(3)
data = accounts.merge(features, on='account', how='inner')
data['tenure_days'] = PREDICTION_DAY - data['signed_up_day']
data['plan_code'] = data['plan'].map({'free': 0, 'pro': 1, 'team': 2})
print(f'max event day used: {int(past["day"].max())} (prediction day {PREDICTION_DAY})')
print(f'{len(data):,} accounts with features')
print()
HONEST = ['events_total', 'active_days', 'days_since_last', 'mean_value',
'recent_login', 'recent_lesson', 'recent_ticket', 'recent_share',
'tenure_days', 'plan_code']
LABEL = 'cancelled_in_window'
# ---- 2. Score the honest feature set. ------------------------------
folds = GroupKFold(5)
honest_auc = cross_val_score(
HistGradientBoostingClassifier(random_state=0), data[HONEST], data[LABEL],
cv=folds, groups=data['account'], scoring='roc_auc')
print(f'honest features: AUC {honest_auc.mean():.3f} +/- {honest_auc.std():.3f}')
# ---- 3. Add the leak, and watch it become "excellent". -------------
data['has_reason'] = data['cancellation_reason'].notna().astype(int)
leaky_auc = cross_val_score(
HistGradientBoostingClassifier(random_state=0), data[HONEST + ['has_reason']],
data[LABEL], cv=folds, groups=data['account'], scoring='roc_auc')
print(f'plus the leak: AUC {leaky_auc.mean():.3f} +/- {leaky_auc.std():.3f}')
print()
# ---- 4. The 0.99 rule, and the question that catches it. ----------
if leaky_auc.mean() > 0.97:
print(f'AUC {leaky_auc.mean():.3f} on a behavioural problem: investigate.')
X_tr, X_te, y_tr, y_te = train_test_split(
data[HONEST + ['has_reason']], data[LABEL], test_size=0.3,
random_state=0, stratify=data[LABEL])
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
importance = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
ranked = pd.DataFrame({'feature': X_tr.columns,
'importance': importance.importances_mean.round(4)}
).sort_values('importance', ascending=False)
print(ranked.head(5).to_string(index=False))
print()
print(f'top feature: {ranked.iloc[0]["feature"]}')
print('Question: would I have this value, with this value, on day 240?')
print('cancellation_reason is written when the account cancels, which')
print('is after the prediction point. The answer is no.')
print()
# ---- 5. The honest model, inspected. ------------------------------
X_tr, X_te, y_tr, y_te = train_test_split(
data[HONEST], data[LABEL], test_size=0.3, random_state=0, stratify=data[LABEL])
model = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
importance = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
ranked = pd.DataFrame({'feature': X_tr.columns,
'importance': importance.importances_mean.round(4)}
).sort_values('importance', ascending=False)
print('honest model, feature importance:')
print(ranked.to_string(index=False))
print()
print(f'An AUC of {honest_auc.mean():.3f} is high enough to deserve the')
print('same question asked of ITS top feature - which is tenure_days,')
print('computable on day 240, so it survives. A perfect 1.000 did not.')
Open this example in the editor and change it. Nothing you type is saved.
The structure of that example is the one to reuse on every behavioural model. Fix a prediction day. Build every feature from events strictly before it. Build the label from the window strictly after it. Then any feature that cannot be constructed under that rule is leakage, and the rule catches all five shapes at once.
When it goes wrong
| Symptom | Cause |
|---|---|
| AUC above 0.99 on messy data | Leakage, until proved otherwise |
| One feature dominates the importance | Interrogate it: is it available at prediction time? |
| Excellent offline, useless in production | A feature that is empty or different live |
| A grouped split did not stop the leak | The feature was built from all the labels |
| Cross-validation beats the held-out set | Preprocessing fitted outside the pipeline |
| Target encoding gave a huge gain | Encode inside the fold, with cross-fitting |
| A date feature helped enormously | It may encode the label's time ordering |
| The model degrades month by month | A feature's meaning drifted upstream |
A check you can run
Take your best feature - the one at the top of the importance list - and ask one question: at the exact moment the prediction is needed, would this column exist, and would it hold this value?
If you hesitate, trace where the column is written in the source system. In my experience one project in three has a leak in its top three features, and the model's real performance is several points below what was reported.