The loss function is the definition of the task. The network will minimise exactly what you write down, including the parts you did not mean to write down. Choosing it is the single most consequential decision in a training script, and it is usually made in four seconds by copying a line from a tutorial.
What a loss function has to do
A loss takes a prediction and a truth and returns one number that is smaller when the prediction is better. That is the whole contract. Two extra properties decide whether gradient descent can use it:
- it has to be differentiable, because the optimiser needs a gradient;
- the gradient has to be informative, meaning it points somewhere useful even when the prediction is badly wrong.
Accuracy satisfies neither. It is flat almost everywhere - nudging a weight does not change how many predictions are correct - so its gradient is zero and training goes nowhere. This is why the thing you report is almost never the thing you optimise.
import numpy as np
truth = np.array([1.0, 1.0, 0.0, 0.0])
# Two predictions. The second is better by any sensible reading.
weak = np.array([0.51, 0.52, 0.49, 0.48])
strong = np.array([0.95, 0.90, 0.05, 0.10])
def accuracy(p, y):
return float(((p > 0.5) == (y > 0.5)).mean())
def cross_entropy(p, y):
p = np.clip(p, 1e-9, 1 - 1e-9)
return float(-np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)))
print(f' accuracy cross-entropy')
print(f'weak {accuracy(weak, truth):8.2f} {cross_entropy(weak, truth):13.4f}')
print(f'strong {accuracy(strong, truth):8.2f} {cross_entropy(strong, truth):13.4f}')
print()
print('Accuracy cannot tell the two apart: both get every answer')
print('right. Cross-entropy can, and the gradient it provides is')
print('what moves the weak model towards the strong one.')
Open this example in the editor and change it. Nothing you type is saved.
The shape of each loss
import numpy as np
import pandas as pd
error = np.array([0.0, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0])
def huber(e, delta=1.0):
return np.where(np.abs(e) <= delta,
0.5 * e ** 2,
delta * (np.abs(e) - 0.5 * delta))
table = pd.DataFrame({
'error': error,
'squared (MSE)': (error ** 2).round(2),
'absolute (MAE)': np.abs(error).round(2),
'huber (d=1)': huber(error).round(2),
})
print(table.to_string(index=False))
print()
print('At an error of 10, squared error returns 100 and absolute')
print('error returns 10. One outlier therefore counts as much as')
print('a hundred ordinary examples under MSE, and as much as ten')
print('under MAE. That ratio is the whole difference between them.')
Open this example in the editor and change it. Nothing you type is saved.
Say it in terms of what the model learns: mean squared error fits the conditional mean, mean absolute error fits the conditional median. That is not a stylistic preference, it is a different answer to the question being asked.
import numpy as np
from scipy import optimize
rng = np.random.default_rng(0)
# Ninety-nine ordinary values, and one data-entry mistake.
values = np.concatenate([rng.normal(100, 5, 99), [100000.0]])
def mse_fit(c):
return float(np.mean((values - c) ** 2))
def mae_fit(c):
return float(np.mean(np.abs(values - c)))
best_mse = optimize.minimize_scalar(mse_fit, bounds=(0, 200000), method='bounded').x
best_mae = optimize.minimize_scalar(mae_fit, bounds=(0, 200000), method='bounded').x
print(f'median of the clean 99 {np.median(values[:-1]):10.2f}')
print(f'constant that MSE prefers {best_mse:10.2f}')
print(f'constant that MAE prefers {best_mae:10.2f}')
print()
print('One bad row dragged the MSE answer a thousand units away')
print('from every real observation. MAE did not move. If your')
print('targets can contain a typo, this is the decision.')
Open this example in the editor and change it. Nothing you type is saved.
The table to choose from
| Task | Loss | What it assumes |
|---|---|---|
| Regression, clean targets | Mean squared error | Errors are roughly normal; a big miss is much worse than several small ones |
| Regression, outliers present | Huber, or mean absolute error | Some targets are wrong, and the model should not chase them |
| Regression, want a range not a point | Quantile loss, one head per quantile | You will report an interval and be judged on coverage |
| Binary classification | Binary cross-entropy on logits | You want a calibrated probability |
| Multi-class, one label each | Cross-entropy over softmax | The classes are mutually exclusive |
| Multi-class, several labels each | Binary cross-entropy per class | A photo can be both "beach" and "sunset" |
| Ranking | Pairwise or listwise loss | Only the order matters, not the scores |
| Severe imbalance | Weighted or focal cross-entropy | The rare class is the one you care about |
The row that catches people is the difference between the last two multi-class rows. Softmax forces the outputs to sum to one, so it cannot express "both". Using it for a multi-label problem makes the classes compete, and the model learns to suppress the second correct answer.
import numpy as np
logits = np.array([2.0, 1.9, -3.0]) # two plausible classes, one not
def softmax(z):
shifted = z - z.max()
exponentials = np.exp(shifted)
return exponentials / exponentials.sum()
def sigmoid(z):
return 1 / (1 + np.exp(-z))
print('softmax ', softmax(logits).round(3), ' sums to', round(float(softmax(logits).sum()), 3))
print('sigmoid ', sigmoid(logits).round(3), ' sums to', round(float(sigmoid(logits).sum()), 3))
print()
print('Softmax says: 52% this one, 47% that one, pick one. Sigmoid')
print('says: 88% this one AND 87% that one, which is what a photo')
print('tagged both "beach" and "sunset" actually needs.')
Open this example in the editor and change it. Nothing you type is saved.
Why the loss is computed from logits
Every framework offers a loss that takes raw scores rather than probabilities - BCEWithLogitsLoss rather than a sigmoid followed by BCELoss, CrossEntropyLoss rather than a softmax followed by a log. Taking the shortcut is not a style choice. It is numerical survival.
import numpy as np
logit = -800.0 # a confident network produces these
naive_probability = 1 / (1 + np.exp(-logit))
print('sigmoid(-800) =', naive_probability)
print('log of it =', np.log(naive_probability) if naive_probability > 0 else '-inf')
print()
# The stable form never materialises the probability. For the true
# label 1, the loss is log(1 + exp(-logit)), computed through the
# identity log(1 + exp(x)) = max(x, 0) + log(1 + exp(-|x|)).
def stable_bce_with_logits(logit, label):
return (max(logit, 0.0) - logit * label
+ np.log1p(np.exp(-abs(logit))))
print('stable loss at logit -800, label 1 :',
round(stable_bce_with_logits(-800.0, 1.0), 4))
print('stable loss at logit 800, label 1 :',
round(stable_bce_with_logits(800.0, 1.0), 4))
print()
print('The naive route produced 0.0, then -inf, then NaN for every')
print('weight in the network. The stable route returned 800.0,')
print('which is a large loss - correct, and still differentiable.')
Open this example in the editor and change it. Nothing you type is saved.
This is the single most common cause of a training run that reports nan after a few hundred steps and never recovers. The clipping that people reach for first treats the symptom; using the logit form removes the cause.
Imbalance: three different fixes for three different problems
import numpy as np
rng = np.random.default_rng(1)
# 2% positives.
n = 20000
labels = (rng.uniform(0, 1, n) < 0.02).astype(float)
print(f'{labels.mean():.1%} positive, {int(labels.sum())} of {n}')
print()
# A model that has given up and predicts the base rate for everyone.
constant = np.full(n, labels.mean())
def bce(p, y, positive_weight=1.0):
p = np.clip(p, 1e-9, 1 - 1e-9)
per_example = -(positive_weight * y * np.log(p) + (1 - y) * np.log(1 - p))
return float(per_example.mean())
print(f'unweighted loss of the give-up model {bce(constant, labels):.4f}')
print(f'weighted loss of the give-up model {bce(constant, labels, 49.0):.4f}')
print()
print('Unweighted, giving up costs almost nothing - 0.098 is a')
print('number that looks like success in a log line. Weighting the')
print('positive class by 49 (the ratio of negatives to positives)')
print('makes the same lazy model expensive, which is the point.')
Open this example in the editor and change it. Nothing you type is saved.
The three fixes, and when each is right:
- Class weighting in the loss. The first thing to try. It changes nothing about the data pipeline and is one argument.
- Focal loss, which down-weights examples the model already gets right so the gradient comes from the hard ones. Worth it when the imbalance is extreme and most negatives are trivially easy - the original use was object detection, where almost every image region is background.
- Resampling the data. Powerful and dangerous: oversampling the minority before splitting leaks the same rows into training and validation, and the validation score becomes fiction. Split first, resample the training half only.
import numpy as np
rng = np.random.default_rng(2)
probability = np.linspace(0.01, 0.99, 9)
label = 1.0
standard = -np.log(probability)
focal = -(1 - probability) ** 2 * np.log(probability)
print(' p(true class) cross-entropy focal (gamma=2) ratio')
for p, s, f in zip(probability, standard, focal):
print(f'{p:15.2f} {s:13.4f} {f:15.5f} {f / s:6.3f}')
print()
print('At p=0.99 - an example already solved - focal loss keeps')
print('0.01% of the gradient. At p=0.01 it keeps 98%. The gradient')
print('stops coming from the easy majority.')
Open this example in the editor and change it. Nothing you type is saved.
Optimisers
The loss says where to go. The optimiser decides how to take the step.
- SGD with momentum. Momentum accumulates a running average of past gradients, which carries speed through flat regions and cancels oscillation across a narrow valley. Still the best final result for large vision models, with a tuned schedule.
- Adam. Keeps a per-parameter estimate of the gradient's mean and variance, and divides the step by the latter. The effect is that rarely-updated parameters get large steps and noisy ones get small steps. It works acceptably without tuning, which is why nearly everything starts here.
- AdamW. Adam with weight decay applied to the weights directly rather than folded into the gradient. In Adam proper the decay gets divided by the gradient variance along with everything else, which makes the amount of regularisation depend on how noisy each parameter is - not what anyone intends. Prefer AdamW for anything substantial.
import numpy as np
import pandas as pd
# A narrow valley: cheap to move along, expensive to move across.
# This is the shape that separates the optimisers.
def loss(point):
x, y = point
return 0.05 * x ** 2 + 2.0 * y ** 2
def gradient(point):
x, y = point
return np.array([0.1 * x, 4.0 * y])
def run(kind, steps=60, rate=0.1):
point = np.array([-8.0, 1.5])
velocity = np.zeros(2)
mean = np.zeros(2)
variance = np.zeros(2)
crossings = 0
previous = np.sign(point[1])
for step in range(1, steps + 1):
g = gradient(point)
if kind == 'sgd':
point = point - rate * g
elif kind == 'momentum':
velocity = 0.9 * velocity + g
point = point - rate * velocity
else:
mean = 0.9 * mean + 0.1 * g
variance = 0.999 * variance + 0.001 * g ** 2
corrected_mean = mean / (1 - 0.9 ** step)
corrected_variance = variance / (1 - 0.999 ** step)
point = point - rate * corrected_mean / (np.sqrt(corrected_variance) + 1e-8)
side = np.sign(point[1])
if side != 0 and side != previous:
crossings += 1
previous = side
return point, loss(point), crossings
rows = []
for label, kind, rate in [('sgd, rate 0.1', 'sgd', 0.1),
('sgd, rate 0.5', 'sgd', 0.5),
('sgd, rate 0.55', 'sgd', 0.55),
('momentum, rate 0.1', 'momentum', 0.1),
('adam, rate 0.1', 'adam', 0.1)]:
point, final, crossings = run(kind, rate=rate)
finite = np.isfinite(point).all()
rows.append({'optimiser': label,
'x (started at -8)': f'{point[0]:.3f}' if finite else 'diverged',
'loss after 60 steps': f'{final:.5f}' if finite else 'inf',
'valley crossings': crossings})
print(pd.DataFrame(rows).to_string(index=False))
print()
print('The steep direction has curvature 4, so plain SGD is stable')
print('only below a rate of 2/4 = 0.5. At 0.1 it is stable and has')
print('covered barely half the distance along x. Raising the rate to')
print('buy speed in the flat direction is not available: 0.5 sits')
print('exactly on the boundary and bounces across the valley')
print('forever, and 0.55 explodes.')
print()
print('Momentum gets there, having crossed the valley 13 times on')
print('the way - the oscillation is real, the average cancels it.')
print('Adam crosses twice and lands short, because its step is')
print('bounded by the learning rate no matter how flat the surface')
print('is. That is the trade: Adam is the one that works without')
print('being tuned, not the one that gets there fastest.')
Open this example in the editor and change it. Nothing you type is saved.
Two numbers worth memorising: 3e-4 for Adam, 1e-2 for SGD. They differ by a factor of thirty, so carrying a learning rate across a change of optimiser is a reliable way to break a training run that was working.
Schedules
The learning rate should fall as training proceeds: large steps to get near the minimum, small steps to settle into it.
import numpy as np
import pandas as pd
total = 1000
base = 3e-4
step = np.arange(total)
constant = np.full(total, base)
step_decay = base * 0.1 ** (step // 400)
cosine = base * 0.5 * (1 + np.cos(np.pi * step / total))
warmup_steps = 100
warm_cosine = np.where(
step < warmup_steps,
base * (step + 1) / warmup_steps,
base * 0.5 * (1 + np.cos(np.pi * np.clip(step - warmup_steps, 0, None)
/ (total - warmup_steps))))
sample = [0, 50, 100, 400, 700, 999]
table = pd.DataFrame({
'step': sample,
'constant': [f'{constant[i]:.2e}' for i in sample],
'step decay': [f'{step_decay[i]:.2e}' for i in sample],
'cosine': [f'{cosine[i]:.2e}' for i in sample],
'warmup + cosine': [f'{warm_cosine[i]:.2e}' for i in sample],
})
print(table.to_string(index=False))
print()
print('Cosine is the default worth starting from. The warmup')
print('column is the one that matters for transformers: at step 0')
print('it is a hundredth of the base rate, because the first few')
print('large steps on a randomly initialised attention layer are')
print('what destabilise the run.')
Open this example in the editor and change it. Nothing you type is saved.
A worked example: the same network, four losses
One dataset, one architecture, and only the loss changed. The point is that the loss decides what the model is for.
import numpy as np
import pandas as pd
from sklearn.linear_model import SGDRegressor, LogisticRegression
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(7)
# Delivery times: mostly 20-40 minutes, but 3% of rows are a
# driver who forgot to close the job and logged nine hours.
n = 4000
distance = rng.uniform(1, 12, n)
traffic = rng.uniform(0, 1, n)
true_minutes = 8 + 2.5 * distance + 18 * traffic + rng.normal(0, 3, n)
corrupt = rng.uniform(0, 1, n) < 0.03
minutes = true_minutes.copy()
minutes[corrupt] = rng.uniform(400, 600, corrupt.sum())
# "Late" means over 50 minutes by the TRUE time - 4.6% of rows.
late = (true_minutes > 50).astype(int)
X = np.column_stack([distance, traffic])
(X_tr, X_te, y_tr, y_te, true_tr, true_te,
late_tr, late_te) = train_test_split(
X, minutes, true_minutes, late, test_size=0.3, random_state=0)
print(f'{corrupt.mean():.1%} of rows corrupted')
print(f'median true time {np.median(true_minutes):.1f} min, '
f'median logged time {np.median(minutes):.1f} min')
print()
# ---- 1. Regression: MSE against Huber, scored on the TRUE times.
rows = []
for name, loss_name, extra in [('mean squared error', 'squared_error', {}),
('huber', 'huber', {'epsilon': 1.35})]:
model = SGDRegressor(loss=loss_name, alpha=1e-4, max_iter=5000,
tol=1e-5, random_state=0, **extra).fit(X_tr, y_tr)
predicted = model.predict(X_te)
rows.append({'loss': name,
'median error vs truth': round(float(np.median(np.abs(predicted - true_te))), 2),
'predicted at 5km, traffic 0.2':
round(float(model.predict([[5.0, 0.2]])[0]), 1)})
print(pd.DataFrame(rows).to_string(index=False))
print(f'the true time at 5km, traffic 0.2 is '
f'{8 + 2.5 * 5 + 18 * 0.2:.1f} minutes')
print()
print('Trained on the same corrupted column, MSE is pulled upwards')
print('by the nine-hour rows and over-predicts by six minutes on')
print('every delivery. Huber lands within a minute of the truth.')
print('Neither model was told which rows were wrong.')
print()
# ---- 2. Classification: unweighted against weighted, on "late".
print(f'late deliveries: {late.mean():.1%}')
rows = []
for label, weight in [('unweighted', None), ('balanced', 'balanced')]:
model = LogisticRegression(class_weight=weight, max_iter=2000).fit(X_tr, late_tr)
predicted = model.predict(X_te)
caught = int(((predicted == 1) & (late_te == 1)).sum())
missed = int(((predicted == 0) & (late_te == 1)).sum())
false_alarms = int(((predicted == 1) & (late_te == 0)).sum())
rows.append({'class weight': label,
'accuracy': round(float((predicted == late_te).mean()), 4),
'late caught': caught,
'late missed': missed,
'false alarms': false_alarms})
print(pd.DataFrame(rows).to_string(index=False))
print()
print('The unweighted model has the higher accuracy - 96.8% against')
print('93.8% - and catches 23 of the 59 late deliveries. The')
print('weighted model gives up three points of accuracy and')
print('catches 54 of them, raising false alarms from 3 to 69.')
print('Accuracy ranks them one way and recall the other, and both')
print('rankings are correct. Which model is better is a business')
print('question - what a missed late delivery costs against what a')
print('false alarm costs - and the loss function is where you')
print('answer it.')
Open this example in the editor and change it. Nothing you type is saved.
Reading the loss curve
The curve is the only honest account of what happened. Six shapes and what each means:
| What the curve does | What it means | What to change |
|---|---|---|
| Falls, flattens, validation tracking it | Working | Nothing; train longer or add capacity |
Falls to nan within a few hundred steps |
Learning rate too high, or a loss computed from probabilities rather than logits | Use the logits form; halve the rate; clip gradients |
| Flat from step one | Rate too low, inputs not normalised, labels misaligned, or the output layer does not match the loss | Check one batch by hand before touching the rate |
| Training falls, validation rises | Overfitting | Stop at the turn; add regularisation or data |
| Jagged, no trend | Batch too small, or rate slightly too high | Larger batch, or lower rate |
| Falls then jumps up and recovers repeatedly | Rate too high for the late phase | Add a decay schedule |
The fourth row has a sibling that is harder to read: training loss far below validation loss from the very first epoch. That is rarely overfitting - it is usually a difference in how the two are computed. Dropout is active in training and inactive in evaluation, and batch normalisation uses batch statistics in one mode and running statistics in the other.
Check your understanding
Compute the loss of a deliberately stupid model and make sure it is the number you can derive by hand. For binary cross-entropy, a model that predicts the base rate p for every example has a loss of exactly the entropy of the labels:
import numpy as np
rng = np.random.default_rng(0)
labels = (rng.uniform(0, 1, 50000) < 0.2).astype(float)
rate = labels.mean()
predicted = np.full(len(labels), rate)
measured = float(-np.mean(labels * np.log(predicted)
+ (1 - labels) * np.log(1 - predicted)))
expected = float(-(rate * np.log(rate) + (1 - rate) * np.log(1 - rate)))
print(f'base rate {rate:.4f}')
print(f'measured loss {measured:.6f}')
print(f'label entropy {expected:.6f}')
print(f'agree: {abs(measured - expected) < 1e-9}')
print()
print('Any trained model whose loss is not clearly below this')
print('number has learned nothing. It is the first line to print')
print('in a training script, and almost nobody prints it.')
Open this example in the editor and change it. Nothing you type is saved.
If your model's loss sits at that number after an hour of training, nothing is broken in your optimiser - the model is predicting the base rate, and the problem is upstream, in the features or the labels.