Deep learning interviews test engineering discipline more than mathematics. The framework handles the calculus; what remains is building a pipeline you can trust and an evaluation that is honest.
Expect questions on why depth needs non-linearity, what each architecture assumes about its data, how training actually goes wrong, and whether you know when a simpler method would have been better.
Answer each one out loud before you open it. An answer you have read is not an answer you can give under pressure, which is the only kind that counts.
Q1 What is a neural network doing mathematically?
Each layer multiplies by weights, adds a bias and applies a non-linear function; stacking them composes an approximator.
Without the non-linearity the whole stack collapses into a single linear transformation and the depth buys nothing at all.
What it is really testing. Whether you name the non-linearity as the reason depth works.
Q2 Why did deep learning take off around 2012 rather than earlier?
Graphics processors suited the arithmetic, and labelled datasets reached the required size.
The core ideas are decades older. What changed was a thousand times the compute and a thousand times the data, not a conceptual breakthrough.
What it is really testing. Whether you know it was hardware and data rather than a new idea.
Q3 What is backpropagation?
An efficient application of the chain rule to compute each weight's contribution to the error.
It is the mechanism that makes training feasible; without it, estimating gradients for millions of parameters would be impractical.
What it is really testing. Whether you can connect it to the chain rule rather than treating it as magic.
Q4 What is gradient descent and what are its variants?
Adjusting weights in the direction that reduces the loss - full batch, stochastic or mini-batch.
Mini-batch is the practical default, trading the noise of single examples against the cost of the full dataset.
What it is really testing. Whether you know why mini-batch is the practical compromise.
Q5 What is representation learning and why does it matter?
The network learns its own features rather than being given hand-designed ones.
That is the actual advantage. Where features are hard to specify - pixels, audio, raw text - learning them beats designing them decisively.
What it is really testing. Whether you can say where it beats hand-designed features and where it does not.
Q6 When is deep learning the wrong choice?
When a domain expert can write down the features, or the dataset is small.
On structured tabular data gradient-boosted trees usually win with far less data and effort, which is a frequent misallocation.
What it is really testing. Whether you are willing to recommend trees instead.
Q7 What assumption does a convolutional network encode?
That nearby inputs are related and a pattern means the same thing wherever it appears.
Weight sharing across positions is why they need far fewer parameters than a fully connected network and why they generalise.
What it is really testing. Whether you can state the assumption rather than describing the mechanics.
Q8 Why did transformers replace recurrent networks for sequences?
Attention lets every position see every other in one step, and the whole sequence trains in parallel.
Recurrence forced information through every intervening step, so long-range dependencies faded and training could not be parallelised.
What it is really testing. Whether you name parallel training as well as long-range dependencies.
Q9 What is attention doing conceptually?
Computing a weighted combination of all positions, where the weights depend on relevance.
Distance costs nothing, which is why a transformer handles a dependency across a thousand tokens as easily as across three.
What it is really testing. Whether you can describe it without reciting the formula.
Q10 Why does a transformer need positional encoding?
Attention is order-agnostic, so position must be supplied explicitly.
Without it the model sees a bag of tokens, and sentences with the same words in different orders become indistinguishable.
What it is really testing. Whether you know attention has no inherent notion of order.
Q11 What is the vanishing gradient problem?
Gradients shrink as they propagate back through many layers, so early layers barely learn.
ReLU activations, careful initialisation, normalisation and residual connections all address it in different ways.
What it is really testing. Whether you can name more than one remedy.
Q12 What do residual connections do?
They add the input of a block to its output, giving gradients a direct path backwards.
That is what made very deep networks trainable, because the signal no longer has to survive every intervening transformation.
What it is really testing. Whether you connect them to trainability of very deep networks.
Q13 What is batch normalisation?
Normalising activations within a batch to stabilise and speed up training.
It also regularises as a side effect. Its theoretical explanation is still debated while its practical value is not.
What it is really testing. Whether you are honest that the explanation is still debated.
Q14 What is dropout and why does it work?
Randomly disabling units during training so the network cannot rely on any single path.
It forces redundant representations, which generalise better. It is disabled at inference, which is a common implementation slip.
What it is really testing. Whether you know it must be disabled at inference time.
Q15 What is early stopping?
Halting training when validation performance stops improving.
It is the simplest regulariser and often the most effective, and it requires keeping the best checkpoint rather than the last.
What it is really testing. Whether you mention keeping the best checkpoint rather than the last.
Q16 What is data augmentation and why is it so effective for images?
Generating new examples by transforming existing ones - rotating, cropping, adjusting colour.
It encodes a true fact about the world: a rotated cat is still a cat. The equivalent for text is much harder because meaning is fragile.
What it is really testing. Whether you can explain why it is harder for text than for images.
Q17 What is transfer learning?
Starting from a network trained on a large general dataset and adapting it to your task.
General features transfer, so a problem needing a million labelled images can often be solved with a few thousand.
What it is really testing. Whether you treat it as the default rather than an optimisation.
Q18 When would you freeze layers versus fine-tune everything?
Freeze early layers when data is scarce or similar; fine-tune more when you have data or a different domain.
Fine-tuning everything on a tiny dataset usually destroys the general features you started from.
What it is really testing. Whether you can give a rule based on data size and domain similarity.
Q19 What is the learning rate and why does it dominate?
The step size for weight updates - too high diverges, too low crawls.
It is the first hyperparameter to tune and frequently the only one that matters much to whether training works at all.
What it is really testing. Whether you treat it as the first thing to tune.
Q20 Why use a learning rate schedule?
A warm-up avoids destabilising steps while weights are random; decay lets the model settle.
A fixed rate for the whole run is rarely best and is often the difference between a model that trains and one that does not.
What it is really testing. Whether you know warm-up and decay serve different purposes.
Q21 How do batch size and learning rate interact?
A larger batch gives a less noisy gradient and generally wants a larger learning rate.
Changing one without the other is a common reason a configuration that worked elsewhere behaves differently.
What it is really testing. Whether you know they are not independent settings.
Q22 What does the loss becoming NaN indicate?
Numerical instability - exploding gradients, a log of zero, or a division by something near zero.
Gradient clipping and numerically stable loss formulations address most cases, as does lowering the learning rate.
What it is really testing. Whether you can name specific causes rather than guessing.
Q23 What is the first thing to try when a model will not learn?
Overfit a tiny subset deliberately.
If the network cannot memorise twenty examples, the pipeline is broken rather than the problem being hard - and that is a very different situation.
What it is really testing. Whether you know this diagnostic. It separates pipeline bugs from hard problems.
Q24 What is an embedding?
A learned dense vector for a category, positioned so similar categories are near each other.
It replaces one-hot encoding, which is enormous and treats every pair of categories as equally distant.
What it is really testing. Whether you can say what one-hot encoding fails to capture.
Q25 Why are embeddings foundational to modern practice?
They let a model generalise from one category to a similar one and make similarity a distance calculation.
Semantic search, recommendation and the input layer of every language model all rest on them.
What it is really testing. Whether you connect them to search and recommendation as well as language.
Q26 What is the difference between softmax and sigmoid outputs?
Softmax produces a distribution over mutually exclusive classes; sigmoid gives independent probabilities per label.
Using softmax for multi-label problems forces the probabilities to compete when they should not.
What it is really testing. Whether you know which to use for multi-label problems.
Q27 What is cross-entropy loss measuring?
The difference between the predicted distribution and the true one.
It penalises confident wrong predictions heavily, which is what pushes the model toward calibrated probabilities rather than just correct ranks.
What it is really testing. Whether you know it penalises confident wrong answers specifically.
Q28 Why do validation metrics sometimes beat training metrics?
Because dropout and augmentation are active during training and disabled at validation.
It looks alarming and is usually benign, but it is worth confirming rather than assuming.
What it is really testing. Whether you know the usual benign explanation rather than panicking.
Q29 What is gradient accumulation for?
Simulating a larger batch by accumulating gradients over several small ones before updating.
It lets you train with an effective batch size that would not otherwise fit in memory.
What it is really testing. Whether you know it exists as a memory workaround.
Q30 What usually limits training - compute or memory?
Memory, more often than people expect.
The model and its activations must fit, which is why batch size is frequently chosen by what fits rather than what is optimal.
What it is really testing. Whether you know memory usually binds before speed.
Q31 What is mixed precision training?
Using lower-precision numbers for most operations while keeping a higher-precision copy of the weights.
It roughly halves memory and speeds up training on suitable hardware, with a loss scaler to avoid underflow.
What it is really testing. Whether you know a loss scaler is needed to avoid underflow.
Q32 Why is reproducibility harder in deep learning?
Random initialisation, shuffling, dropout and non-deterministic parallel operations all contribute.
Seeds must be set for every library, and forcing deterministic kernels usually costs some speed.
What it is really testing. Whether you can list the sources rather than blaming the framework.
Q33 What should be recorded alongside every experiment?
The seed, the configuration, the data version and the full dependency versions.
Without them a result cannot be traced to the conditions that produced it, which makes it unusable as a basis for the next decision.
What it is really testing. Whether you record enough to reproduce a result months later.
Q34 How expensive is inference compared with training?
Far cheaper per run, but paid every time and therefore often the larger total cost.
That arithmetic should be done before deployment rather than after the first invoice arrives.
What it is really testing. Whether you have done the total cost arithmetic rather than only training cost.
Q35 What techniques reduce inference cost?
Quantisation, pruning and distillation.
Quantisation lowers numeric precision, pruning removes weights, and distillation trains a small model to imitate a large one.
What it is really testing. Whether you can name all three and what each trades.
Q36 What is knowledge distillation?
Training a smaller model to match a larger model's outputs rather than just the labels.
The larger model's probability distribution carries more information than a hard label, which is why the small model learns better from it.
What it is really testing. Whether you know why soft outputs teach better than hard labels.
Q37 How interpretable are neural networks, honestly?
Poorly - the available techniques are approximations rather than the model's reasoning.
Saliency maps and attention visualisations give genuine insight and should not be presented as an explanation of why.
What it is really testing. Whether you avoid presenting an approximation as the model's reasoning.
Q38 When should interpretability override accuracy?
Where a decision affects someone's life and an explanation is legally or ethically required.
Credit, hiring and medical decisions are the usual examples, and a reflexively accuracy-maximising culture can forget the question exists.
What it is really testing. Whether you accept a lower score where explanation is required.
Q39 What is catastrophic forgetting?
A network trained on a new task losing performance on an earlier one.
It is why continual learning is hard and why fine-tuning on a narrow dataset can degrade general capability.
What it is really testing. Whether you know fine-tuning can degrade general capability.
Q40 What is the honest environmental and cost argument?
Very large training runs are expensive in money and energy, and reusing a pretrained model avoids most of it.
It is usually also the better engineering decision, which makes it a rare case where the responsible choice is the pragmatic one.
What it is really testing. Whether you can make the argument without either dismissing or overstating it.