What Changed in Deep Learning Was Not the Idea. It Was the Hardware and the Data

The core ideas behind neural networks are old. The artificial neuron dates to the 1940s. Backpropagation, the algorithm used to train essentially every network today, was published in a usable form in the 1980s. Convolutional networks for image recognition worked on handwritten digits in 1989. Then not much happened for twenty years. Neural networks were a respectable but unfashionable corner of machine learning, generally beaten on practical problems by methods that needed less data and less computation. What changed around 2012 was not a conceptual breakthrough. It was that graphics processors turned out to be extremely good at the specific arithmetic neural networks need, and that the internet had produced labelled datasets of a size nobody previously had. The same ideas, given a thousand times the compute and a thousand times the data, started winning competitions by margins that were not close. This history matters because it tells you what deep learning actually is: a family of methods that trade enormous appetite for data and computation against the need to hand-engineer features. Understanding that trade is how you decide whether it is the right tool. What a network is doing A neural network is a stack of layers. Each layer multiplies its input by a matrix of weights, adds a bias, and applies a non-linear function. Stack enough of these and the composition can approximate an extraordinarily wide class of functions. The non-linearity is the part that matters and is easy to skip past. Without it, a hundred stacked layers collapse mathematically into a single linear transformation, and the depth buys nothing at all. The non-linear function between layers is what allows each layer to build on the last rather than merely rescale it. Training works by measuring how wrong the output is, computing how much each weight contributed to that wrongness, and nudging every weight slightly in the direction that reduces it. Backpropagation is the efficient way to compute those contributions through the whole stack via the chain rule, and gradient descent is the nudging. Repeat across millions of examples and the weights settle somewhere useful. That is the whole mechanism. Everything else is architecture, engineering and tuning. Representation learning is the actual advantage In classical machine learning, a human decides which features the model sees. Edges, textures, word counts, ratios — someone looks at the problem and designs the inputs. Deep networks learn the features themselves. In an image network, early layers end up detecting edges and colour gradients, middle layers respond to textures and parts, later layers respond to whole objects. Nobody specified that hierarchy. It emerged because it was a useful way to reduce the error. This is the thing deep learning is genuinely for. Where features are hard to specify — pixels, audio waveforms, raw text — learning them beats designing them, and it beats it decisively. Where features are already meaningful, as in a well-structured table of customer attributes, the advantage largely evaporates, and gradient-boosted trees usually win with far less data and effort. That is the first practical judgement: if a domain expert can write down the features, you probably do not need deep learning. The architectures, and what each assumes Different architectures encode different assumptions about the data, and the assumption is the reason each works. Convolutional networks assume that nearby inputs are related and that a pattern is meaningful regardless of where it appears. A filter detecting an edge is slid across the whole image, sharing the same weights everywhere. That weight sharing is why they need far fewer parameters than a fully connected network for images, and why they generalise. Recurrent networks process a sequence one step at a time, carrying a state forward. They were the standard for language and time series for years, and their weakness is structural: information has to pass through every intervening step, so long-range dependencies fade, and the sequential nature prevents parallel training. Transformers replaced them for most purposes. The attention mechanism lets every position look directly at every other position in one step, so distance costs nothing and the whole sequence can be processed in parallel. That parallelism is what made training on internet-scale text feasible, and it is the architectural reason large language models exist. Overfitting is the default, and regularisation is the response A network with millions of parameters can memorise its training set outright. Left alone, it will. The countermeasures are a standard toolkit. Dropout randomly disables units during training, so the network cannot rely on any single path and is forced to build redundant representations. Weight decay penalises large weights. Early stopping halts training when validation performance stops improving, which is the simplest and often the most effective. Data augmentation generates new training examples by transforming existing ones — rotating, cropping, adjusting colour — which is enormously effective for images because it encodes a true fact about the world: a rotated cat is still a cat. Batch normalisation was introduced to stabilise training and also regularises as a side effect. It is one of several techniques whose theoretical explanation is still debated while its practical value is not. Transfer learning is why this is accessible at all Training a large network from scratch needs data and compute that most people do not have. Transfer learning removes that barrier almost entirely. A network trained on a very large general dataset has learned features that are broadly useful — edges, shapes, textures for images; grammar and semantics for text. You take that pretrained network, replace the final layer with one suited to your task, and train on your own much smaller dataset. The general features transfer; only the task-specific part is learned fresh. The practical effect is that a problem which would have needed a million labelled images can often be solved with a few thousand. For almost anybody not working at a large lab, this is the default approach rather than an optimisation, and starting from scratch is a choice that needs justifying. The training loop and the things that go wrong in it Deep learning has a characteristic set of failures, and most of them have recognisable signatures. The loss does not decrease at all. Usually the learning rate, the single most important hyperparameter. Too high and training diverges; too low and it crawls. It is the first thing to tune and often the only thing that matters much. The loss becomes not-a-number. Numerical instability — exploding gradients, a logarithm of zero, a division by a near-zero value. Gradient clipping and numerically stable formulations of the loss address most cases. Training loss falls, validation loss rises. Classic overfitting; reach for the regularisation toolkit. Both stay flat and high. The model lacks capacity, the features are uninformative, or there is a bug. A good diagnostic: try to overfit a tiny subset deliberately. If the network cannot memorise twenty examples, something is broken, because memorising twenty examples is trivially within its capacity. That last technique is worth adopting as a habit. It separates "my model is not learning" into "my pipeline is broken" and "my problem is hard", which are very different situations with very different responses. Compute, and what it actually costs Deep learning is expensive in a way that classical methods are not, and it is worth being concrete. Training is dominated by matrix multiplication, which is why graphics processors are used — they perform thousands of these in parallel. Memory is usually the binding constraint rather than speed: a model and its activations must fit, and the batch size is often chosen by what fits rather than what is optimal. Inference is far cheaper than training but is paid every time, forever. A model that costs a few hundred to train and a fraction of a cent per prediction becomes a substantial running cost at scale, and that arithmetic should happen before deployment rather than after the first invoice. There is also an energy and environmental cost to very large training runs that is real and increasingly discussed. Reusing a pretrained model rather than training from scratch is the single largest reduction available, and it is usually also the better engineering decision. Interpretability, and the honest position Neural networks are difficult to interpret. A model with millions of weights offers no readable explanation of why it produced a particular output. Techniques exist — saliency maps showing which pixels mattered, attention visualisations, methods that approximate the model locally with something simpler, counterfactual explanations. They provide genuine insight and they are approximations, and treating them as the model's actual reasoning is a mistake. Where an explanation is legally required or ethically necessary — credit, hiring, medical decisions, anything affecting someone's life — this is a serious consideration and sometimes decisive. A slightly less accurate model you can explain may be the correct engineering choice, and a field that reflexively optimises for accuracy can forget that the decision belongs to the domain rather than to the modeller. Where deep learning is right, and where it is overkill It is the clear choice for images, video, audio, and natural language. For these the alternatives are not close. It is usually the wrong choice for small datasets, for structured tabular data, for problems where a rule exists, and for cases where an auditable explanation is required. The default for tabular data remains gradient-boosted trees, and people reaching for a neural network there are often choosing the more fashionable tool rather than the better one. The question worth asking at the start: is my input something a human expert can describe in features? If yes, start with the simpler method. If no — if the input is raw perception — deep learning is likely the only thing that will work. Batch size, learning rate, and how they interact Two settings dominate the training loop, and they are not independent. The batch size is how many examples contribute to each weight update. Larger batches give a less noisy estimate of the gradient and use the hardware more efficiently; smaller batches introduce noise that acts as a mild regulariser and often generalises slightly better. In practice the batch size is frequently chosen by what fits in memory rather than by what is optimal, which is a legitimate constraint but worth recognising as one. The interaction is that a larger batch generally wants a larger learning rate, because the gradient estimate is more reliable and a bigger step is safe. Changing one without the other is a common reason a configuration that worked on one machine behaves differently on another. Learning rate schedules matter more than people expect. Starting with a short warm-up avoids large destabilising steps while the weights are still random, and decaying the rate as training proceeds allows the model to settle rather than bouncing around a minimum. A fixed rate for the whole run is rarely the best choice and is frequently the difference between a model that trains and one that does not. Embeddings, and why they are the quiet foundation Networks need numbers, and much of the interesting data in the world is categorical: words, products, users, locations. The naive encoding gives each category its own column, which is enormous and carries no notion of similarity — every pair of categories is equally distant. An embedding instead learns a dense vector for each category, positioned so that categories used in similar contexts end up near each other. This is worth dwelling on because it is the mechanism behind a great deal of modern practice. Embeddings are what allow a model to generalise from one category to a similar one it has seen less often. They are what make semantic search possible, since similarity becomes a distance calculation. And they are how language models represent words before any of the interesting processing begins. The property that makes them powerful is that the representation is learned from the task rather than specified, which means it captures whatever similarity actually helps rather than whatever similarity a human thought to encode. Reproducibility is harder here, and worth the effort anyway Deep learning has more sources of non-determinism than classical methods, and a result that cannot be reproduced is a result nobody can build on. Random initialisation, shuffling, dropout and augmentation all draw from random number generators, so seeds must be set for every library involved rather than just the obvious one. Some operations on parallel hardware are non-deterministic by default because the order of floating-point accumulation varies, and forcing deterministic behaviour usually costs some speed. Beyond randomness, the environment matters: library versions, driver versions and hardware can all shift results slightly. The practical standard is to record the seed, the full dependency versions, the data version and the configuration alongside every run, so that a result can be traced to the exact conditions that produced it. Experiment tracking tools exist for precisely this and are worth adopting before the project is large enough to need them. What to practise

1. Deliberately overfit twenty examples. If you cannot, your pipeline is broken, and finding that out early saves days.
2. Vary only the learning rate across several runs and plot the loss curves. You will never again treat it as just another hyperparameter.
3. Fine-tune a pretrained model on a few thousand of your own images, and compare with training the same architecture from scratch.
4. Add augmentation to an image task and measure the effect on validation performance.
5. Compare a neural network with gradient-boosted trees on a tabular dataset, counting the time each took as well as the score. Deep learning rewards engineering discipline more than mathematical sophistication. The mathematics is largely handled by the framework; what remains is building a pipeline you can trust, an evaluation that is honest, and the judgement to know when a simpler method would have done.