17. Dropout, Weight Decay, and Noise as Teachers

Level: Intermediate · Article 17/50 · Karpathy-style LLM course

A network with enough parameters can memorize its training set almost perfectly, achieving near-zero training loss while learning nothing useful about the underlying pattern you actually wanted. Regularization is the collection of techniques that make memorization harder on purpose, forcing the model toward representations that generalize because the easy, lazy shortcut has been deliberately closed off.

The shortcut regularization is fighting

Gradient descent doesn’t care about “generalization” — it only cares about reducing the loss on the batch in front of it, right now. If the fastest way to reduce that loss is for a handful of neurons to co-adapt into a fragile, highly specific pattern that happens to fit the training examples (including their noise and idiosyncrasies), gradient descent will happily find that solution, because nothing in the raw loss tells it that solution is fragile. The model ends up relying on brittle, overly specific combinations of features rather than robust, broadly useful ones — which is exactly what “overfitting” means mechanistically, not just as a train/val gap on a plot.

Regularization techniques all share one property: they make that lazy, brittle solution harder to reach or more costly, so the optimizer is pushed toward solutions that don’t depend on fragile co-adaptation.

Dropout: random amnesia as training

Dropout randomly zeroes a fraction p of activations at every forward pass during training, independently, at every layer it’s applied to.

import numpy as np

def dropout(x, p, training=True):
    if not training:
        return x
    mask = (np.random.rand(*x.shape) > p).astype(x.dtype)
    return x * mask / (1 - p)   # inverted dropout: rescale to preserve expected value

Because a different random subset of neurons disappears on every forward pass, no single neuron can rely on any specific other neuron always being present to co-adapt with — it has to become independently useful, or the network learns a redundant, distributed representation where multiple neurons carry overlapping information as insurance against any of them vanishing. This forced redundancy is precisely what makes the network more robust: at test time, when nothing is dropped, you get the benefit of that redundancy without paying the cost of the handicap.

The / (1 - p) rescaling in the code above is called inverted dropout, and it matters more than it looks: it keeps the expected value of the output the same during training as it would be with all neurons present, which means you don’t need to touch the layer’s output scale at all at evaluation time — you simply skip the masking and rescaling entirely.

Weight decay, and why AdamW decouples it

Weight decay penalizes large weight magnitudes directly, nudging every parameter toward zero by a small amount on every step, independent of what the gradient says:

θ ← θ - lr * (grad + λ * θ)     # classic L2-regularized SGD

The intuition: large weights let a network represent sharp, highly specific functions of its inputs — exactly the kind of function that memorizes noise. Constraining weight magnitude nudges the network toward smoother, simpler functions that are less able to fit noise precisely, which usually generalizes better.

The subtlety that gives AdamW its name: in classic Adam, if you fold the λ * θ term into the gradient before computing the m/v moving averages (the naive way to add “L2 regularization” to Adam), the adaptive per-parameter scaling from article 15 ends up rescaling the decay term too — parameters with historically large gradients get less effective decay, which isn’t what you want from a weight-magnitude penalty. AdamW decouples the two: it applies weight decay directly to the parameter, outside the adaptive m/v machinery entirely, so decay behaves like a constant, predictable shrinkage regardless of a parameter’s gradient history. This decoupling is now the standard for essentially all serious LLM pretraining recipes.

Why scale and data dominate for LLMs

Here’s the part that surprises people coming from smaller-scale deep learning: at LLM pretraining scale, with datasets of hundreds of billions to trillions of tokens seen only once or a handful of times each, classic overfitting in the small-dataset sense is much less of the story than it is for, say, an ImageNet classifier trained for 90 epochs on a fixed dataset. When a model rarely if ever sees the exact same training example twice, there’s less opportunity for the specific memorize-the-noise failure mode dropout was originally designed to fight. Data scale, data quality (deduplication, filtering), and total training compute dominate final model quality far more than the strength of dropout you dial in.

This does not mean regularization is irrelevant for LLMs — weight decay remains standard in essentially every serious pretraining recipe, dropout is used in some architectures and fine-tuning setups (especially with smaller datasets, like supervised fine-tuning on a few thousand curated examples, where overfitting is a real and immediate risk again), and both still meaningfully affect training stability. The honest summary: regularization is a real, useful tool, but at LLM pretraining scale it is not the lever that separates a good model from a great one. That lever is data.

Failure mode: dropout left on at eval

A specific, easy-to-make bug: forgetting to switch a model into evaluation mode before running inference or computing validation loss, leaving dropout (and batch normalization’s running statistics) in training mode. In PyTorch this is the difference between calling model.train() and model.eval() — a single method call that most people remember to do, and then forget to do again after some later code path re-enables training mode. The symptom is validation or inference outputs that are needlessly noisy and inconsistent — the same input produces slightly different outputs on repeated evaluation calls, because dropout is still randomly zeroing activations, and metrics that look artificially worse than the model actually is, because you’re evaluating a randomly damaged version of the network instead of the full one.

The tell that should make you immediately suspect this bug: if running the same validation batch twice produces two different loss values, something in your model still has training-mode randomness active. Deterministic outputs on a fixed input, at eval time, is a property you should be able to assert and test for, not just hope for.

Exercise

Why must dropout behave differently at train time versus eval time — specifically, why is “just leave it always on” or “just leave it always off” not an acceptable simplification? Concrete check: your explanation should cover both directions — always-on at eval introduces non-determinism and throws away the ensemble-like benefit dropout provides once trained, while always-off during training removes the entire mechanism (random omission preventing co-adaptation) that dropout exists to provide in the first place. Then verify the inverted-dropout rescaling claim directly: run the dropout() function above on a large all-ones array with p=0.5 a thousand times and confirm the average output value across all those runs converges to 1.0, not 0.5 — that convergence is exactly what “preserves expected value” means in practice, not just in the algebra.

← 16. Initialization Is Not Optional
18. RNNs: The Old Kings and Their Pain →

Course hub · All courses