Layer Normalization

When the scale drifts, training grinds to a halt

The deeper a network gets, the bigger this problem becomes. If one layer has a habit of inflating numbers to one-and-a-half times their size, that habit compounds over twenty layers into something thousands of times the original. Go the other way, and shrinking a little at every layer leaves almost nothing left by the last layer.

Training works by sending error backward from the last layer toward the first, and the value traveling backward goes through the exact same multiplication. So the earliest layers either receive no signal at all, or a signal so large they can't handle it. If the earliest layers can't learn, the whole point of stacking deep disappears.

Layer normalization stops this multiplication from compounding by resetting the scale back to a fixed range at every single layer. That's how a network can stack dozens of layers deep and still have the first and last layer working with numbers of a comparable size.

The reference point is just one position

The "one position" here means the bundle of numbers a single piece in a sentence is carrying. A language model carries thousands of numbers for every single piece. Layer normalization computes its average and spread using only those thousands of numbers, nothing outside them.

Not looking sideways matters. However many sentences get processed together, however long or short a sentence is, each position calibrates using only its own numbers. That's why it behaves exactly the same whether you feed in a single sentence to get an answer or a whole batch during training.

This trait is especially useful for handling text of wildly varying lengths. Approaches that average across multiple examples get thrown off when short and long pieces of text get mixed together, since the baseline keeps shifting.

Two dials to undo it with

Forcing the scale back into a fixed range can erase differences a model worked hard to build. So two dials get attached right after normalizing. One stretches or shrinks the range back out; the other shifts the whole thing up or down.

These dials aren't set by a person — they get set through training. If keeping a large difference at a particular position turns out to help, the model turns that position's dial up on its own. Normalization isn't throwing information away — it's leveling the starting line and leaving how far to stretch entirely up to the model.

Before the layer, or after?

Early designs attached normalization right after a layer finished its computation. This placement caused a lot of instability early in training once layers got very deep, requiring a slow warm-up with a tiny learning rate before ramping up.

The placement widely used now is the opposite. It normalizes first, before entering a layer, and the layer's result gets added onto the untouched value that traveled through a skip connection. This keeps a clean channel running from start to finish, letting even very deep models train reliably.

Whether training succeeds or fails can hinge on where this exact same device gets placed. That where matters more than the calibration itself is one of the lessons learned from working with deep models.