Regularization

Chasing error alone lets values grow

Training only ever moves in the direction that lowers error. But try to hit every single point in the data exactly, and the line standing in for the answer has to bend hard, weaving in and out between the points. Bending that much takes some very large values — a large value is what lets a small change in input swing the result wildly.

A model like this scores extremely well on the data it trained on. But feed it something new and it gets things badly wrong, because it traced the noise mixed into the data as if it were the answer. This is exactly the state called overfitting.

A large value also means the result is leaning hard on that one value. Let that one feature shift even slightly and the answer wobbles. That's why a large value is itself a warning sign.

Fining the size of the values

Regularization adds one more item to the loss training measures. Up to now, training only watched "how wrong am I." Now it watches "how large are my values" too, and moves in the direction that brings both down together.

That turns growing a value to shave a little off the error into a losing trade. The model only grows a value when the error drops by a lot, and otherwise keeps it small. The result is a smoother line standing in for the answer, and scores that hold up on data it has never seen.

The same term shows up in the calculation that moves the values too. Every step, an extra pull toward zero gets tacked onto each value. This approach also goes by the name weight decay.

Setting the strength of the fine

How heavy to make the weight is a choice a person makes. Set it too light and the fine barely registers, so values shoot up as if nothing were there. Set it too heavy and the model can't grow its values at all, missing even the real patterns hiding in the data — everything it outputs turns bland and obvious.

The right strength depends on how much data there is and how big the model is. Less data means memorizing is easy, so the fine gets set heavy; plenty of data lets it run lighter. In practice, several strengths get tried and whichever scores best on data held aside gets kept.

Two ways to charge the fine

How you measure the size of the values changes the outcome. Squaring the values before adding them up charges an especially heavy fine on large values. Values shrink overall without hitting exactly zero, and several of them end up sharing the load evenly.

Adding up the values as they are, with no squaring, behaves differently. It pushes values with little use straight down to zero, so the actual number of values in play shrinks. This approach gets picked when the goal is figuring out which features don't matter. The two are sometimes mixed together too.