Loss Function

Folding every mismatch into one number

Once the model produces an answer, it gets compared against the correct one. The gap can show up in different ways — too high, too low, off on one detail or another. A loss function takes all of that and folds it into a single number using a fixed rule.

There's a reason it has to be one number. Comparing two trays to see which is better only works if there's one comparable value. If the crust looks better on one tray and the center looks better on the other, there's no way to decide what to fix next.

Averaging the penalty across an entire batch of loaves, rather than judging one at a time, follows the same logic. The whole tray needs a single grade before the next tray can be planned.

The penalty rules decide the outcome

The same loaf scores differently depending on how the scorecard is written. Some scorecards add up the mismatch directly. Others square it first, then add. Squaring makes small mismatches barely register while big ones jump out. One badly burnt loaf can swamp the other nine.

So choosing the scorecard isn't a matter of taste — it's a policy decision. If a large mismatch has to be avoided at almost any cost, squaring is the right call. If you don't want a handful of odd loaves to swing the grade, adding the raw mismatch is safer.

Back at the bakery, the scorecard changes depending on what customers hate most. In a neighborhood that gets furious over burnt crust, pile the penalty onto burning, and the oven learns to run a little cooler.

Measuring and fixing are different jobs

A loss function's job ends at measuring. It reports a 12 — it doesn't say which way to turn the dial. Deciding which direction to move, and by how much, belongs to the next step.

That puts one condition on the scorecard: a small change should produce a small change in score. If nudging the temperature by one degree makes the score jump abruptly, there's no way to tell which way to turn it. That's why loss functions are written as smooth rules. Counting the number of correct answers looks great to a person, but a small tweak rarely changes that count, so it's useless for training.

The target shapes the rule

If the answer is a number, the mismatch is measured directly — how many millimeters short the rise fell, say. That's the kind of rule used for predicting numeric values.

If the answer is a label, it works differently. Instead of right-or-wrong, it looks at how confident the model was. It weighs how much belief was placed on the correct option, and the biggest penalty goes to a confident wrong answer. Being wrong while unsure hurts less than being wrong while certain. That pushes the model to sharpen its confidence, not just get the label right.

A low score isn't the same as a good result

A scorecard is a human-made stand-in. What you actually want is bread customers call delicious, but all you can measure is crust color and rise. So it's possible to score well and still taste mediocre.

Models run into the same trap. Loss drops visibly, yet the model acts strange in practice. Sometimes it only lowered its penalty on the practice loaves and falls apart on anything new; sometimes it ignores a quality the scorecard never captured. That's why the loss used for training and the metrics people actually care about get tracked separately, side by side.