Hyperparameter

What gets decided before the start button

The most important dial is the size of one step — how far to move the values by, in proportion to how wrong they were. Too big and it overshoots the target and bounces; too small and progress crawls forever. On the washing machine, this is the water pressure.

How many passes to run gets decided in advance too. Too short and it undertrains; too long and it memorizes the data outright. How many examples to group into one chunk also has to be set. A bigger chunk stabilizes the calculation but eats more memory; a smaller one wobbles more, but escapes a shallow dip more easily.

The model's shape is a dial too — how many layers to stack, how many units per layer, what percentage to randomly switch off during training, how hard to press down to keep values from growing too large. None of these get produced by training itself. A person has to write them in before training can even start.

Where training's values and these live apart

Inside a model sits a separate class of numbers that training sets on its own. Those are called parameters. A person never has to touch them, and couldn't anyway — there are hundreds of millions of them or more.

Hyperparameters sit one level above that. If parameters are the clothes tumbling and rinsing inside the drum, hyperparameters are the dials that decide how that whole process happens. There are far fewer of them, usually around ten, and a person weighs each one deliberately. The names get confused often, but the split is clean once you ask who decides the value.

Finding a good combination

Getting it right in one try is hard, because the dials influence each other. Raise the water temperature and you can shorten the wash time; the same way, a bigger step size usually means fewer passes are needed.

So several runs get tried. The simplest approach sets a handful of candidate values for each dial and tries every combination without missing one. When the number of combinations gets too large, values get sampled at random instead — some dials barely affect the outcome anyway, which often makes this the more efficient choice. There's also an approach that narrows down the next combination to try based on how earlier attempts turned out.

The longer a single run takes, the more this needs a strategy: run short trials on a small slice of the data to get a rough feel first, then run the promising combinations in full.

What gets used to judge the winner

After several runs, the last hurdle is deciding what to judge them by. Judging by the score on training data doesn't work — the run that memorized the data always wins. So they get scored on a separate set of data held back specifically for this.

There's a trap here too. Pick a winner using the same held-back set dozens of times, and you end up with a combination that just fits that particular set especially well — effectively memorizing it by hand. So whatever data will play the role of the final exam gets kept completely untouched the entire time the dials are being chosen.