Local Minimum

The rule of moving downhill only

Training checks which direction is downhill from where it stands, then moves a little that way, over and over. The upside of this method is that it never needs to know the whole landscape. Since it only has to read the slope right underfoot, it can handle an enormous number of values just fine.

But reading only what's underfoot has a cost. If every direction from the current spot happens to be uphill, movement stops. Whether there's a much deeper valley somewhere beyond isn't information the slope underfoot carries at all. It's the exact same reason the yard's water can't cross the dip and reach the drain.

You can't tell from inside the puddle

The tricky part of this problem is that it's hard to even notice you're trapped. Loss going quiet and refusing to drop further looks identical to training having finished properly. Just looking at the number on screen, "we've arrived" and "we're stuck here" look the same.

So in practice, people watch the absolute score. If loss has gone quiet but the value is much higher than expected, a puddle becomes the suspect. It's an even clearer signal if the same settings, run several times, land at very different final scores — one run much better than the rest. There's no way to judge from a single low spot alone; it takes comparing several.

Ways to get past the puddle

The first is changing the starting point. Generate several different sets of random starting values, run each one, and keep whichever result came out best. It's the same as pouring water at several different spots in the yard and seeing where it ends up.

The second is adding momentum. Carry forward some of the speed built up from the direction already traveled, and a shallow dip gets crossed by momentum alone instead of stopping the motion. It's the same as a rolling marble carrying enough speed to clear a small groove instead of settling into it.

The third is jostling things on purpose. Rather than using all the data at once, split it into chunks — the gradient comes out slightly different every time, and that wobble helps nudge things out of a shallow spot. Starting with a big step size and shrinking it as training goes on chases the same effect.

Large models tell a different story

If there were only two or three values, the yard picture would hold up exactly. But a real model has millions of values to adjust. For a spot in that space to actually be a genuine puddle, every single one of those millions of directions has to be uphill at once. A spot like that turns out to be rarer than it looks.

What shows up far more often is an in-between spot where some directions go up and others go down. In a place like that, the gradient is nearly zero, so progress stalls for a while — but eventually a downhill direction gets found and the model escapes. That's why when training in a large model appears to stop, it's usually not truly trapped forever — it's just slowed down for a stretch.