Mini-Batch

Not one at a time, not all at once either

Correct the values after every single example and steps come fast, but the direction swings wildly each time — one odd example can single-handedly decide where things go. It's inefficient too: processing one at a time barely uses the hardware built to crunch many calculations in parallel.

Go the other way and look at all the data before correcting even once, and the direction is very accurate. But with millions of examples, a single step takes forever, and there's no way to even fit all that data in memory at once.

Chunking gets you both. Averaging within a chunk keeps the direction from swinging wildly, and finishing a chunk means finishing a step, so progress stays quick. Nearly every model trained today runs this way.

What sets the chunk size

The first limit is memory. Computing a chunk all at once means holding both the data and every intermediate result in memory simultaneously. Double the chunk size and the memory needed roughly doubles too. Run out of memory partway through training, and shrinking the chunk size is usually the first fix reached for.

The second factor is wobble. A smaller chunk means the direction shifts a little differently every time, and that wobble isn't necessarily bad — it helps shake the model loose from a mediocre spot, and it sometimes even improves the score on new data. A very large chunk smooths the steps out, but each individual step ends up wasted on less new information.

It moves together with the learning rate

Make the chunk bigger and a single step's direction becomes that much more trustworthy. So it's common to bump up the stride to match. Enlarge the chunk but leave the learning rate untouched, and the fewer steps per lap slow training down.

Training a big model across several devices splits it up the same way — each device handles one chunk and the results get combined. The effective chunk size grows with the number of devices in that setup, so without adjusting the learning rate and the warm-up period to match, training tends to break down early.

Leftover data and shuffling

When the amount of data doesn't divide evenly by the chunk size, the last chunk of the lap comes up short. Sometimes that leftover chunk gets used as-is; sometimes it gets dropped to keep every chunk the same size for the computation. Since the data gets reshuffled and rechunked at the start of every lap, whatever got left out this time lands inside a chunk the next time around.

It also matters that a chunk doesn't end up filled with just one category. A pot that's entirely one kind of tomato skews that whole step in one direction. Shuffling the data well ahead of time handles most of this on its own.

Text of different lengths or images of different sizes need extra handling to line everything in a chunk up to the same size — padding a short sentence with blanks, or cropping images to match. Only once that's done can a chunk actually get computed all together.