What is Checkpoint? — one analogy

1The analogy

Sometimes you have to step away from your knitting mid-scarf. Just pull the needle out and the stitches unravel row by row, forcing you to start the whole scarf over. So instead, the live stitches get transferred one by one onto a spare needle, and a note gets jotted down for how many rows are done.

Do that, and coming back days later still means picking the work back up exactly where it left off — the stitch count and the progress are both still intact. If an earlier point turned out to look better, that earlier spare needle can be pulled out and knitting can resume from there instead. A single spare needle captures that whole moment in time. A checkpoint saved mid-training is that spare needle.

2In detail

Why transfer the stitches partway through

Training a large model can take anywhere from days to weeks. Power goes out, a rented machine's time runs out, a small glitch crashes the program — plenty can go wrong along the way. Without a saved snapshot, all the time and money spent up to that moment vanishes right along with the unraveled stitches.

Being able to roll back is a big reason too. Run training long enough and it's common for things to actually get worse past a certain point. If a snapshot got saved at every round, pulling out the best one is all it takes. Without one, there's no choice but to accept the worse result, or start over completely.

It's useful for branching out into different experiments, too. Copy one saved snapshot several times and continue each copy under a different setting, and several branches can be compared without redoing the early part over and over.

What has to travel together to keep knitting

The most important thing is the values inside the model — the numbers training has arrived at so far. With just this, the model at that moment can already produce answers.

But resuming training needs more than that. Which round it currently is, what the step size is scheduled to be right now, how much momentum the tool moving the values has built up so far — all of that has to travel along with it. Without this information, resuming leaves the step size ballooning back to its starting size or the momentum snapping off, and training wobbles for a stretch. It's the same as transferring the stitches without noting how many rows are done — the pattern comes out misaligned. A snapshot meant for producing answers and one meant for resuming training don't hold the same things.

Spare needles run out fast

A snapshot is heavier than it looks. A model's values can number in the hundreds of millions, so one snapshot can run several gigabytes on its own, and adding everything needed to resume can double or triple that. Save one every round and storage fills up fast.

So a rule gets set: keep only the most recent handful and delete the older ones, while keeping the best-scoring snapshot marked separately, untouched. That's also why the snapshot's name usually carries the round number and the score — with several sitting around later, there's no way to tell which is which, and no way to grab the right one.

What gets handed to someone else

What ends up shared with other people, once training wraps up, is this same kind of snapshot too. For that purpose, the information needed for resuming gets stripped out, keeping just the values to make it lighter — most people who receive it only need it for producing answers.

The reverse happens just as often — picking up a snapshot someone else has published and continuing training on it with your own data. Rather than starting from bare ground, that starts from a point that's already learned a great deal, cutting the time and cost by a lot. When a widely used model gets downloaded and adjusted, what actually lands in hand is one checkpoint of that model.

3More precisely

A checkpoint is a file that saves a model's parameters and its training state at a specific point during training. A snapshot meant for resuming training carries, beyond the parameters, the optimizer's internal state, how far along the learning-rate schedule is, the current epoch and step, and the state of the random number generator. Snapshots meant for deployment usually strip everything down to just the parameters to cut the size.

The comparison breaks down in places too. Move a stitch to a spare needle and it comes off the original needle — but a checkpoint gets copied, so training simply continues right alongside it. And unlike putting the stitches back on and perfectly reviving that exact day, resuming training doesn't retrace the original path down to the last detail. The order data gets shuffled in, and tiny differences in the hardware, cause it to drift slightly. A saved snapshot is better understood as "starting over from the same spot," not "walking the exact same path again." One more difference: a scarf only ever has one set of live stitches at a time, but a training run can branch from a single saved snapshot into several separate continuations, each one exploring a different setting from that same starting point.

4Try it yourself

Netron (model structure viewer) ailearn.space Open an actual deployed model file and see how much sits inside each layer. This tool shows the bundle of values a checkpoint holds, though the rest of what's needed to resume training won't show up on screen

5Common misconceptions

It's easy to think a checkpoint means a finished model, but actually it's a snapshot of some point during training, so both unfinished and finished states count as checkpoints.
It's easy to think saving the values alone is enough to resume training, but actually the round number and the tool's running state have to travel together for training to resume smoothly.
It's easy to think more saved snapshots is always safer, but actually each one is heavy, so without a rule for how many to keep and by what standard, storage fills up first.

7One-line summary

In shortA checkpoint is like transferring live stitches onto a spare needle — kept so training can resume from an interrupted point, or roll back to whichever round scored best.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02