Validation Set
Data set aside for checking only, never used to train
- A validation set is data that never once goes into training and is used only for checking. Part of what you have gets set aside from the very start.
- While training runs, this data grades it, deciding when to stop and how to adjust the settings.
- If the training score keeps climbing while the validation score turns and falls, that's the signal that memorizing has begun.
- Adjust settings dozens of times while watching the validation score, and the model slowly starts fitting to that data too. That's why a test set gets kept aside separately, to be pulled out exactly once at the very end.
- The set-aside slice has to be picked without any lean toward one kind of data, and overlap between the two sides inflates the score.
Contents
1The analogy
Say you're restoring a hundred faded film photos. You'd lay a few out in front of you and nudge the color settings up and down until they look right. The photos you actually tuned by hand naturally come out looking great — of course they do, you made them that way. So a few get set aside before you even start. Left untouched in a drawer, they get pulled out once the settings feel roughly right, and applied exactly as they are. If the color comes out wrong here, the settings still have a way to go. A photo you personally tuned looking good isn't proof of skill. The photos used to dial in the settings and the photos held back to check need to be split apart before you begin. When training an AI, this slice kept in the drawer is the validation set.
2In detail
Splitting what you'll learn from and what you'll check with, from the start
Feed a model all of your data and then grade it on that same data, and the score always comes out looking great — you're re-grading something you already tuned to look great. So once data is gathered, before training even starts, one chunk gets pulled aside and set off to the side. The share is usually around one in ten, and can go even smaller when there's a lot of data.
The set-aside data never once gets used to adjust the model's values. It's used purely for grading. That way, its score actually points to skill in front of something new. Let even one drop of the checking data leak into training, and from that point on, the score stops measuring skill and starts measuring memory instead.
The decisions the validation score makes on your behalf
Training gets a little better with every pass. Grade against the validation data at the end of each pass and write the score down, and the trend of those scores ends up making several decisions for you. The most common one is when to stop: if the validation score sits still for several passes running, or starts sliding back down, training gets cut off right there.
How many layers to stack, what learning rate to use, how much dropout to apply — values like these get picked with the validation score too. These aren't values the model learns on its own; they're values a person sets ahead of time, so several candidates get built and run, and whichever leaves the best validation score is the one that's kept.
It can end up fitted to the validation data too
Pull the same photo out of the drawer and tweak the settings a hundred times over, and you can end up with values that only work for that handful of photos. Validation data works the same way. Grade against the same data while adjusting settings dozens or hundreds of times, and it's not the model fitting to the data anymore — it's the person building it. The validation score keeps climbing while performance on genuinely new data stops keeping pace.
That's why data gets split three ways instead of two. A share to learn from, a share to check against while training runs, and a share pulled out exactly once, after every decision has already been made. Only the score measured on that last share is a score nobody has fitted to.
How the split is made changes the score
Cutting straight through the data in order is risky. If the data is stacked by date or by category, the checking share can end up loaded with one particular period or one particular kind. That's why it's usually shuffled well first and split at random, sometimes matching category proportions too as it's split.
The same content ending up on both sides is a frequent trap too. Receive the same document from two different sources, or stretch one photo into several slightly altered versions, and something nearly identical to what was trained on ends up mixed into the checking share. That makes the score come out much higher than it should. For data with a time dimension, pulling the checking share from the later stretch matches the real situation more closely.
When data is scarce
With only a few hundred examples on hand, setting aside one in ten already stings, and the score that comes from that thin a slice swings heavily depending on which data happened to land there. In this situation, instead of fixing one slice in place, the data gets split into several pieces and checked against in rotation, with the scores averaged. This is called cross-validation. It's a way to cut down the score's wobble without throwing any data away.
3More precisely
A validation set is a batch of data that's never used to update a model's parameters, only used to pick hyperparameters and decide when to stop training. The standard shape splits data three ways: training data to learn from, validation data for checking along the way, and test data held for the final report. English-language papers sometimes call validation data a dev set, and in some fields the words for validation and test even get used the opposite way from each other, so it's worth confirming which slice a given piece actually refers to.
The analogy breaks down in one place. Restoring photos lets a person look and point out exactly where things went wrong, but a validation set only ever hands back a single number, not what needs fixing. And a drawer full of photos can be checked once and set aside, while a validation set gets re-graded against the same slice dozens of times over the course of training. Getting reused that often means the risk of fitting to that slice builds up right along with it — something a drawer of photos never has to deal with.
4Try it yourself
5Common misconceptions
It's easy to think validation data ends up helping training anyway, so it's fine to fold it in, but actually the moment it's folded in even once, any score measured on it stops reflecting skill and starts reflecting memory.
It's easy to think a high validation score means it'll do just as well in practice, but actually settings adjusted for a long time against the same slice leave that score inflated too.
It's easy to think more validation data is always better, but actually every bit set aside is a bit less available to learn from, so it needs to be balanced at a reasonable point.
7One-line summary
In shortA validation set is the slice put away in the drawer before training even starts, and because it stays untouched, its score is the one that tells you when to stop.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02