Perplexity
A score for how many directions a model is torn between
- Perplexity is a single number for how many directions a model is torn between when choosing the next word.
- A score of 2 means it's roughly torn between two directions each time; 50 means roughly fifty. Lower is better.
- Measuring it is simple: give it text and average how well it guesses the next word, start to finish.
- Change the way text is split into pieces, or change the text used to evaluate, and the scores stop being comparable.
- A low score doesn't mean good answers. It can't measure whether something is true or useful.
Contents
1The analogy
Pick up the remote in the evening — how many channels come to mind? A household that watches the same show on the same channel every night just presses one number and stops there. On a night when nobody's decided what to watch, the remote flips between five or six channels while everyone checks the screen. However many channels get flipped through, that's the size of that night's indecision.
AI runs into the same kind of hesitation when settling on the next word. After "the capital of France is," there's essentially one channel. After "on weekends I usually," dozens of channels would work and none of them look out of place. Averaging this hesitation across a whole piece of text into one number is perplexity.
A score of 8 means the model is, on average, flipping between roughly eight channels. Given the same text, a model with a lower score changes the channel less.
2In detail
A scale that turns confusion into a headcount
Probability alone doesn't give much of a feel for things. Saying there's a 0.2 chance of guessing the next word right lands far less clearly than saying it's torn between five directions. Perplexity is a scale that turns confusion into a number that reads like a headcount of directions.
The best possible score is 1 — meaning the next word is always fixed. That never actually happens with real text, since even human writing keeps hitting spots that could reasonably go several ways.
A bigger number means more channels worth checking. Run into an unfamiliar subject or an unfamiliar document format and the score jumps.
The way it's measured is a guessing test
The model gets handed a piece of text and reads it one piece at a time from the start. Right before each new piece, it's asked "what do you think comes next," and the likelihood it had already assigned to whatever piece actually shows up gets checked.
Where the guess lands well, the surprise is small; where it's off, the surprise is large. Average that surprise across the whole text and convert it so it reads like a headcount of directions, and out comes perplexity.
The big advantage of this method is that no one has to grade it by hand. Given just the text, it computes automatically, so it can be watched continuously even while training is still running.
Comparing it fairly takes matching conditions
The number itself matters less than how it gets compared. Scores measured on different text can't be compared to each other — a score from news articles and a score from an old novel start from different difficulty levels entirely.
Comparisons also break down when the rule for splitting text into pieces differs. The same sentence split into fine pieces versus split into chunky ones changes how many guesses even have to happen, and different models usually use different splitting rules.
That's why perplexity fits better as a way to watch one model improve over time than as a way to rank different models against each other. A score that steadily drops as training goes on is a sign that learning is going well.
What this scale can't see
A low score means the model guesses the next word well — it doesn't mean the answers are good. Whether something is true, whether it actually answers the question, whether it's useful to the reader: none of that gets captured in this one number.
Following instructions is the same story. Tuning a model to answer the way people actually ask for often nudges perplexity up a little, even as the answers get noticeably more useful to read. The number gets worse while the experience gets better.
That's why perplexity doesn't get used as a standalone report card these days. It's usually placed alongside human ratings, task-by-task accuracy, and head-to-head comparisons between models.
Where it's actually useful
The most common use is watching training progress. A score that drops and then flattens out reads as a sign that there's nothing more to learn, or that something is stuck. A score that suddenly spikes is a cue to check whether something went wrong with the data.
It's also used when tuning a model for a specific field. Comparing a score measured on medical or legal documents before and after tuning gives a sense of how well it adapted to that field's style of writing.
It's used, too, to gauge how unfamiliar a piece of text is to a model. A document with an unusually high score is likely to be a kind the model never saw much of during training.
3More precisely
Perplexity averages the probability a model assigns across every piece in an evaluation text, then inverts that average so it reads like a headcount of directions. Score every piece perfectly and it comes out to 1; the worse the guessing, the larger it grows. It comes from the same root as the error value training tries to shrink, so it tends to move in step with the training curve.
The analogy breaks down in a few places. A channel count is a whole number, but perplexity comes out as a decimal, like 12.4 — that doesn't literally mean the candidates narrowed down to twelve. It's a headcount converted from how evenly spread the likelihood is, so even with tens of thousands of candidates, a score can still come out small if the weight piles onto just one or two of them. And unlike a channel, this number moves depending entirely on what text was used to evaluate it, so a single figure on its own can't say good or bad. If evaluation text leaked into training by accident, the score drops sharply too — not because the model got better at guessing, but because it had already seen that exact text, and the number alone can't tell the two apart. That's exactly why it's worth checking that training data and evaluation data don't overlap whenever this score comes up.
4Try it yourself
5Common misconceptions
It's easy to think a low perplexity means a good AI, but actually it only means the model guesses the next word well, which is separate from whether the answer is true or useful.
It's easy to think scores can be compared directly between models, but actually the comparison only holds if the evaluation text and the piece-splitting rule match.
It's easy to know perplexity as the name of an AI search tool, but actually it's the name of a scale for measuring language models, and that tool borrowed its name from here.
7One-line summary
In shortPerplexity turns how many channels get flipped through while settling on the next word into a number, and the lower it is, the less a model is torn.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02