Top-p Sampling
Drawing only from a candidate pool trimmed from the top
- Top-p lines up candidates for the next word by likelihood and trims the pool from the top down.
- The cutoff isn't a fixed count — it's a running total of likelihood. Add candidates from the top until that total is filled, then stop.
- So how many candidates survive changes case by case — one or two when the answer is obvious, dozens when it's a toss-up.
- Sampling draws from whatever survives the cut; temperature adjusts the gap between the surviving candidates.
- Set it near 1 and almost nothing gets trimmed; lower it and answers get safer but wording starts repeating.
Contents
1The analogy
Pull into an underground parking garage and you don't weigh every open spot as a candidate. You scan from the entrance and draw a line: "only look this far in." You pick a spot within that line and never even glance at a far corner spot past it.
AI can draw a similar line when picking the next word. Line up the candidates from most to least likely, then add them in from the top, one at a time, until a set total is filled — then cut it off there. Whatever's left over loses its candidacy entirely. This trimming the draw pool from the top is top-p.
Draw the same line on two different days and the number of spots inside it can differ. On a nearly full day, maybe only one spot falls inside the line; on a quiet day, more than ten do. It's not the line's position that decides the count — it's how full the garage happens to be that day.
2In detail
Filling from the top, then cutting off
The process runs in three parts. First, line up candidates from most to least likely. Then add them in from the top, one at a time, keeping a running total of their likelihood. Once that total reaches a set share, stop right there and keep only what's been added.
Set the share to 0.9 and it means adding from the top until 90 percent is filled. If the first candidate alone already clears 90 percent, that one candidate is the whole pool. If instead a long line of roughly equal candidates is waiting, even twenty of them might still fall short of 90 percent.
Whatever survives the cut then redivides the shares among just themselves. What was trimmed away gets folded back into the survivors, so the candidate that was originally in second place gets a slightly better shot.
Why the count shifting case by case is the whole point
After "the capital of France is," the next word is essentially fixed already. Wherever the line gets drawn, only one candidate ends up inside it — there's no room for a stray word to sneak in.
After "on weekends I usually," things are different. Dozens of words could plausibly follow, and none of them stand out as obviously wrong. Here, several candidates fall inside the line, so wording ends up varied.
This is exactly why a single top-p value can handle both situations. It narrows itself automatically on obvious spots and widens itself automatically on toss-up ones — the same threshold works fine on a packed day and a quiet day alike.
Different from cutting by a fixed count
There's also a method that just keeps a fixed number of top candidates — always the top 40, say. Simple to build, but it can't read the situation.
Even on an obvious spot, keeping 40 candidates means a pile of awkward ones sit near the bottom. On the flip side, when about a hundred roughly equal candidates are lined up, chopping it down to 40 can cut off perfectly good wording along with the rest.
In the parking picture, that's like deciding ahead of time to only ever check the first five spots — checking just five on an empty day, and still stopping at five on a packed one. Top-p sidesteps this by cutting on a running total instead of a fixed count.
Where to set the value
Most services default to somewhere around 0.9 to 0.95 — a spot that clears out only the very unlikely words while keeping a healthy range of wording. Unless there's a specific reason, it's usually best left alone.
Lowering the value narrows the pool. That helps when facts matter or a document has to follow a fixed format, but push it too low and the same phrases start repeating and the writing gets dull. Set it to 1 and nothing gets trimmed at all — the full candidate pool stays in play.
Moving it together with temperature makes it hard to tell which one caused a change. It's easier to get a feel for it by leaving one at its default and nudging only the other, running the same sentence a few times to compare.
Alongside temperature and sampling
The three sit in one line, each doing a different job. Temperature widens or narrows the gap between candidates. Top-p then trims the bottom off that tidied-up line of candidates. Sampling draws one actual pick from whatever's left.
In the parking picture: temperature decides how differently the scored spots get treated, top-p decides which row to stop checking at, and sampling is parking in one particular spot within that row.
None of the three builds the candidate list itself. The model puts out a fresh list at every single piece, and these settings only decide how that list gets used.
3More precisely
Top-p sampling is also called nucleus sampling. Candidates are sorted by probability from highest to lowest, and the smallest set whose cumulative probability reaches a threshold is kept; the kept candidates' probabilities are then rescaled to sum to one, and the draw happens within that set. Temperature is applied first to shape the probability distribution, and the cutoff happens on top of that already-shaped distribution.
The analogy breaks down in a couple of places. Parking is a one-time decision for the day, but this cutoff happens fresh every single time a piece is chosen — the surviving candidate count can swing from one to twenty within the course of writing a single sentence. And in a garage you can see with your own eyes which spot is closer, while a candidate's likelihood is a value computed inside the model that a person has no way to know in advance, or even guess at reliably. So how much a small change to the value shakes up the result is something that really has to be tried out directly, several times over.
4Try it yourself
5Common misconceptions
It's easy to think top-p sets a fixed candidate count, but actually it cuts by a running total of likelihood, so the surviving count changes from spot to spot.
It's easy to think setting top-p to 1 lets anything come out, but actually it just means nothing gets trimmed — drawing still follows likelihood exactly as before.
It's easy to think lowering both temperature and top-p makes answers more accurate, but actually it just makes wording more predictable, with no guarantee the facts get any better.
7One-line summary
In shortTop-p is like drawing a "only check up to this row" line in a parking garage — it fills candidates in from most likely on down, cuts off once a set share is reached, and only draws from what's left inside the line.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02