What is Token? — one analogy

1The analogy

Nobody eats a foot-long sub sandwich in one bite. You slice it into pieces small enough to chew, then work through them one at a time. An AI reads the same way. It cannot swallow a sentence whole, so the text is cut first and taken in piece by piece. One of those pieces is a token.

The slices are not even. A word the AI has met a million times comes off as one fat piece. Something it rarely sees gets cut thin, into three or four. "Today" arrives in a single bite, while an unusual dish on the menu might arrive as bru, then schetta.

The plate has a limit too. Only so many pieces fit at one time, so when a conversation runs long, the earliest pieces slide off the edge to make room for new ones. That is why an AI can lose track of something you said at the very start while answering perfectly about the last thing you typed.

2In detail

The cutting rule is built on frequency

Nobody wrote the slicing rules by hand. During training, the system scanned an enormous pile of text and looked for letter groups that keep showing up together. Those groups were saved into a list. Anything on the list survives as one whole piece. Anything missing from it gets broken down into smaller pieces that are on the list.

The list usually holds a few tens of thousands of entries. Human language is far richer than that, so rare material always drops to smaller pieces. A very unusual character may even split into two or three fragments on its own.

That is why a common greeting is a single piece while a brand-new slang word or an unfamiliar surname stretches to four. Two sentences of identical length can carry very different piece counts, depending entirely on which words you picked.

Every piece carries a number

Each entry in the list has an index number. When a sentence is cut, every piece is swapped for its number. What the model actually receives is a row of numbers, not letters.

The numbers are name tags for ordering, nothing more. Two neighboring numbers do not mean two related meanings. Meaning appears at the next step, when each number is turned into a bundle of values. That indirect route is also why a model handles a typo so calmly: it works with relationships between pieces, not with the exact shape of the letters.

Writing works piece by piece too

Producing an answer runs the same machinery in reverse. The model looks at everything collected so far, picks the single most suitable next piece, attaches it, and looks again. Text appearing to flow across your screen is really pieces landing one after another.

It may look like typing, but the true unit is the piece, which is why two or three letters sometimes pop out together. A longer answer takes proportionally longer to arrive, and the limit on answer length is counted in pieces rather than characters.

Some languages cost more pieces

In English, a word is usually about one piece. Other languages fare worse. Korean, for one, has a far larger set of characters, and endings shift constantly as they attach, so much of the text never fits the list whole and gets chopped fine.

For the same meaning, Korean tends to spend roughly one and a half to two times as many pieces as English. Less text fits in one go, answers take longer, and costs climb. Any writing system far from the training material's center of gravity pays a similar tax.

Piece counts set the limits and the bill

There is a hard ceiling on how many pieces an AI can hold in view. Once a conversation passes it, the earliest pieces drop out. An assistant that seems to forget a promise it made twenty messages ago is not being careless. Those pieces are simply no longer on the plate.

Charges work the same way. When software calls an AI, the bill counts pieces sent and pieces returned, not characters. A long example pasted in front of your question turns into cost immediately. Most services count input and output separately, so a longer answer means a longer invoice.

3More precisely

A token is a string fragment produced by a tokenizer following a fixed rule set. Because a fragment can be smaller than a word and larger than a letter, these are called subwords. The list of fragments is built by repeatedly merging the character pairs that appear together most often in the training data, a method known as Byte Pair Encoding, or pair merging.

The sandwich comparison breaks down in one important place. A sandwich is cut along ingredients you can see. A tokenizer does not look at meaning at all. It only knows which characters travelled together in the training text. So a cut can land in the middle of a word, a leading space is often folded into the piece that follows it, and a long number frequently splits into two or three chunks for no reason a reader would guess.

Fragment lists are also built separately for each model. The same sentence handed to two different systems can produce different counts, which is why a rule of thumb like "four characters to a piece" only ever holds loosely. If a piece budget matters for your work, run the actual text through the tokenizer that model uses rather than estimating from length.

4Try it yourself

5Common misconceptions

It's easy to think a token is the same thing as a word, but actually one word often splits across several pieces, and several characters can be bundled into one.
It's easy to think counting characters tells you the token count, but actually the count swings with word choice, so two texts of equal length can differ widely.
It's easy to think spaces and punctuation are free, but actually blank space and line breaks are counted as part of the pieces like everything else.

7One-line summary

In shortA token is a bite-size piece of text an AI cuts before reading, and the number of pieces decides how much it can take in and how much you pay.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02