Attention
A way of deciding how much weight to give each input
- Attention decides how much weight to give each input, then blends them in that proportion.
- The weights are not fixed. They get recalculated every time, based on what is being processed right now.
- The weights always add up to 100 percent. Turn one up and the rest automatically shrink.
- Distant words are reached in a single step. The old weakness of information fading as it passed down a chain is gone.
- Watching more things means the number of pairs to weigh grows by the square. That is why cost climbs fast on long text.
Contents
1The analogy
A radio control room has several microphones feeding in at once: the host's voice, the audience, background music. The engineer does not send them all out at the same volume. The moment the host speaks, the host's fader goes up and the rest come down. When the crowd laughs, the audience mic gets a brief lift. This hand movement is what attention does.
What goes out is one signal, but inside it several microphones are mixed in at different levels. Nothing is picked in isolation — the levels are set and then blended.
The levels shift moment to moment. The host being loud a second ago says nothing about the next second. The knobs move again based on what matters right now.
2In detail
How does the engineer know which fader to raise
The engineer reads the script. Attention works out three things instead. The spot being processed puts out a "here is what I'm looking for" request, and every other spot puts out a "here is what I'm carrying" tag. The better the two match, the higher that spot's fader rises.
What actually gets mixed in is separate again. The tag only decides whether a spot matches the request; what flows out once a fader is raised is the content that spot is holding — like matching a search term to a book's title in a library, then reading the book itself, not the title. All three are pulled from the same source in different ways, and how to pull them was fixed during training, not written by hand.
The weights always add up to 100 percent
The raw scores are not used as-is. They get rebalanced so the total comes to 100 percent, which is why you can read them as "40 percent here, 25 percent there."
This rule creates competition. Give one spot a large share and the rest automatically shrink, the same way every fader can't be maxed at once. So the model is always choosing what to give up and what to watch.
Weight can pile onto one spot or spread thinly across many. Piled up means that one spot was decisive; spread out means the whole surrounding mood was drawn on.
Distant words are just as close
Older methods passed a sentence along one word at a time. Anything from the opening had to survive every step to reach the end, and it faded along the way — the usual cause of losing track of a subject in a long sentence.
Attention removes that chain. From the current spot, every earlier spot is visible directly, so a word twenty positions back sits at the same distance as the word right next to it — any microphone is one reach away. Holding onto a name or a condition from early in a long passage is attention's biggest payoff.
More to watch means a bigger board
The convenience has a price. Every spot watching every spot means that doubling the number of spots quadruples the pairs to weigh. Ten microphones are manageable; ten thousand are a different story.
That is why cost and time climb fast on long documents, and why methods that watch only nearby spots, or a few representative ones, or reorder the computation to save memory, keep appearing.
Reading the weight table
Which word leaned on which is left behind as numbers, so this table is treated as a rare window into what a model was looking at. Pronoun references and sentence structure do sometimes show up in it.
Reading it as direct proof of "this is why the model decided X" is risky, though. A high weight doesn't guarantee it was the reason for the decision — good as a clue, weak as evidence.
3More precisely
Attention works from three bundles of numbers: a query, a key, and a value. It measures how well the query matches each key, scales the result down to fit the size of the vectors involved, then runs it through softmax to turn the scores into weights that sum to one. Those weights are used to blend the values, and that blend is the output. The compute and memory needed grow with the square of the input length, which is why long documents get expensive fast.
When generating text, positions ahead are hidden from view, since referencing a word that hasn't been produced yet would mean copying the answer before it exists.
The analogy also breaks down in a couple of places. A control room has an engineer making judgment calls; attention has no one deciding what matters. Numbers tuned during training multiply and add until the weights fall out on their own, with no one choosing them on purpose. And the engineer actually listens and judges the sound, while attention never understands meaning — it only measures how alike two bundles of numbers are, nothing more.
4Try it yourself
5Common misconceptions
It's easy to think attention picks out the one important thing, but actually it spreads weight across everything and blends by that ratio.
It's easy to think a high weight is the reason for a decision, but actually it's only a usable clue, not a full explanation on its own.
It's easy to think attention removed length limits altogether, but actually weighing every pair means cost still grows by the square as length increases.
7One-line summary
In shortAttention is like a control room's faders: it decides how much weight to give each input every time, then blends them in that proportion before passing the result on.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02