What is Compute? — one analogy

1The analogy

Picture a day at a package-delivery warehouse. Ten thousand boxes arrive today, and each one takes five separate touches — scan, sort, load — before it goes out the door. That means fifty thousand touches of work need to happen today. That fifty thousand is compute.

That number doesn't change whether the warehouse is huge or cramped, whether it's staffed by ten people or a hundred. Double the conveyor belts, and the same fifty thousand touches just finish in half a day. Only when twice as many boxes arrive, or an extra touch gets added to the process, does the total amount of work actually grow.

So the warehouse manager tracks two separate numbers: how many touches need to happen today, and how many hours it'll take to get through them. The first number is compute.

2In detail

Work gets counted by multiplications

Most of what happens inside a model is multiplying numbers together and adding them up. Every time a signal passes through a layer, the incoming bundle of numbers gets paired up with that layer's own numbers and multiplied and summed, millions of times over. Add up every layer's share, and you get the total amount of work behind a single answer.

That's why compute isn't measured in seconds or minutes. It's measured by the size of the work itself, the way you'd count boxes. Time is just that size divided by how fast the hardware runs. The same model might take one second on one machine and thirty seconds on another, and the compute is identical either way.

Building it and using it are different orders of magnitude

Using a model is a lot like pulling one box off a shelf and shipping it. One question, one answer, done in a few seconds. Building a model is like reorganizing the entire warehouse over and over. It means producing an answer, measuring how wrong it was, and working backward to fix it, all repeated across the entire dataset countless times.

The fix-it pass takes roughly twice the work of the answer-producing pass. Multiply that by how many times the data gets swept through, and the gap balloons to millions of times over. That's where stories about thousands of computers spending weeks training a single large model come from.

What drives compute up

The first factor is model size. Double the number of trainable numbers, and the multiplication count roughly doubles too. The second is data volume. Sweep through twice as much text, and training takes twice the work.

The third is input length, and this one scales differently. Every piece in a sentence looks at every other piece, so doubling the length nearly quadruples that part of the work. That's why feeding in one very long document costs far more than ten short questions. Getting a longer answer back adds up the same way, since it's one more round of calculation for every extra piece.

Getting the same job done more cheaply

There are two directions: shrink the total amount of work, or speed up the processing. To shrink the work, use a smaller model, distill a large model's knowledge into a smaller one, or use coarser numbers so each multiplication gets lighter. If the same question comes up repeatedly, caching an earlier result and reusing it works too.

Speeding things up means splitting the work across more hardware. But it doesn't split forever. A later layer's calculation needs an earlier layer's result first, so no amount of extra hands can skip that order. Handing pieces of work back and forth between machines also takes time, so past a certain point, adding more hardware stops making much difference.

3More precisely

Compute is usually measured in floating point operations, or FLOPs. The total compute to train a large language model is roughly proportional to the number of parameters multiplied by the number of tokens seen during training. Producing a single answer takes far less — roughly on the order of twice the parameter count.

The analogy breaks down in one place. Delivery boxes can be processed independently of each other, but a neural network's calculations depend on each other in sequence, since one result feeds directly into the next. So doubling the hardware doesn't cut the time exactly in half.

There's another gap worth noting. Often the real reason an answer arrives late has nothing to do with the multiplication count. Pulling numbers out of memory frequently takes longer than the calculation itself, leaving the processing hardware sitting idle waiting for numbers to arrive. Compute is a good yardstick for cost, but it doesn't directly predict speed.

That's also why two systems quoting the same compute figure can still feel completely different to use. One might keep its hardware busy the whole time; the other might spend half its time waiting on memory. The number on the label is only ever half the story.

4Try it yourself

5Common misconceptions

It's easy to think compute is the same thing as how long something takes, but actually the same amount of compute can take wildly different amounts of time depending on the hardware and how many machines it's split across.
It's easy to assume a good graphics card means a big model will run, but actually if there isn't enough room to hold the numbers, it won't even start, no matter how fast the calculation is.
It's easy to think getting an answer back is cheap enough not to worry about, but actually one answer is cheap while hundreds of millions of them add up to far more than training ever cost.

7One-line summary

In shortCompute is the total amount of calculation it takes to run a model, and that amount is what sets both the bill and the wait.

Did this analogy help?

Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02