Streaming

An answer is already built piece by piece

A language model never conjures a whole answer in one go. It looks at everything built so far, picks the single next piece, attaches it, and picks again. A long answer repeats this hundreds or thousands of times.

So the earlier part of an answer sits fully finished well before the whole thing wraps up. Streaming just refuses to hold onto that finished part — it sends it out the moment it's ready. It isn't manufacturing new speed from nothing; it's simply not holding onto what's already made.

It may look like one character at a time on screen, but the real unit is a piece, so two or three characters sometimes pop out together. A short word landing all at once is exactly that.

The wait for the first character shrinks

Response speed is really two separate numbers: the time until the first piece arrives, and the gap between pieces after that. Turning on streaming shrinks the first number a lot. The second stays the same.

Most of the frustration a person feels comes from that first number. Staring at an empty screen for a few seconds and watching text start flowing right away feel completely different, even when the total wait is similar. When text arrives faster than a person can actually read it, reading it feels almost the same as having waited for the whole thing.

The difference grows with a longer answer. A longer answer takes longer to finish, so holding the whole thing back means a longer empty screen too.

It can be cut off mid-flow

Notice the answer heading the wrong direction while it's streaming in, and it can be stopped right there. Rephrasing the question and asking again is usually faster anyway. Whatever part hadn't been generated yet never gets computed at all, so neither time nor cost gets spent on it.

Programs do the same thing. Take only as much as needed and close the connection, or cut it off the moment a forbidden phrase shows up mid-stream. That reacts faster than waiting for the whole answer and checking it afterward.

Some places it doesn't fit

Streaming barely helps anywhere the whole answer has to be there before a judgment can be made. Checking whether something matches a required format before passing it along, pulling a value out of the answer to feed another program, choosing one out of several candidate answers — cases like these. The wait for the full thing is unavoidable either way, so there's little point letting it trickle onto the screen early.

It adds complexity on the receiving end too. Every arriving piece has to be stitched on, a broken connection has to be recovered, and a final signal has to be caught to know it's done. Unless characters are actually being shown on a screen, receiving the whole thing at once is often simpler.