Latency
The time between sending a request and getting an answer back
- Latency is the waiting time from the moment you send a request until you get an answer.
- It splits into two stretches. Time to the first character and time to the last character get measured separately.
- The first stretch mostly comes from how much text you sent in and how long you wait in line. A long document pasted in front of your question becomes pure waiting.
- The second stretch scales almost exactly with how long the answer is, since it's built one piece at a time.
- Streaming the text onto the screen doesn't change the total time, but it cuts the felt waiting enormously.
Contents
1The analogy
Press an elevator button and you wait twice. First, until the doors open — how long that takes depends on which floor the car is currently on and whether someone called it before you. Second, once you're inside, until you reach your floor — and that part comes down almost entirely to how many floors you're climbing.
The two waits are different animals. Doors opening fast doesn't mean you reach your floor fast, and a long wait to board doesn't stop a one-floor ride from being over in seconds. A crowded lobby drags out the first wait without touching the second at all. Ask an AI something, and the time that passes splits the same way. All of that waiting together is latency.
2In detail
The wait splits into two stretches
The first stretch runs from the moment you send the request until the first character shows up on screen. Not a single character of the answer has been produced yet in this window. The request is traveling to a server, waiting its turn in line, and being read from start to finish.
The second stretch runs from that first character until the answer finishes. This is the stretch where the system keeps building one piece and then another, so it stretches out in direct proportion to how long the answer is.
Two three-second waits can hide completely different problems once you open them up. First character at 2.5 seconds with the rest flashing by instantly, versus starting at 0.5 seconds and trickling out slowly — those call for fixing different things.
Why the first stretch drags
The biggest reason is long input text. Before it can start building an answer, the system has to read through everything you sent, and pasting in several pages of a document eats up real time right there.
The second reason is waiting in line. When requests pile up at the same moment, you wait for your turn. Answers starting unusually slowly during busy hours comes down to this.
The third reason is distance traveled. If a request has to reach a machine on the other side of the planet and come back, that round trip adds on top. It's small next to the actual computing time, but in a program firing off short questions constantly, this share adds up fast.
The second stretch is set by length
An answer gets built one piece at a time, and the time to produce each piece stays roughly constant, so doubling the number of pieces doubles this stretch too. That's why a single line — "keep it short" — visibly cuts the wait.
Some languages need more pieces to say the same thing, which drags the answer out that much slower. Asking for a table or a long list makes the piece count balloon, and you feel the slowdown clearly.
Answers built by working through a long chain of reasoning behave the same way. Pieces are being produced even for the part you never see on screen, so that time gets tacked on too. It buys accuracy at the cost of time, so it's worth setting simple tasks to answer briefly by default.
Handles that shorten the wait
Trimming anything unnecessary out of what you send shortens the first stretch. Cutting down a long instruction block that gets pasted in every time, and only feeding in the passages of a document that are actually relevant, is the surest move.
For the exact same question coming in repeatedly, saving the answer and handing it straight back makes the wait nearly disappear. Not reaching for a large model when a small, light one can handle the job is another lever.
Last is streaming the text out. Instead of building the whole answer and showing it all at once, sending it out as it's produced keeps the total time the same but makes it feel far faster, because there's something to read while you wait.
3More precisely
Latency is the time one request takes from start to finish, and its counterpart is throughput, how many requests get through in that same stretch of time. The two often move in opposite directions. Bundling several requests together and computing them as one batch raises how many get processed per hour by a lot, but each individual request that waited for the batch to fill takes longer to come back.
The analogy breaks down in one spot. The moment you step into an elevator, you already know which floor you're headed to; an AI has no idea how long its answer will run until it starts. So it can't tell you in advance when it will finish. Floor count also never changes, but here, sending the exact same question twice can take different amounts of time — it depends on how busy the server happens to be and how much conversation has piled up already. That's why measuring latency properly means measuring it many times rather than once, and looking at both the middle value and the unusually slow ones. Watching only the average makes it easy to miss a problem that shows up once in every ten tries.
4Try it yourself
5Common misconceptions
It's easy to think a faster internet connection means a faster answer, but actually most of the wait comes from computation and queueing, so connection speed barely moves the needle.
It's easy to think a fast first character means the whole thing is fast, but actually the two stretches move independently, so a quick start with a slow finish is common.
It's easy to think the same question always takes the same time, but actually how busy the system is at that moment can swing it by several times over.
7One-line summary
In shortLatency splits into the wait for the first character and the wait for the whole answer, and knowing which stretch is longer tells you where to start fixing it.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02