Caching

It's the second time that gets faster

Caching isn't a technique for making computation faster. It's a technique for skipping computation. That's why it does nothing at all for a question nobody has ever asked. The first request gets computed as usual to produce an answer, with one extra step: a copy of that answer gets kept.

The gap opens up starting with the second request. An answer that used to take a few seconds arrives in an instant, and because no request even goes out to the service, usage doesn't tick up either. The more often the same question comes in, the more this difference piles up.

That's why caching's score gets measured as "how often did it hit." If nine out of ten requests came straight off the rack, that's a well-chosen setup; if it's only one out of ten, all you've bought is the trouble of maintaining the rack.

What can actually be kept on the rack

Only a genuinely identical request can be answered with the same thing again, and the bar for "identical" here is fairly strict. A single different character, or even a slightly different start to the conversation, counts as a different request. That's why an answer to a frequently asked question, a summary of a commonly used document, an intro blurb that always goes out with a listing — anything that goes out the same to everyone — fits well on the rack.

An answer that differs from person to person, on the other hand, is useless to keep. A conversation about your own order history, a question about a file you just uploaded — these don't belong. Watch out too for an answer that has someone's personal information mixed into it accidentally getting handed to someone else. Checking whether an answer would go out the same to anyone is the habit that matters here.

There's one exception: rewriting only the front part of a request rather than the whole answer. A service that sends the same long rules or document at the front of every request can keep the state of having already read that front part and reuse it on the next request. The question tacked onto the end still varies person to person, but since the front part is always the same, that alone saves real time and usage.

Deciding when to throw it out is half the job

A kept answer goes stale over time. If the wording of a notice changed but the rack still has the old answer sitting on it, the service starts handing out a wrong answer very quickly. This is the single most common caching accident.

So an expiration gets set. Content that rarely changes gets a long one; content that shifts constantly gets a short one. Sometimes things get wired up so that fixing the original immediately clears out anything related to it. Without mapping out ahead of time what to throw away when something changes, it becomes very hard later to track down where an old answer is still hiding.

The rack only has so many slots

The space where things get kept isn't free either. Once the slots fill up, something has to be pushed out to make room for something new, and usually whatever hasn't been requested in the longest goes first. That's why a question that only comes in once in a while can get kept and still be gone again by the next time it comes up.

The storage spot often overlaps in several places too — inside the user's own device, in front of the service, at the point where the model itself gets called — each layer kept separately for speed, but each one is also another place an old answer can be hiding. That's exactly why it gets confusing which layer's rack needs clearing when a wrong answer keeps showing up.