Between a user and the truth sits a ladder of caches — the browser, a CDN at the edge, an in-memory store, the database, cold storage at the bottom. Each rung you can answer from is faster and cheaper than the one below it, but it answers with a saved copy that might be out of date. That trade — freshness for speed and cost — is why software feels quick or slow, and runs cheap or expensive. And its hard problem has a name: knowing the exact moment a cached copy became wrong.
A request falls down the ladder until some layer can answer it. A hit high up returns fast and cheap; a miss falls through to the slower, pricier rung below. Toggle layers on and off, set how often each one already has the answer (its hit-rate), then send a request and watch where it stops. The readouts below are the maths over a thousand requests.
The naive model is one hop: every request recomputes the answer from the origin. Simple, always fresh — and, under load, slow and expensive, because the scarcest resource does all the work. A caching ladder answers most requests from a cheap, fast copy long before they reach the origin. The difference between the rungs isn't a tweak; it's orders of magnitude.
One hop, one latency, one cost — every request runs the full query against the database. Easy to reason about and always fresh, but the origin does all the work, so it's slow under load and the most expensive way to run.
Most requests are answered by a fast, cheap copy before they ever touch the origin, which only does the real work once per change. Far faster and far cheaper — at the price of sometimes serving a copy that has gone stale.
| Rung | Answers from | Typical latency | Cost / 1k | Staleness risk |
|---|---|---|---|---|
| Browser cache | the user's own device (RAM / disk)nothing leaves the machine | ~1 ms | $0 | High — hard to purge remotely |
| CDN edge | a point-of-presence near the userhundreds of cities | ~20 ms | ~$0.02 | Medium — bound by its TTL |
| Application cache | in-memory store, in-region (Redis)~1 ms lookup, ~85 ms round-trip | ~85 ms | ~$0.10 | Medium — bound by its TTL |
| Database | SSD + query engine — the source of truth~10 ms query, ~130 ms round-trip | ~130 ms | ~$1.20 | None — always fresh |
| Cold storage | object store / another region / archivecheap to keep, slow to fetch | ~200–500 ms | ~$4.00 | None — always fresh |
Whether to cache a piece of data — and for how long — comes down to two questions: how often does it change, and how bad is it to serve a stale copy? For each item below, make the call: don't cache it, cache it briefly, or cache it for a long time. Then see the reasoning.
The same trade-off explains a system's best day and its worst outage. Four scenarios any team building for real traffic will recognise.
A wire story breaks and traffic spikes to millions of readers in minutes. The same article is served from CDN edges in hundreds of cities; the origin sees only a trickle of cache-fills.
Every user's home feed is different, so there's no shared copy to cache at the edge — a cache hit for one person would be the wrong content for the next.
A price drop shipped, but a 24-hour cache kept serving the old, lower number. Checkout honoured the cached price and the company undercharged every order for a day before anyone noticed.
A hot key expired at midnight. Thousands of requests missed in the same instant and hammered the database in unison to rebuild it — the "thundering herd" — and the origin fell over under the pile-on.
"Just add a cache" is a one-line suggestion that quietly signs you up for a staleness problem. Teams that stay fast without shipping wrong answers treat caching as a deliberate trade — layer by layer, with a plan for how each copy gets invalidated. That's the systems judgement we build with the leaders and teams we coach: turning a diagram on a screen into calls you can defend.
Talk to us about coachingEvery cache answers faster and cheaper with a copy that might be wrong. The question is never "cache or not?" in the abstract — it's how much staleness this particular data can tolerate.
Adding a cache is easy; knowing the exact moment every copy became wrong is the part that ships bugs. Decide how each cached thing gets purged before you cache it — a TTL is a guess, not a guarantee.
Memory is nanoseconds, SSD microseconds, a same-datacenter hop half a millisecond, a cross-region round trip a hundred. Which rung you answer from can change the user's wait by 100x. The numbers are the design.