Systems literacy

Fast software rarely does the work twice. It answers from a cache — and hopes the cache is right.

Between a user and the truth sits a ladder of caches — the browser, a CDN at the edge, an in-memory store, the database, cold storage at the bottom. Each rung you can answer from is faster and cheaper than the one below it, but it answers with a saved copy that might be out of date. That trade — freshness for speed and cost — is why software feels quick or slow, and runs cheap or expensive. And its hard problem has a name: knowing the exact moment a cached copy became wrong.

01 — The ladder

One request, five rungs, five very different answers.

A request falls down the ladder until some layer can answer it. A hit high up returns fast and cheap; a miss falls through to the slower, pricier rung below. Toggle layers on and off, set how often each one already has the answer (its hit-rate), then send a request and watch where it stops. The readouts below are the maths over a thousand requests.

All copies fresh A hit bounces the request back up early. A miss keeps falling.
Median latency (p50)
what a typical user waits
Cost per 1,000 requests
compute + egress
Answered from a cache
could be serving a stale copy
◀ Always fresh (origin only) Often stale (cache-heavy) ▶

02 — The numbers

Why "just ask the database" isn't the whole story.

The naive model is one hop: every request recomputes the answer from the origin. Simple, always fresh — and, under load, slow and expensive, because the scarcest resource does all the work. A caching ladder answers most requests from a cheap, fast copy long before they reach the origin. The difference between the rungs isn't a tweak; it's orders of magnitude.

The naive model

"The server just does it"

One hop, one latency, one cost — every request runs the full query against the database. Easy to reason about and always fresh, but the origin does all the work, so it's slow under load and the most expensive way to run.

Great when: traffic is low, or the data changes so often that a cached copy would almost always be wrong anyway.
The ladder

A stack of caches

Most requests are answered by a fast, cheap copy before they ever touch the origin, which only does the real work once per change. Far faster and far cheaper — at the price of sometimes serving a copy that has gone stale.

Great when: the same answers are read far more often than they change — which is most of what software serves.
Rung Answers from Typical latency Cost / 1k Staleness risk
Browser cache the user's own device (RAM / disk)nothing leaves the machine ~1 ms $0 High — hard to purge remotely
CDN edge a point-of-presence near the userhundreds of cities ~20 ms ~$0.02 Medium — bound by its TTL
Application cache in-memory store, in-region (Redis)~1 ms lookup, ~85 ms round-trip ~85 ms ~$0.10 Medium — bound by its TTL
Database SSD + query engine — the source of truth~10 ms query, ~130 ms round-trip ~130 ms ~$1.20 None — always fresh
Cold storage object store / another region / archivecheap to keep, slow to fetch ~200–500 ms ~$4.00 None — always fresh
The raw scale engineers keep in their heads: an L1 cache reference is ~1 ns, main memory ~100 ns, an SSD read ~150 µs, a same-datacenter round trip ~0.5 ms, and a cross-region round trip ~50–150 ms. Scaled so that 1 ns felt like 1 second, that cross-region hop would take roughly four years. Answering one rung higher up isn't a small win — it's a different order of magnitude. Which is why the field only half-jokes that there are two hard things in computer science: naming things, and cache invalidation.
03 — Should you cache it?

Cacheability is a judgement call, not a default.

Whether to cache a piece of data — and for how long — comes down to two questions: how often does it change, and how bad is it to serve a stale copy? For each item below, make the call: don't cache it, cache it briefly, or cache it for a long time. Then see the reasoning.

Item 1 of 6 Score 0
0 / 6

04 — In the wild

Where the ladder shows up — and where it bites.

The same trade-off explains a system's best day and its worst outage. Four scenarios any team building for real traffic will recognise.

The win

A global news site on a CDN

A wire story breaks and traffic spikes to millions of readers in minutes. The same article is served from CDN edges in hundreds of cities; the origin sees only a trickle of cache-fills.

Why it works: read far more than it changes, and identical for everyone. A CDN turns a traffic spike into a non-event and cuts serving cost by orders of magnitude.
The limit

A personalised feed

Every user's home feed is different, so there's no shared copy to cache at the edge — a cache hit for one person would be the wrong content for the next.

The move: personalisation and shared caching pull in opposite directions. Cache the common parts (layout, static assets, the candidate pool), and compute or privately cache only the personal slice per session.
The bug

The price that went stale

A price drop shipped, but a 24-hour cache kept serving the old, lower number. Checkout honoured the cached price and the company undercharged every order for a day before anyone noticed.

The lesson: the fault wasn't caching — it was invalidation. Anything money touches needs an explicit purge on change, not a hopeful TTL and a shrug.
The failure mode

The cache stampede

A hot key expired at midnight. Thousands of requests missed in the same instant and hammered the database in unison to rebuild it — the "thundering herd" — and the origin fell over under the pile-on.

The lesson: don't let everything expire together. Stagger TTLs, refresh ahead of expiry, or let one request rebuild the value while everyone else keeps serving the slightly-old copy.
05 — Go deeper

The ladder is easy to draw. Living with it is the engineering.

"Just add a cache" is a one-line suggestion that quietly signs you up for a staleness problem. Teams that stay fast without shipping wrong answers treat caching as a deliberate trade — layer by layer, with a plan for how each copy gets invalidated. That's the systems judgement we build with the leaders and teams we coach: turning a diagram on a screen into calls you can defend.

Talk to us about coaching
TAKE ONE

Caching trades freshness for speed

Every cache answers faster and cheaper with a copy that might be wrong. The question is never "cache or not?" in the abstract — it's how much staleness this particular data can tolerate.

TAKE TWO

Invalidation is the hard part

Adding a cache is easy; knowing the exact moment every copy became wrong is the part that ships bugs. Decide how each cached thing gets purged before you cache it — a TTL is a guess, not a guarantee.

TAKE THREE

Know your latency numbers

Memory is nanoseconds, SSD microseconds, a same-datacenter hop half a millisecond, a cross-region round trip a hundred. Which rung you answer from can change the user's wait by 100x. The numbers are the design.