A team with 2,000 tests can ship slower — and more nervously — than a team with 400.
More tests is not more confidence. The shape of a suite decides how fast an answer comes back, how often that answer is a lie, and which kinds of bug it was ever going to find. Those three things trade against each other, and the pyramid is just a name for one particular trade.
Commit before you look.
Two teams work on the same product, and both are proud of their test suites. Team A has 1,800 unit tests and 40 end-to-end tests. Team B has 200 unit tests and 400 end-to-end tests. Both suites are green right now. What is the biggest practical difference between them?
Build a suite and see what it costs you.
Three sliders, three kinds of test. Watch the silhouette change shape, and watch the three things that actually matter move against each other: how long until you get an answer, how often a clean build goes red anyway, and which classes of bug you would have caught.
The numbers behind this model
| Level | Runtime | Flake rate | Saturates at |
|---|---|---|---|
| Unit | 0.01 s | 1 in 1,000,000 | ~400 tests |
| Integration | 0.4 s | 8 in 100,000 | ~80 tests |
| End-to-end | 12 s | 1 in 400 | ~12 tests |
The chance a clean build goes red anyway is
1 − Π(1 − flake)^count. Confidence is modelled per bug class: each level catches a
given class at its own rate (logic bugs 95 % / 40 % / 30 %; contract and wiring
bugs 5 % / 85 % / 70 %; journey and config bugs 0 % / 25 % /
90 %), and coverage within a level saturates as
1 − e−n/k — the hundredth end-to-end test adds far less than the third
did.
These constants are illustrative, not measured from your codebase. The orders of magnitude are the point: an end-to-end test is roughly a thousand times slower and a couple of thousand times flakier than a unit test, and no amount of it substitutes for the levels below.
What is the cheapest test that would have caught this?
Not "which test could catch it" — almost anything can be caught end to end, given enough patience. The question that shapes a suite is which is the cheapest level that reliably would. Call each one before you see the answer.
Called: 0 of 5 · Correct: 0
The pyramid and the cone, side by side.
Both shapes are trying to buy the same thing — confidence that a change is safe. They pay for it in completely different currencies.
| Ice-cream cone | Pyramid | |
|---|---|---|
| Feedback time | Half an hour or more — long enough to context-switch. | Minutes — short enough to stay in the change. |
| False failures | Roughly one clean build in three goes red. | Roughly one in fifteen. |
| Diagnosis | "Something in the checkout journey broke." Now go look. | A named function, a named input, a named expectation. |
| Logic bugs | Caught by accident, if a journey happens to exercise them. | Caught precisely, by the level built for them. |
| Journey bugs | Caught well — this is what the level is for. | Caught well enough, with a fraction of the tests. |
| Effect on behaviour | People re-run until green, and stop reading red. | Red means something, so people act on it. |
Parallelism buys latency, never trust
Sixteen CI machines will take a 33-minute cone down to about two minutes, and the temptation is to call the problem solved. Try it in the builder and watch the second number: the false failure rate does not move at all. You now get an unreliable answer quickly instead of slowly. Hardware fixes the queue; only the shape of the suite fixes the signal.
A test you don't trust is worse than no test
Once a clean build goes red often enough, "re-run failed jobs" becomes a reflex, and at that point the suite costs you its full runtime and returns nothing. A genuine regression can then hide behind three flaky failures for a week and nobody will look. Quarantine a flaky test the day it flakes — out of the blocking suite, into a list with an owner. Deleting it is also a legitimate answer.
The pyramid is a heuristic, not a law
The unit-only preset is a real failure mode too: 94 % of logic bugs caught and about 5 % of wiring bugs, because nothing in that suite ever puts two components in a room together. Some systems — thin services that mostly move data between other systems — genuinely belong closer to an hourglass. The rule that survives is not a ratio, it is buy realism only where nothing cheaper will do. And keep the pre-merge suite under ten minutes, because batch size is the thing that quietly grows when it is not.
How suites go wrong in practice.
None of these start as a decision. Every one of them is the accumulated result of many reasonable choices made one test at a time.
The 40-minute pipeline
The suite grew past a coffee break, so people stopped running it before pushing. Then they started batching changes to amortise the wait — why pay 40 minutes for one commit when you can pay it for six? The batch grew, the blast radius grew with it, and the slow suite ended up making releases riskier rather than safer.
The retry button
CI grows a "re-run failed jobs" habit because a third of failures are noise. It works, mostly. Then a genuine regression sits behind three flaky failures for a week, and every engineer who saw it assumed it was the usual thing. The suite was green the whole time — eventually.
The three-day regression pack
A scripted human pass before every release, run by two people over three days. That is not a cone; it is a cone balanced on nothing at all. It is also the complete explanation for why the release cadence is quarterly — nobody is going to do that fortnightly.
The 90% coverage mandate
A coverage target arrives from above. Coverage duly reaches 92 %, achieved largely by tests that call a function and assert that it did not throw. Line coverage is now excellent and the suite has almost no opinions about behaviour, so the number rose and the confidence did not.
Buy realism only where nothing cheaper will do.
Nobody designs an ice-cream cone. It accretes, one urgent end-to-end test at a time, each one individually justified, until the suite is a 40-minute coin flip nobody believes. Reshaping it is slow work with no demo — which is exactly why it needs to be someone's explicit job. That is part of what we do with the engineering teams we coach.
Talk to us about coachingShape is a bet on speed against realism
Every level buys a different kind of confidence at a different price. The question is never "more tests?" — it is "what is the cheapest thing that would have caught this?"
Flakiness destroys the whole asset
A suite that fails at random costs its full runtime and returns nothing, because people learn to re-run instead of read. Quarantine on the first flake.
Feedback time is a design constraint
A suite slower than attention is a suite that stops being run before merge — and that pushes batch size up, which is where the real risk lives.