The Stonemason's First Settled Foundation: On the Danger of a Perfect Level
We are taught, from the very first moment we deploy a service, that consistency is the ultimate virtue. Our dashboards are a sea of flat, green lines. Our pager stays silent. We chase the mythical five nines, believing that a perfectly level foundation, like a stonemason’s dream, is the pinnacle of reliability. But what if this perfect, unwavering state is not a sign of health, but the first symptom of a profound and hidden fragility?
The common wisdom is to eliminate all noise. We tune our health checks to be hyper-specific, filtering out ‘irrelevant’ blips. We set aggressive thresholds, aiming to catch only ‘real’ failures. In doing so, we build a system that only acknowledges a catastrophe. The gentle, daily tremors that test the integrity of the structure—a slightly elevated latency p95, a few extra database connection resets, a cache miss pattern that drifts by a fraction—are smoothed over and ignored. We silence the very whispers that tell us how the system is actually behaving under load, in the real world.
This creates a dangerous illusion of stability. A foundation that never shifts is a foundation we never learn about. We gain no knowledge of its true load-bearing characteristics, its points of flex, or its hidden fault lines. When the inevitable earthquake of a true traffic surge or a cascading failure finally arrives, the system fails in ways we never anticipated because we never observed its smaller, more honest reactions. The perfect level wasn’t strength; it was a lack of feedback.
Embrace the Telltale Tremor
The counterintuitive path to a more resilient service is to deliberately introduce a degree of observable, controlled imperfection. This isn’t about causing outages; it’s about fostering a richer, more honest relationship with your system’s behavior. It means setting health checks that are sensitive enough to occasionally go yellow, prompting investigation into a minor latency increase rather than waiting for a full timeout. It means allowing—and watching—the natural, small-scale failures that occur during normal operation.
This philosophy shifts the goal from preventing all alerts to understanding every alert. A dashboard with occasional, minor fluctuations is a dashboard that is telling you a story. It’s a system that is constantly communicating its state, its stresses, and its limits. This noisy truth is far more valuable than a silent lie. By learning from these small tremors, we can reinforce the structure before the big one hits. We build not for an abstract ideal of perfection, but for the chaotic, imperfect reality of the web. The stonemason knows that a foundation that has settled, that has been tested by the elements and proven its mettle, is infinitely stronger than one that has merely been declared level on a calm morning.
Notes & further reading
A few pages I came back to while writing this: