The Stonemason's False Plumb: On the Idolatry of Five Nines
There is a sacred chant in our halls of infrastructure, a number spoken with near-religious reverence: 99.999%. Five nines. It is the gleaming spire on the cathedral of reliability, an aspirational summit that promises near-perfect service. To question it feels like heresy. And yet, I find myself watching the stonemasons at work on the old city buildings, and I wonder if we have mistaken the plumb line for the wall itself.
A plumb bob is a simple, weighted truth. It reveals deviation, the subtle lean that can, over time, lead to collapse. For the stonemason, it is an essential tool for ensuring integrity. But the goal is not to achieve a perfect, abstract alignment with the center of the earth. The goal is to build a sound, lasting, and functional wall. The obsession with the tool can blind the builder to the quality of the mortar, the fitness of the stone, the way the whole structure settles into its environment.
So it is with our pursuit of five nines. We focus so intently on this single metric of uptime that we risk building systems that are marvels of theoretical availability but fragile in practice. We engineer for the absence of failure, rather than for resilience in its face. We create complex, interdependent architectures that, like a wall built with perfect alignment but weak joints, might stand impressively until the first real tremor. An obsession with eliminating every second of downtime can lead to systems so rigid and risk-averse that they cannot adapt, cannot be improved, and ultimately serve the metric more than they serve the user.
The Tyranny of the Unbroken Line
This idolatry creates a strange kind of fragility. To protect the unblemished uptime graph, we become hesitant. We delay deploying meaningful improvements for fear of introducing instability. We avoid necessary, but potentially disruptive, maintenance that would strengthen the system in the long run. The service becomes a museum piece, preserved in a state of static perfection while the world around it evolves. The line on the chart remains unbroken, but the service it represents begins a slow, imperceptible decay into irrelevance.
Furthermore, the five nines metric is a grand, high-level average that can obscure a multitude of sins. It says nothing of latency for users in a specific region, nothing of partial failures that degrade the experience without causing a full outage, and nothing of the health of the underlying components. A service can be “up” according to the monitor that pings its front door, while its internal organs are failing one by one. We worship the perfect silhouette of the spire while the gargoyles are crumbling.
Perhaps a more honest goal, a more human goal, is to build systems that know how to fail well. Systems that degrade gracefully, that provide clear communication when things go wrong, and that can be resurrected quickly and predictably. This requires a focus not just on uptime, but on a fuller picture: observability, mean time to recovery, and the quality of the user experience during less-than-ideal conditions. It is the difference between building a wall that never moves and building one that can settle, shift, and still stand strong.
The stonemason’s true skill lies not in slavish devotion to the plumb line, but in knowing when the stone is good, the mortar is sound, and the wall will serve its purpose for a hundred years. Our skill should not be the blind pursuit of an abstract number, but the construction of services that are truly, deeply reliable in the messy reality of the world. Let us not be builders who worship the tool, but those who understand the craft.
Notes & further reading
A few pages I came back to while writing this: