The Carpenter's Unused Level: On the Hubris of the Perfectly Plumbed Beam

There’s a piece of received wisdom in our craft that borders on dogma: every service must be monitored. Every endpoint must be probed. Every metric must be scraped. We build elaborate systems of health checks, convinced that a perfect, continuous stream of ‘200 OK’ is the ultimate sign of a job well done. We are the carpenters, and our services are the beams we’ve painstakingly plumbed, checked and re-checked with our digital levels. But I fear we’ve become so focused on ensuring the beam is straight that we’ve forgotten to ask if the entire structure is sound.

This obsession with perfect, observable plumbness creates a dangerous illusion of control. We watch our dashboards, a mosaic of green tiles, and we feel a sense of security. The service is up. The latency is low. The health checks are passing. But what are we really measuring? We are measuring a service’s ability to respond to itself. A health check endpoint is a carefully constructed performance, a tiny, isolated sliver of code designed specifically to report its own vitality. It says nothing of the database connection pool slowly hemorrhaging connections, nor of the third-party API whose gradual slowdown is poisoning our user experience. It tells us the stage lights work, but not if the play is any good.

The Silent Failure of the Superficial

The most pernicious failures are never the loud, catastrophic ones. They are the slow, silent decays that occur just outside the narrow beam of our health checks. A service can be ‘healthy’ while being utterly useless. It can return a ‘200 OK’ to our synthetic ping from a data center three miles away while delivering a crippled, timing-out experience to an entire continent of real users. Our level told us the beam was straight, but the floor is still sloping dangerously for everyone walking on it.

This is the hubris of the perfectly plumbed beam. We have mistaken the tool for the goal. The health check is not the objective; user experience is. The metric is not the truth; it is a proxy for it, and often a poor one. By focusing all our vigilance on these self-reported signals, we risk building a system that is perfectly observable yet completely opaque to the realities of its operation. We see everything we’ve chosen to see and are blind to everything else.

True observability isn’t about adding more checks to the same old endpoints. It’s about humility. It’s about acknowledging that no synthetic test can fully capture the chaotic, complex reality of a production system. It requires us to look beyond the green tiles and listen for the whispers of degradation in the logs we don’t aggregate, the traces we don’t capture, and the user journeys we don’t monitor. Sometimes, you have to put the level down and just see if the door swings shut on its own.

Notes & further reading

A few pages I came back to while writing this: