The Carpenter's Plumb and the Climber's Headlamp: On Two Ways of Finding True

There is a reason old tools persist. In my grandfather’s workshop, hung on a nail, was a simple plumb bob: a pointed brass weight suspended from a string. It was a tool of absolute reference. Gravity was its only master, and against its unwavering vertical line, any wall, any post, could be judged. It didn't tell you about the humidity in the room or the composition of the plaster; it presented a single, binary truth: straight, or not.

In our world of service reliability, we have our own plumb lines. They are our synthetic transactions, our health checks. We meticulously script a perfect user journey—login, add item to cart, proceed to checkout—and we run it from points across the globe every sixty seconds. This is our absolute. It asserts a fundamental promise: the core path works. The string is taut, the bob points directly to the center of the earth. When it tilts, the alarm is unambiguous. Something is broken. It is the carpenter’s method, establishing truth from a known, fixed point.

But the carpenter works on a controlled site, on a structure meant to be plumb and square. What of the explorer in the un-mapped cave? They have no fixed reference. Their tool is not a plumb line but a headlamp. It casts a beam into the darkness, illuminating only what is directly in front of them: the texture of the rock, a sudden drop, a narrow passage. They move forward by interpreting this immediate, contextual feedback. Their truth is not absolute but relative and emergent.

This is the way of observability. We are not running a scripted perfect journey; we are emitting a cloud of telemetry—logs, metrics, traces—and shining a light on it only when we need to understand the strange, winding path a real user’s request actually took. There is no single ‘straight’ to measure against. Instead, we look for patterns, anomalies, and correlations in the scattered data points. A latency spike in one service coinciding with a memory surge in another, a specific error code appearing only for users in a particular region. The headlamp doesn’t give a yes/no answer; it provides clues for an investigation.

The choice, then, is not about which tool is better, but about the nature of the terrain you’re navigating. The plumb line is essential for validating the integrity of your foundation. You cannot build without it. But when a user reports that ‘the site feels slow,’ or a mysterious failure occurs once in ten thousand requests, you are in the cave. No pre-written script will capture that specific, chaotic journey. You must turn on your headlamp and follow the traces.

Reliability, then, is born from a dialogue between these two disciplines. The plumb line of health checks gives us the confidence that our structure is sound, that the walls we built yesterday are still standing today. The headlamp of observability gives us the humility to admit we cannot foresee every path a user will take or every strange interaction that will occur in the dark. One establishes the known truth; the other helps us discover the truth we didn't know we needed. One is for the architect; the other, for the detective. A robust service needs both the unwavering certainty of the plumb and the adaptive curiosity of the beam.

Notes & further reading

A few pages I came back to while writing this: