The Bell Ringer's Broken Sequence: On the Echo of a Single Missed Stroke
In the quiet English town of Devizes in 1865, a sequence of events unfolded that speaks directly to our work in observability. It was the year of the Great Devizes Clock Failure, a story not of a catastrophic crash, but of a subtle, accumulating drift that culminated in a profound public disorientation. The town’s clock, a mechanical marvel of its day, was governed by a complex system of weights, gears, and a single, crucial bell. Its reliability was a given, a steady heartbeat for the community.
The mechanism was checked, of course. The clock keeper would make his daily rounds, ensuring the weights were wound and the hands appeared to move. This was the equivalent of a basic health check—a ping to the system that returned a simple ‘okay’. The hands moved, therefore the clock was healthy. But this superficial check missed a deeper, more insidious flaw: a tiny, almost imperceptible slippage in the mechanism that governed the striking of the hour bell.
Each day, the clock lost a few seconds. The hands, moving in their silent dance, hid the truth. The failure was not in the timekeeping itself, but in the event—the bell strike—that made that time known to the entire town. For weeks, the clock’s audible heartbeat fell further and further out of sync with the true hour, a latency growing in plain sight yet completely unobserved. The keeper saw the hands move and assumed all was well, blind to the decaying rhythm of the output.
The town only discovered the problem when the clock struck seventeen times at noon. The accumulated error had become so great that the mechanism’s internal counting sequence had completely corrupted. The single metric of ‘are the hands moving?’ had proven to be a catastrophic false positive. The system was up, but it was utterly wrong. It was generating nonsense, confidently.
This historical hiccup is a perfect allegory for the necessity of synthetic and real-user monitoring in tandem. The clock keeper’s health check was purely synthetic—it verified the mechanism could perform its own internal function. But it lacked any real-user check—a verification against the immutable constant of true solar time. It measured activity, not correctness. It monitored the process, but not the purpose.
Our services are the modern town clock. A health check that only confirms a process is running is like confirming the clock’s hands are moving. It tells us nothing of the logic, the latency, or the integrity of the outcome. We must also listen for the bell. We must have checks that verify not just that the service responds, but that it responds with the right data, in the right sequence, at the right time. We need to measure against an external truth, lest we too, one day, confidently strike seventeen times at noon and wonder why the world has stopped listening.
Notes & further reading
A few pages I came back to while writing this: