The Unheard Bell: On the Fallacy of the Silent Check

There is a quiet dogma in our practice, a piece of received wisdom so ingrained we rarely question its echo: a monitoring check that finds nothing is a successful check. We configure our probes, our pings, our health endpoints, and we wait for the satisfying chime of a returned status code. Silence is failure; sound is success. But what of the check that rings a bell in an empty room? We hear the ring and call it victory, never stopping to ask if anyone was there to hear it.

We have become masters of listening for the wrong thing. Our systems are designed to detect the crash, the hard failure, the server that has gone completely dark. This is the loud, obvious catastrophe. But what of the softer decays? The service that responds with a cheerful 200 OK, yet serves subtly corrupted data? The API that answers promptly but with a payload so bloated it chokes the client? The database that accepts writes but has fallen seconds, then minutes, behind its primary? These are the failures that happen with the lights still on, the bell still ringing. They are the most insidious because they are, by the crude metric of our uptime monitors, ‘successful’.

This obsession with the binary—up or down, success or failure—creates a dangerous illusion of control. We stare at dashboards of green checkmarks and feel a sense of security, believing we have tamed the chaos of our distributed systems. But this is a mapmaker’s vanity. We have drawn the coastlines with great precision but left the interior lands blank, marked only ‘here be dragons’. The green checkmark is not a testament to health; it is merely a report that the simplest possible question we could ask was answered. It is the lowest bar, and we have mistaken it for the finish line.

The Sound of Something Working

True observability requires us to listen for more than just the bell. It demands that we learn to interpret the quality of its ring. We must move beyond the mere fact of a response and into the nuance of its character. This means measuring not just latency, but the distribution of latency. It means validating not just that JSON is returned, but that the JSON is correct, complete, and timely. It means checks that are not silent pings but full conversations, asking a question and critically evaluating the answer for truth, not just presence.

The unchecked check, the one that rings out into a void we never bother to peer into, is the most dangerous tool in our shed. It grants us false confidence while the real failure—the slow, the corrupt, the degraded—settles in and makes itself at home. Our duty is not just to ensure the bell rings, but to have someone—or something—sitting in the room, attentive, ready to report not just on the sound, but on the meaning of the silence that follows.

Notes & further reading

A few pages I came back to while writing this: