The Night Watchman's Unlit Lantern: On the Courage of a Silent Alarm

There’s a piece of advice so common in our field it’s practically scripture: when an alarm sounds, drop everything. The pager’s shriek is a siren call to action, a mandate to investigate, to fix, to restore. We build entire philosophies around mean time to resolution, racing to extinguish the smallest flicker of red on a dashboard before it can become a conflagration. But what if this relentless pursuit of immediate response is making our systems more fragile, not less? What if our greatest tool for reliability is not the alarm that screams, but the one we dare to let remain silent?

We are taught to monitor everything that can be measured. Every API endpoint, every database query, every server’s memory footprint. We set thresholds with the precision of a watchmaker, believing that if we can be notified the instant a metric twitches, we can prevent the catastrophe. This creates a world of constant, low-grade anxiety. Alarms become the boy who cried wolf; their urgency is dulled by frequency. We begin to triage not by potential impact, but by the volume of the alert. The truly dangerous failures, the slow, silent ones that don’t trip a simple threshold, are lost in the noise of a thousand petty alerts.

Consider the night watchman of old. His duty was not just to walk his route with a lit lantern, shining a light into every shadow. His deeper responsibility was to listen. To know the difference between the ordinary creak of an old building and the telltale click of a forced lock. By immediately shining his light at every sound, he would blind himself to the deeper darkness, announcing his position and losing the advantage of surprise. His power lay in the disciplined patience to stand in the dark, to assess the nature of the threat before activating his most obvious tool.

Our systems need this same discipline. We need the courage to build monitors that don’t alert by default. Instead of a simple threshold on database latency, what about an alarm that only triggers when latency degrades *and* a specific, high-value business transaction begins to fail? Instead of paging a team for every single server failure in an auto-scaling group, what about an alert that remains quiet until a pattern emerges suggesting the entire deployment is compromised? This requires a more profound understanding of our systems—not just what the metrics are, but what they mean in concert.

This approach is counterintuitive because it feels like neglect. It demands we trust our systems to withstand minor hiccups without our intervention. It forces us to design for resilience from the ground up, rather than relying on human superheroes to swoop in and save the day. The goal shifts from minimizing the time an alarm is active to maximizing the time between truly consequential alerts. It’s the quiet confidence of the watchman who knows that a silent alarm, held in reserve for a genuine emergency, is far more powerful than a dozen false warnings that teach everyone to simply look away.

Notes & further reading

A few pages I came back to while writing this: