The Signalman's Unlit Torch: On the Alarm That Can't Be Heard

We spend so much time crafting our alerts. We tune thresholds, configure escalation policies, and carefully decide on channels—Slack for the nudge, PagerDuty for the jolt. It’s a satisfying kind of craftsmanship, like building a delicate mechanical bell that rings with perfect clarity the moment a gear slips. But what if the bell’s striker is itself broken? In our focus on the message, we risk neglecting the integrity of the messenger. The most elegant, finely tuned alert in the world is useless if it never leaves the lighthouse. The 'unlit torch,' in this metaphor, is the silent alert channel.

The technique I want to discuss is brutally simple, almost embarrassingly so. It’s the practice of testing your alerting pipeline's failure modes. Not just checking that an alert *can* fire, but verifying that it still *does* fire when a critical component in its own delivery chain fails. We test our services for resilience; we must apply the same logic to our means of observation.

Here’s the concrete how-to. Pick one critical alert—the one that pages you at 3 a.m. for a database outage. Now, meticulously trace its entire path. A latency spike triggers a metric in Prometheus. That fires an alert to Alertmanager. Alertmanager matches it to a routing rule, which sends a notification via a webhook to your paging service. Your paging service then sends an SMS to your phone. Great. Now, what happens if the network segment between your Prometheus server and Alertmanager goes down? Does your paging service get a 'connection refused' and give up? Does it retry? Is there a secondary route? More pointedly, would anyone ever know?

The solution is to regularly inject failures into this pipeline and witness the result. The simplest method is a 'dead man's switch' for your alerting system itself. Create a low-priority, non-paging alert that is guaranteed to fire every single day—a simple cron job that pushes a metric to trigger a 'heartbeat' alert. The content of the alert is irrelevant; its sole purpose is to prove the pipeline is alive. If that heartbeat alert stops arriving at its final destination (your email, a low-traffic Slack channel, a dedicated dashboard), you know your torch has gone out.

This heartbeat is your canary in the coal mine for observability itself. When it fails, it signals a deeper, more insidious problem than any single service outage: the problem of blindness. A database going down is a crisis; your ability to know about the database going down also failing is a catastrophe. The unlit torch is a more profound failure than the fire it was meant to signal. Implementing this practice forces a humbling recognition: our beautiful observability stacks are not ethereal oracles. They are composed of servers, networks, and configurations, all of which are fallible and must be watched by something even simpler, and more fundamental.

Notes & further reading

A few pages I came back to while writing this: