The Gardener's Watering Can and the Fire Sprinkler: On the Deluge and the Drop of an Alert

In the world of uptime monitoring, we often talk about the systems that trip the alarm, but rarely about the nature of the alarm itself. When a health check fails, an observability metric goes critical, or latency spikes through the roof, what happens next? I find myself thinking of two very different tools for delivering a crucial substance: the gardener's watering can and the ceiling-mounted fire sprinkler. One is for nurture, the other for suppression. In how we alert our teams, we often confuse which we are using, and to our service's peril.

The fire sprinkler is a system of total, automated response. It is triggered by a single, unambiguous condition: heat. When that condition is met, it unleashes a deluge. There is no nuance, no staged escalation. It is designed for a binary, catastrophic reality. In our dashboards, this is the PagerDuty blast at 3 a.m., the all-hands-on-deck SMS, the siren that presumes the house is already on fire. It's for the database cluster going offline, not for the 95th percentile latency creeping from 150ms to 155ms.

The gardener's watering can, by contrast, is a tool of measured, contextual care. The gardener walks the rows, observes the soil, feels the leaves. She sees a plant that is merely thirsty, not aflame. The response is proportional: a gentle shower directed at the roots, not a torrent across the entire garden. This is the alert that lands in a low-priority Slack channel, the subtle annotation on a Grafana graph, the scheduled digest email that says, "The basil in row three looks a bit parched; consider watering it tomorrow." It is information for nurture, not a signal for panic.

The trouble arises when we install sprinklers everywhere and call it observability. Every blip becomes a blaze, every minor hiccup a five-alarm fire. Teams, soaked by constant, overwhelming deluges, develop alert fatigue. They start ignoring the sprinklers, even when real smoke appears. The system, designed for reliability, breeds its own form of chaos through over-reaction. The nurturing signal—the gentle warning that allows for thoughtful, pre-emptive care—is drowned out.

Conversely, the danger of the watering can approach is in its passivity. A gardener who only ever gently mists a smoldering root ball will lose the whole plot. Some conditions truly are infernos and demand an immediate, overwhelming response. The art, then, lies not in choosing one tool, but in knowing precisely which plot of your service garden is which. Is this metric a sign of thirst, or a precursor to combustion? Is this health check failure a wilting leaf, or the first lick of flame?

Our goal should be to design alerting systems that understand the difference. To have the courage to let most things be a gentle, informative trickle into the soil of our team's awareness, reserving the deluge for the genuine, structural emergencies. It’s a shift from monitoring as a system of triggers, to observability as an act of horticultural discernment. We must learn to walk the rows with our can in hand, but know, without a doubt, where the sprinklers are wired and when we must all step back and let them roar.

Notes & further reading

A few pages I came back to while writing this: