The Unwatched Pot: On the Boil That Never Comes

Conventional wisdom in our field is a chorus of vigilance: watch everything, measure everything, alert on everything. We instrument our services with a thousand tiny gauges, convinced that the sheer volume of data will grant us ultimate control. We treat our systems like a watched pot, staring intently, waiting for the slightest tremor to signal the boil. But what if our constant scrutiny is the very thing preventing the water from heating? What if, in our quest for perfect observability, we have built a system so burdened by its own measurement apparatus that it can no longer function with grace?

The counterintuitive truth is this: the most reliable component in your system might be the one you’ve chosen not to monitor. We are taught that every endpoint must be probed, every latency percentile tracked, and every log line parsed. This creates a significant, often unacknowledged, tax. Each health check is a request, consuming CPU cycles, network bandwidth, and I/O. Each log event requires processing and storage. Each metric scraped adds to the payload. In aggregate, this constant self-observation becomes a non-trivial load, a ghost in the machine that is never accounted for in our capacity planning because we mistake it for the machine itself.

We have conflated observability with reliability, assuming that more of the former automatically begets the latter. But a system choking on its own telemetry is not reliable; it is fragile. It’s the pot that never boils because all the heat is being used to power the thermometer. The pursuit of a perfect record can obscure the very performance it seeks to measure, creating a hall of mirrors where we are no longer watching the service, but only watching our watchmen.

This isn't an argument for blindness, but for thoughtful, minimalist sight. It is a call to audit our monitoring not for coverage, but for cost. Which checks are truly predictive? Which metrics are actionable? Which logs would we actually miss in a crisis? Often, a single, well-placed synthetic transaction that validates a core user journey is worth a thousand internal heartbeats. It measures the system as a user experiences it, not as an engineer imagines its internals. It allows the pot to heat in peace, only signaling us when the water—the thing we actually care about—is ready. Sometimes, the most profound reliability is achieved not by adding another sensor, but by having the courage to turn one off.

Notes & further reading

A few pages I came back to while writing this: