The Clockmaker's First Adjustment: On the Necessary Drift of a Slightly Unreliable Timepiece
There is a doctrine in our world of service reliability that is treated as gospel, an unassailable truth upon which entire monitoring philosophies are built: consistency is paramount. We are taught to hunt down every jittery millisecond, to smooth every latency spike, to calibrate our checks until the heartbeat of our systems ticks with the unwavering regularity of a metronome. We strive, in essence, to build the perfect clock. But what if this quest for perfect consistency is leading us astray? What if a completely reliable check is a dangerously misleading one?
The most common advice is to make health checks as deterministic as possible. Use a lightweight, predictable endpoint that returns a simple, static response. The logic is sound: by minimizing variables, you isolate the health of the service itself. A failure of this simple check is a clean, unambiguous signal of a genuine problem. We polish this check until it shines, insulating it from the messy, unpredictable reality of the service it represents. We have built a chronometer that keeps perfect time in a sealed case, safe from the elements. But a service in production is not a sealed case; it is a machine exposed to the real world, subject to the gusts and storms of user traffic, third-party dependencies, and data flow.
This is where the counterintuitive idea emerges. A check that is too reliable, too insulated, can become a liability. It creates a blind spot. The service could be suffering from insidious, real-world ailments—memory leaks triggered by specific query patterns, database connection pools exhausting under load, sluggish API responses from a downstream service—while the pristine, minimal health check continues to report a serene, perfect state. The clock in the sealed case ticks on, oblivious to the fact that the building around it is creaking under a gale.
A more truthful approach, I argue, is to intentionally introduce a small, controlled amount of “unreliability” into your monitoring. Not by making checks flaky, but by designing them to be more representative of actual service behavior. Instead of a simple ping to a /health endpoint, design a check that performs a lightweight but real transaction. Have it execute a read against a database, call an internal service endpoint with a dummy payload, or validate a cache entry. This check will inherently be more “noisy.” Its response time will vary. It might occasionally spike in latency, reflecting genuine, low-level contention in the system. It will drift, like a clock exposed to changes in temperature and humidity.
By observing this drift, you gain a far richer understanding of your service’s true health. A gradual increase in the baseline latency of this “dirty” check is an early warning sign of resource saturation long before a simple ping would ever fail. The occasional, minor spike teaches your alerting system, and your team, what normal variation looks like, preventing alert fatigue when a truly anomalous spike occurs. This slightly unreliable timepiece doesn’t just tell you the time; it tells you about the weather. It forces you to pay attention to the system's behavior in its entirety, not just the isolated pulse of a single, sanitized component. The goal is not to build a perfect clock, but to build a sensitive barometer.
Embracing this philosophy means accepting that monitoring is not about achieving perfect signal purity. It’s about creating a dialogue with your system, one that acknowledges its complexity. A check that never wavers is a check that is likely not listening closely enough. Sometimes, the most reliable indicator of a system’s robustness is its graceful, observable handling of a little bit of planned-for chaos.
Notes & further reading
A few pages I came back to while writing this: