The Brewer's First Stale Wort: On the Stagnation of a Constant Check
In our pursuit of reliability, we have enshrined the constant health check as an unassailable good. We configure our probes to query endpoints every thirty seconds, a rhythmic pulse that assures us our services are alive and well. This cadence has become received wisdom, a default so ingrained we rarely question its purpose. But what if this very constancy, this metronomic ticking, is lulling us into a false sense of security, creating a stale understanding of what ‘healthy’ truly means?
The problem lies not in the check itself, but in its sameness. A probe that hits the same endpoint with the same parameters at the same interval is like a brewer who only ever tastes the wort from the same vat at the same hour. The liquid might consistently meet a narrow definition of ‘acceptable’, but it tells you nothing of the fermentation happening elsewhere in the brewery, the subtle shifts in temperature, or the slow development of off-flavors that haven't yet reached your single sample point. Our systems are dynamic, living entities. Traffic patterns change, cache populations warm and cool, database connections ebb and flow. A static check is blind to this rhythm.
This false constancy breeds a dangerous complacency. We come to trust the green checkmark, the uptime percentage ticking towards a perfect 100. We are monitoring for the absence of failure, not for the presence of health. A service can be profoundly unwell—suffering from rising latency, increasing error rates, or degrading performance—while still returning a simple 200 OK to a basic ‘/health’ endpoint. We have successfully monitored its heartbeat while entirely missing its creeping illness.
True observability requires us to break this monotonous rhythm. Instead of a constant, shallow pulse, we need a symphony of checks that vary in depth, frequency, and perspective. A canary release, a synthetic transaction that walks a critical user journey, a dependency check that runs less frequently but probes deeper into a downstream service—these are the tools that reveal the true state of the system. They are the brewer tasting from every barrel, noting the air pressure, and smelling the yeast.
The goal is not to abandon health checks, but to liberate them from the tyranny of constancy. We must design them to be as dynamic as the systems they monitor. Let us challenge the received wisdom of the thirty-second interval. Sometimes a service needs a check every second; sometimes it only needs one every five minutes. The rhythm should serve the function, not the other way around. Otherwise, we are not brewers nurturing a complex fermentation; we are merely clockmakers, satisfied that the pendulum is still swinging, unaware that the time it tells is increasingly wrong.
Notes & further reading
A few pages I came back to while writing this: