The Gardener's Unraked Soil: On the Value of Overgrown Health Metrics

The received wisdom of observability is a doctrine of neatness. We are taught to define our health checks with surgical precision: an endpoint must return a 200 status code within 200 milliseconds; a database replica must be less than five seconds behind the primary; queue depth shall not exceed 100. These are the tidy, raked paths of our service garden. We monitor the green checks and dash for the red alerts. The philosophy is seductive: a healthy system is one where every metric is perfectly in-bounds, a sterile lawn with no weeds.

But a garden isn't truly alive until it's a little wild. A perfectly raked plot shows no sign of burrowing insects, no hidden compaction, no subtle drift in soil pH. It reveals only the absence of the gross failures we already thought to look for. In the same way, a suite of hyper-specific health checks creates a powerful illusion of control while potentially obscuring a slower, more systemic decay. We become excellent at detecting the snapped branch, but blind to the tree's gradual rot.

The Signal in the Overgrowth

What if the most critical sign of a system's vitality isn't found in the crisp pass/fail of a synthetic probe, but in the unkempt, 'overgrown' metrics we relegate to second-tier dashboards? The 99th percentile latency that creeps by milliseconds each week, like a vine slowly claiming a wall. The gently rising baseline memory footprint of a service after a 'minor' library update. The subtly changing ratio of cache hits to misses that no single alert will ever trigger. This is the unraked soil, rich with data about the real, lived-in state of our systems.

Our obsession with binary health checks trains us, and our escalation policies, to ignore everything that isn't a five-alarm fire. It cultivates a team culture that responds only to sirens, not to the changing scent of the air or the different quality of the silence. A system can be '100% uptime, all checks green' while its architectural debt accrues compound interest, hidden in those very metrics we consider merely 'performance' rather than 'health'. The line between performance degradation and impending failure is a cartographer's fading border—it exists only until you look too closely.

This isn't a call to abandon health checks, but to radically reconsider their role. They should be the fence around the garden, not the entirety of the cultivated space. We must spend more time wandering in the overgrowth: observing trends in the 'non-critical' data, correlating the gentle drift of seemingly unrelated metrics, and asking what the slow tells us that the outage never will. The true resilience of a service is often built in understanding why it hasn't failed yet, not just in knowing the instant it does. Sometimes, the most important thing to monitor is not the path, but what's quietly thriving—or dying—just off the edge of it.

Notes & further reading

A few pages I came back to while writing this: