The Watermill's Iced Gears: On the Creeping Ilk of Stagnant Metrics

For me, deep winter is a season of watching things slow to a halt. Out by the old creek at the edge of town, the waterwheel that turns so reliably from spring through autumn is locked fast. It’s not a dramatic failure, no shattered timber or broken axle. It’s a gradual conquest: a thin skin of ice at dusk thickens overnight into a clear, immovable casing. The gears are caught mid-motion, the water’s flow muffled under a solid sheet. The whole mechanism is preserved, perfect, and utterly useless.

It’s a sight that always brings me back to the terminal, to the dashboards that glow with a similar, deceptive stillness. We celebrate ‘uptime’ as a binary state—a service is up, or it is down. The green checkmark is our holy grail. But winter teaches a more insidious lesson: a thing can appear operational while its vital processes have ceased. The mill is ‘up’ in the sense that it’s still standing, but the grinding of grain, the very purpose of its existence, has stopped. The metrics that show an ‘OK’ status are like the frozen superstructure—they tell you the machine is present, but nothing of the lifeblood flowing through it.

This is the ilk of stagnant metrics. It’s the health check that pings a server and gets a 200 OK, but never measures if the database connection pool is slowly exhausting, like a pond freezing from the edges inward. It’s the latency graph that shows a flat, acceptable line, masking the gradual degradation of a third-party API that’s beginning to stutter, a sluggishness that hasn’t yet crossed our arbitrary threshold for alarm. We are lulled into a false sense of security by the ice, mistaking the preservation of form for the continuation of function.

Observability, then, is the practice of measuring the flow itself, not just the position of the wheel. It’s the thermocouple dipped below the ice to feel for the current, the sensor that tracks the water’s temperature and velocity, not just the wheel’s rotation. It requires us to instrument the pathways of our services for meaningful work: Are transactions completing? Are messages flowing through queues at the expected rate? Is the response time for a critical user journey consistent, or is it developing a barely perceptible drag?

We must learn from the watermill. The coming thaw will eventually crack the ice and set the gears turning again, but the lost time, the unmilled grain, is a cost already paid. In our systems, we can’t wait for the spring. We need to build our checks not for the stark contrast of summer and winter, but for the subtle, creeping chill that precedes the freeze. We must monitor for the signs of stagnation, for the slowing of the flow, long before the gears of our services lock solid and the silent alarm of a green checkmark becomes the only epitaph for a system that has, for all intents and purposes, already stopped.

Notes & further reading

A few pages I came back to while writing this: