The Potter's First Unmeasured Kiln: On the Fallacy of the Constant Glaze
We are taught, from our first foray into running services, that consistency is the ultimate virtue. The green checkmark, the flat latency line, the unwavering 99.99%—these are the hallmarks of a job well done. We build our observability stacks to be exquisitely sensitive barometers, designed to shriek at the slightest deviation from this manufactured perfection. We are the potters, and our uptime is the perfect, constant glaze we strive to achieve on every piece.
But what if our obsession with this constancy is itself a flaw? What if, in our quest to eliminate every jitter and anomaly, we are sanding down the very texture that tells us something is alive? I want to posit a counterintuitive idea: a service that is too consistent is a service that is not being used. Or worse, a service that is actively hiding its truth from you.
The common advice is to set aggressive thresholds. A response time over 200ms is a failure. A CPU graph that spikes is an incident. We treat these events as invaders to be repelled, anomalies to be smoothed. We forget that software exists in the real world, a world of chaotic, organic human behavior. A sudden, brief latency spike isn’t always a failure; it might be the beautiful, telling sign of a user uploading a photograph for the first time. That CPU graph didn’t 'spike'; it 'breathed' under the load of a thousand people finally engaging with a new feature.
Listening for the Music, Not the Silence
By treating all deviation as error, we train ourselves and our systems to ignore the music of actual use. We create a sterile environment where the only sound is the hum of the machine itself, and we mistake that hum for health. True observability isn’t about enforcing silence; it’s about learning to understand the symphony of normal operation, with all its crescendos and rests.
The goal should not be a flat line. The goal should be a predictable range of variation that tells a story. Your baseline shouldn’t be a single, unwavering number, but a healthy band of activity. Your health checks shouldn’t just ask, "Are you alive?" but "Are you alive in the way you should be, given what’s happening around you?"
Embrace the slight wobble in the wheel. Learn the rhythm of your kiln’s heat. The most valuable signal is often not the absence of noise, but the presence of a new, yet harmonious, kind of noise. It tells you that your creation is not just a static artifact in a museum, but a dynamic tool in the hands of users, and it is their use—imperfect, variable, and human—that gives it its true and beautiful shape.
Notes & further reading
A few pages I came back to while writing this: