The Unseen Anchor: On the Peril of Measuring Everything

We are told, with the fervor of a new gospel, to instrument everything. To leave no metric uncollected, no log line unwritten, no trace unspanned. The promise is a map so detailed it becomes the territory itself—a perfect digital twin of our systems, shimmering in the observability platform of our choice. This is the zenith of operational awareness, the path to reliability. But I want to propose a heretical thought: this exhaustive measurement is not a map to clarity; it is an anchor, dragging us down into a sea of noise.

The common advice is born from a good place. The fear of a blind spot, of the dreaded "unknown unknown," is a powerful motivator. So we instrument out of anxiety, collecting data points like beachcombers collecting shells, convinced that the one we leave behind will be the most beautiful and crucial. We end up with petabyte-scale data lakes that are, in practice, data swamps—murky, overwhelming, and impossible to navigate with any urgency when the alarms are blaring.

This is the first peril: the paralysis of abundance. When every possible signal is recorded, the truly critical signal is drowned out. An engineer faced with a thousand charts and a million log lines during an incident isn't empowered; they are besieged. The cognitive load of sifting through this avalanche of data delays diagnosis, elongates mean time to resolution, and ultimately makes the service less reliable, not more. We built a library where every book is screaming for attention simultaneously, ensuring none can be read.

The second, more insidious peril, is that this practice can actively erode engineering intuition. The relentless focus on quantifiable data comes at the expense of the qualitative, systems-level understanding that veteran operators develop. It’s the difference between knowing the precise RPM of every engine component and knowing how the car *feels* when it’s about to have a problem. That gut feeling, born of experience and a deep, holistic grasp of a system’s behavior, is an invaluable tool. By insisting that every insight must be data-derived, we risk atrophying that instinct, creating a generation of engineers who can read a dashboard flawlessly but cannot sense the storm coming.

This is not an argument for ignorance. It is an argument for intentional, thoughtful instrumentation. Measure what matters, not what can be measured. Start with the core user journey—the golden signals of latency, traffic, errors, and saturation. Then, and only then, add specific metrics to test explicit hypotheses about known weaknesses. Your observability stack should be a curated telescope for viewing the cosmos of your system, not a security camera recording every leaf that falls in the forest.

True reliability isn’t forged in the firehose of total data. It is cultivated through focus, through understanding which questions are important to ask, and having the discipline to ignore the rest. Sometimes, the most observant watchman is not the one with a thousand screens, but the one who knows the sound of the night and can pinpoint the single note that is out of place.

Notes & further reading

A few pages I came back to while writing this: