The Illusion of the Perfect Baseline: On the Seduction of a Steady State
In the quiet halls of service reliability, there is a near-sacred mantra: know your baseline. We are taught to hunt for it, document it, and then vigilantly guard against any deviation. We plot our p95 latencies on dashboards and set our alert thresholds a safe, respectful distance above that holy line of normalcy. The baseline becomes the story of health, the definition of "good." But what if our reverence for this steady state is not just misguided, but actively harmful? What if the perfect baseline is a phantom, and chasing it makes us blind to the reality of a living system?
Our obsession with a stable baseline assumes that a healthy system is a static one. It presumes that yesterday's performance envelope is the only valid one for tomorrow. This is a comfortable fiction. In truth, a service that is truly alive—being developed, deployed, scaled, and used by humans—has no single baseline. Its heartbeat has a natural arrhythmia. The latency profile at 3 AM on a Tuesday is not the same as at 3 PM on a launch day. The "normal" error rate for a cache-miss is different from the "normal" for a database failover. By enforcing a single, averaged story of performance, we smooth over these essential rhythms and declare the system's natural breath to be an anomaly.
The Tyranny of the Green Dashboard
This creates a subtle tyranny. When the graphs are flat and the thresholds untouched, we experience a deep, seductive calm. The dashboard is green; all is well. But this calm is the enemy of understanding. It encourages us to stop looking, to stop asking why the 3 AM latency dip occurs, or what the slight, daily upward creep in response size might foretell. We become keepers of the green light, not students of the system. We celebrate the absence of alarms as success, rather than seeking a deeper, more nuanced knowledge of the entity in our care.
Worse, the perfect baseline becomes a crutch for poor instrumentation. If our only goal is to detect deviation from "normal," we are incentivized to monitor superficial symptoms—response codes, simple latency—while remaining ignorant of the underlying physiology. It's like checking only a patient's pulse and declaring them healthy, never asking about their diet, sleep, or stress. A system can be compliant with its historical baseline right up to the moment it suffers a novel, catastrophic failure that our metrics were never designed to see.
The alternative is not chaos, but a shift in philosophy. Instead of worshipping a mythical steady state, we should cultivate a sense of the system's contested terrain. Our focus should move from "Is this different from before?" to "What is this system capable of right now?" This means embracing a richer, more contextual observability that correlates load with behavior, that understands seasons and cycles, and that seeks to explain why performance is what it is, not just if it has changed. It means setting alerts not just on deviation from the past, but on convergence toward known bad states, regardless of how we got there.
Let go of the baseline. In its place, seek the narrative. A line on a graph has no story; it is merely a residue of activity. The true reliability of a service lies not in its constancy, but in our growing, evolving comprehension of its mutable nature. The goal is not a silent alarm, but a continuous, informed conversation with the machine.
Notes & further reading
A few pages I came back to while writing this: