The Almanac's Dog-Eared Page: On the Risk of a Static Baseline

My grandfather kept an old farmer’s almanac on a shelf by the back door. Its spine was cracked, and one page in particular—the one with the regional planting calendar—was so dog-eared it felt like cloth. For years, that single page was the unquestioned authority. When to sow, when to reap. It represented a baseline, a pattern of stability drawn from decades of averaged weather data. It was, in its own analog way, a form of uptime monitoring for the entire growing season. The expectation was set: follow the almanac, and the crops will come.

In our world of digital services, we create similar baselines. We set monitoring thresholds for latency, define acceptable error rates, and establish performance budgets. These are our dog-eared pages. They are essential. Without them, we’re farming in the dark. A p95 latency of 200ms, a 99.9% uptime target—these figures become the bedrock of our operational reality. We check our dashboards against them the way my grandfather consulted his almanac, looking for the reassuring confirmation that things are ‘normal’.

But the risk, as any farmer who has lived through a changing climate will tell you, is in the static nature of that baseline. The almanac’s average first frost date is a composite of a hundred past winters; it cannot account for the unprecedented early freeze that wipes out an entire harvest. It gives a false sense of security, a belief that the past is a perfect predictor of the future. Our system metrics can suffer the same fate. That 200ms threshold, established during a period of low traffic or simpler application logic, becomes a relic. It doesn’t account for the new machine learning feature that adds 50ms under the hood, or the gradual creep of technical debt that slows database queries by imperceptible increments each quarter.

The Slow Drift of Normal

The real danger isn't a sudden, catastrophic failure—the ‘storm’ that everyone is watching for. It’s the slow, incremental drift of what we consider ‘normal.’ Our alerts remain silent because we haven't crossed the old, dog-eared threshold, but the user experience has silently degraded. The service is up, but it’s sluggish. It answers requests, but with a lethargy that slowly trains users to expect less. This is the peril of a static baseline: it quietly accommodates decay.

True observability, then, isn't just about having a baseline; it's about having a conversation with it. It requires us to periodically re-read the entire almanac, not just the familiar page. We need mechanisms that detect anomalies against a moving average, that recognize when the pattern of the last week has deviated meaningfully from the pattern of the last month, even if both are technically ‘green.’ It demands a practice of periodically questioning our own thresholds. Is 200ms still the right number? What does the 99th percentile look like now compared to a year ago? Are we monitoring for the conditions of yesterday or the expectations of tomorrow?

The dog-eared page was a testament to my grandfather’s trust in tradition. But his real wisdom came from looking out the window each morning, from feeling the soil, from knowing when the almanac’s advice no longer matched the reality on the ground. Our systems require the same vigilance. Our baselines are necessary maps, but we must never forget that the territory is constantly, if subtly, changing. The goal is not to find a perfect, permanent threshold, but to cultivate a practice of continuous recalibration, ensuring that our definition of ‘healthy’ never becomes a faded headline from a past season.

Notes & further reading

A few pages I came back to while writing this: