The Lighthouse Keeper's Dimmed Lamp: On the Necessary Blindness of a Partial Outage
We build our systems to be seen. We string up health checks like lanterns along a dark coast, each one a promise of vigilance. The common wisdom is absolute: every service, every endpoint, every critical function must be monitored. To leave a single component in the dark is an act of negligence, a blind spot waiting to become a catastrophe. But what if this quest for total illumination is itself the danger? What if, in our zeal to see everything, we have forgotten the strategic value of a controlled dimming?
The counterintuitive truth is that not every failure deserves an alarm. In fact, some failures should be intentionally obscured from our central dashboards. This isn't about ignoring problems; it's about triaging them with wisdom rather than raw data. A service designed for graceful degradation, for instance, might deliberately shed non-essential features during times of high load or upstream instability. If our health checks are wired to scream bloody murder the moment a single, non-critical API for a low-priority widget fails, we haven't built a resilient system. We’ve built a noisy one that treats a stubbed toe with the same urgency as a heart attack.
This creates a paradox of observability: by monitoring everything with equal fervor, we can ironically make the system less observable. The signal of a truly critical failure is drowned in a sea of minor, self-correcting alerts. The team becomes numb, conditioned to ignore the pager’s cry. The lighthouse beam, meant to warn of ship-shattering rocks, is now so constantly sweeping across every pebble on the beach that its urgent signal is lost.
The wiser approach is one of deliberate, architectural blindness. It requires designing services not only to fail gracefully but to report their failures gracefully. A health check should be a nuanced statement of capability, not a binary shriek. It should be okay for a check to return a ‘degraded’ state—a dimming of the lamp—without triggering a full-blown incident. This forces a more thoughtful conversation about what ‘up’ truly means and allows the system the breathing room to heal itself from minor ailments without our panicked intervention. Sometimes, the most reliable service isn’t the one that never has a problem, but the one that knows how to be quietly, temporarily unwell without causing a panic.
Notes & further reading
A few pages I came back to while writing this: