The Unwitting Arms Dealer: How Our Obsession with Resilience Fuels Complexity

Everywhere you turn, the doctrine is the same: build for resilience. Create systems that bend but don’t break. Engineer services that can withstand the unexpected shock, the traffic tsunami, the cascading failure. This is the unquestioned gospel of reliability engineering, and we have built a sprawling industry of uptime monitoring, health checks, and circuit breakers to serve it. We are all fervent evangelists, preaching the good word of redundancy. But I’ve begun to wonder if, in our righteous quest for resilience, we have inadvertently become arms dealers in a war against ourselves, constantly supplying our systems with more sophisticated weaponry to fight the very complexity we introduce.

Consider the standard path. A service starts simple. Then, to make it "resilient," we add a failover database. To monitor that, we introduce health checks. The health checks need to be observed, so we add an observability platform. The platform generates alerts, so we build a paging system with routing logic to avoid alert fatigue. Each layer, each new service, is a node that can fail, a source of latency, a new variable in the sprawling equation of "is it working?" Our solution to complexity, time and again, is to add more moving parts.

We champion patterns like circuit breakers and retries to make services robust in the face of flaky dependencies. And they work, splendidly. But they also create a dangerous buffer. A downstream service, poorly designed and slow, can be propped up indefinitely by a clever retry logic and back-off strategy. The pain of its inefficiency is absorbed by the client. The incentive to fundamentally fix the underlying slowness, to simplify the interaction, diminishes. The resilience pattern becomes an enabler, allowing architectural debt to accumulate silently, shielded from the immediate, painful feedback that would otherwise force a reckoning.

This isn’t just about technical debt. It’s about a reflexive engineering culture. We reach for the circuit breaker before we ask if the communication channel needs to be so chatty. We scale horizontally with Kubernetes pods before questioning if a single, well-optimized process could handle the load. Our dashboards fill with graphs for p95 latency and error rates, but we have no metric for "conceptual weight"—the cognitive load required to understand the path a simple request now takes through a dozen resilient, monitored, but deeply intertwined services.

The Seduction of the Safety Net

The promise of observability and health checks is a seductive one: total awareness, total control. But this safety net can make us reckless architects. Knowing we can monitor every hiccup and automatically fail over gives us permission to build more brittle, more distributed, more intricate systems. The safety net doesn't just catch us when we fall; it encourages us to build towers of dazzling height and fragility.

What if, instead of always reaching for a more resilient hammer, we occasionally asked if the thing we’re building needs so many nails? The most reliable system is the one you don't have to monitor with frantic urgency because its operation is simple, its interactions few, and its failure modes comprehensible to a single human brain. The ultimate form of uptime might not be achieved by adding more layers of watchful guardians, but by a kind of radical simplification that makes those guardians mostly redundant.

This isn’t an argument against monitoring or resilience. It’s a plea for perspective. We must be gardeners, not just guards. We should spend as much energy pruning, simplifying, and clarifying our systems as we do building fortifications around them. Otherwise, we risk winning every battle against downtime while slowly losing the war to an ever-expanding, self-justifying jungle of our own creation.

Notes & further reading

A few pages I came back to while writing this: