The Watchman's Lantern: On the Light That Illuminates Only Itself

There is a quiet, almost sacred, belief that runs through our practice of building reliable services: if we can just see everything, we can fix anything. We string up our dashboards like festive lanterns, each one a promise of visibility, a tiny sun banishing the shadows where failures hide. We call this observability, and we treat it as an unalloyed good. But I’ve come to wonder if our brightest lanterns are sometimes casting the most deceptive shadows, illuminating their own intricate designs while leaving the true nature of the night outside our walls a mystery.

This is the paradox of the internal metric. We instrument our services with exquisite care, tracking CPU cycles, memory allocation, and garbage collection pauses down to the nanosecond. We have beautiful, real-time graphs showing the frantic heartbeat of every microservice. The watchman’s lantern burns bright, and we can see the watchman himself in perfect detail—the sweat on his brow, the steady rise and fall of his chest. We know he is alive and vigilant. And yet, we can remain utterly blind to the silent army massing just beyond the circle of his light.

Our services do not exist in a vacuum. They are participants in a conversation with the outside world, a world we do not control and cannot fully instrument. That world speaks in a language of flaky mobile networks, misconfigured DNS resolvers on local ISPs, aggressive corporate firewalls, and browser extensions that rewrite scripts. Our internal metrics, for all their precision, are silent on these matters. They report that the server is healthy, the database is responding, and the application logic is executing flawlessly. The watchman is healthy. But from the user’s perspective, the bridge is out.

This creates a dangerous comfort. A sea of green checks and flat latency graphs can foster a false sense of security, making the eventual user-reported outage feel like a betrayal by the universe itself. "But the graphs show nothing!" we exclaim, frustrated. Of course they show nothing. They were never designed to see that particular problem. We polished the lens of our lantern so perfectly that we forgot it can only show us what we put directly in front of it.

True reliability, then, is not just the art of internal observation, but the humility of external validation. It requires a conscious effort to step outside the warm glow of our own dashboards and into the darkness. It means embracing the messy, imperfect signals from the real world: synthetic monitoring that traces a user’s actual journey from across the globe, client-side performance metrics that capture the true end-to-end experience, and canary deployments that test the water before we all jump in. The watchman’s lantern is vital, but it is not enough. We must also listen for the sounds in the dark that it cannot show us.

Notes & further reading

A few pages I came back to while writing this: