The Lighthouse Keeper's First Unlit Hour: On the Tyranny of the Green Light
We build our services, we instrument them, and we watch the dashboards light up. A sea of green status indicators, a symphony of 200 OK responses from our health checks. It feels like success. The lighthouse beam is sweeping, the signal is strong, and we rest easy. But I want to ask you a question that keeps me up some nights: what if all that green is a lie?
It’s a quiet, insidious problem. Our uptime monitors are often brilliant at telling us when a service has crashed completely—when the lighthouse has gone dark. But what about the hour the lens cracks, distorting the beam just enough that a ship might misread its signal? The service is still ‘up.’ It’s returning a successful HTTP 200. It’s passing its basic health check. But it is, for all meaningful purposes, broken.
This is the tyranny of the green light. We become beholden to a binary pass/fail state, blind to the gradual decay happening just beneath the surface. A database connection pool might be slowly leaking, adding milliseconds of latency with each new request. A caching layer might be serving stale data, corrupting the user experience without triggering an alert. A third-party API we depend on might have started returning subtly wrong answers, which we faithfully pass along. The health check, in its simplistic purity, sees none of this. It only confirms its own narrow world is functioning.
This is where the art of observability must conquer the crude science of mere monitoring. A health check is a checklist; observability is a conversation. It asks not just ‘is it up?’ but ‘is it well?’ It requires us to move beyond the simple ping and into the rich territory of latency distributions, error rates even within successful status codes, and the composition of the traffic itself. It means crafting synthetic transactions that don’t just check for a login page, but that actually log in and verify the data on the next page is correct.
The true goal isn’t a dashboard free of red. It’s a system whose every nuance we understand, whose murmurs of distress we can hear long before they become screams. We must be the lighthouse keeper who, upon seeing the steady green light, still climbs the tower to feel the heat of the lamp, to listen for the grind of the gears, and to look out at the sea to ensure the beam is truly reaching its destination. The goal is not just to keep the light on, but to ensure it is always guiding safely.
Notes & further reading
A few pages I came back to while writing this: