The Lighthouse Keeper's Log vs. The Weather Vane's Spin: On the Duality of Measurement
There is a quiet, fundamental schism in how we understand the reliability of our services. It is the difference between knowing a thing is up, and knowing how it is up. This division is as old as measurement itself, and it plays out in our tools every day, personified by two distinct approaches: the definitive check and the ambient signal.
The definitive check is the lighthouse keeper’s log. At a precise interval, the keeper ascends the spiral stairs, checks the flame, notes the oil level, and winds the mechanism. The result is a binary, historical fact: at 23:00 GMT, the light was operational. Our health checks and pings are this keeper’s ritual. They are a deliberate, scheduled action that returns a clear, unwavering answer—a 200 OK or a timeout. This is powerful, essential knowledge. It is the bedrock of uptime, the unassailable record that a service was, without question, alive at a specific moment.
But what of the moments between the checks? What of the strain on the mechanism just before the keeper’s next visit? This is where the ambient signal, the weather vane’s spin, comes into play. The vane does not announce its observations; it simply reacts, constantly, to the pressures of the world. It offers a continuous, analog readout of condition.
Our equivalent is the stream of telemetry: latency percentiles, memory consumption, thread pool saturation, error rates. These are not yes-or-no questions but a chorus of how. A service might return a 200 OK while its 99th percentile latency is slowly creeping toward the stratosphere, a silent gale building that the simple health check cannot perceive. The weather vane shows you the wind is shifting long before the storm hits the lighthouse.
Neither approach is superior; they are complementary halves of a whole truth. Relying solely on the keeper’s log leaves you blind to gradual degradation, the infamous “slow bleed” that users feel long before the alarms sound. You only know a thing has died, not that it is dying. Trusting only the weather vane’s spin, however, can overwhelm with a cacophony of data points without the clear, declarative state that dictates immediate action. Is the service up or isn’t it? Sometimes, you need that simple, brutal fact.
True observability is found in the dialogue between these two. It is the keeper glancing from his logbook to watch the vane’s frantic spinning, understanding that the light is on for now, but the changing conditions demand his attention. It is the synthesis of the definitive, momentary truth with the continuous, contextual narrative. One tells you the state at a point in time; the other tells you the story of what led there and what likely comes next. To build reliably, we must listen to both.
Notes & further reading
A few pages I came back to while writing this: