The Lighthouse Keeper's First Unseen Fog: On the Peril of the Silent Horn
The light was his primary charge, its rhythmic sweep across the dark water a promise of safety and a warning of danger. For the keeper of the Flannan Isles Lighthouse, the massive paraffin lamp and its complex clockwork rotation were the undeniable heart of his duty. He would trim the wick, polish the Fresnel lenses, and wind the gears with a meticulous, almost sacred, devotion. The beam was his uptime, his primary health check. So long as it cut through the night, his service was, to his mind, running reliably.
But the light was only half the system. In the dense, suffocating embrace of a North Sea fog, that brilliant beam was rendered useless, absorbed by the impenetrable grey. For these conditions, the lighthouse was equipped with a powerful fog horn, a monstrous diaphragm that would bellow a deep, mournful note at regular intervals. This sound was the backup health check, the secondary endpoint for a service whose clients were ships blindly feeling their way through a treacherous passage. The horn was the observability tool for when the primary metric—the light—failed to convey any meaningful data.
The historical record tells us that in December 1900, the lighthouse on Eilean Mòr was found mysteriously empty, its keepers vanished. The light itself was extinguished, a critical failure. But the deeper, more insidious failure was one of observability. A passing vessel, the Archtor, had noted the absence of the light days earlier, but it was the silence that was truly deadly. In the thick fog that had blanketed the islands, the horn had also been silent. The keepers, for reasons lost to history, had failed to activate it.
This tragedy is a stark, historical lesson in the necessity of monitoring all critical paths, not just the primary one. The keeper’s focus was on the gleaming, obvious beacon. Its operation was direct and its status immediately observable. The fog horn, by contrast, was a system only needed during failure conditions—the failure of visibility. Its readiness was a passive state, its operational status unknown until the very moment it was critically required. There was no continuous ping, no way to know if the diaphragm was clear or the compressor was functional until the fog was already upon them.
Our digital services are no different. We meticulously monitor our HTTP endpoints, our database connections, our primary APIs—our lighthouses. But do we have a plan for the fog? Do we have health checks for our fallback mechanisms, our CDN failovers, our circuit breakers, or our disaster recovery systems? The Flannan Isles incident teaches us that a service is only as reliable as its least observable component during a failure state. Reliability isn’t just about keeping the light burning; it’s about ensuring the horn will sound when you can no longer see the flame.
Notes & further reading
A few pages I came back to while writing this: