The Lighthouse Keeper's First False Flame: On the Peril of a Single Beacon
In the annals of maritime history, the Eddystone Lighthouse stands as a monument to human resilience. But before its iconic stone tower, there was a wooden one, built by a man named Henry Winstanley. His story, one of the first great experiments in persistent service reliability, offers a stark lesson for any of us who watch over digital systems today: a single point of observation is a single point of failure.
Winstanley was a painter, an inventor, and a showman. In the late 17th century, after losing a ship to the treacherous Eddystone rocks, he undertook to build a lighthouse. He succeeded, creating a fantastical structure adorned with ornate carvings and even a comfortable room for himself. So confident was he in his creation that he famously declared he wished to be inside it during "the greatest storm that ever was." He trusted his build implicitly. His monitoring strategy was simple: he was there, inside it, and he could see the flame burning. The service was up.
And then, in 1703, the Great Storm came. For Winstanley, it was the ultimate health check. The lighthouse’s single, central wooden pillar held firm for a time, and from his perspective, the flame was lit. The primary metric—luminous output—was green. But he was monitoring the wrong thing. He was watching the flame, not the foundation. He was checking the service, but not the structural integrity of the entire system upon which it depended.
The storm surge and waves did not attack the lantern directly; they undermined the very rock it stood upon. The entire structure, with Winstanley inside, was wiped from the face of the Earth. There were no survivors. His monitoring was fatally flawed because it was myopic. He had a green checkmark for his flame, but no observability into the latent stresses on the rock, the integrity of the joints, or the rising water pressure at the base. He had no synthetic checks from the perspective of a ship at sea in the storm, no way to know if the light was even visible through the tempest.
We build our own Eddystone lights every day. A health check endpoint returning a 200 status code is our burning flame. It tells us the application is running, but it says nothing of the database connections slowly pooling, the memory leak in a background thread, the latency spike in a third-party API, or the disk space quietly vanishing on a primary node. Like Winstanley, we can be lulled into a false sense of security by a single, passing check.
True observability isn’t just about seeing the flame; it’s about understanding the entire rock, the sea, and the weather. It requires a symphony of metrics, logs, and traces from every critical component. It demands synthetic checks that simulate real user journeys, not just uptime pings. We must build systems with multiple, redundant beacons of insight, ensuring that when the great storm finally arrives, we aren’t watching a single, steady flame as the entire foundation crumbles beneath us.
Notes & further reading
A few pages I came back to while writing this: