The Lighthouse Keeper's Damp Match: On the Spark You Keep in Reserve

I was seventeen, volunteering for the summer at a remote marine biology station on the Pacific coast. My most romantic duty, I thought, was the weekly check of the old, automated lighthouse on the farthest tip of the headland. It was a classic sentinel, all white paint and rusting iron, but its light was electric, its rotation governed by a simple timer. My job wasn’t to tend the light itself, but to check the backup—a cantankerous diesel generator housed in a damp concrete shed at its base.

Every Thursday, I’d unlock the heavy door to the generator room. The air inside was always thick with the smell of salt and old oil. The checklist was taped to the wall: Check fuel level. Inspect for leaks. Crank the engine manually to prevent seizing. And the final, crucial step: Test start.

I remember one afternoon particularly well. A low, grey fog had rolled in, muffling the sound of the waves. I went through the ritual, my flashlight beam cutting through the dimness. Fuel was fine. No leaks. I strained against the manual crank, feeling the satisfying resistance of the cold engine. Then I reached for the red ‘Start’ toggle. I flipped it. The starter motor whined, strained, and then… nothing. Not even a cough. Just a tired, grinding sound that faded into the hum of the fog. I tried again. Same result. A cold knot tightened in my stomach. The main light was fine, but the promise of this machine, the entire purpose of my being here, felt like a lie.

In that moment, the abstraction of a ‘backup system’ vanished. It was replaced by the visceral reality of a dead piece of machinery in a silent, lonely shed. The lighthouse beam above me continued its slow sweep, a silent tenant oblivious to the fragility of its own support. The system was up, but its resilience was an illusion. My weekly checks had been a form of observability, but they were only measuring the generator's latent state, not its actual capacity to spark into life when called upon.

I learned two things that day, lessons that have stuck with me through a career spent building and watching digital services. First, a health check that doesn’t simulate a real failure condition is just theatre. Pinging an endpoint tells you it’s alive; it doesn’t tell you if it can handle the storm. Second, and more importantly, resilience isn’t a property you can assume. It’s a potential energy that must be regularly, genuinely tested. You have to actually strike the match in the damp dark to know if it will light.

We fixed the generator, of course. It was a corroded connection, a tiny failure hidden from routine view. But the real fix was the change in our protocol. From then on, the test wasn’t just starting the generator; it was starting it and then switching the lighthouse to it for a full five minutes, letting the primary power rest. We introduced a deliberate, controlled failure to prove the integrity of the backup. It was no longer about avoiding use; it was about ensuring readiness through use. The true measure of the sentinel wasn't its silent vigil, but the certainty that its voice would be there, loud and clear, the moment the primary song faded.

Notes & further reading

A few pages I came back to while writing this: