The Unlit Wick: On the Folly of an Unobserved Candle

I once spent an entire afternoon watching a candle that wasn’t lit.

It was years ago, in a small, windowless server room that hummed with the sound of a dozen machines doing… something. My job was to babysit a data migration, a process that was supposed to be fire-and-forget. The senior engineer who handed me the runbook had smiled, clapped me on the shoulder, and said, ‘It’s all automated. Just watch the final log for the ‘SUCCESS’ flag. Should be about four hours.’ Then he’d left for the day.

So I watched. The script spat line after line of indecipherable timestamps and transaction IDs into a black terminal window. It was a torrent of data, a performance with no intermission. After an hour, my eyes glazed over. The words lost meaning. It was just noise. I had a single, binary metric for success—a specific string of text—and until it appeared, I was in the dark.

That’s when I noticed the candle. It was a fat, red emergency candle sitting on a shelf near the UPS, a relic from a more analog time. And a thought, born of sheer boredom, occurred to me: What is the state of this candle? Is it ‘up’ or is it ‘down’? Its purpose is to provide light in a failure state, but its readiness is entirely unknown. It could be a hollow shell of wax with no wick. The wick could be too short. It could be a dud.

I was staring at the digital equivalent. My entire monitoring strategy was that unlit candle. I had one signal, one binary outcome, and a vast, silent chasm of process in between. I had no idea if the data was transferring at a trickle or a flood, if latency was spiking, if the target database was healthy enough to receive it. The system was running, but was it running well? I couldn't say. I was just waiting for a flame to appear or not.

When the ‘SUCCESS’ message finally flashed on the screen, my relief was tempered with a cold unease. It had worked, but only through luck. I had been given a single point of observation and told it was enough. It wasn't.

That afternoon taught me that true reliability isn’t just about celebrating the light. It’s about constantly, gently, observing the wick. It’s about knowing the length of the fiber, the consistency of the wax, the stability of the holder. It’s the subtle art of watching the system live and breathe through a dozen different lenses—latency, throughput, error rates, resource health—so that long before you ever need its light, you know, without a doubt, that it will catch fire.

We don't monitor to see if something is finished. We monitor to understand how it is living. Anything less is just sitting in a dark room, staring at an unlit candle, and hoping.

Notes & further reading

A few pages I came back to while writing this: