The Dust on the Mirror: On the Image That Reflects an Empty Room

Years ago, I was responsible for a small, internal tool that hummed away in a forgotten corner of our infrastructure. Its only job was to periodically check a handful of external APIs we depended on, and it had one output: a single, static webpage with a table. Green for up, red for down. I had set up a basic health check for the checker itself—a simple ping that returned a 200 OK. For three years, its dashboard was a serene, unchanging meadow of green. It was the most reliable thing we had.

Then, one Tuesday afternoon, everything broke. Or so the alerts screamed. Our main service was failing, cascading into user-facing errors. The team scrambled, diving into metrics, logs, and traces—the full observability suite. In the chaos, I reflexively pulled up the old checker's dashboard. Still green. A perfect, placid row of emerald cells. A sigh of relief; at least the external world was intact. But the relief was short-lived. A colleague, digging deeper, found the root cause: one of those very APIs had been returning garbled nonsense for hours. Our system choked on it. Yet the checker’s cell for that API glowed a confident, verdant green.

I SSHed into its ancient host. The process was running. The health check endpoint responded instantly. But when I looked at the last log entry, the timestamp was from the previous Thursday. The program had frozen solid, five days prior, in the middle of its last successful run. It had died in a state of perfect health, like a photograph of a smiling face. The ping endpoint was a separate, tiny server that knew nothing of the main application's state. It reported the container’s aliveness, but not the service’s *liveness*. It was a mirror that had ceased reflecting the room years ago, but because we still saw our own expectant faces in its dusty glass, we assumed the room behind us was unchanged.

We had mistaken a heartbeat for a purpose. The health check was a solipsistic loop, confirming only its own ability to say "I am here." It told us nothing of the watchful gaze, the comprehension, the intent of the service it was meant to guard. Latency was zero, uptime was 100%, and yet it had been functionally dead for days, faithfully reporting a fictional world.

That static green table was the ultimate illusion of observability. It presented data—clean, binary, reassuring—while hiding the complete erosion of meaning. We had instrumented the act of looking, but not the quality of the sight. Now, when I design a check, I ask a different question. I don’t ask, "Are you running?" I ask, "Did you *see* something, and did it make sense to you?" The difference is the difference between a mirror and a window. One shows you only yourself; the other must reveal the storm gathering outside, even if it means you have to wipe the pane clean to see it.

Notes & further reading

A few pages I came back to while writing this: