The Lighthouse Keeper and the Watchful Crows

There’s a quiet but critical schism in the world of keeping services reliable. It’s the difference between the keeper of a lighthouse and a flock of watchful crows. One operates on a schedule, a system of deliberate, pre-planned checks. The other relies on an organic, reactive alertness to subtle shifts in the environment. In our terms, this is the classic divide between synthetic monitoring and real user monitoring (RUM). Both are vital for observability, but they serve as the conscious and subconscious minds of a system, each with its own strengths and blind spots.

The lighthouse keeper is our synthetic monitor. It’s a model of predictability. From a controlled location, a scripted sequence of actions is performed at precise intervals—every five minutes, a beacon sweeps the shore. It checks the lights, tests the foghorn, and confirms the rotation is true. This is the scheduled health check pinging our login endpoint, ensuring our API returns a 200 status code. Its greatest gift is consistency; it tells us the system is technically alive and accessible from a specific vantage point, even in the dead of night when no ships are passing. It’s our first line of defense, offering a baseline of uptime that feels reassuringly solid.

But the lighthouse has a narrow field of view. It knows nothing of the ships themselves. It can’t tell if a vessel took an unusually long time to come into view around the headland, or if smaller boats are struggling with a sudden, localized swell just outside the main channel. The keeper’s log only records the state of the lighthouse itself. This is the core limitation of synthetic checks: they monitor the system in isolation, not the system in use.

This is where the watchful crows come in. They represent real user monitoring. They don’t run on a schedule; they are the actual users navigating the waters. They perch on the rigging of every passing ship, cawing with data. A sudden chorus of squawks might indicate a strange current slowing progress (increased latency). A change in the pattern of their calls could signal that ships from a particular port are having trouble reading the lighthouse’s signals (a regional CDN issue). RUM gives us the messy, real-world truth. It tells us not just if the endpoint is up, but how it feels to the people and systems depending on it.

The crows, however, are fickle. If no ships come, they are silent. During a storm, their warnings might be lost in the gale. RUM data is only as good as the traffic generating it, making it a poor tool for detecting failures during low-usage periods or for brand-new features with no user base yet.

Truly understanding the health of our services isn't about choosing one over the other. It’s about listening to both the steady, rhythmic pulse of the lighthouse keeper’s log and the chaotic, insightful chatter of the crows. The keeper assures us the mechanism is sound; the crows tell us the sailors’ experience. One without the other gives us only half the picture. Reliability is found not in a single truth, but in the conversation between the planned and the actual, the signal and the noise.

Notes & further reading

A few pages I came back to while writing this: