The Farmer's Well and the Dowser's Rod: On Probes of Known and Unknowable Depth

There’s a deep-seated tension in the craft of running reliable services. It’s the tension between what we can measure and what we must infer. It occurs to me that this is much like the ancient divergence between two approaches to finding water: the farmer who digs a well and the dowser who reads a forked rod. One operates on the plane of the certain and the mechanical; the other, on the intuitive and the suggestive.

The farmer’s well is our synthetic monitoring. It is a deliberate, constructed probe into the system’s health. We decide on a location—a specific endpoint, a critical transaction like “user login”—and we dig down, creating a repeatable, controlled test. We lower our bucket at regular intervals, measuring precisely how long it takes to fill, whether the water is clear, and if the rope holds. The data is unambiguous: 200ms latency from the US-East region, a 99.99% success rate. It gives us a clear, quantitative baseline. When the well runs dry or the water turns muddy, we know, with certainty, that a specific, probe-able path is broken. It is reliable, but its vision is limited to the exact spot we dug.

The dowser’s rod, in contrast, is our real-user monitoring (RUM). The dowser doesn’t dig; they walk the land, feeling for subtle tremors in the rod that suggest water flows deep below. This monitoring captures the actual, messy experience of every user, each following their own winding path through the application. The rod twitches not for a single, predetermined transaction, but for a slow page load for a user in a remote location, a JavaScript error triggered by a specific browser version, or a stalled API call due to a cellular network blip. The data is vast, statistical, and profoundly contextual. It doesn’t tell you a service is down; it tells you that something feels wrong, a degradation in the user’s reality that our well might never have intersected.

The peril for the farmer is complacency. A dozen wells might be full of clean water, yet an entire field could be withering because a subsurface stream, unseen and untested, has changed its course. Your synthetic checks are all green, but your conversion rate is falling because a new feature is confusing users on mobile devices. You trusted the well, but the land itself told a different story.

The peril for the dowser, however, is ambiguity. The rod’s twitch could mean a vast aquifer or a shallow puddle. A spike in latency from RUM data could be a genuine server-side issue, or it could be one user on a congested train Wi-Fi. Without the firm, binary truth of the well to correlate against, you can spend hours chasing phantoms, misinterpreting the signal of a single user for a systemic failure.

The art, then, is not in choosing one over the other, but in understanding their conversation. The farmer needs the dowser’s intuition to know where to dig new wells. When the rod consistently trembles in a formerly quiet corner of the application, it’s a signal to sink a new synthetic probe there. Conversely, the dowser needs the farmer’s certainty to validate their sensations. When the rod shakes, the first question should be: “What do the wells say?” If the wells are steady, the problem may be localized, a trick of the light on the landscape rather than a rupture in the aquifer. Together, they move beyond simple uptime and towards true reliability—a service that is not just running, but truly serving, across the entire, unpredictable terrain of user experience.

Notes & further reading

A few pages I came back to while writing this: