The Gardener and the Surveyor: On Two Ways to Know the Land

There are two primary modes of knowing the health of a service, and they differ as much in philosophy as in practice. One is the Gardener, who walks the land daily, fingers in the soil. The other is the Surveyor, who takes precise measurements from fixed, immutable points. In our pursuit of reliability, we often deploy both, but rarely acknowledge the tension between their worldviews. Understanding this contrast is key to building a truly resilient system.

The Surveyor's approach is what we formally call synthetic monitoring. It is the prescribed health check, the uptime probe from three global regions, the scripted transaction that asserts the login flow works. It is excellent, providing a consistent, objective baseline. Like a surveyor’s tripod and theodolite, it gives us repeatable data: 200ms latency from Ashburn, a 99.99% success rate on the API endpoint. This is the clean, plotted map. It tells us if the known landmarks are where they should be. But it can only ever check for what we’ve thought to check.

The Gardener’s method is observability. It is less about predefined checks and more about cultivating a rich, living ecosystem of telemetry—logs, traces, metrics—and then wandering through it, noticing what’s different. The gardener doesn’t just check if the rosebush is upright (the Surveyor’s job); they notice a slight yellowing on a lower leaf, an unusual absence of aphids, the way the soil compresses. This is the domain of the anomalous error pattern no one scripted for, the gradual memory leak in a background service, the user’s strange journey that somehow succeeds despite our assumptions.

The peril lies in trusting one mode utterly. A system monitored only by the Surveyor becomes a collection of green checkmarks that breeds false confidence. The map is perfect, but the territory has changed. A new failure mode, unimagined by the script-writer, goes silently undetected. Conversely, a system tended only by the Gardener risks becoming an overgrown, inscrutable jungle of data. Without the fixed points of the Surveyor’s checks, how do you even know where to start your walk? You might spot a fascinating fungal growth on a log, but miss the fact that the entire north field has quietly stopped producing fruit.

The synergy, then, is not in choosing a side, but in letting them inform each other. The Surveyor’s red alert—a failed health check—is the starting pistol for the Gardener’s deep dive into traces. The Gardener’s discovery of a strange, slow database query spotted in a trace becomes a new, precise check the Surveyor can install. The Surveyor defines the known knowns; the Gardener explores the known unknowns and, with luck, stumbles upon the unknown unknowns.

Reliability is not achieved by merely having both tools, but by fostering a dialogue between these two sensibilities. Let the Surveyor’s rigid map define the borders of normalcy, and let the Gardener’s intimate, intuitive knowledge of the land reveal everything that map cannot show. The land is always more complex than the plot, and the service is always more alive than its health checks.

Notes & further reading

A few pages I came back to while writing this: