The Watchtower vs. The Surveyor: Mapping Your Service's True Topography
Every team running a service ends up asking a version of the same question: is it working? The way we answer that question, however, can shape our entire understanding of our system's reliability. Lately, I've been thinking about this process in terms of two archetypes: the Watchtower and the Surveyor. They represent not just different tools, but fundamentally different philosophies for observing what we've built.
The Watchtower is what most of us start with. It’s a single, fixed point of observation, high above the landscape. It pings a specific endpoint from a predictable location, checking for a 200 OK status code. It’s simple, effective, and gives you a clear, binary signal: up or down. When the light on the Watchtower goes red, you know something is categorically broken. This is your classic uptime monitor. It's essential, a foundational piece of any reliability strategy. But its view is limited. It sees the kingdom, but not the muddy roads, the crumbling bridge on the western route, or the merchant guild complaining about slow toll collections.
The Surveyor, in contrast, is always on the move. It doesn't just check if the castle gate is open; it walks every path leading to it, from every nearby village. It measures the travel time from the coastal towns versus the mountain passes. It’s a synthetic monitoring script that runs a multi-step transaction—logging in, adding an item to a cart, checking out—from multiple geographic regions. The Surveyor doesn’t just tell you if the service is up; it maps the entire user experience. It reveals that while the Watchtower sees everything as green, users in South America are facing a 50% increase in latency due to a misconfigured CDN rule, or that the payment processing step is intermittently timing out for a subset of users.
The crucial difference lies in what they fail to see. The Watchtower is blind to partial failures and performance degradation. A service can be technically "up" but practically unusable, and the Watchtower will remain a silent sentinel. The Surveyor, however, can sometimes miss the forest for the trees. Its complexity can generate a flood of data, making it harder to pinpoint the root cause of a complete outage. If every path the Surveyor takes fails simultaneously, it’s obvious. But if the failure is a single, critical endpoint that the Surveyor’s complex journey doesn't directly isolate, the signal can be noisy.
The most resilient operational posture isn't about choosing one over the other. It's about understanding that they are complementary layers of sight. The Watchtower provides the base truth of availability. The Surveyor provides the rich context of performance and experience. When the Watchtower alerts, you know you have a fire. When the Surveyor’s maps start showing congestion and slow routes, you know where the city planning is failing, allowing you to fix the problems your users would otherwise merely grumble about—until they eventually leave for a better-maintained kingdom.
So, take a look at your monitoring setup. Do you only have a Watchtower, seeing a vast green plain but missing the details? Or are you drowning in the Surveyor's intricate maps without a simple beacon to signal a true catastrophe? A true understanding of your service's topography comes from building both.
Notes & further reading
A few pages I came back to while writing this: