The Ferryman's First Shifting Shoal: On the Myth of the Constant Current
An old ferryman knows the river is never the same. He knows that the channel he navigated at dawn may be silted over by noon, that a submerged log from last night’s storm waits just beneath the surface, and that the current he relies on is a fickle thing, changing with the rain miles upstream. He would never trust a single measurement of the water’s depth or speed, taken once, to guide his craft all day. Yet, in our digital realms, we often do the equivalent. We set a health check to a single endpoint, see a 200 status code, and declare the service ‘healthy’ for the next hour. This is the myth of the constant current.
We build our services to be dynamic, elastic, and complex. They scale up and down, they re-route traffic, they cache and expire data. A single, static ping to a homepage tells you one thing: that at one precise millisecond, one specific path through your infrastructure was clear. It says nothing of the database connection pool slowly hemorrhaging connections five layers deep. It is blind to the rising latency in a critical third-party API that hasn’t yet tripped a timeout. It cannot see the memory leak in a background worker that only manifests under a specific, unmonitored query pattern.
This is the shifting shoal. The conditions of the river are in constant flux, and a single, fixed measurement point is worse than useless—it breeds a false confidence. The green checkmark becomes a tranquilizer, assuring us that all is well while the undercurrents pull the foundation from beneath us. The service isn’t a static monument; it’s a living, flowing system. To understand its health, we must move beyond the singular ping and become cartographers of its entire ecosystem.
True observability is the act of casting many lines into the water, all the time. It is measuring not just if a request succeeded, but how it succeeded. It’s tracing its entire journey, noting the depth (latency) at every stage, feeling the pull of the current (throughput), and watching for debris (errors) along the way. It’s understanding that a service’s health is a multi-dimensional state, defined by the interplay of hundreds of these tiny measurements.
The modern ferryman doesn’t just have a pole to test the depth; he has a sonar array painting a real-time picture of the riverbed ahead. He feels the vibration of the engine (resource usage) and watches the weather reports upstream (dependencies). His vigilance is continuous, not periodic. Our approach to monitoring must be the same. We must abandon the comfort of the single, green checkmark and embrace the nuanced, flowing reality of the systems we build and tend. The river never stops changing, and neither should our watch.
Notes & further reading
A few pages I came back to while writing this: