The Tiller and the Gimbaled Compass: On Steering by Reactive Force and Inertial Truth
In the quiet, critical work of keeping a service afloat, we often reach for the tools that promise the loudest, clearest signal. For many, this is the external health check: the synthetic transaction, the scheduled ping, the tiller of our vessel. It’s a direct, external force. When it pulls, we react. A 500 status code is a sharp yaw to port; a timeout is a hard correction to starboard. The logic is beautifully simple: something outside the ship reports a problem, so we heave to and fix it. This approach is undeniably powerful, offering a clean, customer-facing perspective. But it is, by its nature, a reactive force. It tells you the ship is off course only after the wind has already caught the sail.
Contrast this with the internal instrument, the gimbaled compass housed deep within the vessel’s hull. This is the realm of detailed, self-reported observability: the meticulously instrumented code, the internal metric for cache hit ratios, the trace of a database connection pool’s ebb and flow. Like a gyroscopic compass suspended to remain level despite the ship’s pitch and roll, these metrics seek an inertial truth about the system’s own state, independent of the chaotic sea outside. They don’t wait for a customer request to fail. They whisper of strain on the rudder post long before the tiller goes slack, of a slow leak in the bilge before the deck lists.
The core distinction lies in their relationship to failure. The tiller—the external check—is binary. It reports a failure of function. The gimbaled compass reports a degradation of form. One answers “Is it working for them?” The other suggests “Why will it soon stop working for us?” A health check might flag a service as ‘down’ because an API endpoint times out. The internal metrics, however, could have shown you the ninety-fifth percentile latency creeping upward for hours, the memory footprint growing in a sawtooth pattern of slow leaks, the thread pool utilization sitting stubbornly at 98%. The failure was not an event; it was a process made visible, if you were looking at the right instruments.
Relying solely on the tiller breeds a culture of firefighting, of responding to the last gust. You steer wildly, correcting the symptom. A pure focus on the internal compass, however, risks naval gazing—perfectly understanding a system that is, from the outside, already unresponsive. The art, then, is not in choosing one over the other, but in understanding their dialogue. The sharp tug on the tiller (a failed health check) must prompt an immediate look at the compass and other internal gauges to diagnose the ‘why’ behind the ‘what.’ Conversely, a concerning drift on the internal compass must be validated—does this actually affect the tiller’s response? Does this metric degradation have a customer-facing consequence?
True reliability is navigated in the tension between these two truths. The external check provides the unvarnished reality of user experience, the definitive pull on the tiller. The internal observability provides the context, the stability, the predictive whisper. One steers the ship; the other tells you about the integrity of its hull and the set of its sails. To sail a reliable service, you need both the force that corrects your course and the truth that explains why you drifted in the first place.
Notes & further reading
A few pages I came back to while writing this: