The Potter's Centered Clay: On the Competing Pulls of Probes and Traces

The goal, for any of us running services, is a centered one: a system that spins smoothly, reliably, predictably. But the methods we use to achieve this balance pull in two distinct directions. In our workshop, we have two primary tools: the active probe and the passive trace. One is like the potter’s hands, constantly touching the clay to feel for wobbles. The other is like the impression left on the wheelhead after the work is done, a silent record of every vibration and tremor. Choosing between them is not about finding the 'best' tool, but understanding the nature of their pull on the clay of our systems.

Active health checks are our deliberate probes. We build a synthetic transaction, a canary in the coalmine, and send it down a predetermined path. We ask a simple, direct question: "Are you alive?" from a specific vantage point outside the system. It is a binary intervention, a tap on the shoulder. Its great strength is its simplicity and its proactive stance. It tells us, with certainty, if a critical path is passable *before* a real user ever encounters a failure. It’s the potter's thumb testing the rim for evenness. But this clarity comes with a cost. The probe only knows the path we told it to check. It lives in a world of our own making, and it is blind to the novel failures, the bizarre edge cases, or the slow degradations that real users, with their chaotic and unpredictable behavior, will inevitably uncover.

Contrast this with the passive trace. A trace does not ask questions; it merely observes. It is the residue of actual work, the fine dust of a real request as it is processed by a dozen different services. Where a probe is a single tap, a trace is the complete vibrational signature of the entire wheel during the creation of a single pot. Its power lies in its fidelity to reality. It shows us not just that a failure occurred, but the exact sequence of events leading to it—the slow database query that triggered a timeout, which then cascaded into a circuit breaker opening. It reveals the complex, emergent behavior of the system under true load.

Yet, the trace is a historian, not a sentry. It can tell you, in exquisite detail, why a user had a bad experience fifteen minutes ago. It is a tool for forensic analysis, not for prevention. By the time you read the trace, the damage is done. Relying solely on traces is like a potter who only examines the finished, fired pot for cracks, rather than feeling the clay for weaknesses as it spins.

And so, we are pulled in two directions: the outward pull of the probe, which seeks to prevent catastrophe by constantly testing known boundaries, and the inward pull of the trace, which seeks to understand complexity by deeply examining what has already transpired. The centered system, the one that spins true, is not built by choosing one over the other. It is built by using the probe to establish a baseline rhythm—a steady, reassuring beat that says the vital signs are strong. And it is built by using the trace as the subtle feedback felt through the hands, the sensitive instrument that detects the tiny, unexpected wobbles introduced by new clay, a change in moisture, or a slight unevenness in the wheel. One sets the pace; the other guides the form. Our craft demands we listen to both.

Notes & further reading

A few pages I came back to while writing this: