The Unseen Guest at Every Table: On the Protocol That Never Eats
We set a place for every service at the table. We lay out the finest china of our APIs, the polished silver of our databases, and the crystal glassware of our user interfaces. We expect our guests—the users, the requests, the data packets—to arrive, to partake, and to leave satisfied. But there is one guest we often forget to set a place for, one who never actually eats. This guest is the health check itself.
We treat our health checks as servants, not as dignitaries. We send them scurrying to the kitchen door for a quick, functional glimpse at the stove’s heat. ‘Is the service up?’ we ask. A binary yes or no is all we demand before we shoo them away. We rarely consider the journey these simple probes take through our own house. They walk the same hallways, knock on the same doors, and are subject to the same drafts and creaking floorboards as any honored guest. Yet we never think to ask if their path was clear, or if they had to wait too long in the foyer.
The Taste of Latency
This is the curious question: what does the health check experience? If a user’s request finds a table set with a five-second delay, we hear the complaint loud and clear. But if a health check encounters that same five-second delay and still returns a ‘200 OK’, we declare the service healthy and move on. The protocol that never eats has just tasted the exact same latency as a paying customer, but its report is silenced by our own simplistic expectations. We asked if it could find the dining room, not if the soup was already cold.
This creates a peculiar blindness. A service can be functionally ‘up’ yet performance-wise on life support, and our most basic monitoring will miss it entirely. The health check becomes a ghost, moving through a system that is increasingly sluggish, reporting back only that the walls are still standing. It feels the latency, it senses the queueing, it experiences the resource exhaustion—it is the canary in the coal mine that we’ve taught to only sing one note: ‘I’m not dead yet.’
To truly see the health of our service, we must start listening to the full story of our unseen guest. We must set a finer place for it, with instruments that measure not just its arrival, but its journey. How long did it take to get from the front door to its seat? Did it have to retry knocking? Was the hallway congested? This shift—from verifying existence to observing experience—turns the health check from a simple servant into our most intimate informant. It is the one guest who tours the entire estate before anyone else sits down, if only we bother to debrief it properly.
Notes & further reading
A few pages I came back to while writing this: