The Ferryman's First Uncounted Passenger: On the Weight of the Unseen Load
There is a particular kind of dread that lives in the silence between the splash of the oar and the creak of the rope. It’s a feeling I know well, from my earliest days running the service that managed user authentication for a dozen other internal tools. My dashboard was a sea of green, a perfect, placid lake. Every health check passed, every latency graph was a flat, serene line. I was the diligent ferryman, and my boat, by all observable metrics, was sound.
I remember the moment the metaphor broke. It was a Tuesday, or perhaps a Wednesday—the kind of day that blurs into the infrastructure. A colleague from another team casually mentioned in the kitchen that they’d been seeing ‘weird auth blips’ for about a week. Just a handful of failed logins, a spike in 401s so minor it hadn’t tripped any of their alerts. They’d assumed it was on our end. My stomach sank. I checked my boards. My beautiful, silent, green boards. They showed nothing. No failed pings, no latency spikes, no CPU load. The ferry was running on time, and the manifest was clean.
But a passenger had fallen overboard, and I hadn't heard the splash.
Rushing back to my terminal, I dug past the synthetic health checks—the automated pings from known IPs that simply asked, "Are you there?" and received a cheerful "Yes!" I went hunting for the real traffic, the messy, human chaos of actual user requests. And there it was: a tiny, almost pathetic thread of failures. Less than 0.1% of traffic. It was a race condition in a token refresh handler, a ghost in the machine so faint it was drowned out by the noise of our own successful checks. Our monitoring was perfectly tuned to watch the boat, but it had never thought to count the souls on board.
That was the day I learned the difference between being up and being reliable. Uptime is a binary state; it’s the ferry being in the water. Reliability is the qualitative measure of the journey. It’s about knowing if the ferry is overcrowded, if the current is pulling it off course, if a single passenger is silently slipping beneath the waves. We had been so focused on the vessel itself—its structural integrity, its speed—that we had forgotten its purpose: to carry weight.
I rebuilt our checks that afternoon. I added observability that didn’t just listen for the boat’s horn but listened for the gasps. I instrumented for business logic—counting successful logins alongside failed ones, measuring the weight of the load, not just the rhythm of the oars. The sea of green lost its innocence for me that day. Now, its calm surface doesn't signify peace, but a question: what unseen weight are we carrying today, and are we strong enough to bear it?
Notes & further reading
A few pages I came back to while writing this: