The Ferryman's Unseen Passenger: On the Weight of the Unchecked Request

There is a particular kind of dread known only to those who run a service others depend on. It’s not the sharp, metallic fear of a full-blown outage, but something slower and colder. It’s the dread of the unknown request, the one you never thought to watch for, silently piling stones into the hull of your boat.

I remember the moment I became a ferryman. It was years ago, a simple internal tool for processing image uploads. It was a small, sturdy thing. I’d built it with care: health checks for the API, uptime monitors for the server, latency metrics for the storage layer. My dashboard was a placid lake of green indicators. I watched the logs stream by, a gentle rhythm of POST and GET, 200 and 201. All was well. I was confident. I was blind.

The first sign was a sluggishness, a growing latency in responses that the metrics attributed to normal variance. The boat was just riding a little lower in the water, that’s all. Then, one Tuesday, it nearly sank. The database connections pooled out. The service choked. After a frantic restart, I went digging, not through the errors, but through the quiet, successful requests. The ones that had passed all my checks, paid their toll, and boarded the ferry without a glance.

And I found him. The unseen passenger. A single, legacy script from another team, running on a monthly cadence I never knew existed. It wasn’t making a few requests; it was making thousands. And each request, upon success, was asking for a verbose, nested JSON response it didn’t need—a full manifest of every processed file, a log of every step. My service, ever the diligent ferryman, was generating this enormous payload every single time, thousands of times a month. It was a polite, successful, utterly devastating request.

My observability had been a performance for an audience I assumed I knew. I was watching the ferry cross the river, counting the passengers I could see, but I never thought to weigh the boat. I never looked for the invisible cargo each passenger might be carrying aboard. That single, unchecked endpoint was the weight that was slowly swamping us.

Now, I am a different kind of ferryman. I still watch the passengers, but I also listen to the strain of the ropes and the groan of the timbers. I measure not just if the request arrived, but what it asked us to carry. The health of a service isn’t just in its beating heart, but in the quiet, cumulative weight of every single thing it is asked to do, especially the things you never think to expect.

Notes & further reading

A few pages I came back to while writing this: