The Miller's First Unmeasured Stream: On the Deception of the Average Response Time
There is a comforting simplicity to the average. It reduces the chaotic, sprawling reality of a thousand data points into a single, digestible number. In the world of service health, the average response time has become this comforting figure. It is the miller who trusts the placid, averaged surface of the stream, never dipping a hand below to feel the violent, hidden currents that could tear his waterwheel apart. We have placed far too much faith in this single metric, and in doing so, we have built a dangerous illusion of performance.
The fundamental flaw of the average is its inherent cowardice. It is a statistic that flees from the truth of suffering. A single catastrophic request, taking ten seconds to complete, is brutally suppressed by a thousand that took only fifty milliseconds. The math is correct, of course: (1000 * 0.05 + 1 * 10) / 1001 = ~0.059 seconds. The average barely flickers. But for that one user, the experience was a frozen screen, a timeout, a failed transaction. Their reality is not averaged away; it is a total failure. To them, the service was broken, no matter what the dashboard's gentle green line claims.
This obsession with the mean creates a culture of 'good enough,' blinding us to the outliers that truly define user perception. We are lulled into a false sense of security, polishing the smooth, averaged path while ignoring the potholes that break axles. The users who encounter these potholes are not statistical anomalies to be smoothed over; they are canaries in the coal mine, the first and most important signal of a system beginning to strain under a load it cannot evenly distribute.
Instead of the average, we must listen to the discordant symphony of percentiles. The 95th and 99th percentile response times are the truths the average tries to hide. They speak of the experience of our slowest users, the ones on poor connections, the ones triggering a complex database query, the ones whose requests land on a server struggling with a garbage collection cycle. This is where the real work is. A low average with a high p99 is not a healthy system; it is a system with a deeply hidden illness, one that only reveals itself to a unfortunate few, but reveals its flawed architecture nonetheless.
Our goal should not be a low average, but a small gap between the average and the extreme. A system where the p99 is only marginally higher than the mean is a system that is predictably fast for everyone, not just the statistically lucky. It is a system that has been engineered for equity of experience, not just efficiency on paper. Let us stop being millers who only glance at the stream's surface. Let us be hydrologists who measure its depth, its flow, its turbulence, and its hidden currents. For it is in those extremes, not in the placid average, that the true health of our services is determined.
Notes & further reading
A few pages I came back to while writing this: