The Carpenter's Level and the Crooked Shelf: On the Deception of the Average Response Time
There is a quiet satisfaction in placing a level on a newly hung shelf and seeing that bubble settle perfectly between the lines. Everything looks true. In the world of service monitoring, our 'level' has long been the average response time. We configure our checks, watch the dashboard, and if that average stays within a green band, we assume all is plumb and square. The service, we conclude, is healthy. But just as a single average measurement can mask a shelf that is subtly warped, so too can this number conceal the true experience of every user trying to use your service.
The average, or mean, is a useful abstraction, but it is also a great deceiver. It flattens distinction. Consider a simple health check that pings an API endpoint one hundred times. Ninety-nine of those requests return in a brisk 150 milliseconds. One request, however, gets tangled in a garbage collection cycle or contends for a saturated database connection, taking a full two seconds to complete. The arithmetic mean of these one hundred requests is roughly 165 milliseconds—still well within our acceptable threshold. The dashboard remains a serene green. Yet, for that one user represented by the slow request, the service was effectively broken. They experienced a frustrating delay, a timeout on their end, or a broken UI component. The average, in its quest for a single truth, has erased their entirely real, negative experience.
Measuring the Crookedness
The technique, then, is to stop trusting the level alone and start feeling for the warps. This means shifting our focus from averages to percentiles. Instead of asking, "What was the average response time?" we must ask, "How fast did the slowest requests feel for the majority of our users?" The 95th percentile (p95) and the 99th percentile (p99) are the tools for this job.
Here is the concrete how-to: when you review your latency metrics, ignore the average for a moment. Drill down into the p95 and p99 values. If your p50 (the median) is 150ms, that’s great—half your users had that speed or better. But if your p95 is 1800ms, it means that 5 out of every 100 requests were slower than 1.8 seconds. That’s a significant portion of your user base encountering palpable slowness. The p99 will show you the true tail of the distribution, the very worst experiences that are still common enough to matter. This is where you find the frayed edges of your service, the specific endpoints or database queries that are causing a small but meaningful number of users to have a bad day.
Adopting this perspective requires a change in your monitoring posture. You must configure your dashboards and alerts to highlight these percentile values. An alert on a soaring p99 is often a more urgent and actionable signal than a gradual creep in the average. It points directly to a specific, acute problem affecting real users, not just a statistical abstraction.
The goal is not to achieve a perfect, warp-free board—that’s an impossible standard for any complex system. The goal is to know where the imperfections are, to understand their impact, and to decide which ones are serious enough to plane down. By looking past the average and embracing the percentile, we stop gauging the general state of the service and start advocating for the experience of every single user, especially those who would otherwise be hidden by the deceptive stillness of a comfortably green number.
Notes & further reading
A few pages I came back to while writing this: