Build a Smoke Test Endpoint to Slash MTTR
You get paged. The dashboard is a sea of red. Your service's 99th percentile latency has spiked, and error rates are climbing. The immediate question isn't just "What's broken?" but more pressingly, "Is the core functionality even working?" This initial triage – figuring out if the fundamental user journey is completely down or just limping – can eat up precious minutes. You need a definitive, binary answer, fast.
We overload the term "health check" to mean everything from "is the app running" to "can it connect to the database." But these granular checks don't tell you if a real user transaction succeeds. That's where a dedicated, internal-only smoke test endpoint comes in. It's a single API route designed to simulate a critical user action from start to finish, giving you a clear, immediate signal of overall service health.
Here’s how to build one. First, identify the absolute core transaction. For an e-commerce site, it might be "add item to cart." For a SaaS app, it could be "authenticate user and load a key document." This isn't about testing edge cases; it's about the happy path that must work.
Next, implement the endpoint, perhaps at /internal/smoke-test. This endpoint should programmatically execute that entire transaction. It should create a test user or use a dedicated test account, call the relevant services, write to and read from the database, and then tear down any test data it created. Its response should be a simple JSON payload with a boolean "success": true and maybe a "duration_ms" value. Crucially, this endpoint should bypass caches to force a real operation.
Now, point your uptime monitoring tool at this endpoint. Set an alert to fire if it returns anything other than an HTTP 200 with "success": true, or if the duration exceeds a strict threshold. This alarm now has a incredibly specific meaning: the most important thing your service does is broken. There's no ambiguity. When this alert fires, you know you're dealing with a true P1 incident, and you can bypass checking a dozen other metrics. You've just turned a chaotic investigation into a targeted response, dramatically cutting your Mean Time To Resolution (MTTR) for the most severe outages.
This technique provides a brutal, honest assessment of your system's operational state. It cuts through the noise of partial failures and gives you a stark, undeniable truth about whether your service is actually working for users. It's the canary in the coal mine, redesigned for modern web services.
Notes & further reading
A few pages I came back to while writing this: