The Unsent Postcard: Creating a Health Check That Acknowledges Reality
We spend an inordinate amount of time teaching our services to cry for help. We configure monitors to scream when a disk is full, an endpoint times out, or a process dies. These are the obvious alarms, the fire bells in the night. But what about the quieter failures? The ones where the service is, by every technical metric, alive and well, yet has become functionally useless to the people who rely on it? It’s the difference between a house that’s structurally sound and a house whose front door is inexplicably locked. The lights are on, but you simply cannot get in.
I was recently reminded of this by a small, third-party authentication service one of our applications depends on. Our standard health check, a simple HTTP GET to its status endpoint, returned a proud 200 OK. Green across the board. Yet, our users were reporting they couldn't log in. The authentication service itself was up, but its connection to a critical downstream identity provider had silently failed. Our service was sending out cheery postcards saying "Wish you were here!" while stranded on a desert island with no way to receive a reply.
This is where the concept of a synthetic transaction elevates a health check from a simple pulse monitor to a genuine assessment of capability. The technique is simple in theory but requires a shift in thinking: instead of asking "Are you up?" we design a check that asks "Can you do the specific, valuable thing I need you to do?"
Writing the Postcard and Demanding a Reply
For our authentication service, the fix wasn't to add more probes to its internal state. It was to script a miniature version of a real user's journey. Our new health check does the following, on a loop, from a monitoring node outside our core infrastructure:
First, it attempts to authenticate using a dedicated set of test credentials that mirror a real user's login flow. It doesn't just ping the login endpoint; it posts the credentials and expects a specific, successful response, including a valid session token. This proves the core logic and database connectivity are sound.
Second, and this is the crucial part, it uses that newly acquired token to make a second, authorized request to a protected user profile endpoint. If this succeeds, it confirms that not only can the service authenticate, but it can also manage sessions and authorize subsequent requests—the complete chain of trust is intact. Finally, it gracefully logs out, cleaning up the test session.
This multi-step transaction takes only a few seconds, but it tells a complete story. It's no longer an unsent postcard; it's a postcard mailed with a return receipt requested. We're not just checking for a heartbeat; we're checking for a coherent conversation.
Implementing this required creating a dedicated test identity and accepting a small amount of synthetic traffic in our logs. The trade-off is trivial compared to the clarity gained. Now, when this health check goes red, we know with certainty that user logins are broken, not just that a server is rebooting. It acknowledges the messy reality of modern service dependencies. It’s a check that understands that being available and being useful are not always the same thing, and in the pursuit of reliability, we must vigilantly monitor for both.
Notes & further reading
A few pages I came back to while writing this: