The Scribe's Quill Scratch: On the Character of a Failed Check

The alarm goes off. A service check has failed. Your monitoring system dutifully declares something is "down." This is the moment we usually rush to fix the thing, to make the red light turn green again. But in that haste, we often erase the most valuable piece of evidence: the precise manner of the failure. We treat it as a binary event, when it is, in fact, a rich narrative. I’d like to suggest a simple, almost antiquated technique: before you restore service, write down how it failed, in plain human language, as if you were a scribe recording an event for an audience that wasn’t there.

This isn’t about automated logs or stack traces. Those are the footprints. I’m talking about the gait. Did the health check endpoint time out, or return a 500 error? If it was a 500, was it a database connection refused, or a null pointer exception in the login service? Did latency spike uniformly for all users, or just for those in a specific region? Did the failure happen the instant a new deployment finished, or an hour later when cache entries began to expire? These distinctions are the character of the failure. They are the scratch of the quill, the specific shape of the broken line that tells you what pressure was applied and where the nib caught on the parchment.

The Anatomy of a Scratch

Implementing this is straightforward but requires discipline. When a critical alert fires, your team’s first action—before the debugger or the rollback—should be to open a shared, ephemeral document (a dedicated channel, a virtual whiteboard, even a text file) and append a new entry. Title it with the timestamp and the service. Then, in a few sentences, describe the symptom as observed from the outside. Force the language to be concrete. Instead of "the API is slow," write "health check p95 latency to US-East-1 jumped from 120ms to 4.2 seconds, while EU-West remained at 110ms. The /v2/status endpoint started returning 503 after 30 seconds."

This practice does two profound things. First, it decouples the observation from the diagnosis. It creates a clean record of the symptom, uncontaminated by the eventual root cause we might retroactively impose. Second, and more importantly, it trains your observational muscle. You start to look for these nuances instinctively. You begin to configure your probes not just to pass/fail, but to capture the texture of the failure—timeout thresholds, specific HTTP status codes, partial response validations.

Over time, this collection of scratches becomes a palimpsest of your system’s frailties. You’ll notice patterns: the database faults have a certain character, the network partitions another. You’ll see that the "blip" last Tuesday had the exact same failure signature as the major outage a month prior, just at a smaller scale—a previously invisible correlation. This record becomes your system’s biography, written in the hand of its ailments.

Reliability isn’t just about preventing failure; it’s about understanding it intimately. A green checkmark tells you nothing. A red one only tells you to look. It’s the description of the red—the scratch, the sputter, the particular silence—that teaches you what your system truly is, and what it fears. So, keep a log of the scratches. They are not just records of breakdowns, but the very text from which you learn to build more resilient prose.

Notes & further reading

A few pages I came back to while writing this: