The Lighthouse and the Lantern: On the Reach of the Beacon Versus the Glow of the Proximate
There are two distinct philosophies in how we watch over our services, two methods of casting light into the operational darkness. One is the lighthouse: a singular, powerful beacon sweeping across a vast expanse from a fixed, distant point. The other is the lantern: a softer, more intimate glow carried with us, illuminating the immediate ground at our feet. In the realm of uptime and observability, these translate to the external synthetic check and the internal health endpoint.
The external check is our lighthouse. It is operated from a network far removed from our own, a synthetic client that periodically makes a request, measures the response time, and verifies the content. It sees our service as any user on the internet would. Its great strength is its reach and its impartiality; it can detect a routing mishap in a cloud provider or a DNS misconfiguration that would be entirely invisible from within the system's own walls. It is the truth of the outsider. But its beam is broad and intermittent. It can tell us the light is out, but it cannot always see why, or distinguish between a blown bulb and a fogged lens.
The internal health endpoint, by contrast, is the lantern we carry inside the service itself. It is a dedicated, private API that answers a simple, profound question: how do you feel from where you sit? It can report on the state of the local database connection, the availability of a caching layer, the free space on a disk, or the internal latency of a critical function. Its light is continuous and deep, revealing the texture of the stones beneath our feet. It offers a richness of context the lighthouse can never provide.
Yet the lantern's light only extends so far. It cannot see the network paths that lead to its door. A health endpoint can report itself as perfectly healthy while the world outside cannot reach it, trapped behind a misconfigured firewall or lost in a BGP black hole. It is a truth, but an internal one, potentially blind to the external realities that define a user's experience.
The wisdom, then, lies not in choosing one over the other, but in understanding the unique sight each provides. The lighthouse keeper trusts the sweeping beam to warn ships of the rocky coast, but he also carries a lantern to check the integrity of the lighthouse's own stairs and mechanisms. One guards against external peril, the other against internal decay. A reliable service requires both: the far-reaching, periodic certainty of the external beacon, and the continuous, proximate glow of the internal self-check. Together, they illuminate the complete picture, from the vast, dark sea all the way down to the ground upon which we stand.
Notes & further reading
A few pages I came back to while writing this:
- Surprise, AZ
- The Bell-Ringer's Pause: On the Critical Silence Between Strikes
- Elk Grove, CA
- The Frost Heave: On the Slow Strain of a Wintering Service
- Pasadena, CA
- The Silent Bell: On the Reliability of the Unrung Alarm
- New Haven, CT
- Stamford, CT
- Washington, DC
- one area's overview
- a practical rundown
- Little Rock, AR
- Gilbert, AZ