The Signalman's First Unheard Bell: On the Fallibility of the Human Loop
Before the age of automated pings and synthetic transactions, the reliability of a network depended on a far more fragile component: a person. In the early days of the telegraph and railway, the signalman was the living, breathing health check for a sprawling, interconnected system. Stationed in isolated boxes along the line, his duty was to monitor the state of the tracks, communicate with adjacent stations, and ensure the safe passage of trains through his section. The entire enterprise relied on his constant, unerring vigilance.
The primary tool for this observability was the electric bell, a simple circuit connecting one signal box to the next. A series of coded rings would announce a train’s departure, its type, and its intended path. This was the ‘ping’ of the 19th century—a binary signal indicating life and intent. But this system contained a critical, single point of failure that any modern SRE would recognize instantly: the signalman himself. He had to be present, awake, and attentive to hear the bell. There was no automated logging, no alert escalation to a secondary on-call engineer, no post-incident review. If the bell rang and he was distracted, dozing, or simply away from his post, the signal went unheard. The state of the world, from the perspective of the next station down the line, was dangerously incorrect.
This human latency could be catastrophic. A famous inquiry into a railway accident often cited ‘the signalman’s attention being otherwise engaged’ as a root cause. The bell had sounded, but the acknowledgement was never sent. The system assumed a ‘green’ status because it had not received an explicit ‘red,’ operating on a default pass rather than a confirmed healthy state. This is the historical antecedent of our modern ‘false positive’ in a health check—the assumption that because an error hasn’t been explicitly reported, everything must be functioning perfectly.
What the railway engineers eventually learned, and what we must continuously reinforce in our digital systems, is that any monitoring loop which relies on a human to close it is inherently unreliable. The move from the solitary signalman to automated block systems and, eventually, to fully computerized signaling was not just about efficiency; it was about building observability that was independent of human fatigue, distraction, or error. It was about creating a system that could declare its own status without waiting for a person to listen for its ring.
Our modern dashboards and alerting systems are the electric bells of our time. They are only as effective as the loops they complete. The lesson from the signal box is that we must design for the inevitable moment of inattention. We must build systems that fail safe, that escalate autonomously, and that do not assume a state of health based on a silent bell. For in that silence, much like the signalman who missed a crucial ring, we might just be hearing the approach of a catastrophe we failed to foresee.
Notes & further reading
A few pages I came back to while writing this: