The Myth of the Single Source of Truth

We build our systems on a promise, a comforting fiction whispered in architecture review meetings and etched into our monitoring dashboards: the Single Source of Truth. It is the pristine database cluster, the golden record, the one canonical dataset to which all services must bow. We design our health checks to ping it, our uptime monitors to worship at its altar. If it is up and responding with low latency, we reason, then the core of our system is healthy. This is a dangerous and seductive illusion.

The problem is not with the data repository itself, but with the faith we place in its solitary verdict. We mistake a single component’s operational health for the holistic truth of our system’s state. A database can be utterly healthy—processing queries with lightning speed, returning 200 status codes to every health check—while the ecosystem around it is crumbling. A cache layer could be serving catastrophically stale data. A background ETL process might have failed silently hours ago, leaving the data technically consistent but functionally useless. The database, our supposed ‘source of truth,’ is oblivious. It is a perfectly healthy heart unaware that the lungs have stopped breathing.

This obsession creates a brittle form of observability. We are lulled into a false sense of security by a steady stream of green checkmarks from one critical node. We become like a watchman who only ever checks the main gate, confident that all is well because the lock is firm, while the walls have been quietly scaled elsewhere. Our monitoring becomes a form of theater, a performance of reliability that masks deeper, more systemic fragilities.

True observability doesn’t emerge from a single point, no matter how vital. It is a symphony played by a distributed orchestra of signals. It is the correlation between the database’s latency and the application’s error rate. It’s the business metric—a sudden, inexplicable drop in completed transactions—that tells a truer story than any HTTP 200 ever could. It is the health of the pathways between services, the consistency of the data in flight, not just its state at rest in the designated temple.

We must abandon the quest for a single source and instead embrace a practice of seeking convergent truths. Health is not a binary output from a privileged component; it is a emergent property of a complex, interacting system. Our vigilance should be directed not at a lone beacon, but at the entire landscape, listening for the dissonance between different instruments that signals the real failure, the one that happens between the lines of our perfect pings.

Notes & further reading

A few pages I came back to while writing this: