The map that could not be trusted

The short version

Bottom line. A generated dependency map reported no consumers for a database six stacks were using, and served that answer with a healthy 200 and a fresh timestamp.

• What it cost — months of a silently incomplete map that every availability signal read as healthy, because the failing run refreshed the freshness stamp too.
• What changed — the render now stages, gates and only then promotes, with a per-source completeness tolerance. A whole-graph threshold cannot catch one source dying: every database edge is under five per cent of the graph.
• What it costs — the estate sometimes serves a map a day old rather than a fresh one that is slightly wrong. Staleness announces itself; incompleteness does not.

The problem

The estate publishes its service-dependency graph as a generated artifact: nodes and edges derived from sources that are already maintained for other reasons, so the map cannot drift without something else breaking first.

The failure of a generated map is not that it goes down. It is that it goes silently incomplete. If a derivation source becomes unreachable, the render skips everything that source would have contributed and emits a beautiful, confident, half-empty graph. The web server returns it with the same 200 as a complete one, and — this is the part that closes every other escape route — the bad run refreshes the freshness timestamp, so the staleness signal reads healthy too. Availability alerting is structurally blind to it.

This was not hypothetical. The derivation queried the database over a hostname that did not exist and resolved to nothing. Every run failed that source silently, and the published map reported “database consumers: none” while six stacks were using it.

The design

The render was restructured into stage → gate → promote, and it never writes into the served path directly. Two independent completeness layers sit between staging and promotion, and both fail closed:

Layer	What it catches	How it fails
Structural	A source that died — errored or returned nothing at all. Also asserts node coverage against the probe inventory	Aborts before anything is staged
Calibrated	A source that answered thinly — reachable, returned some data, returned much less than last time	Rejects the staged map; the previous one stays live

The second layer is per-source rather than whole-graph, and that is the load-bearing detail. A whole-graph tolerance cannot catch one source dying: losing every database edge is under five per cent of the graph, which any sane global threshold would wave through. Each source gets its own axis.

On rejection the previous map stays live, a metric moves, and a notification goes out. Publishing a degraded map is worse than publishing a stale one, because a stale map announces its age on its own header while a degraded map announces nothing at all.

The trade-off

The obvious objection is that the estate now sometimes serves a map that is a day old when it could have served a fresh one that was merely slightly wrong. That is the deliberate choice, and it is not free: someone reading a stale map can be misled about a service added yesterday.

Two things make it the right side of the trade. Staleness is visible — the map’s own header shows its age and turns amber past a threshold — whereas incompleteness is invisible by construction. And freshness is not left to hope: it has an SLO with a real error budget and a written policy for what happens when the budget is spent, rather than an alert that fires on every slow day and gets muted (ADR-0371).

The second cost is calibration. A per-source tolerance has to be derived from what that source actually produces, and a borrowed number would produce a gate that reviews cleanly and cannot work. The tolerances came from measuring each source’s own variance, not from a default.

The evidence

• The failing source was found by the structural layer on its first armed run — the same condition that had been failing silently for months.
• The gate is exercised by a proven-red test rather than trusted: it is driven over a staged graph with one source removed and required to reject.
• The freshness SLO and its budget are queryable on the estate’s own dashboard, and the error-budget policy names the action taken at each burn level.

Full reasoning: ADR-0359 — a derived, fail-closed dependency explorer and ADR-0321 — run-to-completion workloads are gated on their produced artifact, not their exit code.

You can walk the sanitised graph on the estate topology page.