0359 — servicemap: a derived, fail-closed service-dependency explorer

Status

Accepted — 2026-08-08. Realises the servicemap design (Epic #2260). Builds on ADR-0059 (the Alloy probe inventory as the service source of truth) and ADR-0321 (the artifact acceptance gate). Frontend dependency governance is split into ADR-0360.

Context

The estate’s per-service documentation states each service’s endpoints, alerts and dashboards but shows nothing of its position. A reader on services/app/paperless.md could not see what Paperless depends on without opening a different page and finding it by eye. The estate-wide dependency map existed — generate_service_deps.py had derived it for some time — but it was one flat Mermaid diagram on one page, carrying no live state and appearing on no service page.

The prompting artefact was a beautiful hand-drawn architecture diagram. That is exactly what must be rejected: a hand-drawn estate diagram is stale the day after it is drawn, and worse, it is confidently stale — it looks authoritative while describing a system that has moved.

There is a sharper reason to care about correctness here than aesthetics. While building this we found that the existing derivation had been querying PostgreSQL over ssh postgresql — a host that does not exist and resolves NXDOMAIN. Every run had failed that source silently, and the published estate map had been reporting “PostgreSQL consumers: none” while six stacks were using it. PostgreSQL is not central in this estate at all; it is a per-stack sidecar container on docker01. The map had been wrong, confidently, and nothing said so.

That is the failure mode that shaped this design. A generated map’s interesting failure is not that it is down — it is that it is silently incomplete. nginx serves a truncated graph with exactly the same 200 as a complete one, and a bad run refreshes the freshness timestamp, so every availability and freshness signal reads healthy. An empty result and a genuinely dependency-free estate are indistinguishable at the output.

Decision

Build servicemap as its own service on the proven vectormap pattern — a stock nginx tier serving a generated bundle, a render job on a timer, and a sidecar for live data — with four decisions that follow from the context above.

1. Derive, never draw; one derivation, three renders. generate_service_deps.py derives the topology from sources already maintained for other reasons — the Alloy probe inventory (the service node set), the Ansible inventory (hosts and the observability plane), live PostgreSQL on docker01, and services.yml for the handful nothing can discover. From that single derivation it emits the graph JSON the D3 frontend loads, a per-service Mermaid subgraph injected into each generated service page, and the estate-wide docs map. Nothing is maintained twice, so the map cannot drift without something else breaking first.

2. The render asserts its own output before publishing, and fails closed. Two independent layers, because they catch different things:

• Structural — --require-sources aborts the run when any derivation source errored or returned nothing, before anything is staged, and asserts node coverage against the probe inventory.
• Calibrated — artifact_gate.py measures the staged map against the last known-good render.

Crucially the calibrated layer measures per source, not per graph. A whole-graph tolerance cannot catch one source dying: during development, a render with PostgreSQL unreachable lost all six database edges — 4.7% of a 127-edge graph — and sailed through a 0.9 whole-graph floor as “within tolerance”. Measured on its own axis, that same loss is −100%. Each derivation source therefore has its own measurement.

On breach the render keeps the last good map, increments a failure counter and pages. Publishing a degraded map is treated as worse than publishing a stale one, because a stale map announces its age on the freshness gauge while a degraded map announces nothing.

3. Topology baked, health live — and grey rather than a false green. The daily render bakes structure; a sidecar resolves probe_success live from obs-prometheus:9090 so nodes are coloured by current health. When the sidecar cannot reach Prometheus it publishes no health rather than stale health, and the whole map greys out. In the frontend, health is encoded as fill + glyph + written label, never fill alone: the up/down pair is green vs red, the classic deuteranopia confusion (measured CVD ΔE 4.1).

4. No credential anywhere. The sidecar joins the pki network and reads Prometheus by name on-host — the unauthenticated in-mesh route the observability stack already uses. Nothing in Infisical, nothing to rotate, nothing in the blast radius. Both containers are stock images with code mounted read-only (RULE 6 category 3), so Renovate keeps bumping the tags and Trivy/Dependency-Track keep scanning against a live upstream CVE feed.

The SLI is freshness, not availability: 99% over 30 days that the published map is no older than 24 hours. Availability is measured and alerted as baseline Pillar 1/3 but is deliberately not the SLI — a perfectly available map showing last month’s estate would score 100%.

Alternatives considered

Alternative	Why rejected
A hand-drawn architecture diagram	Stale the day after it is drawn, and confidently stale. The whole value here is that the diagram is derived.
A docs-pipeline HTML artifact instead of a service	HTML is dark to pit-memory retrieval, and it would refresh only on a docs push.
A new per-stack catalog-info.yaml (the Backstage model)	Richest semantics, but a new hand-maintained artefact needing its own conformance gate — the exact thing this design exists to avoid.
Live runtime state only	Cannot show intent or ownership, and shows nothing that is currently down.
Fully static map (no live health)	A node reads green while the service is down — a Prime Directive 12 proxy trap you would be tempted to trust.
Fully live per request	Slowest and most fragile, and the map goes blank precisely when you want to look at it.
A single whole-graph completeness tolerance	Measured and rejected during implementation. It accepted a map that had lost an entire derivation source.
Keeping the postgresql host in the derivation	The host does not exist. The instances are now derived live from the running containers and their compose-project labels, which is also a better mapping than the name-matching heuristic it replaces.
A custom-built sidecar image (what vectormap-search does)	Consistent with the sibling, but silently takes ownership of base-image currency — the outcome RULE 6 cat 3 exists to prevent.

Consequences

Good. The dependency picture now reaches both readers of the corpus from one source — Arron via the interactive map and the MkDocs service pages, the agent via retrievable Mermaid in markdown. A confidently-wrong map is structurally hard to publish. Correcting the PostgreSQL derivation fixed a live, months-old inaccuracy in the published estate documentation as a side effect, and the per-stack instance model it revealed is a more honest picture of blast radius than “one shared cluster” ever was.

Costs and caveats.

• The completeness floor calibrates against the last accepted render, so a deliberate large retirement will red the gate. That is the gate working; the runbook documents re-seeding the baseline explicitly rather than widening a tolerance.
• The drift alert’s 2h tolerance is a starting estimate, to be tuned against observed churn during the early-life-support window rather than defended as if it were measured.
• Only services carrying an Alloy probe appear. A service outside the probe inventory is invisible here — but it is equally invisible to the dashboards and the rollup, so the gap is not new.
• The sidecar is deliberately dependency-free stdlib, so there is no pip requirements file to SBOM. Vulnerability coverage for the Python side is therefore the stock container image via Trivy → Dependency-Track, which is where the packages actually are. This is a narrowing of the design’s “pip requirements” wording in the safe direction: fewer dependencies, not fewer scans.
• servicemap has no downstream consumers and holds no unique state, so its blast radius on failure or removal is nil.

Consequences — added at review (2026-08-10)

Derivation breadth is the binding constraint on the map’s value, and it was under-set at tranche 1. The decision to derive rather than draw is vindicated — the map has never been wrong — but the delivered derivation reads four sources and only two of them emit dependency edges, so 14 of 129 edges carry architectural information and the rest are placement and observability plumbing. The estate’s real wiring sits in sources that were already maintained and simply not read: the compose files, the Caddyfile, the Alloy scrape config, pvesh, the PBS API and tunnel.tf.

The lesson generalises beyond servicemap: “derived, never drawn” is only as good as the number of sources you derive from. A derived map with too few sources is not neutral — it is confidently partial, which is the same failure mode as a hand-drawn one, arrived at more honestly. The completeness floor protects against a source dying; nothing protects against a source never being added.

Tracked as Issues #2317 (depth: compose + Caddy + Alloy scrape) and #2318 (breadth: pve01 guests, backup chain, exposure path) under Epic #2260, parked in the backlog 2026-08-10.