ADR-0064: Service-owned alert runbooks live on the service page, fleet-wide alerts stay in operations

This ADR records why the per-alert “KB” runbooks split out of the single operations/alerting reference and onto each owning service’s catalog page, while genuinely fleet-wide alerts stay put. It amends ADR-0062 (which mandated a kb link and parked every per-alert explainer in one flat reference) and builds on ADR-0063 (umbrella pages) and ADR-0059 (the generated catalog).


Status	Accepted
Date	2026-06-28
Deciders	Arron + Claude
Amends	ADR-0062
Builds on	ADR-0063, ADR-0059
Epic	ADO #1289 (Alert runbooks → service pages + umbrella catalog)

Context — one flat reference made the alerting doc do two unrelated jobs

ADR-0062 mandated a kb link on every alert and authored all ~50 per-alert explainers as anchors in operations/alerting.md. That co-located two different kinds of content: the alerting architecture (dependency chains, inhibit rules, Pushover enrichment, how to add a rule — genuinely cross-cutting and correctly operational) and a flat per-alert runbook catalog. Most of those runbooks are not cross-cutting at all: “sonarr reports a health issue”, “a Plex mobile sync failed”, “pit-memory semantic search degraded” are facts about one service, and a reader who lands on the sonarr page expecting to find what its alerts mean instead finds nothing — the knowledge lives a repo-section away under operations. The KB link in the page that fired (visible in the alert screenshot that prompted this work) pointed at operations/alerting/#arr-health-issue, not at sonarr.

Decision — runbooks follow ownership; the rule’s kb routes to the owner

A service-owned alert’s runbook lives on its service (or umbrella) catalog page. The split rule: service-scoped and the service has a generated page → the runbook moves; otherwise it stays in operations. ~32 alerts move (the *arr apps, Plex, the podcast pipeline, n8n, Paperless, Gotenberg, pit-memory, vectormap, Dependency-Track, Infisical, the Cloudflare tunnel, Pi-hole, Zabbix); ~18 fleet-wide alerts stay (host liveness, disk, memory, hypervisor, fleet collector, clock, containers, cross-service readiness, kernel journal, secret rotation).
The runbook prose is authored as data in ansible/scripts/doc_gen/alert_runbooks.yml — one entry per alert: heading, anchor (identical to the old slug, so deep-links survive), the owning pages, the literal kb: value, and the body. The catalog generator renders each entry into an ## Alerts section on every page it names. This keeps the generated page as-code (ADR-0059): the prose has a machine-readable home rather than being hand-pasted into a generated file.
The rule’s kb: annotation points at the owner page, not at operations. For a rule that fans out across instances (the shared *arr system_health_issues/queue_total rule, whose $labels.job is radarr/sonarr/lidarr-cd/lidarr-mp3), the URL is templated by label — …/services/media/{{ $labels.job }}/#<slug> — so each instance’s page resolves. Where the firing label does not map 1:1 to a page (Pi-hole’s $labels.pihole), the kb: is a static link to the umbrella page (ADR-0063).
operations/alerting keeps the architecture and the fleet-wide alert reference. It is no longer the home of service runbooks; its reference section now documents only alerts owned by no single service.
The CI guard expands templated kb links. validate_notification_catalog.py reads alert_runbooks.yml: a {{ … }} kb URL is verified by checking the anchor resolves on every concrete page the template declares, so a fan-out link cannot silently break for one instance. The guard also treats alert_runbooks.yml as the source of truth for the page’s ## Alerts anchors — validating a rule’s kb: anchor against the runbook entry’s declared anchor (unioned with the committed markdown), not against the async-regenerated page alone. This decouples the guard from regen timing so a new alert+runbook passes CI on the first run rather than failing until the regen rebuilds the page (ADR-0085).

Alternatives considered

Leave everything in operations/alerting (status quo, ADR-0062). Rejected: it forces the reader away from the service to learn what the service’s own alert means, and grows one unbounded flat list.
Move every alert to a service page. Rejected: host/disk/kernel/clock/readiness alerts have no single service owner; inventing synthetic homes for them fights the meaning of the services/ tier. The split is by ownership, not wholesale.
Hand-author an ## Alerts section directly in each service page. Rejected: service pages are generated (ADR-0059) and overwrite hand edits; the prose must live in a data file the generator reads.
*Give every arr instance its own rule to avoid templating. Rejected: one regex rule is the existing, simpler design; Go-templating the kb by $labels.job (already used for the summary) reuses it for the link.
A second new operations/infrastructure-alert-runbook.md for the fleet alerts. Rejected as unnecessary churn: the fleet alerts already live correctly in operations/alerting; only the service-owned ones needed to move.

Consequences

Positive: the KB link on a service’s alert now lands on that service’s page, beside its identity, endpoints, and dashboard — for human and agent alike. operations/alerting shrinks to its real job (architecture + fleet alerts). Old operations/alerting/#<slug> deep-links for moved alerts are superseded by the service-page anchors (same slug), and the blocking CI gate proves every link — including the fan-out templated ones — resolves.
Negative / limits: authoring a new service alert now touches alert_runbooks.yml (not operations/alerting.md), and the author must set the right owner pages; a wrong page name fails the generator’s orphan_pages guard rather than shipping a dead link. A reader following an old bookmark to operations/alerting/#arr-health-issue lands on the (still-valid) page but no longer finds that anchor there — it now lives on the service page.
As-built: 32 rule kb: annotations rewritten (26 Prometheus + 6 Loki); 18 fleet anchors retained in operations/alerting; 3 umbrella pages added (ADR-0063); promtool check rules and the notification-catalog guard both green.