ADR-0062: A mandatory kb explainer link on every notification, resolved from a notification catalog as code

This ADR records why pitlab makes a documented kb (“what & why”) link and a dashboard (“live view”) link mandatory on every notification, and resolves them from a single notification_catalog.yml keyed by (component, event) rather than hard-coding URLs at each emit site. It extends ADR-0034 (the machine-readable header) and ADR-0019 (enriched Alertmanager alerts), and is governed alongside the Notification Standard.

Status	Accepted
Date	2026-06-27
Deciders	Arron + Claude
Extends	ADR-0034, ADR-0019
Amended by	ADR-0064 (service-owned runbooks move to the service page; only fleet-wide alerts stay in operations/alerting)
Epic	ADO #1275 (Notification kb-link mandate & catalog); Issues #1276–#1283

Context — notifications were self-contained for triage but not for understanding

ADR-0034 made every notification carry a greppable header (host/component/event/severity/ts), and ADR-0019 gave Alertmanager alerts deep-links (PromQL/Grafana/Karma). But the ref field was only recommended, so a conforming notification could still arrive with no link at all — exactly what happened with the daily schedule-reconciler clean digest: a correct, well-formed info notification that told the reader what fired but offered no path to why it exists or what it means. A human glancing at the phone, or the agent handed the text as investigation input, had to leave the notification and reconstruct the context from elsewhere. The header answered “what/where”; nothing answered “what does this mean and why should I care”, and many script notifications had no live dashboard either.

Decision — kb + dashboard mandatory, resolved from a catalog, enforced in CI

1. A kb link is mandatory on every notification — a docs.pitbun.com URL whose heading anchor explains that specific event (what fired it, why it matters, first action). kb is a distinct field from ref (which remains for ADO ids / cross-references); the two are not merged.
2. A dashboard link is mandatory too — the live view. When a component has no dedicated dashboard, the helper falls back to a new parametric host-USE dashboard (uid: host-use, ?var-host=<host>) derived from --host, so even dynamic hosts (reboot-coordinator on bun3d/mesh01/pitups) resolve to a meaningful node-resource view rather than a generic page.
3. Granularity is per-event. Each (component, event) maps to its own anchor — reconcile_done and drift_found land on different sections.
4. The mapping lives in notification_catalog.yml as code (in the ansible repo, deployed beside pushover_notify.py). The helper resolves kb+dashboard by (component, event) — call sites already pass --component/--event, so most needed no change to gain both links. --kb/--dashboard override per call.
5. Resolution is fail-safe. A missing catalog, absent yaml module, parse error, or an uncatalogued pair never drops the notification: the helper renders kb=MISSING, logs a warning, and still sends. An operational signal is never lost for a missing link.
6. All three emitters conform via their native mechanism. Script/cron → the catalog. Prometheus/Loki alert rules → a kb annotation beside runbook/dashboard, rendered by the Alertmanager Pushover template. Zabbix triggers → a kb event tag surfaced by {EVENT.TAGS."kb"} in the media-type message.
7. A blocking CI gate enforces it (validate_notification_catalog.py, mirroring the schedule-tz guard): every kb anchor (catalog + alert-rule annotations + zabbix tags) must resolve to a real heading in pitlab-docs; every literal (component,event) emitted must be catalogued; a bare (no --component/--event) call fails the build.

Alternatives considered

• Keep ref recommended, enforce by review (status quo, ADR-0034). Rejected: review demonstrably let a linkless notification ship. The mandate needs a fail-closed gate, not a convention.
• One catalog as the source of truth for all three surfaces. Rejected: alert rules already keep runbook/dashboard inline (ADR-0019); externalising only kb would split a rule’s definition across two files. Each surface uses its native carrier; the CI gate validates them uniformly.
• Make the helper refuse to send without a kb. Rejected: a misconfigured script would then go silent and a real failure could be missed. kb=MISSING + warning keeps the signal and surfaces the gap.
• Hard-map per-host dashboards for the fallback. Rejected: not every host has its own dashboard, and reboot-coordinator fires from arbitrary hosts. One parametric host-USE dashboard covers every node-exporter host.
• Source the catalog from NetBox now. Deferred (see below): NetBox #1032 is infra hybrid-SoT (hosts/IPs/provisioning) and does not model per-component CIs, so it cannot supply this data today.

Consequences

• Positive: every notification — script, alert, or Zabbix — now carries an explainer and a live view, for human and agent alike. The catalog is one greppable source of truth; per-event anchors document the fleet’s notifications in one place (Notification Catalog, Alert reference). The gate makes drift impossible to merge silently.
• NetBox seam (the future producer): the helper consumes a rendered catalog file. Today ansible hand-authors it; if NetBox later grows service CIs with doc/dashboard custom fields, a NetBox export job becomes the producer with zero change to the helper, the notification contract, or the CI gate. NetBox is the earmarked upstream, not a dependency of this work.
• Negative / limits: the kb mandate adds a small authoring tax (a new event needs a documented anchor, or it ships kb=MISSING and fails CI on the literal pair). Pushover allows one supplementary URL button, so the kb gets the button and the dashboard is an inline linkified dashboard= line. The Zabbix kb on vendor-template triggers is empty by design (only pitlab-authored triggers carry the tag).
• As-built notes (reconciled 2026-06-28):
  • The catalog + helper deploy fleet-wide, not just to control01/pve01/docker01 — the reboot_coordinator role meta-depends on pushover_notify, so every reboot-fleet host (xt010/xt020/plex01/zabbix/mesh01/urbackup01/xt035/xv035/dispatcharr01/iventoy01 + the three) carries /etc/pitlab/notification_catalog.yml.
  • The ADO approval-pending trigger’s kb is applied by an idempotent managed change in zabbix_ado_monitoring.py (apply_kb_tags, Issue #1285), not a manual bootstrap as first scoped — it pushes the kb tag to the approval trigger and the fail-trigger prototype (discovered instances inherit via the next LLD cycle).
  • The blocking CI gate became registered as a self-hosted-pool pipeline only after a one-time agent-pool authorization; a newly created ADO YAML pipeline defaults to a hosted pool and stalls until authorized (Issue #1287 tracks the same misconfig on ansible-reboot-coordinator).
  • Cross-repo trigger (reconciled 2026-07-04, Epic #1448): the guard resolves anchors in pitlab-docs and alert-rule kb annotations in docker-stacks, but originally triggered only on ansible-repo pushes — so a docs-side anchor rename/removal or an alert-rule kb change broke the guard silently, latent until an unrelated ansible push tripped over it (bounded only by the weekly Saturday drift run). It now declares both siblings as resources.repositories with main-branch triggers (pipelines/validate_notification_catalog.yml, mirroring hass_config.yml), so a break fires the guard at the commit that causes it. Proven live: a pitlab-docs push triggered the guard via ci.triggerRepository=pitlab-docs. The drift run is retained as backstop.
  • Call-site scan scope + argv detection (reconciled 2026-07-10, Issue #1632): distinct from the cross-repo trigger above — the guard fired on docker-stacks pushes but its bare-call/coverage scan still walked only the ansible repo, so the five docker-stacks/scripts/ pipeline-gate emitters (testplan / synthetic / smoke / loki-log gates + the shared runner) shipped bare and undetected until one paged Arron with a non-conforming alert. The scan now also walks docker-stacks; is constant-aware (a helper path held in a PUSHOVER = ".../pushover_notify.py" constant and invoked as [..., PUSHOVER, "--flag"] is a call site, not a mere mention — the original blind spot); and reads the Python argv-list literal form ("--component", "x") using value-quoting as the literal-vs-variable discriminator (a quoted literal is verified against the catalog; an unquoted variable is left as a dynamic skip, never misread as a spurious pair). The stronger scan immediately surfaced three further uncatalogued-but-conforming emitters (zombie-reaper, the two homepage gates), now catalogued. Lesson: a policy-as-code gate must scan every repo that can host the pattern, not only its owning repo — trigger coverage and scan coverage are separate guarantees.
  • Zabbix kb-tag scan widened to playbooks (reconciled 2026-07-10, Issue #1638): the same “scan every host of the pattern” lesson, one layer down. collect_zabbix_kb originally read kb tags only from scripts/zabbix_*.py, so kb tags authored in the configure_*_zabbix.yml / zabbix_*.yml playbooks — the pve01 ZFS/IO tags (Issue #1633) and the fleet-wide backfill (Issue #1638, +28 tags) — shipped entirely outside the ANCHOR check: a playbook kb pointing at a 404 would have passed CI. The collector now also globs the playbooks and reads both YAML forms ({tag: kb, value: URL} and the per-trigger kb: URL field). This closed the gap that let the Zabbix-trigger half of the mandate drift unguarded while the catalog/script/alert-rule halves were enforced.