ADR-0078: Alerting severity taxonomy and routing contract

Status: Accepted

Date: 2026-06-30

Context

pitlab raised alerts from three surfaces, each with its own severity vocabulary and nothing reconciling them:

• Prometheus / Loki rules used severity: warning|critical — plus, in at least one rule, an off-taxonomy severity: high (PitMemoryIndexEmpty).
• Zabbix uses its native 6-tier scale: Not classified / Information / Warning / Average / High / Disaster.
• The alerting-notification-standard already defined a 3-tier info | warning | critical → Pushover priority 0|1|2 mapping for script/cron notifications, but no equivalent governed alert rules.

This unreconciled state produced a live defect. Alertmanager matches the severity label as an exact string and routes first-match-wins (alertmanager-routing); it had explicit matchers for warning and critical but none for high. The high alert therefore fell through to the default route, whose receiver was pushover-critical (emergency). A low-importance “memory index is empty” signal paged as a repeating, Do-Not-Disturb-overriding emergency — silently, because an off-taxonomy severity does not error, it mis-routes. There was also no single standard defining the shape of a conforming alert rule (required labels and annotations, including the mandatory kb link from ADR-0062/0064), so rule authoring drifted per-author.

Decision

Adopt a single Alerting & Severity Standard (alerting-notification-standard) that:

1. Defines a canonical 3-tier taxonomy — info | warning | critical — as the only valid values for the Prometheus severity label, aligned with the notification standard’s Pushover priority mapping (0|1|2). Any other value (high, error, major, domain words, …) is forbidden.
2. Reconciles all three surfaces with a cross-surface mapping table: Prometheus severity maps as the identity; Zabbix’s 6 tiers collapse onto the 3 canonical tiers via the “Balanced” profile (Not classified+Information→info; Warning+Average→warning; High+Disaster→critical), with the live Balanced nuance that Zabbix High pages at priority 1 to avoid double-paging against action #10.
3. Mandates explicit per-tier Alertmanager route matchers for info, warning, and critical. A tier is never assigned by default-route fallthrough; the default route is a safe backstop only.
4. Requires a conforming alert-rule shape — PascalCase alert name; severity (canonical) + identifying labels (host/service/instance/target); and annotations summary, description, runbook, dashboard, and a resolving kb (mandatory per ADR-0062/0064).

Alternatives considered

• Adopt the Prometheus 2-tier scale (warning|critical) only. Rejected: it discards the genuine operator value of Zabbix’s richer native granularity and provides no info tier for record-only signals. A 3-tier canonical scale keeps Zabbix’s nuance mappable while staying aligned with the existing notification standard.
• Keep the per-surface scales unreconciled (status quo). Rejected outright — this is exactly what caused the defect. With no mapping and no closed taxonomy, an off-taxonomy value silently mis-routes to the worst-case channel.
• Make the Alertmanager default route carry a tier. Rejected: relying on fallthrough is the mechanism that mis-paged high. Explicit per-tier routes make the failure visible (a no-match alert is obviously misconfigured) rather than silently emergency-paged.

Consequences

• The off-taxonomy severity: high was changed to critical in homelab.rules.yml (shipped to docker-stacks main) — the live PitMemoryIndexEmpty mis-routing defect is fixed.
• Alertmanager configuration (alertmanager.yml.j2) needs explicit info, warning, and critical route matchers, each declaring its own grouping/repeat/mute (no inheritance — alertmanager-routing).
• The mandatory kb annotation was absent from many live rules. kb backfill across existing alert rules and extending the validate_notification_catalog.py CI gate to cover Prometheus/Loki rule annotations is follow-on work, tracked separately — the mandate stands now regardless of enforcement coverage.
• Authors now have a single conformance checklist for new alerts; reviewers and /wrapup enforce it alongside the alerting-notification-standard and metrics-dashboards-standard checklists.

Consequence recorded 2026-08-22 — a tier may route to 'null', but it still needs its explicit route

ADR-0465 stopped the warning tier paging. That does not relax this ADR’s per-tier routing contract — it tightens the reading of it. A tier that must not page keeps its explicit route and points it at the 'null' receiver; deleting the route instead would leave those alerts matching no child route, at the mercy of whether some earlier continue: true tap happened to match, and otherwise falling through to the root receiver — the emergency channel, which is precisely the high mis-route this ADR closed. The severity label vocabulary is unchanged: still exactly info, warning, critical.

The mirror hazard is now live. This ADR guards against severity inflation; with two of the three tiers no longer paging, deflation — labelling a real fault warning to keep it quiet — silences it until someone looks. Tier choice is by whether the condition degrades further while unattended, never by the paging behaviour wanted.