ADR-0326: Never wake the operator — no notification exceeds Pushover priority 0

Nothing in pitlab is urgent enough to override sleep. Every path to Pushover is pinned to priority 0, which removes the last two mechanisms that could break through Do-Not-Disturb, and the ceiling is enforced at a chokepoint rather than trusted to call sites.

Status

Accepted — 2026-08-05. Amends ADR-0264.

Context

ADR-0264 established quiet-hours (21:00–07:00) as rest protection, but deliberately left a carve-out: “Emergency still breaks through — quiet-hours is a tier filter, not a kill switch”. A Pushover priority-2 emergency, or a priority-1 high, would still override Do-Not-Disturb.

The question surfaced concretely while wiring up notifications for the fleet patch coordinator. Asked directly whether a failed 04:00 patch cycle would wake him, the honest answer was yes — all five of the coordinator’s failure paths used priority 1, which bypasses quiet hours. The operator’s ruling was unambiguous: “Never wake me up. Nothing is that serious in pitlab.”

Auditing the estate for what could actually break through found three independent paths, and a fix to any one would have left the others live:

1. Scripts via pushover_notify.py — SEVERITY_PRIORITY mapped warning → 1 and critical → 2, with a legacy default of 1; 21-plus call sites also passed --priority 1 explicitly.
2. Zabbix — posts to Pushover directly through its own media type, mapping Warning/Average/High → 1 and Disaster → 2.
3. Alertmanager — posts directly too. pushover-warning set priority: "0", but pushover-critical set no priority at all — and Alertmanager’s Pushover default is 2 (Emergency), a one-minute retry loop for up to an hour. The single receiver most likely to fire overnight was the loudest thing in the estate purely by omission.

The justification for the ruling is a property of the estate rather than a preference about noise: the systems that can fail overnight all fail SAFE. The patch coordinator halts the rollout and rolls the host back to its pre-patch snapshot; backups and restore drills abort leaving the prior artifact intact; deploy gates refuse to promote. Nothing degrades further while unattended, so there is no 3am action that could not wait until 07:00.

Decision

No notification originating anywhere in pitlab may exceed Pushover priority 0. Priority -1 (quiet) remains available; 1 and 2 are forbidden.

Each of the three paths is pinned independently, because none of them shares an enforcement point with the others:

• pushover_notify.py — SEVERITY_PRIORITY becomes {info:0, warning:0, critical:0}, and a MAX_PRIORITY = 0 clamp downgrades any explicit --priority 1/2 at the point of send, logging the clamp to stderr.
• zabbix_pushover_media_standard.py — every tier set to "0", priority_disaster included.
• alertmanager.yml.j2 — priority: "0" set explicitly on every pushover_configs block, never left absent.

Severity is not abolished — it still drives routing, grouping, repeat intervals, the incident record and the failover email. It simply no longer drives loudness.

Alternatives considered

• Sweep the 21-plus --priority 1 call sites and leave the helper permissive. Rejected. It fixes today’s scripts and silently regresses the moment anyone adds --priority 1 to a new one — the failure mode being that the regression is invisible until it pages at 3am. A chokepoint clamp makes a future violation a no-op. This is the “a declared control needs an enforcement point” discipline from ADR-0317 applied to notification policy.
• Keep priority 2 for a genuinely catastrophic tier (active data loss, hypervisor down). Rejected on the operator’s explicit instruction, and defensible on the merits: the overnight-capable systems fail safe, and a tier that fires once a year is a tier nobody trusts or tunes. A carve-out is also exactly what ADR-0264 tried and what this ADR is removing.
• Rely on Pushover’s own per-device quiet hours. Rejected — that is user-side configuration invisible to this repo, unversioned, and silently bypassed by priority 1/2 anyway. Policy belongs where it can be reviewed in a diff.
• Drop the coordinator’s pages only (the immediate trigger). Rejected as too narrow; the audit showed Alertmanager’s pushover-critical was the bigger exposure and was not the thing being asked about.

Consequences

• A genuine emergency born at 02:00 will not wake anyone and is seen at 07:00 — 07:20 since 2026-08-21, when ADR-0264’s amendment moved the quiet-hours boundary so the merged morning report lands ahead of the overnight burst (Epic #2604). Stated plainly because it is the whole cost of the decision; accepted knowingly on the fail-safe reasoning above.
• Overnight visibility now rests on things that do not need to be loud: the FreeScout incident record (which ADR-0318 made immune to the quiet-hours mute), the Zabbix dead-men, and the morning summary reports.
• MAX_PRIORITY is the single lever if the policy is ever revisited — changing it in one place restores paging for every script path at once. The Zabbix and Alertmanager paths would still need their own edits, which the standard records.
• The Pushover emergency retry/expire parameters in pushover_notify.py become dead code paths, retained rather than deleted so the clamp is a policy decision at one line rather than a capability that has to be rebuilt if reversed.
• Anything that genuinely needs the operator awake now has no mechanism to achieve it. That is intentional, and any future request for one should reopen this ADR rather than add a second carve-out.