ADR-0264: Sleep quiet-hours is a rest-protection mute, distinct from planned maintenance

Status: Accepted (amended 2026-08-21 — boundary moved 07:00 → 07:20) Date: 2026-07-26 Extends: ADR-0078 (severity taxonomy / routing) Amended by: ADR-0326 (priority-0 ceiling, 2026-08-05) and by the 2026-08-21 amendment below (Epic #2604, Issue #2606) Epic: #2046 (Actionable alerts + sleep quiet-hours) · Issues: #2048, #2049

Context

pitlab had a single overnight mute — the nightly-maintenance interval (Alertmanager) mirrored by Zabbix maintenance id=1 — covering 01:55–07:00 Australia/Melbourne. Its stated purpose was planned-maintenance suppression: the PBS backup window, patching, the reboot coordinator, the Sunday 02:58 UniFi auto-upgrade. But it was also, de facto, the only thing stopping a non-critical alert from waking Arron at 03:00 — a rest-protection job it was never sized for (it started at 01:55, leaving 21:00–01:55 unprotected).

These are two different concerns that happened to share one window:

• Planned-maintenance mute — scoped to specific hosts and a specific work window, set before intentional disruption and removed after (the mute-first discipline).
• Rest-protection (quiet-hours) mute — a standing, blanket, every-night window so routine noise does not cost sleep.

Conflating them meant the rest window was mis-sized, and there was no guard proving the standing blanket mute was not accidentally left covering the daytime — a mute stuck ON silently swallows every page with no error anywhere.

Decision

1. Define sleep quiet-hours as a first-class, distinct mute class: a standing 21:00–07:00 Australia/Melbourne, blanket, 7-nights rest-protection window. It replaces the 01:55–07:00 nightly-maintenance interval and fully subsumes it (all overnight maintenance churn stays covered), so there is still exactly one nightly window, now correctly sized.

2. Both surfaces, same span, DST-aware:

   • Alertmanager: the interval is renamed nightly-maintenance → quiet-hours and widened, declared on every leaf route (Alertmanager does not propagate mute_time_intervals from parent to child). The 21:00→07:00 span crosses midnight, which one times entry cannot express, so it is split into two same-day sub-ranges (21:00–24:00 + 00:00–07:00) OR’d under location: Australia/Melbourne.
   • Zabbix: maintenance id=1 → start_time 75600 (21:00), period 36000 (10h). Server-local AEST/AEDT. (Amended 2026-08-21: period 37200 — 10h20m, ending 07:20. See the amendment below.)
   • DST is automatic on both: the wall-clock 21:00/07:00 is bound to Melbourne local time, so the window follows the AEST/AEDT changeover with no code change.

3. Standing planned-maintenance mutes are unchanged. Per-host dynamic patch silences (the fleet patch coordinator’s scoped AM silence + Zabbix maintenance) remain the mute-first mechanism for intentional disruption; quiet-hours is orthogonal to them.

4. A stuck-mute guard is mandatory (Issue #2049):

   • Static CI gate (docker-stacks/scripts/quiet_hours_guard.py, wired into the observability deploy) asserts both surfaces declare the canonical 21:00–07:00 Melbourne span and are byte-identical as a covered-minute set — drift fails the deploy.
   • Runtime canary (quiet_hours_canary.py, daily ~13:00 AEST) queries the live surfaces and pages if the mute is still active at midday (stuck-on) or has drifted. Alert-on-failure only; quiet.hours.canary.age is its dead-man.

Amendment 2026-08-21 — the boundary moves to 07:20, and incident_always widens to pay for it

Epic #2604 (Dawn triage — act, then one report at 07:20) · Issue #2606. The window end moves 07:00 → 07:20 Australia/Melbourne. The span is now 21:00–07:20, 10h20m.

Why. A 06:50 autonomous triage actuator and the 07:00 health sweep now feed one merged report, delivered at 07:20. The point of a single report is that it is the first thing Pit sees. With the mute lifting at 07:00, every alert held overnight discharged twenty minutes ahead of the report, so he met the burst before the summary that explains it — which is the failure the merged report exists to remove. The mute end and the report time are therefore the same instant by construction, not by coincidence.

What moves, and why it is five artifacts rather than four. The span is declared in five places and they move together or a gate reds:

Artifact	Change
Alertmanager quiet-hours time_interval (docker-stacks .../alertmanager.yml.j2)	second sub-range end_time '07:00' → '07:20'
Zabbix maintenance id=1 (ansible playbooks/configure_zabbix_server.yml)	maintenance_period 36000 → 37200
The hard assert pinning the canonical span in that same playbook	== 36000 → == 37200
quiet_hours_guard.py static CI gate (docker-stacks/scripts/)	CANON_SPAN_MIN 10*60 → 10*60+20
quiet_hours_canary.py runtime canary (ansible/scripts/)	CANON_SPAN_MIN likewise

The fifth is the one worth naming. ADR-0264 §4 describes the guard and the canary as the two halves of one control, but only the guard was in the lock-step list anyone was working from. The canary carries its own copy of the canonical span and pages on "window drifted away from 21:00-07:00" — so moving the other four and leaving it behind would have fired a false stuck-mute page every day at 13:00, from the control whose entire job is to be trustworthy about mutes. A declaration whose enforcement lives elsewhere decays invisibly (FC-13); five is the real number.

The widened accepted risk, and the mitigation that is not optional. The residual risk below grows by twenty minutes — a genuine outage born in the window is now unseen until 07:20 rather than 07:00, and the 06:50–07:20 slice is newly muted, which is precisely when the dawn actuator is acting. That extra exposure was accepted only alongside a mitigation, because the more expensive half of the risk was never the twenty minutes: it is the ten hours that were already there. An alert that fires and clears inside the mute is lost from every muted route entirely (see the residual-risk section), and only 8 of 139 rules carried the incident_always exemption that preserves the FreeScout record (ADR-0318).

So the sweep required by this amendment applies incident_always: "true" to every rule meeting both conditions:

1. its firing means a scheduled or periodic job run terminally failed — errored out, produced nothing, did not run, or is stuck and will not complete — as distinct from continuous-service liveness, saturation, capacity, posture, drift, SLA-ageing or device availability; and
2. the condition can self-clear inside the ~10h20m window — an error-rate window rolling off, a stuck worker draining, or the next run of a sub-window-cadence job succeeding.

Condition 2 is what makes the sweep narrow rather than a blanket relabelling. A rule that is still firing when the mute lifts is delivered and loses nothing, so a multi-day staleness alert (ClaudePodStale, VectormapStale, TopchartsListsStale, ArronpitmanSiteBuildStale) is deliberately excluded — labelling it would add an incident record for an alert that already arrives. Fourteen rules qualified, taking coverage 8 → 22 of 139:

OrpheusTTSErrorRateHigh, PlexConversionErrors, N8nWorkflowExecutionFailures, ArrQueueStuck, PaperlessIngestionStalled, PitMemoryReindexErrors, PitMemoryIndexStaleVsDocs, Docker01CacSweepStale, MCPHealthCheckStale, DockerHubHeadroomExporterStale, ADOMetricsCollectorStale, IncidentZabbixPollerCycleErrors, IncidentZabbixPollerRecallErrors, IncidentWebhookRecallErrors.

Two rules were considered and rejected on the criterion, which is the evidence it does real work: FreeScoutHealFlapping fires when the heal job succeeded three times (recurrence, not a terminal failure), and FreeScoutHealFailed states in its own description that the web tier “will not self-recover” — it persists to the lift and is delivered. Paging is unchanged for all of them: recording an incident is silent and wakes nobody, so it never needed the mute that paging does.

Stated bounded acceptance, unchanged in kind: rules judged not-terminal, or terminal but persistent, remain lossy only in the fire-and-clear case, and that loss is now confined to a population that was derived by the stated property above rather than assembled by recall.

Also fixed under this amendment, because it lived in the same delivery path: the morning_health_publish.py AMBER/RED notifier passed Pushover priority=1, in breach of ADR-0326’s hard priority-0 ceiling. It was saved only by the MAX_PRIORITY clamp inside pushover_notify.py — a latent non-conformance that would have become real the moment the chokepoint changed. The caller now passes 0.

Accepted residual risk

A genuine outage born during 21:00–07:20 (21:00–07:00 as originally decided) is not paged until 07:20 on the muted (non-emergency) surfaces. This is deliberate — the whole point of rest protection — and bounded:

• Emergency (Pushover priority 2) still breaks through. Retired by ADR-0326 — nothing breaks through; the priority-0 ceiling is absolute. Retained struck-through because it was load-bearing rationale for accepting the risk at the time. The quiet-hours mute is a normal AM mute_time_intervals / Zabbix maintenance; a page-until-acted emergency (ADR-0263) is delivered by Pushover’s emergency retry, which overrides Do-Not-Disturb. Truly critical, act-now conditions (a blocking gate; outage/data-loss triggers routed emergency) are not silenced — only the routine warning/average tier is.
• In compliance terms this is an accepted control gap on the Detect function (NIST CSF Detect / CPS 230 operational-resilience monitoring) for the overnight window on the non-emergency tier, with T0 detection deferred to 07:20. Recorded here and in the design doc.

Alternatives considered

• Keep it named nightly-maintenance. Rejected: the name asserts a planned-maintenance semantic the window no longer has; the rename makes the config self-documenting and matches the new standard section (tidy-up discipline).
• Widen only Alertmanager, leave Zabbix at 01:55. Rejected: split-brain muting (a Zabbix agent trigger pages 21:00–01:55 while the Prometheus side is quiet). The two surfaces must agree — hence the CI gate that proves it.
• No canary, trust the CI gate. Rejected: the CI gate proves the committed config is correct but not that the live mute is behaving; a stuck-on mute (or an out-of-band edit) is exactly the silent failure — so a runtime canary that would actually fire on it is required (Prime Directive 12: prove a gate red before trusting it green).

Consequences

• One correctly-sized nightly window; Arron’s sleep is protected 21:00–07:20 (21:00–07:00 as first decided) without losing overnight-maintenance coverage.
• Since the 2026-08-21 amendment the window end is coupled to the merged morning report’s delivery time: moving one without the other reintroduces the pre-report alert burst, so they are changed together.
• Alerting gains a durable distinction — quiet-hours (rest) vs planned-maintenance (work) — captured in the Alerting & Notification Standard.
• The stuck-mute failure mode is now guarded on both the config (CI) and the live (canary) plane.