Alerting & Notification Standard

This is the single domain standard for how pitlab raises alerts and delivers operational notifications — one home for the three decisions that used to live as separate micro-standards: how every Pushover notification is structured so a human and the agent can both act on it (notification content), how every alert is authored, severity-classified, routed, and tuned (alerting & severity), and how a labelled Prometheus counter must be instrumented so a rate()/increase()/delta() alert over it can actually fire (alertable counter instrumentation). Each is preserved below as a self-contained, subject-named rule section with its conformance checklist verbatim; the consolidated Enforcement table carries every machine, review, and advisory obligation across all three. This domain-unit consolidation is governed by ADR-0252; the per-decision rationale lives in the ADRs cited inline in each section (ADR-0034, ADR-0078, ADR-0245).

How this standard is organised — three rule sections, one domain

This standard folds three former standards into one domain per the domain-unit rule (the unit of a standard is the domain, ADR-0113 / ADR-0252). Read the section for the concern you have; each is self-contained with its own conformance checklist:

• Notification content — the machine-readable header, catalog-resolved kb/dashboard links, severity→priority mapping, and the single pushover_notify.py emitter for every script/cron/agent notification (ADR-0034).
• Alerting & severity — the closed three-tier severity taxonomy, per-tier Alertmanager routing, the required alert-rule shape, the cross-surface mapping, and the alert-quality tuning rules (ADR-0078).
• Alertable counter instrumentation — priming every labelled counter child at 0 and backing a discrete-run alert with a last-run-status gauge so a rate()/increase()/delta() alert can actually fire (ADR-0245).

---

Notification content standard — self-contained, machine-readable Pushover notifications

This is the authoring standard for every operational notification pitlab sends to Arron’s phone via Pushover. It exists because notifications have two consumers: a human glancing at a phone, and the Claude Code agent that may be handed the notification text as the input to an investigation. A notification that carries only prose forces both to leave it and re-derive the where, what and how from another tool. Write to this standard and the notification is self-contained — host, component, event, severity and timestamp are machine-parseable, a documented kb explainer (“what & why”) and a dashboard (“live view”) are one tap away, and the first triage step needs no other tool. This is the script/cron counterpart to ADR-0019 (which enriched Alertmanager alerts) and is governed by ADR-0034 and ADR-0062 (the mandatory kb/dashboard links, resolved from a catalog as code, enforced in CI). It defines content structure; ADR-0033 governs the channel (Pushover, never email).

Scope — every script, cron and agent notification, not just alerts

The standard applies to all Pushover notifications emitted anywhere in pitlab: cron scripts, Ansible-deployed jobs, agent actions, rotation and backup outcomes, watchdogs. Prometheus alerts routed through Alertmanager already satisfy it via the enriched receiver templates (ADR-0019) — those are the reference implementation. Everything else emits through the pushover_notify.py helper and must pass it the structured flags below. A bare --title/--message notification is non-conforming.

The required shape — a machine-readable header, then human detail

Every notification has two parts: a header block of key=value lines an agent can grep, then a blank line, then the human-readable detail. The helper renders this for you from its flags.

host=docker01
component=rclone-files-sync
event=sync_failed
severity=warning
count=3
ts=2026-06-22T23:05:00+10:00

3 of 412 files failed to sync to Backblaze b2:pitlab-files after 2 retries.

kb=https://docs.pitbun.com/docker/rclone-files-backup/#sync-failed
dashboard=https://grafana.pitbun.com/d/host-use/host-use?var-host=docker01
ref=https://docs.pitbun.com/operations/offsite-backup/

The title is a separate, also-structured line: [pitlab] <component>: <outcome>. The kb and dashboard footer lines are bare URLs (Pushover auto-linkifies them, so both are tappable, and an agent can grep ^kb= / ^dashboard=); the kb link is also the Pushover supplementary URL button.

Required and optional fields — what every header must carry

Field	Required	Meaning	Form
host	yes	the host/service the event concerns	DNS shortname (docker01, pve01, control01) — never an IP
component	yes	the emitting script/service	kebab-case (feedback-hygiene, ado-pat-rotate, ups-watchdog)
event	yes	the machine event name	snake_case, stable across runs (sync_failed, rotation_ok, dedup_candidates)
severity	yes	triage urgency	info | warning | critical
ts	yes	when it happened	ISO-8601 with offset, Melbourne local — rendered by the helper
count / metric	where apt	the salient number	key=value (count=3, evicted=5)
kb	yes	explainer (what & why) for this event	docs.pitbun.com URL with a heading anchor — resolved from the catalog, not hand-passed
dashboard	yes	the live view	Grafana URL — from the catalog, else the host-USE fallback derived from host
ref	optional	ADO id or a cross-reference doc, distinct from kb	URL or #NNNN
url	optional	override the Pushover supplementary URL button (defaults to kb)	URL (+ url_title)

event names are part of the contract — keep them stable so an agent can route or correlate on them across runs. Pair an outcome with its inverse where both fire (rotation_ok / rotation_failed). kb and dashboard are mandatory (ADR-0062) but you do not pass them per call — the helper resolves them from the catalog by (component, event); see below.

Severity drives priority — one mapping, no per-script drift

severity sets the Pushover priority automatically, so a script never hard-codes a number:

severity	Pushover priority	Use for
info	0 (normal)	FYI outcomes, completed sweeps, candidates to review
warning	1 (high)	degraded state, partial failure, action expected soon
critical	2 (emergency)	outage, data-loss risk, immediate action

An explicit --priority still wins when a script genuinely needs -1 (quiet) — but severity must always be present for the header.

kb and dashboard are resolved from the catalog, not hand-passed

kb and dashboard are mandatory, but a script does not spell them out at each call site. The helper looks them up in notification_catalog.yml (deployed to /etc/pitlab/notification_catalog.yml beside the helper) keyed by (component, event) — the two flags the call already passes. The catalog is the single source of truth, hand-authored in the ansible repo today and earmarked to be produced by NetBox once it models service CIs (ADR-0062, the producer seam).

• kb is per-event and required in the catalog: a docs.pitbun.com path to a heading anchor that explains that event. Add a catalogued event without a kb and the blocking CI gate fails the build.
• dashboard is optional in the catalog: a component’s own Grafana URL when it has one, else the helper falls back to the parametric host-USE dashboard (/d/host-use?var-host=<host>, derived from --host) — meaningful even for dynamic hosts (e.g. reboot-coordinator on bun3d/mesh01).
• Resolution is fail-safe. A missing catalog, absent yaml, parse error, or an uncatalogued (component,event) never drops the notification: the helper renders kb=MISSING, logs a warning, and still sends. Never lose an operational signal for a missing link.
• --kb / --dashboard override the catalog for a one-off; the other two emitters carry kb in their native carrier — Prometheus/Loki alert rules via a kb annotation, Zabbix triggers via a kb event tag.
• The catalog governs only the pushover_notify.py (script/cron) path. A Zabbix trigger does not consult notification_catalog.yml: it carries its explainer inline as a per-trigger kb event tag, authored as code in the trigger definition (configure_*_zabbix.yml / zabbix_*_monitoring.py) and surfaced by kb={EVENT.TAGS."kb"} in the media type 89 template. Same mandate and same anchor-placement rules (below), different resolution mechanism — the catalog keys (component,event), the Zabbix tag is the URL itself. Both are proven by the same CI gate.

A kb anchor lives with the thing it explains — service-owned vs fleet-wide

A kb link must not only resolve — it must point at the owner of what fired (ADR-0064). The CI gate enforces resolution; placement is an authoring judgment this standard governs. Classify every new notification before writing its kb:

• Service-owned — the signal belongs to one service (or umbrella): ArrHealthIssue on sonarr, PaperlessConsumerErrors, a script that maintains one service. Author the runbook on that service’s catalog page, never in operations/alerting:
  • Alert rules → add the entry to ansible/scripts/doc_gen/alert_runbooks.yml (owner pages, anchor, body); the catalog generator renders it into the page’s ## Alerts section. Point the rule’s kb: at …/services/<domain>/<service>/#<slug>. For a rule that fans out across instances, template the path by label — …/services/media/{{ $labels.job }}/#<slug> — so each instance lands on its own page (the shared *arr rule).
  • Script/cron + Zabbix → set the catalog kb (or the Zabbix kb tag) to the same …/services/<domain>/<service>/#<slug> anchor.
• Fleet-wide — the signal belongs to no single service: host liveness, disk, memory, the hypervisor, clock, containers, cross-service readiness, the kernel journal, secret rotation. Author the runbook in operations/alerting and point kb there.

The test: if a reader followed this link, is the page they land on the home of the thing that broke? If the broken thing has a service page, that page is the answer — not the fleet alerting reference. A service-owned runbook parked in operations/alerting still resolves (so CI stays green) but violates this standard; a reviewer should bounce it. See ADR-0063 (umbrella pages) and ADR-0064 (this split).

How to emit a conforming notification — the helper does the rendering

pushover_notify.py (on control01 + pve01) renders the header from structured flags. Pass them instead of pre-formatting prose:

# Direct
pushover_notify.py --title "[pitlab] ups-watchdog: on battery" \
  --host pve01 --component ups-watchdog --event on_battery --severity warning \
  --field runtime_s=540 --ref https://docs.pitbun.com/operations/ups/ \
  --message "UPS on battery; estimated 9 min runtime. Graceful-shutdown armed."

# Piped (cron stdout becomes the human body)
some_job 2>&1 | pushover_notify.py --title "[pitlab] media-cleanup: done" \
  --host docker01 --component media-cleanup --event cleanup_ok --severity info

The flags (--host --component --event --severity --field KEY=VALUE --ref --kb --dashboard --url --url-title) are backward-compatible: a legacy --title/--message-only call still sends, it is just non-conforming and must be migrated. You normally pass only --component/--event and let the catalog supply kb/dashboard; --kb/--dashboard are overrides for one-offs.

Emit in a form the CI guard can see. Reach the helper either by a direct pushover_notify.py … call or by assigning it to a per-script constant and invoking that — PUSHOVER=/usr/local/bin/pushover_notify.py (shell-unquoted, shell-quoted, or python-quoted are all recognised) then "$PUSHOVER" --component … --event …. The static guard (validate_notification_catalog.py) windows the call region from the helper name or the recognised constant; a call reached through some other indirection it cannot localise (an env-var default, an eval, a dynamically-built path) is invisible to the guard and — for a bare-form file — now trips the coverage / non-attribution check rather than passing silently. The blind spot that motivated this rule (an unquoted $PUSHOVER constant the guard failed to recognise) silently masked 13 real violations across 8 scripts until Epic #1879 / ADR-0222. Do not invent a novel invocation idiom to save a line — use one of the two recognised forms.

Good vs bad — the same notification, two ways

A non-conforming notification (prose only — the reader must open another tool to find the host and the next step):

Title: Feedback dedup candidates Body: 9 overlapping feedback pair(s) to review/merge: …

The conforming version (self-contained for human and agent triage):

Title: [pitlab] feedback-hygiene: dedup candidates Body: host=control01 / component=feedback-hygiene / event=dedup_candidates / severity=info / count=9 / ts=… — then the detail — then ref=…

Constraints — body limit and parse-safety

• Pushover body is 1024 characters. The header costs ~120; keep the human detail compact and truncate long lists (e.g. top 8 items) rather than letting Pushover silently cut the body. State the truncation.
• Header keys are ^[a-z0-9_]+= at line start — lowercase, no spaces — so an agent can parse with a trivial grep. Values may contain spaces; keys may not.
• One event per notification. High-cardinality sources group by their differentiating field upstream (mirrors the ADR-0019 group_by constraint) so each notification carries one event, not a truncated bundle.

---

Alerting & severity standard — canonical taxonomy, routing, and alert-rule shape

This is the authoring standard for every alert pitlab raises — Prometheus/Loki rules through Alertmanager and Zabbix triggers alike — and the single reconciliation of pitlab’s three competing severity vocabularies. It exists because three surfaces (Prometheus, Alertmanager, Zabbix) each carried their own severity words, nothing mapped them, and a stray off-taxonomy high severity slipped past Alertmanager’s matchers and was paged as a phone-shattering emergency for days before anyone noticed. This standard fixes that class of defect at the root: one canonical severity taxonomy, an explicit per-tier routing contract, a required alert-rule shape, and a cross-surface mapping so a Zabbix Disaster and a Prometheus critical mean the same thing and page the same way. It is the Pillar 3 (Alerting) companion to the alerting-notification-standard (which governs content and channel) and is governed by ADR-0078.

Canonical severity is a three-tier taxonomy: info | warning | critical

pitlab has exactly three severity tiers, and they are the only valid values: info, warning, critical. They match the tiers the alerting-notification-standard already defines and the Pushover priorities they drive. There is no fourth tier, no synonym, and no surface-specific extra value. Choosing a tier is choosing how the event pages:

Tier	Meaning — pick by impact	Typical paging behaviour
critical	Page now: user-facing outage, data-loss risk, a fault that will become one if untouched, or a blocking gate that only Arron can clear (e.g. a pending ADO approval — the page-until-acted archetype).	Pushover priority 0 — see never-wake-me. Severity still drives routing, grouping, repeat and the incident record; it no longer drives loudness.
warning	Degraded or partial failure: needs attention this day / next day, not this minute.	NEVER sent to Pushover — recorded and reviewed, not paged, except six named alerts. See warning is recorded, not paged.
info	Record only: completed sweeps, FYI outcomes, candidates to review.	NEVER sent to Pushover — suppressed at the chokepoint and recorded to Loki instead. See info is never paged.

The discipline is to reserve critical for things genuinely worth waking up for. Severity inflation — labelling a degraded-but-serving condition critical — is the failure mode this tier guidance exists to prevent: when everything pages as emergency, nothing does.

Informational severity is never sent to Pushover — it is recorded, not paged

severity=info never reaches the phone. pushover_notify.py suppresses it at the chokepoint, prints the event to stderr — which journald ships to Loki — and exits 0. Suppression is a success, not a failure: a silenced info is not a delivery error, and callers that treat a non-zero exit as “notification broken” stay correct.

Pit’s ruling, 2026-08-18: “Tell me about something when it’s wrong. If you need to record that something is working, write it in the background somewhere, but I don’t need the noise.” Pushover is the alert channel (ADR-0033) — it exists for a condition needing attention. info states by definition that nothing is wrong, so it has no business interrupting. The trigger was overnight noise: ~20 scheduled jobs run inside the 21:00–07:20 quiet window, and the patch coordinator’s cycle_start (04:00) and cycle_end-on-success (~04:30), plus the memory eviction sweep’s digests (03:20), were all landing as info pushes. Neither the Alertmanager quiet-hours route nor the Zabbix maintenance window can mute those, because a script calling pushover_notify.py directly bypasses both mute layers entirely.

This weakens no failure path. The affected call sites are written severity="info" if ok else "warning", so only the healthy branch is silenced.

⚠️ Severity is not a synonym for priority. Under the never-wake-me ceiling every tier maps to priority 0, which made the tiers look interchangeable — and one caller (/go’s deferral_failed) was tagged info for exactly that reason while reporting a genuine failure. Since info is now suppressed, mislabelling a fault as info silences it outright. Choose the tier by whether something is wrong, never by the paging behaviour you want.

Enforced at the chokepoint, not per caller — the same argument as the priority ceiling: a per-call-site sweep fixes today’s scripts and silently regresses the moment someone adds a new --severity info. Governed by ADR-0425.

The warning tier is recorded, not paged — six named alerts are the only exception

severity=warning does not reach the phone on either surface. In Alertmanager the severity: warning catch-all route resolves to the 'null' receiver; in Zabbix the Pushover user-media severity bitmask is 49 (Not classified, High, Disaster), which excludes both native Warning and native Average. This is ADR-0425’s rule applied one tier up, and it is governed by ADR-0465.

Pit’s ruling, 2026-08-22: “I’m getting way too many notifications right now and I’m fatigued, so I’m not looking at the alerts.” An alert nobody reads has already failed, so volume is a reliability problem, not a comfort one. Measured over the 30 days to that date, the warning class was ~58% of ~63 Pushover pushes a day — 80% of Zabbix’s 860 sends (Warning 330 + Average 357) and 57% of Alertmanager’s 1041. Note that Zabbix Average maps to canonical warning under the cross-surface mapping, so it is in scope by taxonomy rather than by exception; in practice it carried battery-replacement reminders, docker01 load-average and Dependency-Track vulnerability-count changes.

Six named alerts still page at warning tier — ContainerDown, ContainerHighMemory, ContainerOOMKilled, N8nWorkflowExecutionFailures, PaperlessConsumerErrors, DiskSpaceLow. Each has its own alertname route naming pushover-warning, and those routes are evaluated before the catch-all, so they are unaffected. They were ~42 notifications/30d combined. Adding to this list is a deliberate act: the test is that the alert names a specific fault rather than a symptom class, and that it is low-volume enough to stay readable.

This is a paging change, not a visibility change. Warning-tier events still fire, still appear in the Zabbix problem list and the Alertmanager UI, still count in Grafana, still open FreeScout incident records through the incident-webhook tap (which runs earlier in the route tree with continue: true and is untouched), and still surface in /health and the 07:20 morning report.

⚠️ Severity deflation is now the failure mode to watch. ADR-0078’s original hazard was inflation — labelling a degraded-but-serving condition critical. The mirror hazard now exists: labelling a genuine fault warning silences it until someone looks. Choose the tier by whether the condition degrades further while unattended, never by the paging behaviour you want.

The actionable / page-until-acted archetype pages emergency and repeats until acted

Most alerts describe a fault that auto-resolves once the underlying problem clears — they page once and recover on their own. A distinct class is the actionable / page-until-acted alert: the condition persists until a human acts and, while it persists, something is blocked. A pending ADO pipeline approval is the archetype — the deploy stays blocked until Arron clicks approve, and nothing clears it but him. This is the third case of the broadened critical definition above (“a blocking gate that only Arron can clear”) and is governed by ADR-0263. Two rules make it correct:

• Repeat-mechanism rule — one repeat mechanism, never two. “Keep nagging until acted” can come from Pushover’s native emergency retry (priority 2 → one push that self-repeats every retry seconds until acknowledged or expire) or a Zabbix escalation (an action that re-notifies every N minutes — many separate pushes). Pick one. For the ADO approval it is the Pushover emergency retry (via the Disaster→priority-2 Balanced mapping), so the Zabbix action stays one-shot (single operation, no escalation step). Stacking both double-pages the same event.
• Recovery-symmetry rule — sustained fire, instant recovery. A page-until-acted trigger should fire on a sustained condition (a grace window absorbs the routine quick-action case — the ADO approval uses min(…,15m)>0, so an approve-within-minutes never pages) but recover the instant the human acts. Achieve the fast reset with an explicit recovery expression (Zabbix recovery_mode=1 + recovery_expression last(…)=0, ~1–2m), not by waiting for the sustained fire-window to age out (which would leave a “resolved” page lingering 15m after the click). Asymmetric windows — slow to fire, fast to clear — are the correct shape here.

Emergency persistence differs by source surface — a documented, deliberate divergence

“Emergency” does not mean the same duration of nagging on both surfaces, and that is intentional — but it must be stated, not accidentally drifted (this is the F2 reconciliation):

Emergency source	Pushover retry / expire	Re-page after expire
Zabbix Disaster (media type 89, Balanced profile)	retry=60s, expire=1200s (20 min)	No — the action is one-shot; the problem stays visible in the Zabbix problem list.
Prometheus critical (Alertmanager pushover-critical)	Pushover/Alertmanager emergency default (retry≈60s, expire=3600s/1h)	Yes — Alertmanager repeat_interval: 4h re-delivers a fresh emergency every 4h until the alert resolves.

The Zabbix profile suits host/agent Disasters where a 20-minute insistent burst plus the persistent problem row is enough; the Prometheus profile suits service outages that must keep paging until fixed. If a future decision wants a single uniform emergency-persistence profile across both surfaces, change it here first (the source of truth) and reconcile the media-type + Alertmanager values to match — do not let them drift silently.

severity label values other than the three tiers are forbidden

The Prometheus severity label MUST be exactly one of info, warning, critical. Any other value — high, error, major, minor, page, a domain word like downloads, anything — is forbidden. The reason is not stylistic: Alertmanager routes on the exact string, and an unrecognised value silently falls through to the default route (see the routing contract below), which is precisely how a real incident happened.

The high incident. A Prometheus rule (PitMemoryIndexEmpty) was authored with severity: high — an off-taxonomy value. Alertmanager matched severity="warning" and severity="critical" explicitly but had no matcher for high, so the alert fell through to the default route, which pointed at the pushover-critical (emergency) receiver. A low-importance “the index is empty” signal therefore paged as a repeating, DND-overriding emergency. The fix was a one-word change — high → critical (shipped to homelab.rules.yml on docker-stacks main) — but the lesson is structural: an off-taxonomy severity does not error, it mis-routes silently to the worst-case channel. This is why the taxonomy is closed and why routing must be explicit per tier rather than relying on fallthrough.

The cross-surface severity mapping reconciles all three vocabularies

Three surfaces emit alerts with three different severity vocabularies. They all reduce to the canonical tier, which in turn fixes the Pushover priority. Author to the canonical tier; this table is the contract for how each surface maps onto it.

Source surface	Native severity	→ Canonical tier	→ Pushover delivery
Prometheus / Loki (severity label)	info	info	not delivered — recorded (ADR-0425)
	warning	warning	not delivered — recorded, bar six named alerts
	critical	critical	delivered, priority 0
Zabbix (6-tier native, “Balanced” profile)	Not classified	info	delivered, priority 0 — see note
	Information	info	not delivered — excluded by the media mask
	Warning	warning	not delivered — excluded by the media mask
	Average	warning	not delivered — excluded by the media mask
	High	critical	delivered, priority 0
	Disaster	critical	delivered, priority 0

The column states delivery, not loudness: under ADR-0326’s ceiling every tier that is delivered at all is priority 0, so priority no longer distinguishes the tiers and the useful question is whether a tier reaches the phone. Zabbix gates that per user media with a 6-bit mask (currently 49), not per media type — see ADR-0465. Not classified is deliberately left delivered despite mapping to info: it is where a trigger lands when nobody set a priority, it sent 0 problems in 30 days, and leaving it on means a mis-configured trigger announces itself rather than failing silent.

For Prometheus the mapping is the identity — the severity label is the canonical tier, which is why off-taxonomy values are forbidden. Zabbix keeps its richer native 6-tier scale at the trigger (it is useful operator granularity) but collapses onto the three canonical tiers through the “Balanced” profile asserted by ansible/scripts/zabbix_pushover_media_standard.py. Note one deliberate divergence the Balanced profile encodes and that this table simplifies for the canonical mapping: in live Zabbix, High pages at Pushover priority 1, not 2 — because Zabbix action #10 already re-notifies High every 30 min, so emergency would double-page; only Disaster reaches emergency. See zabbix-pushover-media for the exact per-severity priority_* parameters.

Alertmanager must have an explicit route matcher per severity tier

Every canonical tier — info, warning, critical — MUST have its own explicit route matcher in alertmanager.yml.j2. An alert’s tier is never assigned by default-route fallthrough. The default route is a backstop for genuinely unmatched alerts (a misconfigured rule), not the mechanism by which a tier reaches its channel.

This is a direct consequence of how Alertmanager actually matches (see alertmanager-routing): routing is first-match-wins, and a value with no matching route falls through to the default route’s receiver. The high incident was exactly this — no matcher for the value, fallthrough to the emergency receiver. The defence is that the default route must resolve to a safe receiver and must never be relied on to carry a real tier.

A tier that must not page still needs its explicit route — pointed at the 'null' receiver, never deleted. Deleting it is not equivalent: an alert matching no child route is at the mercy of whether some earlier continue: true route happened to match, and if none did it falls through to the root receiver, which is the emergency channel. 'null' states the intent and cannot drift.

Per alertmanager-routing, remember that child routes do not inherit group_by, repeat_interval, or mute_time_intervals — only receiver. Each per-tier route declares its own grouping/repeat/mute explicitly, and the quiet-hours mute must be listed on every leaf route.

Quiet-hours (rest protection) is a distinct mute class from planned maintenance

pitlab runs exactly one standing blanket mute: sleep quiet-hours, 21:00–07:20 Australia/Melbourne, every night, declared on both Alertmanager (the quiet-hours mute_time_interval, on every leaf route) and Zabbix (maintenance id=1). It exists so routine non-critical noise does not cost sleep, and it subsumes the overnight maintenance churn (backups, patching, reboot coordinator, Sunday UniFi auto-upgrade) that used to justify a separate window. It is governed by ADR-0264, whose 2026-08-21 amendment moved the end from 07:00 to 07:20 so the merged morning report (Epic #2604) lands before the overnight burst discharges. Keep two mute classes conceptually separate — conflating them mis-sizes the rest window and hides drift:

• Quiet-hours (rest protection) — standing, blanket, 7-nights, un-scoped, never removed. A timing policy about when Arron sleeps, not about any particular work. Its risk is being stuck ON during the day (every page swallowed) — guarded by the stuck-mute guard: a static CI gate proving both surfaces declare the identical canonical span, and a ~13:00 canary proving the live mute is inactive at midday.
• Planned-maintenance mute — scoped to the specific host(s) and the specific work window, set before intentional disruption and removed after (the mute-first discipline; e.g. the fleet patch coordinator’s per-host AM silence + Zabbix maintenance). Its risk is being left ON after the work — guarded by an explicit duration + removal step. These are unchanged by quiet-hours and remain the mechanism for intentional disruption.

Three implementation rules for the quiet-hours window:

• Five artifacts, one span, DST-aware — they move together or a gate reds. Alertmanager (quiet-hours time_interval) and Zabbix (maintenance id=1 maintenance_period) must declare the identical 21:00–07:20 window; the hard assert in configure_zabbix_server.yml, CANON_SPAN_MIN in the static gate docker-stacks/scripts/quiet_hours_guard.py, and CANON_SPAN_MIN in the runtime canary ansible/scripts/quiet_hours_canary.py each pin it independently. The canary is the one that gets forgotten — it carries its own copy of the span and pages on drift, so moving the other four alone fires a false stuck-mute page every day at 13:00. DST is automatic — the wall-clock 21:00/07:20 is bound to Melbourne local time (AM location:, Zabbix server-local), so it follows AEST/AEDT with no code change. A git-time drift between the surfaces is a CI-gate failure, not a silent split-brain.
• The midnight wrap needs two sub-ranges. Alertmanager cannot express a times entry where start_time > end_time, so 21:00→07:20 is split into 21:00–24:00 + 00:00–07:20 OR’d under one location.
• Nothing breaks through — quiet-hours is now absolute. This reverses the earlier carve-out, under which a priority-2 emergency overrode Do-Not-Disturb. Operator ruling, 2026-08-05: “never wake me up — nothing is that serious in pitlab”. No notification in this estate may exceed Pushover priority 0; see never-wake-me for the enforcement and the accepted risk. (ADR-0264 is amended by ADR-0326.)
• A muted alert is DELAYED, not dropped — unless it self-clears. mute_time_intervals suppress notification during the window and deliver afterwards if the alert is still firing at the lift. So a persisting alert costs latency only (the 2026-08-05 patch-cycle failure fired 04:22 and opened its incident at 07:00). An alert that fires and resolves inside the window is lost from every muted route entirely, because nothing remains firing at the lift to deliver. Know which shape a rule has before relying on the mute being harmless — it is also the test that decides whether a rule needs incident_always.

No notification may exceed Pushover priority 0 — never wake the operator

Operator ruling, 2026-08-05: “never wake me up — nothing is that serious in pitlab”. No notification originating anywhere in this estate may be sent at Pushover priority 1 or 2. Priority 1 bypasses the user’s Do-Not-Disturb; priority 2 additionally re-alerts until acknowledged. Both are forbidden. 0 still delivers immediately and still sounds during waking hours — it is simply held silently overnight — and -1 remains available for genuinely low-value chatter.

The justification is a property of the estate, not a preference about noise: the systems that can fail overnight all fail SAFE by design. The patch coordinator halts the rollout and rolls the host back; backups and restore drills abort and leave the prior artifact intact; deploy gates refuse to promote. None of them degrade further while unattended, so there is no 3am action that cannot wait for 07:00. Paging into sleep bought nothing and cost rest.

Three independent paths reach Pushover, and each is pinned separately — a fix to one does not cover the others:

Path	Enforcement
Scripts via pushover_notify.py	SEVERITY_PRIORITY is {info:0, warning:0, critical:0} and a MAX_PRIORITY = 0 clamp rejects any explicit --priority 1/2 at the chokepoint
Zabbix media type	every tier in zabbix_pushover_media_standard.py set to "0", including priority_disaster
Alertmanager receivers	priority: "0" set explicitly on every pushover_configs block

The Alertmanager case is the subtle one: an absent priority field is not harmless, because Alertmanager’s Pushover default is 2 (Emergency). pushover-critical had no priority set, so the one receiver most likely to fire at 3am was the loudest thing in the estate purely by omission. Never rely on an absent field being quiet.

The clamp lives in the helper rather than being swept across the 21-plus call sites deliberately: a per-caller sweep fixes today’s scripts and silently regresses the moment someone adds --priority 1. Capping at the chokepoint makes a future violation a no-op instead of a 3am page — the declared-control-needs-a-gate discipline applied to notification policy.

Accepted risk, stated plainly: a genuine emergency born at 02:00 — data loss actively in progress — will not wake anyone and is seen at 07:20 (07:00 until the ADR-0264 amendment of 2026-08-21 moved the window end). That trade was made knowingly on the reasoning above. Governed by ADR-0326, which amends ADR-0264.

Terminal job-failure alerts keep their incident record inside quiet-hours

The lab’s scheduled jobs run almost exclusively inside 21:00–07:20 by design, and the self-clearing ones (error-burst and stuck-worker rules) are exactly the shape the bullet above says the mute drops. A render that errors at 02:00 and recovers by 03:00 left no page, no incident, and nothing to find in the morning.

Two conditions decide it, and both must hold (Epic #2604 amendment to ADR-0264, which widened coverage from 8 to 22 of 139 rules when the window grew to 07:20):

1. the rule’s firing means a scheduled or periodic job run terminally failed — errored out, produced nothing, did not run, or is stuck and will not complete — as distinct from continuous-service liveness, saturation, capacity, posture, drift, SLA-ageing or device availability; and
2. the condition can self-clear inside the window — an error-rate window rolling off, a stuck worker draining, or the next run of a sub-window-cadence job succeeding.

Condition 2 is what keeps this narrow. A rule still firing when the mute lifts is delivered and loses nothing, so a multi-day staleness alert (ClaudePodStale, VectormapStale, TopchartsListsStale) is deliberately excluded — labelling it would only duplicate a record that already arrives. Likewise FreeScoutHealFailed, whose own description says the tier “will not self-recover”.

A rule meeting both conditions MUST carry the label:

labels:
  incident_always: "true"

That label routes it to the incident webhook through a dedicated route placed before the muted tap, carrying no mute_time_intervals and continue: true — so the FreeScout record is always captured while every sibling route, including the Pushover routes, evaluates unchanged. Paging is deliberately NOT exempted: recording an incident is silent and wakes nobody, so it does not need the mute that paging does. The muted tap carries the negative matcher incident_always != "true" to keep the two mutually exclusive; without it both match outside quiet-hours and double-deliver to the same webhook.

Two authoring consequences: the exemption is opt-in, so a new job-failure rule gets no overnight record unless labelled; and exp_labels in a promtool unit test is an exact match, so labelling a rule that has a test requires updating that test in the same commit. Governed by ADR-0318.

Required Prometheus alert rule shape — name, labels, annotations

Every Prometheus/Loki alert rule MUST conform to this shape. Author to it; reviewers bounce rules that miss any element.

• Alert name — PascalCase, subject-naming, stable across runs (it is the correlation key): ArrHealthIssue, PaperlessConsumerErrors, PitMemoryIndexEmpty. Not snake_case, not a sentence.
• labels.severity — exactly one of info | warning | critical (canonical taxonomy; off-taxonomy values forbidden).
• Identifying labels — at least one of host, service, instance, target so the alert names where it fired. These join the alert to its dashboard and probe_success{service,domain} target.
• annotations.summary — one-line human outcome.
• annotations.description — the detail, with the salient metric value templated in.
• annotations.runbook — what to do about it.
• annotations.dashboard — the Grafana live-view URL for triage.
• annotations.kb — the knowledge-base explainer link (what & why). Mandatory per alerting-notification-standard (ADR-0062/0064), pointing at the owner of what fired (service page for service-owned signals, operations/alerting for fleet-wide).

- alert: PitMemoryIndexEmpty
  expr: pit_memory_chunks_indexed == 0
  for: 5m
  labels:
    severity: critical          # canonical tier — NOT 'high'
    service: pit-memory
    host: docker01
  annotations:
    summary: "pit-memory index is empty — docs_search will return no results"
    description: "pit_memory_chunks_indexed=={{ $value }} on {{ $labels.host }} for 5m — docs_search returns nothing."
    runbook: "Re-run reindex_docs; check the indexer container logs in Loki."
    dashboard: "https://grafana.pitbun.com/d/pit-memory/pit-memory"
    kb: "https://docs.pitbun.com/services/observability/pit-memory/#index-empty"

Current gap — kb backfill. kb is mandated and is meant to be CI-enforced by validate_notification_catalog.py (the same blocking gate the alerting-notification-standard describes), but it was absent from many live alert rules at the time this standard was written. kb is required on every rule; the backfill of existing rules and extending CI validation to cover Prometheus/Loki rule annotations is follow-on work, not a relaxation. A rule without a resolving kb is non-conforming.

A rate()/increase()/delta() rule over a labelled counter requires primed children — see the Alertable Counter Instrumentation Standard

A conforming rule shape does not guarantee the rule can fire. When an alert uses rate(), increase(), or delta() over a labelled counter, the emitting service MUST prime every label child at 0 at startup — otherwise a first-ever event is born at 1 with no 0→1 delta and the rule is silently dead through the very failure it exists to catch (the claude-pod eight-day miss, ADR-0244). The full emitter-side rule — priming, the discrete-run last_run_status gauge, and the workload-shape distinction — is the Alertable Counter Instrumentation Standard (ADR-0245). Treat it as the instrumentation half of this rule-shape contract: author both together.

A low-value alert is tuned to its actionable case, not deleted

When an alert is noisy or low-value, the first move is to tune it — raise for:, tighten the threshold, add a user-impact condition, or adjust severity/routing (the three sections below are the mechanics) — never delete it as a shortcut to silence. Deleting a real signal to stop noise converts a false-positive problem into a false-negative one: the pages stop, but so does detection of the genuine case. The live value may still sit on a dashboard tile, but a tile is a pull — nobody is watching it at 3am — whereas an alert is a push. Deletion is reserved for a signal that is genuinely redundant (the same condition already pages from another rule) or meaningless (it measures nothing actionable), and that redundancy/meaninglessness must be stated, not assumed because the alert is inconvenient.

PlexTranscodeSaturated — deleted, then tuned instead (ADR-0095, 2026-07-02). The alert paged on self-clearing parallel-transcode bursts (min(plex_transcode_job_speed) < 1.2 for: 10m; observed FIRING→RESOLVED in 10m at the boundary, no operator action). The first reaction was to delete it — which would have removed the ability to catch a sustained over-commit that genuinely stalls downloads. It was restored and the for: raised 10m→30m instead: the burst clears well inside 30m, real saturation still pages. Signal kept, noise gone — the correct outcome of the “dampen with for:” section below, reached by not deleting first.

Size the metric lookback window to the condition’s real duration, not padded for safety

An increase() / rate() lookback window or a Zabbix nodata() window sets how long the alert lingers after the underlying event stops, because the windowed expression stays true until the event scrolls out of the window. A window padded “to be safe” therefore smears a single transient into a multi-hour alert that re-pages on every repeat_interval. The rule: the lookback window approximates how long the condition genuinely persists, not a safety margin. For a counter that increments on a discrete failure (increase(errors_total[W]) > 0), W is the resolve-lag you will tolerate — minutes for a warning, not hours.

The [6h] smear (Epic 1369, 2026-06-30). PitMemoryReindexErrors used increase(pit_memory_reindex_errors_total[6h]) > 0. A single transient reindex error — a load-induced blip lasting seconds — kept the alert firing for the full 6 hours until the increment aged out of the window, re-paging across the afternoon for an event that was already over. Shortened to [15m]: the same blip now self-resolves ~15 minutes after the errors stop, while an active failure (errors still incrementing) keeps firing. IncidentZabbixPollerCycleErrors ([15m]→[10m]) and the docker01 agent.ping nodata(...,2m)→nodata(...,5m) were the same class of fix.

Dampen flapping with for:, not by widening the lookback window

The two knobs are not interchangeable. The lookback window controls post-event linger (above); for: controls how long a condition must hold before it pages, which is the correct tool for flap dampening. A check that flips true for 1–2 minutes then self-clears (a momentary probe failure, an agent missing one poll under load) should be ridden out with a longer for:, never by widening the lookback window — widening the window makes the linger worse, not the flapping better.

The critical that flickered (Epic 1369). CFTunnelProbeDown (probe_success == 0) paged critical on repeated 1–2 minute tunnel blips that self-cleared, because for: 2m barely outlasted them. Raised to for: 5m: transient edge reconnects are absorbed, a genuine outage (which persists well past 5m) still pages, at the cost of ~3 minutes slower notification — an accepted trade to kill false criticals. The mirror anti-pattern is alerting on a self-heal succeeding: FreeScoutCacheOwnershipDrift paged every time the 1-minute watcher corrected cache ownership before an outage — i.e. on the safety net working. Re-pointed from increase(...[30m]) > 0 to increase(...[1h]) >= 3 so a lone benign correction passes silently and only a looping root-writer (the actionable condition) pages.

A rate window has a FLOOR as well as a ceiling — it must span its source’s update cadence

The two clauses above both push a lookback window shorter, because both are about an alert that fires too much. There is a floor underneath them, and crossing it fails in the opposite direction — silently, permanently, and invisibly to every existing check.

The rule: a rate() / increase() / delta() window must be at least twice the interval at which its source series actually advances. Not the scrape interval, and not the collector’s poll period — the interval at which the underlying value changes. For a counter fed by a batch job, an event-driven exporter, or a textfile collector that only writes when there is something to write, those are three different numbers and only the last one matters. Measure it (changes(series[24h]), or the gap distribution from the source’s own store); never assume it.

Below that floor, rate() over the window returns no data at all — not zero. In an and chain, one no-data term takes the whole expression with it, so the alert cannot fire against any breach, however severe. promtool check rules passes, review passes, and a promql_expr_test unit test passes. A dead alert and a healthy service emit byte-identical evidence: silence.

The multiplier of two is not decoration. A window exactly equal to the cadence can land between two updates and see nothing; two intervals guarantees it contains one. Where the source is bursty rather than periodic, size against the measured p95 gap, not the mean.

A burn-rate alert declares the cadence it was sized against, in a source_cadence_seconds annotation, and counter_birth_guard.py fails the deploy if any window in the expression is under twice it. The declaration exists because no linter can measure a counter’s real update interval — the same limitation, and the same escape-hatch answer, as the counter-birth ack.

And it is proven RED against a breach series before it is trusted green — through alert_rule_test, never promql_expr_test. The distinction is load-bearing rather than stylistic: promql_expr_test evaluates an expression written inside the test file, so it tests a copy that can silently diverge from the shipped rule, and it has no notion of an alert being pending, so it cannot see for: at all. A rule whose expression is briefly true but can never hold for its for: duration is, to promql_expr_test, indistinguishable from a healthy one.

The SLO that could not fire (FreeScout incident #1758, ADR-0394). The CI queue-time SLO’s two burn alerts used Google’s canonical 1h/5m and 6h/30m window pairs — a table derived for a service taking thousands of requests a second. The SLI counted ~124 pipeline runs a day, from a counter that advanced roughly once an hour. Both alerts were dead from birth, and the SLO sat in undetected breach at 80% attainment against a 95% target for the entire life of the Epic that delivered it. Its unit tests passed throughout — they were promql_expr_test blocks holding hand-copied expressions, still asserting a clamp_min form hours after that form had been deleted from the rules.

Alert on a smoothed statistic of a bursty gauge, not its instantaneous value

When the underlying metric is a bursty instantaneous gauge — one that is legitimately at its “bad” value for much of the time even when the system is healthy — alerting on the raw value pages on normal operation. Reduce the per-subject series to the statistic that actually expresses the SLO over a window (max_over_time / avg_over_time / quantile_over_time) before comparing to the threshold, and pick the statistic by what “healthy” means: if healthy means “achieves rate X at least sometimes”, use max_over_time (the peak); if it means “sustains average X”, use avg_over_time. This is distinct from the two knobs above — the lookback window tolerates resolve-lag and for: dampens flapping, but neither fixes a statistic that is wrong at every instant; a gauge that reads “bad” 90 % of the time when healthy will defeat any for:.

PlexTranscodeSaturated — instantaneous min on a bursty gauge (ADR-0114, 2026-07-04). Plex reports transcode speed (×realtime) instantaneously: a healthy transcode encodes ahead in bursts (5–14×) then idles at 0× while the buffer drains, reading 0× for 55–90 % of scrapes, and a copy remux reads 0× forever. min(plex_transcode_job_speed) < 1.2 was therefore ~always true whenever streams ran and fired on ordinary busy evenings — the for: 30m from ADR-0095 could not save it because the healthy-0× states persist. Fixed by measuring each job’s peak over 15m — min(max_over_time(plex_transcode_job_speed[15m]) and on(key,context) (plex_transcode_job_throttled{video_decision="transcode"} == 0)) — so a job that never bursts above 1.2× for a sustained 30m (a real stall) pages, while healthy idle does not. Live proof: instantaneous min = 0× vs smoothed = 3.5×.

A saturation alert against a hard ceiling owes a trend companion — a level cannot distinguish a plateau from a climb

Where a resource has a hard ceiling and an alert warns on proximity to it — container memory against mem_limit, a filesystem against its size, a queue against its bound — that level alert MUST be paired with a second rule on the rate of approach. A level is a scalar with no time axis, so a subject settled at 87 % forever and a subject passing through 87 % on its way to exhaustion produce the identical firing. The level alert therefore delegates the entire question of is this bad to whoever reads it, and the reader answers it with whatever window is convenient — which is usually one sample. The trend companion is what puts the answer in the alert instead of in the responder’s judgement.

Shape the companion as a burn rate, not a slope: predict_linear(<metric>[<fit>], <horizon>) >= <ceiling>, with a utilisation floor so a small subject growing fast in percentage terms does not page while nowhere near its limit, and a deriv(...) > 0 guard so a noisy downward fit cannot cross. Set the horizon from how long remediation actually takes, not from the fit window — these leaks are slow, and a horizon equal to the fit window typically warns hours before exhaustion on a fault that took days to build. A warning that arrives too late to act on is not worth having. State the fit, the horizon, the floor and why each was chosen, and pin every guard clause with a mutation-tested unit case: a clause nobody has seen turn the suite red is not known to work.

pit-memory — 87 % read as a plateau, three days into a climb (ADR-0404, 2026-08-15). ContainerHighMemory fired correctly at 85 % of a 512 MiB cap. The container was sampled three times across one minute — 446.5 / 446.5 / 446.5 MiB — recorded as “FLAT, a plateau, not a leak”, and filed as ambient noise. Over the preceding 60 hours the same series ran 118 → 188 → 282 → 384 → 423 → 464 MiB, monotonic, with the cgroup already having hit its ceiling 20 times. Every number in the assessment was right; the conclusion was wrong, because a plateau and a climb differ only in a dimension one minute does not contain. The remedy was not a tighter threshold — that fires earlier on both shapes — but ContainerMemoryExhaustionPredicted, a 6h fit projected 24h ahead, which selects exactly one container out of 83 and names the right one.

Size the threshold against the statistic the expression computes, not the raw values you observed

The three sections above pick the statistic, the lookback window and the for:. None of them picks the number, and a threshold chosen from the wrong distribution defeats all three. The rule: before committing a threshold, measure the aggregate the expression actually evaluates — over the poll interval the item actually uses — on both the unhealthy subject and the healthy ones, and place the threshold in the gap between them. A value read off raw samples during an investigation is a different distribution from avg_over_time(...[15m]) of those samples at the collector’s real cadence: smoothing pulls the aggregate toward the mean, so a threshold set from raw peaks lands above the smoothed signal’s own range and clips its top edge — producing a rule that flaps on the very condition it was written for, which reads as a noisy alert rather than a mis-sized one.

Sizing is a two-sided constraint, and a threshold satisfying only one side is not sized: it must sit below the unhealthy subject’s observed floor (so the alert holds) and above the healthiest comparable subject’s ceiling (so it stays quiet). State both margins when adding the rule. Where the metric exists on sibling subjects — per-AP, per-host, per-container — those siblings are the control group and make the gap measurable rather than guessed; where there is no sibling, the subject’s own healthy baseline over a representative window serves the same purpose.

The external-airtime trigger that clipped its own signal (ADR-0296, 2026-08-03). WiFi 2.4GHz congestion is externally driven was set at avg(15m) > 25 from ad-hoc 30-second sampling whose median was 32.5 % — a defensible-looking number from real data. Live, it flapped three times in fifteen minutes. The statistic the trigger evaluates behaves differently from the samples it was sized on: measured over 20 consecutive 60-second polls, the rolling avg(15m) ran 20.9–24.2 % on the congested AP against 4.5–5.2 % and 1.9–2.1 % on its two sibling APs. > 25 therefore sat above the real signal’s entire range. Retuned to > 15 — six points below the congested AP’s floor, three times above the loudest healthy sibling — it produced one state change in the following twenty minutes. Note the tell: the flapping looked exactly like a for:/window problem, and widening either would have masked a threshold that was simply on the wrong side of the data.

An alert must be able to reach GREEN — name the clearing condition and confirm the estate can reach it

The three sections above size the statistic, the window and the number so the alert fires correctly. None of them asks the opposite question, and it is the one that kills alerts in practice: under what reachable condition does this rule resolve? State it explicitly when adding the rule, and confirm it is achievable at the estate’s actual rate of change — not merely arithmetically possible. A rule whose clearing condition requires work the estate cannot sustain is red from birth, and a permanently-red alert is not a control: it is trained-in noise that also blinds the surface it sits on, because “still red” and “newly red” look identical.

Three failure shapes recur, and all three pass every other clause in this standard:

• The clearing condition is unreachable. The alert is correct, fires on a true condition, and nothing anyone will realistically do makes it stop.
• The metric moves on its own. The rule watches a slope, but the underlying gauge drifts upward (or downward) as a mechanical consequence of accrual, ageing or growth — so the derivative is dominated by drift rather than by conduct, and the condition is satisfied forever with nothing wrong.
• The cleared state needs more of a fixed resource than exists. Not unwilling work and not drift — arithmetic. The rule demands a distribution, separation or spare capacity the estate cannot supply because the underlying pool is smaller than the demand, and no conduct changes that.

So: before adding a rule, COUNT what the cleared state requires against what exists. Radios against non-overlapping channels, services against ports, replicas against nodes, jobs against maintenance windows, IPs against a subnet. This is a one-line check and it is the only one that catches the third shape — expression review cannot, because the expression is correct. Where the count comes up short, the alertable condition is not “the resource is contended” (always true) but “the fleet is using fewer of the available slots than it has”, which is both fault-bearing and reachable. State the count in the rule’s description so the next reviewer does not re-derive it.

WiFi 5GHz co-channel interference — an alert on a condition physics forbade clearing (ADR-0411, 2026-08-16). ADR-0407 correctly replaced a channel-equality compare — which could not fail regardless of the fault — with true spectral-overlap detection, then alerted on overlap > 0. Australia offers exactly two non-DFS 80 MHz blocks (36–48, 149–161); the estate has three APs on channel=auto with the controller’s auto-RF planner avoiding DFS. Three into two forces an overlap, so the trigger fired at creation and logged no resolve event in its first 24 hours because none was possible — and the ADR’s own stated remedy (“move Lounge to 149–161”) merely relocated the overlap onto Bedroom, which already occupied that block. Nobody caught either, because nobody counted blocks against radios. Replaced by distinct blocks in use < 2: quiet on the unavoidable overlap, red when the planner collapses radios into fewer blocks than were available. Note the inversion tax — a < n predicate fires on a low reading, so absent data reporting 0 would fire, and the item must throw to UNKNOWN rather than report zero (ADR-0305 applied to a polarity it did not contemplate: absence is safe under > 0 by luck and unsafe under < n by construction).

So: before alerting on a metric’s rate of change, MEASURE its baseline drift — never assume it is flat. A standing backlog, an ageing queue, an accruing counter-like gauge and a growing corpus all move without anyone failing at anything. Where drift exists, the threshold must sit clear of it (state the measured drift and the multiple), and the rule’s unit-test baseline series must BE that measured drift, not a flat line. A flat fixture under a drifting metric is the specific way this defect survives review: the test agrees with the rule about something they are both wrong about.

Where a genuine standing breach cannot be cleared this week, the answer is not a slope rule and not a silence — it is this estate’s own remedy for an un-remediated finding: an owner-accepted risk with an expiry, carried in the rule itself as a date on which it arms (see the vulnerability register pattern, ADR-0302). That keeps the debt visible and dated instead of either paging daily or being quietly muted.

DTRemediationBacklogGrowing — a slope rule over a gauge that ages upward (ADR-0361, 2026-08-08). The rule deliberately avoided an absolute threshold on the Dependency-Track remediation backlog, because 3,800+ findings were already past SLA and > 0 would have been red from birth — correct reasoning. It chose sum(delta(dt_sla_remediation_overdue_count[14d])) > 0 instead, on the assumption that a standing backlog is flat and only moves when the estate fails to convert. It is not: a finding joins that gauge purely by ageing past its SLA, measured at ~110/day, so holding the delta at zero meant converting 851+ findings per fortnight indefinitely. It fired within 48 hours of the metric first existing and could never have cleared — the same red-from-birth failure it was written to avoid, one derivative up. Its unit test passed throughout, because case 1 asserted a flat backlog stays silent and called that fixture “today’s live state”; the real series never was flat. Replaced by a dated ceiling (armed 2026-12-01, clears under 250) plus a step-change guard sized at ~2.6× the measured ageing rate.

A runbook for an alert that can be READ after it clears must give a RETROSPECTIVE diagnostic path

Every clause above governs whether the rule fires correctly. This one governs whether the runbook can be followed — and for a whole class of alerts it cannot, because the runbook’s steps sample state that no longer exists by the time a human reads them. ssh <host> 'uptime; cat /proc/pressure/io; ps -eo state,comm', docker stats, ss -tnp and every other instantaneous read answers “is the host unhealthy right now?”, which is a different question from “was the host unhealthy when the alert fired?” — and on a transient the answer is reliably “no”. That reading is not a null result: it is indistinguishable from the condition never having existed, so it actively pushes the operator toward the wrong conclusion.

Three alert shapes routinely put minutes-to-hours between the event and the reading, and any one of them triggers this clause:

• Self-recovering / transient — the condition clears on its own, often before the notification is even opened. A bounded linger (max_over_time(...[10m])) buys overlap for inhibition; it does not keep the underlying state alive for an operator.
• Quiet by design — severity: info routed to null, visible only in Karma or the incident feed, so it is read whenever someone happens to look.
• Fireable inside quiet hours — the 21:00–07:20 blanket mute (ADR-0264) means an overnight alert is first read at 07:20, up to ten hours after the event. This applies to nearly every rule in the estate, which is what makes the clause general rather than niche.

So: where an alert can be read after its condition has cleared, annotations.runbook MUST lead with a time-anchored query against retained data — a PromQL offset / subquery, a LogQL range over the incident window, a Zabbix history read. A live command may follow as a supplement for the case where the operator catches it firing, but it may not be the only path. The test is blunt: if I read this alert an hour late, can I still tell what happened? If the answer is no, the runbook is non-conforming for that alert.

The retrospective form is usually already available and simply unreferenced — the series are scraped and retained whether or not anyone points at them. Prefer naming the exact query over describing it, so the operator pastes rather than derives it under pressure:

# was it IO, or was it run-queue? — anchored on a spike that fired ~12 minutes ago
max_over_time(rate(node_pressure_io_stalled_seconds_total{instance="docker01"}[2m])[10m:30s] offset 12m)
max_over_time(node_procs_running{instance="docker01"}[10m] offset 12m)
max_over_time(node_procs_blocked{instance="docker01"}[10m] offset 12m)

Docker01TransientStall — a live-only IO fallback on an alert engineered to be transient (2026-08-11). The rule is severity: info, routed to null, and lingers only ~10m past the spike; its IO fallback was ssh docker01 'uptime; cat /proc/pressure/io; ps -eo state,comm | awk "$1~/D/"'. Read nine minutes after a real breach the host was clean on every one of those — but the spike itself had node_load1 at 12.17, 19 runnable processes, 0 blocked, CPU 23% busy and PSI io-stalled at 0.004 s/s. The instantaneous reading could not have distinguished “IO was never the cause” (true, and now evidenced) from “the alert is spurious” (false). The retrospective PromQL above answered it in one query against data that had been sitting in Prometheus the whole time. Note the shape of the original defect this repeats: the alert’s first diagnosis survived eight months precisely because it was quiet enough that nobody checked (ADR-0358) — a runbook that cannot be executed late is the same blind spot one layer up.

A health check asserts the SHAPE of a response, never an exact mutable value

A check that compares a probe’s output to a literal value that legitimately changes — a version string, a build hash, a release date, a config revision — couples the alert to the release cadence rather than to health. It fails twice over: the next routine upgrade puts it in permanent PROBLEM that cannot self-clear, and, far worse, the check silently becomes a no-op — once the comparison can never again be satisfied, a genuinely broken service produces exactly the same alert state as a healthy one, so the check is blind while still looking green-adjacent on the board. Assert instead what a healthy response structurally looks like: non-empty, parseable, matching a shape regex (^[0-9]+[.][0-9]+[.][0-9]+$), within a range. If the value itself genuinely matters, the source of truth for “what version should be running” is the pinned tag in Git, not a monitoring threshold.

The qdrant version pin (incident #1449, 2026-08-03). Qdrant (docker01): API returned unexpected response asserted last(/docker01/qdrant.api.ping)<>"1.18.2". Renovate’s routine 1.18.x bump deployed v1.18.3 at 20:53 AEST; the trigger went to High PROBLEM and stayed there, unclearable, for ~10h — during which the check could no longer distinguish a broken Qdrant from a healthy one. (Availability coverage held throughout via the nodata() sibling trigger and probe_success{service="qdrant"}, which is the only reason this was a blind spot rather than an outage.) Fixed to find(/docker01/qdrant.api.ping,,"regexp","^[0-9]+[.][0-9]+[.][0-9]+$")=0, which passes for any real semver and treats an empty/HTML/null payload as the fault. Note [.] not \.: Zabbix accepts only \" and \\ as escapes inside a quoted string constant. The class is now enforced in code — tasks/zabbix_ensure_objects.yml fails closed at reconcile time on any trigger expression comparing an item to a version literal, so it cannot be reintroduced from any consuming playbook.

A condition inferred from a FAILED LOOKUP must distinguish lookup failure from the absence it infers

Where an alert’s condition is reached by looking something up and not finding it — joining an id back to its owner, resolving a name in a registry, matching a record against an index, correlating an effect to its cause — the not-found result is evidence about the lookup, not about the world, until the lookup is shown to have been capable of succeeding. A detector that treats the two as the same thing reports a fault whenever its own join breaks, and does so in the most convincing possible form: a specific, well-formed finding naming a real entity.

This is distinct from ADR-0305 and the < n polarity clause above, which govern a missing metric sample and are satisfied by making the source throw to UNKNOWN. Here the sample is present and correct — a change genuinely happened — and it is the inference drawn from an empty join that is unsound. No sentinel value fixes it, because nothing is missing at the metric layer.

So a rule of this shape MUST:

• Name the precondition under which the lookup could have succeeded, and evaluate it. Typical preconditions: the index was populated for the window being queried, the correlating identifier had not been reissued, no observation gap spans the diff.
• Emit a distinct, non-alerting outcome when the precondition fails — “could not determine” rather than the fault verdict. Silence is not sufficient: an unresolvable case suppressed without a name is indistinguishable from a clean one.
• Count that outcome as its own signal, so a detector spending its cycles unable to judge is visible rather than quiet. This is the alert-on-finding dead-man argument one level in: the dead-man proves the detector ran, this proves it could actually decide.
• Derive any precondition threshold from a measurement of the system it governs, never a borrowed constant.

The HA ownership detector — three false-positive classes, one shape (ADR-0410, 2026-08-16). The detector recovers what caused a light to change by joining the entity’s context.id back to the automation whose run carries it, and every cause is None verdict reads that failed join as the positive claim nothing in Home Assistant claims this change. ADR-0405 closed two classes and cut it from ~20 pages/day to one page in the next 16 hours — and that page was also false. automation.patti_s_lamp_2_daytime ran legitimately on motion at 07:31:08 and took a child’s lamp to 255; a deliberate homeassistant.restart at 07:31:51 reissued every automation context id; the 07:32 poll died on Connection refused; by 07:33:01 the causing run existed nowhere, and the detector paged unexplained-change on its highest-value branch — the one written for the 2026-08-09 incident — against an automation doing exactly its job. The fix is this clause: preconditions (a poll gap >90s against a 60s timer, or ≥90% of the automation context table reissued in one poll — measured at 1–5 of 47 normally versus 47 of 47 across the restart), a non-alerting unattributable verdict carrying which precondition failed, and an unattributable_last_poll counter collected in Zabbix with no trigger until a baseline exists to justify one.

One root event should page once — correlated alerts need inhibition, not independent tuning

When several alerts fire from a single underlying event, tuning each rule in isolation does not fix the fatigue — the event still produces a burst. The structural fix is Alertmanager inhibition (or grouping) so one root cause yields one notification, not a fan-out. Recognise the pattern by timestamp correlation: distinct alerts whose firing times cluster on the same moments share a root cause.

The docker01 load-blip fan-out (Epic 1369). Over one 4-hour window, agent.ping nodata flaps (09:01/09:11/10:21/12:28), pit_memory_reindex_errors (08:35/12:30), and zabbix_incident_poller_cycle_errors (08:35/09:00/10:20/12:30) all clustered on the same instants — three to four alerts per event. No kernel soft-lockup occurred; docker01 simply had brief load spikes under which the agent missed a poll and the two services’ API calls timed out. Window/for: tuning shortened each alert, but the fan-out itself (one blip → many pages) is resolved by an Alertmanager inhibit rule, not per-rule tuning: a quiet Docker01TransientStall alert (high load + idle CPU, routed to null) is the inhibit source for the two collateral symptoms, suppressing them only while a stall is concurrently firing.

Memory-headroom alerts use a three-tier USE-method ladder with macro-driven per-host thresholds

Every monitored Linux host carries a memory-headroom early-warning ladder: three rungs built on the USE method (utilization + saturation) that give lead time before the OOM killer fires, rather than only notifying after a kill. This clause generalizes the control01 OOM-hardening prototype (ADR-0231) into a fleet standard (ADR-0232); the live per-host thresholds and rationale are the Fleet memory-headroom baselines table.

The three rungs, each a Zabbix trigger on the stock Linux by Zabbix agent memory items plus a PSI UserParameter:

Rung (USE leg)	Expression shape	Canonical tier	Fires when
Headroom — utilization	min(/host/vm.memory.size[pavailable],10m) < {$MEM.PAVAIL.WARN}	warning (Zabbix P3) — advisory	Free-memory % held below the host’s floor for 10m — the “keep N% free” buffer is gone.
Imminent — absolute	max(/host/vm.memory.size[available],5m) < {$MEM.AVAIL.MIN.MB}·MB	critical-adjacent (Zabbix P4 High, pages)	Absolute available held below the host’s MB floor for 5m — OOM is close, intervene now.
Saturation — PSI	min(/host/mem.psi.some.avg60,10m) > {$MEM.PSI.SOME.MAX}	warning (Zabbix P3) — advisory	Processes stalled on memory reclaim ≥ the % floor for 10m — the earliest honest pre-OOM signal.

Five rules make the ladder correct and fleet-portable:

• ”% floor OR absolute-MB floor, whichever breaches first.” The two utilization/imminent rungs are complementary, not redundant: on a large host the % rung leads (400 MB free is imminent but a tiny %), on a small host the absolute rung leads (45% free is fine but 150 MB is not). Both are always present; whichever condition a given host hits first is the one that fires.
• The imminent floor is absolute, not a percentage. OOM is driven by absolute free memory, and a fixed MB floor stays valid if the guest is ever resized — a percentage floor silently moves when RAM changes.
• Sustained windows only — a single sample must never fire. Every rung wraps its item in min()/max() over a window (10m advisory, 5m page) so a transient spike is ridden out, exactly the “dampen with the window/for:, not the instantaneous value” discipline the sections above require. An un-windowed memory trigger is non-conforming.
• Thresholds are Zabbix user macros, set per-host and data-driven. {$MEM.PAVAIL.WARN}, {$MEM.AVAIL.MIN.MB}, {$MEM.PSI.SOME.MAX} carry sane defaults (20 / 400 / 10) on the house template and are overridden per host from a script that reads ~14d of trend history and sets each floor a margin below that host’s normal operating floor — so the rung fires on abnormal headroom loss, not on the host’s normal steady state (a ZFS-ARC host like pve01 legitimately reads low % free, so its floor is ARC-adjusted downward). Role-aware baselines are the whole point: a fixed fleet threshold either page-storms the busy hosts or never fires on the idle ones.
• The PSI rung is deployed only where the kernel exposes it. /proc/pressure/memory requires CONFIG_PSI (and, in an LXC, host-kernel support); a host without it gets the utilization + imminent rungs and the PSI rung is skipped and logged — never silently dropped.

Deployment is config-as-code: a house Zabbix template owned in-repo carries the macro defaults and the PSI item/trigger (linked to PSI-capable hosts); the utilization and imminent triggers are host-instantiated referencing the stock vm.memory.size[*] items with the per-host macros. The stock Linux by Zabbix agent template is never edited (updates would clobber it) — the ladder is additive and macro-overridable, exactly as the as-code discipline requires. A per-service Grafana dashboard for the fleet ladder is a deliberate, ADR-recorded waiver (Pillar 2), not an omission — see ADR-0232.

Work dispatched to an asynchronous executor must be asserted to reach a terminal state, on a schedule

Every clause above alerts on a thing — a process, an endpoint, a certificate, a queue depth, a scheduled job’s cadence. None of them alerts on a work item. That is the blind spot this clause closes, and it is governed by ADR-0307.

The rule. Any system that hands a unit of work to an asynchronous executor — something that takes the item now and reports the outcome later, out of band — MUST assert on a schedule that every dispatched item reached a terminal state, and MUST treat the absence of a terminal state past a deadline as a failure, not as still-pending.

The absence is the whole point. “No outcome yet” and “the outcome will never come” are the same observation; only a deadline separates them. A system that waits for a failure report cannot detect an executor that dropped the item silently, because a dropped item reports nothing — which is indistinguishable from health on every surface that watches liveness. This is Prime Directive 12’s absence of errors is not absence of activity applied to work rather than to processes: up=1, probe_success=1 and an empty error log are all fully consistent with the work having vanished.

The 51-day silence. A Radarr grab of Romeo + Juliet (1996) reached no terminal history row — no import, no failure, no ignore. The film sat monitored-and-missing for 51 days, surfacing only because Arron noticed the title reappear in Plex. Nothing paged, because nothing emitted: the queue no longer held it, Radarr’s Health page has no such check, and the blackbox probe was green throughout. A 60-day scan then found 66 further silent stalls against ~1477 grabs — 4–5 %, none previously observed (ADR-0298).

A conforming assertion has four parts — vocabulary, deadline, in-flight exclusion, executor scope

Naming the rule is not enough; each part is a way the assertion goes wrong in practice.

Part	Obligation	The failure it prevents
Terminal vocabulary	Enumerate the executor’s terminal outcomes read off the live API, and exclude events that describe a side effect rather than the item’s fate.	Counting a side-effect event as terminal masks the stall. Radarr writes movieFileDeleted reason=Upgrade at import time; treating it as terminal passes a stalled upgrade grab as resolved.
Deadline	A per-surface age past which absence is failure, sized to the executor’s genuine working time.	With no deadline the two indistinguishable cases stay indistinguishable — the defect itself.
In-flight exclusion	Exclude items the executor still legitimately holds (a live queue/run read), so “working” never reads as “lost”.	Every in-progress item pages.
Executor scope	Index terminal outcomes across the group of dispatchers that share the executor, not the dispatching instance alone.	A sibling that consumes the same executor can terminalise an item the other dispatched. Scoping per-instance reports it stalled — and if the check remediates, it redoes completed work.

The executor-scope part is the least obvious and was the most expensive to get wrong. When Lidarr coverage was added, all three of lidarr-cd’s apparent stalls (3 of 66 grabs) had been imported by its sibling lidarr-mp3 ~40 s after the grab, because both instances consume the same SABnzbd. A per-instance index — the shape Radarr and Sonarr had used safely for months, precisely because their content types never overlap — would have produced a 100 % false-positive rate and re-grabbed three albums already on disk. Scope the index by who shares the executor, and prove that scoping with a committed negative-control test.

Remediate rather than only alert where the corrective action is unambiguous and idempotent

Where the fix for a stalled item is the same every time, idempotent, and derivable from the item itself, the assertion MUST perform it rather than merely report it. An alert-only control converts a silent 51-day gap into a noisy 51-day gap and leaves a to-do list; that is the definition of toil. Two bounds keep automated remediation safe, both learned from the reference implementation:

• Cap the retry on the ITEM, not the dispatch. A remediation typically produces a new dispatch id, so a cap stored against the dispatch never engages — every retry looks like a first offence and an unobtainable item ping-pongs forever. Keep the alert-once dedup keyed on the dispatch id and the retry counter keyed on the item.
• Alert once per stalled dispatch, ever. A permanently-unobtainable item must not page daily; past the cap, report it as capped and leave it alone.

Automated remediation is also an Autonomous Remediation Authority mutation — the blast radius rules there apply on top of this clause.

An alert-on-finding assertion is silent when healthy, so it owes a dead-man

A terminal-state assertion is normally quiet — it says nothing when every item terminates. That makes “no page” indistinguishable from “the assertion is dead”: the exact fault it exists to close, one level up. So every scheduled assertion stamps a success-only heartbeat and registers a <check>.age Zabbix dead-man per the Standard-Enforcement Standard. Stamp the heartbeat from the checking code’s success path only — never from the cron wrapper, whose exit status a | systemd-cat pipe would mask (ADR-0297).

A dead-man threshold must allow at least TWO missed runs

Success-only stamping (the clause above) is correct, and it has a consequence the threshold must be sized for: the heartbeat ages on a job that FAILED exactly as it ages on a cron that never fired. Those are different faults with different owners — a failed run is the job’s own failure alerting to report; a cron that stopped firing is the dead-man’s. If the threshold is under 2× the job’s cadence, one transient failure trips the dead-man too, so a single blip costs two pages for one event and the operator learns to ignore both.

So: threshold ≥ 2 × cadence, plus a small buffer. For a daily job that is 50h, not 36h. Sizing it at 1.5× is what fired HA long-lived token dead-man not running (>36h) on 2026-08-06 after a single HTTP 502 aged the heartbeat to 45.5h — while the token was valid throughout and the job’s own wrapper had already paged for the failure.

Two caveats, both learned the same day:

• Do not apply the 2× rule to a weekly or monthly job. Doubling a 168h or 720h cadence means a genuinely dead cron goes unnoticed for two weeks or two months — worse than the false page it prevents. There the remedy is a retry of the job’s transient path, or stamping the heartbeat unconditionally so it means “the cron fired” and the job’s own failure alerting owns the failure case.
• A retry is only safe where the worker separates a transient failure from a verdict. ha_token_deadman.py does — rc=1 is documented as “an operational anomaly, not a token verdict”, so retrying it cannot mask a dead token. external_attack_surface_scan.py returns 1 if findings else 0, where rc=1 is the verdict; a retry there would suppress a real security finding. Read the worker’s exit codes before adding a retry anywhere.

Enforcement is per-surface watchdogs, not one generic reconciler

Each dispatch surface gets its own assertion, colocated with the system it watches. A single generic reconciler was considered and rejected: the four parts above are irreducibly per-surface — terminal vocabularies share no schema (downloadFolderImported vs succeeded vs albumImportIncomplete), deadlines differ by orders of magnitude (minutes for a pipeline run, 48 h for a usenet grab), in-flight is read from a different API each time, and remediation ranges from a re-search to nothing safe at all. A generic engine would be a plugin registry whose plugins carry all the real logic, plus one shared failure domain across every surface. What generalises is this clause, not an implementation.

The live dispatch-surface inventory — every surface is asserted or is a stated gap

Coverage is only meaningful if the surface list is explicit. Each row was verified against the live system, not assumed; adding a dispatch surface means adding a row.

Dispatch surface	Executor	Terminal-state assertion	Deadline
Radarr / Sonarr grab	SABnzbd / qBittorrent	stalled_grab_watchdog.py (daily) — re-searches, then pages once	48 h
lidarr-cd / lidarr-mp3 grab	shared SABnzbd	same watchdog, group-scoped index (Issue #2152)	48 h
ADO pipeline run	self-hosted agent pool	Zabbix ado.runs.stuck.count + ADO pipeline run wedged in flight trigger (ADR-0125); queue congestion is a separate, non-paging ado.runs.queued.* signal	20 min (in-flight)
Paperless document ingestion	Paperless consumer + n8n flow	PaperlessIngestionStalled — backlog non-zero and not draining	1 h
Guardian-audio episode queue	fgr worker	AppProcessQueueStuck — depth non-zero, nothing completed	30 min
verge topic render queue	verge worker	VergeTopicQueueStuck	30 min

Two candidates were investigated and are not gaps: Infisical is pull-only in this estate — no asynchronous sync integration exists to strand an item, and the synchronous fetch path is directly alerted by InfisicalEnvFetchFailure; n8n executions are accounted (started_total = success_total + failed_total live) and each consumer of an n8n dispatch carries its own downstream assertion (the two rows above plus verge_last_daily_run_timestamp staleness), so the direct assertion would be redundant rather than absent.

Two valid alert sources — pick by where the metric originates

Pillar 3 is satisfied by either of two emitters; choose by where the signal lives, do not duplicate one signal across both:

• Alertmanager rules — for Prometheus and Loki metrics. Rules live as code under docker-stacks/stacks/observability/templates/prometheus/rules/. This is the path for anything with a Prometheus metric or a Loki metric/recording rule.
• Zabbix triggers — for agent items (zabbix-agent2 collected metrics, host-level checks). These page through Zabbix media type 89 per zabbix-pushover-media.
• Loki ruler rules — a sub-case of the metrics path for log-content signatures: a known root-cause error line that must page directly rather than via a downstream symptom. These live under docker-stacks/stacks/observability/templates/loki/rules/; see the log-content alerting section of the logging-standard.

The same canonical taxonomy and Pushover-priority mapping govern all three — that is the whole point of the reconciliation above.

Grafana-managed alerting is not a sanctioned emitter and is disabled at the config layer

Grafana is a dashboards-only surface. Its built-in unified alerting engine is not a valid alert source and is disabled as code — obs-grafana sets GF_UNIFIED_ALERTING_ENABLED=false in docker-stacks/stacks/observability/docker-compose.yml.j2 (Grafana v11 has no legacy alerting, so this disables all Grafana alerting). Do not create alert rules or contact points in the Grafana UI: a Grafana-managed rule is a third, unreconciled emitter that pages on its own contact point, bypasses Alertmanager’s routing / silences / inhibition, escapes the canonical severity taxonomy, and — created in the UI — lives only in grafana.db, invisible to review and reproduction (RULE 7 drift). This is exactly the bypass ADR-0103 closed after two UI rules were paging Pushover directly. Visualising alert state on a Grafana dashboard (e.g. active-alerts-health, alerts-mobile, which query Alertmanager/Prometheus as datasources) is fine and encouraged — showing an alert is not raising one. If you need to alert on a signal you were about to build a Grafana rule for, author it in the sanctioned path above instead.

Conformance checklist for a new alert rule

An alert conforms when all of these hold — the checklist /wrapup and reviewers apply alongside the alerting-notification-standard and metrics-dashboards-standard checklists:

• severity is exactly info | warning | critical — no off-taxonomy value (no high, error, major, …).
• The chosen tier matches impact: critical = page now / outage / data-loss; warning = degraded, attend soon; info = record only.
• (Prometheus) Alert name is PascalCase and stable; (Zabbix) the trigger’s native severity maps cleanly onto a canonical tier per the cross-surface table.
• Identifying labels present (host and/or service / instance / target).
• Annotations include summary, description, runbook, dashboard, and a resolving kb (mandatory — alerting-notification-standard / ADR-0062).
• Alertmanager has an explicit route matcher for this severity tier — the alert does not depend on default-route fallthrough.
• The rule lives as code in the correct path (Prometheus rules / Loki rules / Zabbix trigger config) and is committed (RULE 6 / RULE 7).
• Source chosen by metric origin (Alertmanager for Prometheus/Loki, Zabbix for agent items) — the signal is not duplicated across both, and it is not authored as a Grafana-managed rule (Grafana is dashboards-only, ADR-0103).
• If a known root-cause log signature exists, a Loki ruler rule pages on the cause, not a downstream symptom (see logging-standard).
• If the condition is reached by a lookup that came back empty (a join, correlation, registry or index resolution), the rule names and evaluates the precondition under which that lookup could have succeeded, emits a distinct non-alerting outcome when it fails, and counts that outcome (failed lookup, ADR-0410).
• The increase()/rate()/nodata() lookback window approximates the condition’s real duration — it is the tolerated post-event resolve-lag (minutes for a warning), not a padded safety margin that smears a transient into hours.
• Flap dampening is done with for:, not by widening the lookback window; a self-clearing 1–2 minute blip is ridden out by for:, and the rule does not page on a self-heal merely succeeding.
• The window also clears its floor: every rate()/increase()/delta() window is at least twice the measured interval at which the source series actually advances (not the scrape or poll interval). A burn-rate rule declares that cadence in source_cadence_seconds and is proven RED against a breach series via alert_rule_test — never promql_expr_test, which tests a copy of the expression and cannot see for: (window floor, ADR-0394).
• The threshold value was sized against the aggregate the expression computes (not raw observed values), with both margins stated — below the unhealthy subject’s floor and above the healthiest comparable subject’s ceiling (ADR-0296).
• A noisy or low-value alert was tuned (for: / threshold / user-impact condition / severity), not deleted — deletion was used only for a genuinely redundant or meaningless signal, with that justification stated (ADR-0095).
• The clearing condition is named and reachable — state under what condition the rule resolves and confirm the estate can actually get there at its real rate of change. A rule that cannot reach green is red from birth and is not a control (clearability, ADR-0361).
• Where the cleared state needs a fixed resource (channels, ports, nodes, windows, addresses), the count was done — demand against supply — and stated in the rule description. Short supply means the alertable condition is under-use of the available slots, not contention itself (clearability, ADR-0411).
• If the expression fires on a LOW reading (< n), absence has been made safe — the source throws or the rule is census-gated, so missing data goes UNKNOWN rather than PROBLEM (ADR-0305, ADR-0411).
• If the rule watches a rate of change, the metric’s own baseline drift was measured, not assumed — the threshold sits clear of that drift with the multiple stated, and the unit test’s baseline series is the measured drift rather than a flat line. A gauge that accrues, ages or grows on its own defeats any slope rule (clearability).
• A standing breach that genuinely cannot be cleared now is carried as a dated, owner-accepted risk that arms the rule on its expiry — not as a slope rule and not as a renewable silence (ADR-0302).
• If the alert can be read after its condition clears — self-recovering, quiet/null-routed, or able to fire inside the 21:00–07:20 mute (which is nearly all of them) — its runbook leads with a time-anchored retrospective query against retained data (PromQL offset/subquery, LogQL range, Zabbix history), with any live ssh/cat/ps command as a supplement only. A runbook whose sole steps sample instantaneous state cannot be followed late, and a clean live reading is indistinguishable from the condition never having occurred (retrospective runbook, ADR-0390).
• The expression asserts the shape of a healthy response (non-empty / parseable / regex / range), never an exact mutable value such as a version, build hash or config revision — a value pin fires on every routine upgrade and silently blinds the check (incident #1449).
• If the metric is a bursty instantaneous gauge (healthy at its “bad” value much of the time), the rule alerts on a smoothed statistic over a window (max/avg/quantile_over_time) chosen to express the SLO — not the raw instantaneous value, which no for:/window tuning can rescue (ADR-0114).
• If the signal is one of several that fire from a single root event (timestamp-correlated), inhibition/grouping is considered so the root event pages once rather than fanning out.
• If the change introduces or touches a system that hands work to an asynchronous executor, a scheduled terminal-state assertion exists for that surface, carrying all four parts — a live-API-verified terminal vocabulary (side-effect events excluded), a deadline, an in-flight exclusion, and an index scoped to the executor-sharing group — plus a <check>.age dead-man if it is alert-on-finding, and a row on the dispatch-surface inventory (ADR-0307).
• If the change adds or touches an automation that destroys state to remediate a condition (auto-prune, auto-evict, auto-reclaim, auto-delete), it satisfies all four destructive auto-remediation clauses — alerts below its action threshold, aborts on no progress, prefers recoverable removal, and treats a guard trip as an alertable event rather than a success (ADR-0350).
• A memory-headroom trigger uses the three-tier USE ladder — windowed (min/max, never instantaneous), macro-driven ({$MEM.PAVAIL.WARN} / {$MEM.AVAIL.MIN.MB} / {$MEM.PSI.SOME.MAX}) with a data-driven per-host override, ”% OR absolute-MB floor whichever first”, PSI rung only where /proc/pressure/memory exists — and does not edit the stock Linux by Zabbix agent template (ADR-0232).

Destructive auto-remediation alerts below the threshold at which it acts

An automation that destroys state to remediate a condition — auto-prune, auto-evict, auto-reclaim, auto-delete — is only supervisable if a human is told before it acts. media_cleanup.sh began deleting media at 75% pool usage while the capacity alert for that pool fired at 85%: its entire normal operating range sat inside the monitoring blind spot, so every run in the lab’s history happened silently by construction. When its deletions silently stopped reclaiming space (orphaned ZFS snapshots pinned 7.87 TiB, so every rm -rf freed zero bytes) the loop escalated rather than stopped, destroying 587 items in one run. Governed by ADR-0350.

Four clauses, all mandatory:

Clause	Requirement	Why
Alert below the action threshold	An alert must fire at a threshold crossed before the automation acts	An automation whose action threshold is lower than its alerting threshold operates entirely unobserved
Abort on no progress	Measure whether the remediation is working, in the unit that matters, and stop when it is not	Repeating a destructive action that demonstrably did nothing is the failure mode, not the recovery
Prefer recoverable removal	Route destruction through a recovery window (recycling bin, separate pool, soft-delete) where one is cheap	Irreversibility must be a deliberate, justified choice, not a default
A bounded stop is alertable	On a guard trip, emit a failure signal and withhold the success marker its dead-man consumes	A run that stopped early because it could not do its job must never read as healthy

Measure progress in the unit that changes, not the unit that is convenient. The original loop re-read zpool list -o cap — an integer percent, where one percent of a 42 TiB pool is ~420 GiB, far too coarse to distinguish “this delete reclaimed space” from “this delete did nothing”. The guard reads zpool get -Hp free in bytes. A progress check whose resolution is wider than the progress it measures is not a check.

Prefer a guard that measures the outcome over one that enumerates causes. A snapshot is only one thing that can pin blocks — a clone, an open file handle on a deleted inode, or a freeing backlog do the same — so “did this actually free space?” covers the whole class where “is there a snapshot?” covers one member of it.

Progress must be measured in a counter only your own action changes

“Measure the unit that changes” is necessary but not sufficient, and the amendment is load-bearing. media_cleanup’s corrected byte-level guard obeyed the original wording literally — it read zpool get -Hp free, the right unit at the right resolution — and was still unsound, because pool-free is a SHARED counter. Concurrent intake moved it 0.6–0.9 GiB every five seconds, so a genuine 4.06 GiB reclaim measured live as negative progress and the guard would have aborted on essentially every run. A guard that fires on healthy runs gets widened or removed, and then it protects nothing.

So the counter must be one that only the automation’s own action can move. The working form asks two exact questions per item — is this path genuinely gone, and were its blocks re-pinned into a snapshot taken mid-run — neither of which any other writer can perturb. Governed by ADR-0357.

When auditing, apply the amended clause, not the original. A job measuring a shared counter passes clause 2’s letter and fails its intent.

Clauses 1 and 2 bind condition-triggered remediation; clauses 3 and 4 bind everything destructive

The four clauses were written against a pressure-relief automation, and applied indiscriminately they misfire. The ADR-0350 audit (Issue #2255) examined the four candidates the ADR itself named and found only one of them is remediation-shaped; the rest are scheduled retention. Establish the shape first, then apply the clauses:

• Condition-triggered remediation — reads a condition, and destroys state because the condition is true, repeating until it is not (media_cleanup pressure relief). All four clauses bind.
• Scheduled retention — destroys state on a timer according to a fixed policy, with no condition and no loop (docker-prune, pvesnap-prune, PBS prune/GC, the recycling-bin purge). Clauses 1 and 2 are N/A by construction, and marking them “fail” is an audit error: there is no action threshold to sit below, so clause 1 has nothing to alert on, and there is no escalation path, so clause 2 has nothing to abort. Clauses 3 and 4 bind in full — the removal must still be recoverable where that is cheap, and a failed or bounded run must still be alertable and still withhold its success marker.

The distinction is not a loophole. It is what stops a reviewer bolting a meaningless pre-alert onto a nightly image prune, and — more importantly — what keeps attention on the clauses that do bind, which is where the audit found every real defect.

Conformance checklist for a destructive automation

Applied by /code-review and /wrapup’s Pillar-3 (Alerting) review whenever an automation that deletes, prunes, evicts, or reclaims is added or changed. Enforcement here is by review, not by gate, and ADR-0350 says so plainly: no static check can pair an action threshold in a shell script with an alert threshold in a Zabbix database.

• Shape declared. Is this condition-triggered remediation or scheduled retention? State which, because it decides whether clauses 1–2 apply at all.
• (Remediation only) Alert below the action threshold. An alert fires at a value crossed before the automation acts, with enough margin for a human to intervene. Name both numbers and confirm the alert’s threshold is the lower one.
• (Remediation only) Aborts on no progress, never escalates. A bounded consecutive-failure counter stops the loop, and there is a hard per-run cap independent of it.
• (Remediation only) The progress counter is exclusively ours. Confirm no other writer can move it. A pool-wide, host-wide, or otherwise shared figure fails this even when the unit and resolution are right.
• Recoverable removal, and the recovery window is REAL. Not merely “it moves to a bin” — verify the window actually starts when the item arrives and actually closes later. Two ways this silently degrades to zero: the expiring component keys off a timestamp the move did not reset, or the bin’s owner lacks permission to delete what was put there, so the window never closes and the bin grows without bound.
• Irreversibility, where chosen, is justified in a comment. A terminal rm -rf is acceptable as the end of a recovery window; it is not acceptable as the whole removal path.
• A bounded stop emits AND withholds. Both halves. A withheld heartbeat alone is correct but slow — check the dead-man’s grace period is proportionate to the run cadence, not a placeholder. A daily job behind a nine-day dead-man has an eight-day blind spot.
• The failure signal exists at all. For an unattended script this is the PITLAB_RUN_RESULT=fail marker (ADR-0219) with a Loki rule keyed to it. For a vendor scheduler with no marker to emit (PBS, PVE), it is an item asserting on the job’s own task status and an age dead-man on last success — the two catch different failures and neither substitutes for the other.
• Every clause is evidenced against the live system, not the code. Read the object back: the trigger exists and evaluates, the item’s lastclock is non-zero, the recycle bin actually drains. A green playbook is not evidence (ADR-0385).

---

Alertable Counter Instrumentation Standard — priming labelled counters so an alert can fire

This standard governs how a Prometheus counter must be instrumented in the emitting service so that an alert defined over it with rate(), increase(), or delta() will actually fire on the first real event. It exists because a labelled counter’s child series is created lazily — so a first-ever failure is born at value 1 with no 0→1 delta, rate()/increase() report nothing, and any rate(...)>0 alert over it is silently dead. That footgun muted ClaudePodGenerationErrors through two real claude-pod failures for eight days (ADR-0244), and a fleet audit found the same latent hole in verge, orpheus, and vectormap. It is the emitter-side instrumentation companion to the Alerting & Severity Standard (which governs alert-rule shape): a rule can be perfectly shaped and still never fire if the metric backing it is born dead. Governed by ADR-0245.

The failure mode: a labelled counter child is born at 1, so rate()/increase() never see the first event

prometheus_client (and every conforming client library) creates a labelled child series lazily on its first .inc(). A counter with a label whose value has never errored has no series at all until the first increment — and that first increment brings the series into existence already at 1. rate(), increase(), and delta() all need a prior sample to diff against, so a series that appears at 1 and stays flat shows no increase. An alert written rate(errors_total{...}[W]) > 0 therefore stays silent through the very first failure — exactly the failure it exists to catch.

An unlabelled counter does not have this problem: a bare Counter("x_total", ...) registers its single series at 0 at process start, so the first .inc() is a visible 0→1 delta. The footgun is specific to labelled counters.

MUST prime every labelled counter child at 0 at startup

Every service that emits a labelled counter which backs — or could back — a rate()/increase()/delta() alert MUST instantiate each label child at process startup, before any real increment. Instantiating a child (calling .labels(...) without .inc()) creates its series at 0, so the first real failure is a visible 0→1 delta and the alert fires as intended. Enumerate the exact label values the code can emit — do not guess a superset.

episodes_errors = Counter("verge_episodes_errors_total", "Processing errors", ["stage"])
# Prime every stage child at 0 so a first-ever error is a visible 0->1 delta.
for _stage in ("scrape", "llm", "speech_prep", "tts"):
    episodes_errors.labels(stage=_stage)

The priming loop lives immediately beside the counter definition, at module import / app startup, and enumerates every label value the increment sites use. For a multi-dimensional counter, prime the full cross-product the code can actually produce, not every theoretical combination. This is the canonical prometheus_client fix — preferred over rewriting the alert expr (increase() shares the same first-sample blindness; an offset comparison mishandles counter resets).

Manual text-exposition and dict-backed counters are already safe when every child is emitted at 0

A service that renders its own Prometheus exposition (hand-written # HELP/# TYPE + metric{label="..."} value lines from a state dict) is safe by construction if and only if it initialises every label child in the dict and emits all of them every scrape. The incident-webhook / zabbix-incident-poller family (m = {"recall_errors": 0, "cycle_errors": 0, ...}, all keys emitted unconditionally) and the freescout_heal.sh textfile collector (rewrites the whole .prom each call, emitting freescout_heal_total{action="clear|restart|failed|reconcile"} each defaulting to 0) are conformant: their children exist at 0 from the first scrape. The rule generalises — the child must be present at 0 before the first real event, however the metric is produced. A manual exposition that emits a label line only once its counter is non-zero has the same born-at-1 hole as the lazy client child.

Back a discrete-run failure alert with a directly-alertable last-run-status gauge

For a workload whose unit of work is a discrete, atomic run with a single pass/fail outcome — a scheduled batch job, a cron-driven generation, a periodic pipeline (claude-pod’s weekly episode; a nightly export) — a rate()/increase() error alert is fragile even once primed: the signal lives only inside the lookback window W, so a failure older than W is invisible and the window must be widened to span the run cadence (which smears transients — see the Alerting Standard lookback-window rule). Such workloads MUST additionally emit a last-run-status gauge — <svc>_last_run_status set to 0 on success and 1 on failure at the end of every completed run — and alert on it directly:

- alert: ClaudePodLastRunFailed
  expr: claudepod_last_run_status == 1
  for: 5m

A gauge is immune to the counter-birth problem, so a single failed run is directly alertable; the alert holds firing until the next successful run flips it to 0. Initialise the gauge to 0 at startup (last_run_status.set(0)), and where a restart would otherwise blank a companion last_success_timestamp gauge that a staleness alert guards on > 0, seed it from durable state (e.g. the newest record’s timestamp) so a restart never disables the staleness backstop (ADR-0168, ADR-0244).

Continuous request servers use the primed rate/ratio — a per-request gauge is the wrong signal

A continuous or queue-draining service — a request-serving API (orpheus TTS, vectormap search) or a worker draining an async queue (verge topic processing) — has no discrete run with a single outcome, so a last_run_status gauge would flap 0/1 per request and carry no meaningful failure signal. For these, priming the labelled counter is the complete and correct fix: a primed rate(errors_total[W]) > 0 (or an error-ratio alert, rate(errors[W]) / rate(total[W])) fires on the first real error and is the sanctioned single-event signal. Do not bolt a run-status gauge onto a request server; prime the counter instead. The choice is by workload shape: discrete-run → gauge and primed counter; continuous/request/queue → primed counter (ratio optional).

Conformance checklist for a counter-backed alert

Before an alert defined with rate()/increase()/delta() over a counter is considered done:

• Is the counter labelled? If yes, the emitting service primes every label child at 0 at startup (or, for manual/dict/textfile exposition, emits every child at 0 from the first scrape).
• The priming loop enumerates exactly the label values the increment sites use — verified against the source, not assumed.
• For a discrete-run workload: a <svc>_last_run_status gauge is emitted (0/1 at run end), initialised to 0 at startup, with a companion == 1 alert; any last_success_timestamp gauge is seeded from durable state on restart.
• For a continuous/request/queue workload: the primed rate/ratio alert is the signal; no per-request status gauge is added.
• Proven, not assumed (Prime Directive 12): the primed children are present at 0 in live Prometheus after deploy, and the alert would fire on a synthetic single increment.

---

Enforcement

This is the consolidated enforcement declaration for the whole Alerting & Notification domain — the notification-content, alerting-&-severity, and alertable-counter-instrumentation rule sections above resolve to one table (per the Standard-Enforcement Standard, the ## Enforcement section is the single source of truth the meta-gate resolves against). The notification-content and config-shape clauses are largely machine-enforced at deploy time; the severity-judgement, kb-placement, alert-tuning, and emitter-side counter-priming clauses — which no static check can decide — are human review checkpoints keyed to the conformance checklists above.

Obligation	Class	Layer	Mechanism	Dead-man
Every notification emits through pushover_notify.py with structured --component/--event — a bare --title/--message call is non-conforming (ADR-0034/0062)	machine	deploy	validate_notification_catalog.py BARE-CALL check (the NotificationCatalog job of CI ansible_ci.yml + pre-commit)	n/a
Every literal (component, event) a call site emits resolves to a notification_catalog.yml entry	machine	deploy	validate_notification_catalog.py COVERAGE check (CI + pre-commit)	n/a
The guard must actually see every emitter — a bare-form file that carries notification flags but exposes no windowable call region (unrecognised invocation idiom) fails loud, never passes in ignorance (ADR-0222)	machine	deploy	validate_notification_catalog.py non-attribution coverage guard; recognises all three helper-path constant forms (python-/shell-quoted, shell-unquoted) so the common $PUSHOVER idiom is windowed	n/a
kb is mandatory and every kb link (catalog, Zabbix kb tag, alert-rule kb annotation) resolves to a real docs heading anchor (ADR-0062)	machine	deploy	validate_notification_catalog.py ANCHOR check (CI + pre-commit)	n/a
A kb points at the owner of what fired — service-owned runbook on that service page, fleet-wide signal in operations/alerting (ADR-0064)	review	—	/code-review + PR review against this standard; reviewer bounces a service-owned runbook parked in operations/alerting (resolution gate stays green, so review is the only control)	n/a
severity is present and drives Pushover priority via one mapping — no per-script hard-coded priority number	advisory	—	justification: rendered by the single pushover_notify helper by construction; no static check of the runtime-sent header	n/a
Parse-safe compact body — 1024-char limit with stated truncation, lowercase key=value header keys, one event per notification	advisory	—	justification: authoring discipline over runtime output, no reliable static signal	n/a
Every Prometheus/Loki alert rule is well-formed (valid PromQL, templated annotations render) and deploys only from committed code	machine	deploy	promtool check rules staged fail-closed in deploy.yml (observability stack)	n/a
The Alertmanager routing config (per-tier routes, receivers, templates) is valid before it reaches the live path	machine	deploy	amtool check-config on the staged render in deploy.yml (fail-closed)	n/a
Every alert rule carries a mandatory kb annotation whose docs page and anchor resolve	machine	deploy	validate_notification_catalog.py (the NotificationCatalog job of CI pipeline ansible_ci.yml) scans Prometheus/Loki rule kb: annotations	n/a
Zabbix native severity maps onto the canonical tier and Pushover priority via the Balanced profile	machine	deploy	zabbix_pushover_media_standard.py idempotently asserts the priority_* map on media type 89 (config-as-code)	n/a
Which Zabbix severities reach Pushover at all is code, not a hand-set UI field — the per-user media severity mask is 49 (Not classified, High, Disaster), excluding Warning and Average (ADR-0465)	machine	deploy	zabbix_pushover_media_standard.py ensure_user_media_severity() idempotently asserts the mask on every user carrying Pushover media, re-sending the user’s other media rows verbatim so the High/Disaster email failover (media type 102, ADR-0162) survives user.update’s replace-not-merge semantics	n/a
The severity: warning Alertmanager route resolves to the 'null' receiver, and only the six named alertname routes page at warning tier (ADR-0465)	review	—	/code-review of alertmanager.yml.j2; the routing is provable with amtool config routes test --config.file=<rendered> severity=warning, which must resolve to null — run against the staged render alongside the existing amtool check-config gate. Not machine-gated: adding a named exception is a deliberate judgement about whether an alert names a specific fault or a symptom class	n/a
Every Zabbix trigger (alert source) lives as code, never hand-created via the UI/API	machine	deploy+scheduled	configure_homeassistant_zabbix.yml and its per-service siblings define triggers as code; zabbix_monitoring_reconciler.py daily drift audit flags any uncoded live trigger	monitoring.reconciler.age
severity is exactly info/warning/critical and the rule shape conforms (PascalCase stable name, identifying labels, summary/description/runbook/dashboard)	review	—	“Conformance checklist for a new alert rule” (this standard) applied by /wrapup and /code-review — no CI check covers rule-annotation taxonomy yet	n/a
An alert readable after its condition clears (self-recovering, quiet/null-routed, or fireable inside quiet hours) carries a time-anchored retrospective diagnostic in its runbook, not only a live host read (ADR-0390)	review	—	“A runbook for an alert that can be READ after it clears” + conformance checklist item, applied by /code-review and /wrapup. A machine heuristic is available (self-recovering rule shape + a runbook containing only ssh/cat/ps and no promq/LogQL query) but would carry day-one false positives over prose, so it is deliberately deferred to Issue #2379 — the failure is silent by construction, since a live read on a cleared transient returns healthy whether or not the cause was real	n/a
Every severity tier has an explicit Alertmanager route matcher — no default-route fallthrough	review	—	Conformance checklist item, applied by /code-review of alertmanager.yml.j2 route matchers	n/a
No Grafana-managed alert rule is authored — Grafana is dashboards-only	review	—	Conformance checklist item + /code-review (GF_UNIFIED_ALERTING_ENABLED=false is set in the observability compose; a UI-created rule is caught only by review)	n/a
Alert-quality tuning fits the condition — right lookback window, for: for flap-dampening, smoothed statistic for bursty gauges, tune-not-delete, inhibition for correlated fan-out	review	—	Conformance checklist tuning items + per-decision ADR review (ADR-0095/0114) via /code-review	n/a
A burn-rate alert declares its source’s update cadence and every window spans at least 2x it — below that floor rate() returns no data and the alert cannot fire at all (ADR-0394)	machine	deploy	counter_birth_guard.py batch-cadence class (Validate stage of docker-stacks-observability); requires source_cadence_seconds on any 1 - (rate/rate) burn shape and fails the deploy on any window under 2x it. Proven red against the real pre-repair rules, not only fixtures	n/a
A burn-rate or SLO alert is proven RED against a breach series through alert_rule_test before it is trusted green — promql_expr_test tests a copy of the expression and cannot see for:	review	—	“A rate window has a FLOOR” + conformance checklist item, applied by /code-review and /wrapup. Not machine-checkable: the gate cannot tell a faithful assertion from a vacuous one, which is the failure that let incident #1758 run its full course	n/a
A signal an incident review reveals as missed gets an alert authored before the work closes	review	—	/wrapup Pillar-3 (Alerting) review checkpoint	n/a
An automation that destroys state alerts below its action threshold, aborts on no progress, prefers recoverable removal, and treats a guard trip as alertable (ADR-0350)	review	—	“Conformance checklist for a destructive automation” (this standard) applied by /code-review + /wrapup Pillar-3 — no static check can pair an action threshold in a shell script with an alert threshold in Zabbix; the withheld-heartbeat clause is the one observable part, caught by the job’s existing <check>.age dead-man	per-job <check>.age
The four destructive clauses are applied by automation SHAPE — clauses 1-2 bind condition-triggered remediation only, clauses 3-4 bind every destructive automation including scheduled retention (ADR-0388)	review	—	“Clauses 1 and 2 bind condition-triggered remediation” (this standard); the reviewer states the shape before applying the checklist, so a retention job is not failed for lacking a pre-alert it cannot have	n/a
A destructive automation’s recovery window is verified to START on arrival and CLOSE on expiry, against the live system (ADR-0388)	review	—	Conformance checklist item applied by /code-review; the two silent-degradation modes (an expiry keyed off a timestamp the move did not reset, an owner without permission to delete what was placed in the bin) both leave code that reads correct, so only live evidence decides	n/a
Every kb URL authored as a KB_* module constant in scripts/zabbix_*.py resolves, not only the inline tag-dict form (ADR-0388)	machine	deploy	validate_notification_catalog.py py_const_re scan, added under Issue #2255 — the constant form had been unscanned since Issue #1551, exempting the whole xt035 PBS trigger family from the anchor gate	n/a
One signal is not duplicated across both emitters — source is picked by where the metric originates	advisory	—	justification: cross-surface duplication is a design judgement — a reviewer can only catch it if they already know the other emitter exists, so no reliable machine or human gate	n/a
Every labelled counter backing a rate()/increase()/delta() alert primes all its label children at 0 at startup	review	—	/code-review of the emitting service and its alert rule against the conformance checklist above — no static check decides emitter-side priming yet	n/a
Manual / dict / textfile-collector exposition emits every label child at 0 from the first scrape	review	—	/code-review of the exposition code against the conformance checklist	n/a
A discrete-run failure alert is backed by a last_run_status gauge (0/1 at run end) with a == 1 alert	review	—	/code-review plus the /wrapup Pillar-3 (Alerting) checkpoint when a batch/cron service is added or changed	n/a
A continuous/request/queue service uses the primed rate/ratio, not a per-request status gauge	advisory	—	justification: workload-shape is a design judgement — a reviewer catches a misapplied run-status gauge only with service context, so no reliable machine or human gate exists	n/a
Every system that dispatches work to an asynchronous executor has a scheduled terminal-state assertion, with absence of a terminal state past a deadline treated as failure (ADR-0307)	review	—	“Conformance checklist for a new alert rule” (this standard) applied by /wrapup and /code-review — enumerating what is a dispatch surface is a design judgement no static check can make; the dispatch-surface inventory is the reviewer’s worklist	n/a
An alert whose condition is reached by a lookup that returned nothing names and evaluates the precondition under which that lookup could have succeeded, emits a distinct non-alerting outcome when it fails, and counts that outcome (ADR-0410)	review	—	“Conformance checklist for a new alert rule” (this standard) applied by /wrapup and /code-review — whether a condition rests on an inference from an empty join is a design judgement no static check can decide; the failed lookup clause is the reviewer’s test	n/a
Every *arr grab (Radarr, Sonarr, lidarr-cd, lidarr-mp3) is asserted to reach a terminal history event, re-searched once, and paged once (ADR-0298/0307)	machine	deploy+scheduled	ansible/scripts/stalled_grab_watchdog.py — daily 11:40 AEST via stalled_grab_watchdog_cron.sh, deployed by sync_control01.yml	media.grab.stall.watchdog.age
A terminal-state assertion’s resolved-item index is scoped to the executor-sharing group, not the dispatching instance — proven red before green (ADR-0307)	machine	deploy	ansible/scripts/test_stalled_grab_watchdog.py — blocking self-test step in the sync_control01.yml pipeline; asserts the per-instance shape produces the false stall and the group-scoped shape suppresses it	n/a
A trigger/alert expression asserts the shape of a healthy response, never an exact mutable value (version, build hash, config revision) — a pin fires on every routine upgrade and silently demotes the check to a no-op, so a broken service reads identical to a healthy one (incident #1449, ADR-0446)	machine	deploy+scheduled	Two layers, because one alone leaves the majority uncovered. At deploy: the fail-closed assert in ansible/tasks/zabbix_ensure_objects.yml rejects a pinned expression at reconcile time — but only for triggers flowing through that task, not the template-provisioned and monitoring-provisioner-created majority. Scheduled: ansible/scripts/stuck_alert_watchdog.py control B sweeps all enabled triggers on the live server daily (5,084 at authoring, 0 violations — a regression guard, not a backlog). Discrimination proven red by ansible/scripts/test_stuck_alert_watchdog.py, a blocking step in sync_control01.yml, whose negatives include decimal-equality expressions because mutation testing showed the first draft could not fail without them	stuck.alert.watchdog.age
An ADO pipeline run reaches a terminal result, or is reported wedged in flight past the age threshold (ADR-0125)	machine	deploy	ansible/scripts/zabbix_ado_monitoring.py idempotently asserts the ado.runs.stuck.count/.detail items (now inProgress-only) and the ADO pipeline run wedged in flight trigger as code, plus the separate ado.runs.queued.* items and Information-severity ADO pool congestion trigger — queue congestion is measured but never pages (tag ado_queued, which the Pushover action’s tag-name routing on ado_stuck cannot match) — detection itself is continuous Zabbix server-side polling, so there is no cron to age out; out-of-band drift on the trigger is carried by the “every Zabbix trigger lives as code” row above and its monitoring.reconciler.age dead-man, not claimed again here	n/a
An alert-on-finding terminal-state assertion (silent when healthy) carries a success-only heartbeat and a <check>.age dead-man	machine	deploy+scheduled	zabbix_cron_heartbeats_monitoring.py JOBS registration + the heartbeat UserParameter deployed by sync_control01.yml	media.grab.stall.watchdog.age