ADR-0034 — All notifications carry a machine-readable header, emitted through one helper

Status	Accepted — implemented (2026-06-22)
Date	2026-06-22
Deciders	Arron + Claude
Extends	ADR-0019 (enriched Alertmanager alerts)
Complements	ADR-0033 (channel policy — Pushover, never email)

Context — alert notifications were enriched, but script/cron notifications were ad-hoc prose

ADR-0019 made Prometheus alerts self-contained for triage: the Alertmanager Pushover receivers render host, severity, labels, a Melbourne-local timestamp, a correlation id and deep-links, so a human or agent can act straight from the notification. But that enrichment lives only in the Alertmanager receiver templates. Every other notification — cron scripts, rotation and backup outcomes, watchdogs, agent actions — went through the pushover_notify.py helper as a free-text --title + --message, each script inventing its own prose. The feedback-memory-hygiene work (ADR-0031) added more of the same, which surfaced the gap: the agent is increasingly handed notifications as investigation input, and prose-only notifications force it to leave the text and re-derive the host, the event and the next step. There was no written standard for notification content, only for the channel (ADR-0033).

Decision — a structured header on every notification, rendered by the one helper

All pitlab notifications conform to the Notification Standard: a machine-readable header block of key=value lines (host, component, event, severity, ts, plus salient metrics and a ref), then the human detail, under a [pitlab] <component>: <outcome> title. severity (info/warning/critical) maps to Pushover priority (0/1/2) in one place, so no script hard-codes a number.

Conformance is made the path of least resistance rather than a convention to remember: pushover_notify.py — already the single emitter used by every script on control01 and pve01 — gains optional structured flags (--host --component --event --severity --field KEY=VALUE --ref --url) that render the header. The flags are backward-compatible (a legacy call still sends), so migration is incremental, but a bare --title/--message notification is defined as non-conforming. Alertmanager alerts already satisfy the standard via ADR-0019 and are the reference implementation; they are left as-is.

Alternatives considered

• Leave each script to format its own prose (status quo). Rejected: every script re-implements where/what/how, new scripts start thin, and nothing is parseable — the exact problem ADR-0019 solved for alerts, left unsolved everywhere else.
• Emit structured JSON as the body. Rejected: the notification has a human consumer too, and a phone showing raw JSON is hostile. A key=value header is greppable by an agent and readable by a human, and fits the 1024-char Pushover body far better than JSON’s punctuation overhead.
• A separate structured side-channel (e.g. a webhook/Loki event) parallel to the human Pushover. Rejected as over-engineering for the need: the agent consumes the same Pushover text a human does, so enriching that one text serves both. Loki already captures script stdout for forensics; the notification is the triage surface.
• Enforce via a CI linter that greps scripts for bare notifications. Deferred, not rejected: the helper-as-single-emitter plus review is sufficient today. A grep gate in the ansible pipeline that flags pushover_notify calls lacking --event is a cheap future hardening if drift appears.

Consequences

• Positive: every notification is self-contained for triage by human and agent alike; severity-to-priority is consistent fleet-wide; the shape evolves in one file (the helper) and one spec (the standard). The agent can route or correlate on a stable event name. No new dependency or channel.
• Negative / limits: the standard is enforced at write time by review, not by an automated gate, so a careless bare notification can still ship until the optional CI grep is added. The 1024-char Pushover limit means the header trades a little room against the human body — acceptable, since the header is ~120 chars and long lists are truncated by design.
• Migration: existing script callers (ado-pat-rotate, media-cleanup, rclone-files-sync, ups-watchdog, memory-eviction, decom-phase2, bun3d-vm-rebuild, pitlab-hookscript-deploy, feedback-hygiene) are brought to the standard in this change; the helper renders the header for all of them. Future scripts inherit conformance by calling the helper with the flags.