The alert that paged as an emergency

The short version

Bottom line. Three alerting surfaces used three severity vocabularies and nothing reconciled them, so a low-importance signal fell through to the default route and paged as a repeating emergency.

• Why it was expensive — an off-taxonomy severity does not error. It mis-routes. No red build, no rejected rule, no log line: the system did exactly what it was told.
• What changed — one taxonomy mapped across all three surfaces, routing defined against that contract, and a default route that no longer pages — because whatever reaches a default is by definition the thing nobody classified.
• What it costs — per-pipeline failure paging was deliberately switched off. A single pipeline can now fail without interrupting anyone, which is only defensible because a failed deploy leaves the previous version live.

The problem

Alerts arrived from three surfaces. The metrics and log rules used one severity vocabulary. The infrastructure monitoring platform used its own native six-tier scale. The notification standard defined a third, three-tier scheme mapping to notification priorities — and it governed scripts and cron jobs but not alert rules.

Nothing reconciled the three. The routing layer matches a severity label as an exact string, first match wins, and it had explicit matchers for two values and none for a third that one rule was using.

So that rule fell through to the default route, whose receiver was the emergency one. A low-importance signal — an index being empty — paged as a repeating, do-not-disturb-overriding emergency.

The mechanism is worth naming precisely, because it is the reason this class of fault is expensive: an off-taxonomy severity does not error. It mis-routes. There is no red build, no rejected rule, no log line. The system does exactly what it was told, and what it was told was outside the vocabulary anyone had written down.

There was also no single definition of what a conforming alert rule looks like — which labels are required, which annotations, whether a link to the resolving runbook is mandatory — so rule authoring had drifted per author.

The design

One taxonomy, three surfaces, an explicit mapping between them (ADR-0078). Every surface’s native severity maps onto one contract, and routing is defined against that contract rather than against whatever string a rule happened to carry. The default route stops being an emergency receiver — a default that pages is a trap, because the thing that reaches a default is by definition the thing nobody classified.

A conforming rule has a defined shape, written into the alerting and notification standard: required labels, required annotations, and a mandatory link from the alert to the runbook step that resolves it (ADR-0062, ADR-0064). Every notification also carries a machine-readable header emitted through one helper, so the notification surface has one implementation rather than one per script (ADR-0034).

Volume is designed, not tolerated. Per-pipeline failure paging was deliberately disabled in favour of a single global alert, because a fleet of pipelines each paging on its own failure is a guaranteed route to a muted channel (ADR-0125). And a fault with a common root cause raises one alert rather than one per symptom, through dependency chains and inhibition (ADR-0001) — the change that turned forty-one notifications from a single host restart into one.

The trade-off

Disabling per-pipeline failure paging is the uncomfortable one, and it should be. It means a single pipeline can fail and nobody is paged about that pipeline. The failure still raises a visible record and still turns a board red; it just does not interrupt anyone.

That is a deliberate exchange of detection latency for signal quality, and it is only defensible because of what sits underneath it: a failed deployment leaves the previous version live, so the cost of finding out an hour later is bounded. If a failed deploy degraded the running service, the trade would be wrong.

The second cost is that a reconciled taxonomy has to be enforced, or it decays back into three vocabularies within months. The reconciliation is therefore checked, not documented — which is the same pattern as everywhere else here: a rule with no machine check behind it is a suggestion.

The evidence

• The mis-routing was reproduced from the routing configuration itself: the label had no matcher, and first-match-wins sent it to the default receiver. It was a configuration reading, not an inference from the symptom.
• The single-fault-single-alert change is measurable against its own before state: one host restart, forty-one notifications, one root cause.
• Notification volume is now bounded by design rather than by muting: nothing exceeds the lowest priority, and nothing pages between 21:00 and 07:00 at all (ADR-0326).

The last of those is the constraint that makes the rest necessary. An estate that can page at 3am can carry a noisy rule for months. One that cannot has to fix it.