ADR-0125: One global ADO pipeline stuck alert — disable per-pipeline failure paging

Status: Accepted (Implemented 2026-07-06, Epic #1564) Date: 2026-07-06

Context

The ADO pipeline monitoring subsystem (Zabbix host ado-pipelines, config-as-code in ansible/scripts/zabbix_ado_monitoring.py, ADR-0036) had grown three paging paths, two of which paged per pipeline:

• action #7 “ADO Pipeline Failures” — the ADO pipeline failed: {#PIPELINE_NAME} trigger prototype fires once per failing pipeline, and action #7 sent a Pushover per pipeline plus a matching “recovered” Pushover when it went green.
• action #8 “ADO Approvals” — single scalar, one alert.
• action “ADO Stuck Runs” — the detective stuck-run subsystem (Issue #1296), already a single global alert firing when min(ado.runs.stuck.count,10m)>0, thresholds notStarted>20m / inProgress>60m.

An alert-volume audit over ~2 weeks found 50 of 56 ADO-related Pushovers came from the per-pipeline failure path (#7), dominated by transient CI failures churning Pipeline Failed→Pipeline Recovered pairs across ~12 pipelines (pitlab-docs ×6, ansible-validate-notification-catalog ×5, …). The stuck alert fired only 3 times. Arron’s stated requirement: “I don’t want individual alerts about individual pipelines — I just want to know when any pipeline is stuck for 20 minutes or more, via one single alert.”

The desired end-state — one global alert when any run is wedged ≥20m — already existed as the stuck subsystem; the noise was the per-pipeline failure path, which Arron reads as “stuck” (a stuck run that times out ultimately reports failed, so it also pages per pipeline).

A run-duration check (last ~120 runs) confirmed the slowest healthy pipeline is docker-stacks-infra at ~18m max (avg 8m); every other pipeline completes well under 20m. So a 20-minute inProgress threshold sits safely above the slowest healthy run — it will not false-page on normal execution.

Decision

Make the single global ADO pipeline run stuck alert the only pipeline page. Disable the per-pipeline failure action (#7) and lower the stuck inProgress threshold from 60m to 20m.

1. Lower STUCK_INPROGRESS_MIN 60 → 20. With notStarted already at 20m, any run wedged (queued or running) past 20 minutes now trips the one global alert. Safe margin above the 18m slowest-healthy run.
2. Disable action #7 (status 0 → 1), config-as-code, via a new idempotent apply_fail_action_disabled() managed change in zabbix_ado_monitoring.py. The failure trigger prototype is left enabled, so a red pipeline still raises a Zabbix problem (visible in the problem list / dashboard for reference) — only the Pushover routing is silenced. Fully reversible: flip status back to 0.

The stuck alert lists every wedged run (names + deep-links) in one Pushover body via {ITEM.LASTVALUE2}, so “which pipelines” is answered inside the single message without per-pipeline alerts.

Alternatives considered

• Consolidate #7 into one rolled-up “N pipelines failing” alert (keep failure paging, collapse to a single message). Rejected: still pages on every transient fast-failure, which is the bulk of the noise and which the ADO UI already shows; Arron wanted the stuck condition paged, not every red build. Offered as a future opt-in if fast-failure visibility is later wanted.
• Lower inProgress below 20m (e.g. 15m). Rejected: docker-stacks-infra legitimately runs ~18m; sub-20m would false-page it.
• Disable the failure trigger prototype (not just the action). Rejected: keeping the trigger preserves the per-pipeline problem in Zabbix for dashboard/forensic reference at zero paging cost.

Consequences

• One pipeline page, as requested. ADO pipeline run stuck (Average) is the sole interrupt: fires once when any run is notStarted/inProgress >20m, lists all stuck runs.
• Trade-off — fast failures no longer page. A pipeline that fails quickly (test failure, bad commit dying in <20m) is neither notStarted nor inProgress long enough to trip the stuck alert, and #7 is silenced. Such failures are visible only in the ADO UI and the Zabbix problem list, not via Pushover. This is the accepted cost of eliminating the failed→recovered churn; the wedged/blocked-deploy case (the one worth interrupting for) remains covered. This narrows the paging surface established in ADR-0036 — it does not reverse the “alerting stays in Zabbix, no exporter” decision, only which conditions page.
• Reversible. Re-enabling per-pipeline failure paging is a one-line status flip in zabbix_ado_monitoring.py.
• Prime Directive 8 (Alerting): the alertable condition (a stuck/blocked pipeline) remains covered by an alert; the change removes redundant per-instance paging, not coverage of the signal Arron cares about.