ADR-0329: CI trigger gaps are detected by a reconciler, and pipeline liveness is derived from the ADO API

Status: Accepted Date: 2026-08-06

Context

Issue #2198 was raised as “CI triggers silently swallowed — a pushed fix can sit unbuilt on a RED main indefinitely”, recording two apparent losses within one hour on 2026-08-06 and naming batch: true (ADR-0311) as the suspected mechanism: a batch head consumed by a completed or failed run, with the successor never queued.

The builds API does not support that reading, and the correction is the load-bearing part of this decision. Queried across all statuses rather than completed-only:

Case	What was reported	What the API records
pitlab-docs (def 18)	run 7348 failed; 38 min later no run queued, queue empty	run 7363 queued batchedCI at 22:04:42Z — five seconds after 7348 finished at 22:04:37Z — carrying sha 3bfa284c. It sat notStarted for 60 minutes and started at 23:04:43Z. The manual “recovery” run 7364 was queued at 22:07:45Z, three minutes after the auto-trigger had already fired.
ansible-control01-sync (def 30)	push b1295d23 touched scripts/, no run queued	run 7357 queued batchedCI at 21:54:37Z — five seconds after 7344 finished — carrying sha 0aff641b, which has b1295d23 as an ancestor. The manual run 7365 came 14 minutes later.

So batch: true behaved exactly as specified, and a failed run does not suppress its successor. Both runs existed continuously; both manual queues were the redundant duplicates the operating principles explicitly warn against. The defect was in the looking, not in the triggering — a queued run is invisible to a completed-only view, and “no run” and “a run you cannot see” produce identical output.

Two genuine gaps remain underneath the misdiagnosis, and they are what this ADR addresses:

1. Nothing verifies the estate’s most load-bearing assumption. Auto-rollback, the post-deploy test-plan gates, the pre-push validator and every config-as-code guarantee all rest on pushed == will be built. No control tested it. A commit that genuinely never built would look precisely like a commit that did.
2. A 20–60 minute queue makes “HEAD has no queued run” a normal state for the better part of an hour, with no signal distinguishing legitimate batch deferral from a real loss. ado.runs.queued.* covers congestion at Information severity and deliberately never pages (ADR-0322); it is a different question.

Separately, Issue #2199 step 8 needed the eit_reconciliation and standards_enforcement_reconciliation pipelines to stop stamping /var/lib/pitlab/<check>.last on the agent’s own disk. That pattern encodes a single-agent assumption — the Standard-Enforcement Reconciliation doc states verbatim that “the pitlab ADO agent runs as pit on control01, which owns /var/lib/pitlab” — which a two-runner pool invalidates: the stamp lands on whichever host won the job, so each host’s copy is stale half the time and both dead-men flap.

Both problems are the same shape: poll the ADO builds API per pipeline and act on the result.

Decision

One script, ansible/scripts/ado_ci_reconciler.py, with one declared registry (ado_ci_reconciler.yml), run by a 20-minute control01 cron, serving both concerns.

Coverage is ancestry, not equality. Under batch: true a run is labelled with only the last sha of the batch it coalesced, so commit C is built iff C is an ancestor of (or equal to) some run’s sourceVersion. Equality matching would report every coalesced push as unbuilt. This is the same semantics ado_pipeline_run.py already implements for pipewait, deliberately reused rather than re-derived.

Three independent suppressors, each of which a naive implementation gets wrong. A gap is reported only when all three hold, and every one of them was proven necessary by a false positive on the first live run:

• Contemporaneous trigger paths, not today’s. Evaluating an old commit against the current paths.include invents gaps that never existed. ansible-service-catalog is the worked example: its YAML gained scripts/service_catalog_regen.sh on 2026-07-26 (the #1967 fix, whose own comment records that an edit to that script “sat committed-but-not-live”). Judged by today’s filter, two commits from the preceding week look like swallowed triggers. They are not — the pipeline genuinely did not watch that path when they landed. The reconciler resolves each commit against the pipeline YAML as it stood at that commit.
• CI-skip tokens — but only in the subject line. Azure Pipelines skips a push whose commit message carries [skip ci] and its variants, and this estate generates those constantly — every doc auto-refresh commit has one. The first live run reported both tf-cloudflare pipelines as gaps for commit 93bdb628, “docs: auto-update Cloudflare documentation [skip ci]”, a commit that asked not to be built. The subject-versus-body refinement below is what makes this suppressor safe rather than a hole.
• A pending run is never a gap, at any age. A run sitting notStarted means a build is coming. Reporting it would recreate the exact misdiagnosis this tool exists to prevent.

Detect and page; do not auto-queue. --queue exists but is off by default and absent from the cron. Queueing a “gap” into a single-agent pool manufactures the congestion that produces the apparent gaps, and the incident that prompted this work consisted of two manual queues that were themselves the only spurious artifacts in the record. Revisit once the two-runner pool (Issue #2199) has capacity to absorb it.

The threshold is a queue-latency allowance, not a taste setting. 90 minutes = the worst observed queue wait (60 min) plus the longest run (~25 min) plus margin. A threshold below the observed queue latency would fire on healthy batching and be muted within a week. It is expected to come down once the queue-time SLI has a post-cutover baseline (Issue #2206).

Heartbeats stamp the run’s finishTime, never the poll time. Stamping “now” would make the dead-man unfalsifiable — it would refresh on every poll whether or not the pipeline had run again, which is an assertion satisfiable without the feature it claims to test (Prime Directive 12, and ADR-0317). The stamp is additionally monotonic, so an ADO page returning stale data cannot rewind a heartbeat into a false alert.

One dead-man guards three controls. ado.ci.reconciler.age (>90 min, ≥2 missed runs plus margin) exists because a silently dead reconciler would freeze eit-recon.last and standards-enforce.last at their last good values — so both sibling dead-men would keep reading healthy while nothing was checking anything. That compounding blind spot is the specific failure the self-heartbeat makes impossible.

Nothing about a trigger is duplicated into the registry. Branches and path filters are read from each pipeline’s own YAML on origin/main, resolved through the ADO definition’s repository + process.yamlFilename. A copied filter list would drift from the YAML silently, and the reconciler would then reason about triggers that do not exist. The registry declares only what cannot be derived: repo checkout locations, the heartbeat mapping, the threshold, and explicitly-reasoned exclusions.

UNRESOLVED is not clean. A pipeline whose repo is missing or whose YAML cannot be read exits 2, not 0 — it was not checked, and an unchecked check must never read as a passing one.

A CI-skip token in the body is a mistake, and this tool proved it on itself

The commit that introduced ado_ci_reconciler.py was never built. Its message described the skip-token suppressor and, in doing so, contained the literal text [skip ci] in the body. Azure Pipelines honours a skip token anywhere in the message, so it queued nothing — no run, no red build, no signal. pipewait --sha-only returned DISCOVERY_TIMEOUT, the authoritative “trigger never fired”. Writing about the mechanism triggered the mechanism, and a change whose entire purpose was to catch unbuilt commits became one.

The first version of the reconciler treated that commit as a legitimate skip and correctly reported “no trigger expected” — faithful to ADO, and useless. So the suppressor is now split by position:

Token position	Treatment	Why
Subject line	genuine skip, silent	every deliberate use in the estate takes this form
Body only	reported as CI_SKIPPED_IN_BODY, immediately	nothing about the commit says the skip was intended

Measured across 120 days, 298 commits carry a skip token and every single one has it in the subject — ansible 229, tf-proxmox 37, tf-cloudflare 32, and zero in a body, in any repo. Every real use is a generated doc-refresh commit. A body-only token is therefore not an ambiguous signal to weigh; it is a mistake, and the position split turns an unfalsifiable judgement into a measured one.

This class is reported without waiting out the 90-minute threshold, unlike an ordinary gap. The threshold exists to let a queued run arrive; here there is no run and never will be, so waiting only delays the fix. Recovery is one unskipped commit — the pipelines re-checkout origin/main, so a single follow-up builds the whole tree.

A pre-push hook was rejected as the control: the ansible repo has no hook infrastructure, hooks are per-clone and untracked, and — decisively — a CI gate cannot catch this by construction, because the commit it must judge is precisely the one that runs no CI. Only an out-of-band poller can see it, which is exactly what this reconciler already is.

Proven before trusted

Two layers, both re-runnable, because “41 pipelines checked, 0 unbuilt” and “the detector is blind” are the same output:

• --self-test — 29 offline assertions over the path matcher, the trigger parser and the skip-token reader, run automatically before every other mode so a detector that cannot match a path can never report a clean estate. Proven red by disabling one matcher rule: 3 assertions failed, exit 1.
• --red-test — against the live estate, on the live code path (audit_definition() is shared, so the test cannot pass a parallel reimplementation). It picks the busiest clean pipeline, withholds its newest runs, and asserts the audit then reports exactly those commits unbuilt — then restores them and asserts clean again. Measured on pitlab-docs: 0 unbuilt at baseline → 89 unbuilt with 68 runs withheld → 0 unbuilt restored. Both directions asserted, so neither a stuck-green nor a stuck-red detector passes.

With the detector proven, --audit --since-days 14 reported 41 CI-triggered pipelines checked, zero trigger-matching commits never built — the systematic disproof of the swallowed-trigger hypothesis, reconstructed from the record rather than argued from one incident.

The audit also found real drift on its first run: two enabled ADO pipeline definitions whose YAML no longer exists on main (ansible-podcast-zabbix id 44, ansible-five-great-reads-podcast id 47), orphaned by the ADR-0323 decom, which removed the code but left the definitions registered and triggerable. That class now has its own orphan_definition page.

Alternatives considered

• Scope batch: true off the pipelines where a missed build is unacceptable. Rejected: it treats a mechanism that was proven innocent, and it would spend queue slots — the scarce resource — to fix a problem that does not exist. The queue latency that does exist is addressed by capacity (Issue #2199), not by de-batching.
• Alert on queue latency instead. Rejected: a different question, already answered. ado.runs.queued.* carries congestion at Information severity by deliberate design, precisely so it cannot page; folding “unbuilt change” into it would put a correctness signal behind a noise filter.
• Compare HEAD against the newest run only, without path filters. Rejected: reports roughly 90 phantom gaps per push, since most pipelines are correctly path-scoped. A check that fires on everything discriminates nothing and is routed around within a week.
• Evaluate commits against the current trigger paths. Rejected on measurement: it manufactured a three-commit false gap on ansible-service-catalog from a path-coverage bug that had already been found and fixed.
• Two separate crons, one per concern. Rejected: the gap reconciler and the heartbeat poller are the same shape — poll the builds API per pipeline, act on the result — over the same registry. Two near-identical pollers would drift in their pipeline lists, and each would need its own dead-man.
• Keep the pipelines stamping their own heartbeats and replicate the file to both runners. Rejected: it splits each heartbeat across two hosts and makes both stale half the time, which is worse than the single-agent version it replaces.
• A Zabbix trapper fed by zabbix_sender, or a Zabbix HTTP-agent item polling ADO server-side. Rejected: the gap test needs local git ancestry and per-commit file lists, which no server-side item can compute. The <check>.age dead-man shape is the estate standard and is retained.

Consequences

• The estate gains its first control over pushed == will be built, and a standing artifact (--audit) that can answer the question historically rather than anecdotally.
• Two pipelines lose their final heartbeat step, removing an agent-local dependency and unblocking the multi-runner cutover. The <check>.age dead-man shape is unchanged, so no monitoring consumer moves.
• One new dead-man, ado.ci.reconciler.age, is attributed by the existing zabbix_cron_heartbeats_monitoring.py regex rule in monitoring_config.yml — no registry change was needed, verified rather than assumed.
• Three new notification events (trigger_gap, orphan_definition, run_failed), all severity warning, capped at Pushover priority 0 by ADR-0326.
• Accepted limitation: the reconciler answers “did this pipeline build this commit”, not “was this change deployed by something”. Where two pipelines overlap on a path, a gap in one is reported even if a sibling deployed the same artifact. That is the correct per-pipeline semantics — an overlapping deploy is a coincidence, not a guarantee — but it means a genuine finding may sometimes have a benign explanation.
• Accepted limitation: an ADO API outage reads as UNVERIFIED (exit 2) and pages run_failed, adding one indirection between “CI is broken” and “we cannot see CI”. This is deliberate: the alternative is a green report on unchecked state.

As-built reconciliation (2026-08-07)

Three corrections the first deployed runs forced, recorded here because each was a real defect in this decision as originally written — the reconciler found them by running, not by review:

• An orphaned definition is a FINDING, not an inability to check. It was first classified as unresolved — the bucket meaning “could not check this pipeline” — which forced rc=2. Because rc=2 withholds the self-heartbeat, the reconciler would have never stamped ado-ci-reconciler.last (tripping its own dead-man within 90 minutes) and paged run_failed every 20 minutes forever, for a condition only a human deleting the definitions can clear. The rc contract is now explicit — 0 checked-and-clean, 1 checked-and-found-something, 2 could-not-check — and the self-heartbeat is stamped on 0 and 1, withheld only on ≥2. A heartbeat that answers “is the reconciler running?” must not go stale because the reconciler found something.
• Pages fire on CHANGE, not on every run. Even with the rc fix, a standing finding paged every 20 minutes — roughly 72 notifications a day about something nobody can fix faster. Pages are now keyed on the finding’s content: first sighting pages, repeats are suppressed, a changed finding pages immediately, and an unchanged one re-pages after 12 h. Content-keying matters so a new orphan is not swallowed because an unrelated one is already being tracked.
• The skip-token suppressor splits by position — see the section above. The original treated any token as a deliberate skip, which is what let this ADR’s own implementing commit go unbuilt.

The two orphaned definitions this work discovered — ansible-podcast-zabbix (44) and ansible-five-great-reads-podcast (47) — were deleted on 2026-08-07 with Arron’s approval, completing the ADR-0323 decom. The reconciler now reports 0 gaps, 0 orphans, 0 unresolved.

The heartbeat cutover it depended on is now complete (2026-08-07)

This decision described the API-derived heartbeats as the replacement for the two pipelines’ own agent-local stamps, but at the time it was written the pipelines still stamped. Issue #2199 removed both stamp steps (ADR-0330), so the reconciler is now the sole source of eit-recon.last and standards-enforce.last rather than a parallel writer. Verified live at cutover: standards-enforce.last = 1786061476, exactly run 7514’s finishTime, and eit-recon.last = 1786061009, exactly run 7509’s — both dead-men green across the change.

The same session found a third instance of the single-agent assumption this ADR names, in a guise nobody had enumerated: three pipeline steps write Prometheus textfile metrics into control01’s Alloy collector directory, which is the only exporter of them. On a runner those files land on the runner and control01 exports nothing — and unlike a stale heartbeat this fails with no red run and no alert at all, taking ADR-0180’s synthetic probe_success with it. The generalisation is wider than heartbeats: any pipeline artifact consumed off-agent carries the assumption, not just liveness stamps.

--queue is now ON (2026-08-07, Arron’s explicit decision at the Issue #2199 wrap-up). This ADR set the precondition — “revisit once the two-runner pool has capacity to absorb it” — and it is met: control01 has left the pool, both runners take work, and concurrency is proven from the API.

The reasoning that kept it off was never “auto-queue is unsafe”; it was that auto-queueing into a saturated pool manufactures the congestion that produces the apparent gaps. Capacity removes that objection, and the detector’s conservatism — which is what actually makes the automation safe — is unchanged and already proven: a pending run is never a gap at any age, coverage is ancestry not sha equality, only commits touching a pipeline’s own trigger paths count, a subject-line skip token suppresses, and a gap must outlive the 90-minute threshold. The 14-day audit found 41 CI-triggered pipelines and zero unbuilt trigger-matching commits, so this arms automation against a condition that currently never fires — which is precisely when to arm it rather than after an incident.

Rollback is one flag in scripts/ado_ci_reconciler_cron.sh; --queue remains off by default in the Python, so removing it from the wrapper restores detect-and-page.

Standards gap this exposes

The decommissioning runbook does not remove the ADO pipeline definition. ADR-0323 swept the service across Prometheus, Alloy, Grafana, Zabbix, Pi-hole, Caddy, Cloudflare, Infisical, the exposure register and the repo — and left two enabled ADO definitions pointing at deleted YAML. A definition that survives its code is invisible drift: still registered, still triggerable, guaranteed to fail if anything queues it, and permanently absent from every “is this pipeline live” coverage check. This is the same shape as the dangling-wikilink gap ADR-0325 added to the runbook, one layer out. Proposed amendment: the decom runbook gains an explicit “delete or disable the ADO pipeline definition” step, and the reconciler’s orphan_definition event is its conformance check — the gate that watches the guard.

Epic #2187 / Issues #2198, #2199.