ADR-0205 — Durable ADO pipeline-wait pattern: discover by SHA, follow detached, classify the result

The agent repeatedly needs to wait for an ADO pipeline run it did not queue — the individualCI run that fires after git push origin HEAD:main — and know whether it passed or failed. The naive approach (poll in-session, grab “the latest run”) fails three ways in this environment: the slow self-hosted pool takes minutes to register a run so polling gives up early (“miss the trigger”); the harness culls in-session background Bash at turn boundaries so a long wait dies mid-flight (“lose the wait”); and “the latest run” belongs to whichever concurrent cc-pool session pushed last. This ADR fixes the pattern: discover the run by (pipeline-id, SHA), follow it in a setsid-detached poller that outlives turn boundaries, and classify every terminal state — surfaced as ado_pipeline_run.py wait + the pipewait.sh launcher.


Status	Implemented 2026-07-19 — `wait` op on `ado_pipeline_run.py`, `pipewait.sh` launcher, deployed to `/usr/local/bin` via `sync_control01.yml`; Epic #1824, Issue #1825
Date	2026-07-19
Builds on	ADR-0141 (build-hook / pipeline-gate test standard), the “Pipelines Are Slow — Wait With Patience” operating principle, the Entra-app Bearer flow (Epic #269)
ADO	Epic #1824 (Reliable ADO pipeline-wait pattern); Issue #1825 (build & test)

Context

Three independent failure modes made in-session pipeline waiting unreliable:

Trigger-registration lag. The individualCI trigger on the self-hosted pitlab pool can take minutes just to register a run after a push. A poll loop that assumes the run exists immediately concludes “trigger didn’t fire” and either gives up or fires a redundant manual az pipelines run — a documented past mistake.
Turn-boundary culling. The Claude Code harness culls its own run_in_background Bash tasks at turn boundaries. A poll loop long enough to outlast a real deploy is killed before the run finishes, and the result is silently missed.
Wrong-run ambiguity. Five concurrent cc-pool sessions push to main. “The latest run” is frequently another session’s. Matching must be by the pushed commit SHA (sourceVersion), not recency.

A prior belief — captured in agent memory — held that az pipelines runs list (no --status) hides in-progress runs. Live verification on 2026-07-19 (az-cli 2.88, devops ext 1.0.3) showed the default list does include notStarted and inProgress runs. Rather than depend on that CLI’s filtering semantics either way, the wait path queries the Build REST API (_apis/build/builds?definitions=<id>&branchName=…), which exposes sourceVersion, status, and result directly and is filterable server-side.

Decision

Add a wait operation to the existing ado_pipeline_run.py (which already uses the Entra-app Bearer flow, avoiding the often-stale devops-extension PAT), plus a thin pipewait.sh launcher that makes the wait durable.

Discover by (pipeline-id, SHA). Poll _apis/build/builds filtered to the pipeline + branch; match the build whose sourceVersion starts with the given SHA (full or ≥7-char short); pick max(id) because in-progress builds have a null finishTime and only the monotonic id reliably identifies the newest match. The query MUST pass queryOrder=queueTimeDescending — the Build API defaults to finishTimeDescending, which sorts a just-triggered run (null finishTime, because it is notStarted/inProgress) below every completed build, off the $top window, so the very run being discovered is invisible. This is not hypothetical: it was caught during live testing — the naive default-order query missed an in-progress run for 220 s while the run plainly existed, the exact “miss the trigger” failure this tool exists to eliminate. Ordering by queue time puts the newest run first regardless of finish state.
Two phases, distinctly. A discovery phase waits (default 600 s) for the run to register — heartbeating “no run yet” — and if it never appears exits DISCOVERY_TIMEOUT (exit 4), a real finding that the trigger never fired, not a hang. A follow phase then polls the discovered build to status=completed (default 2400 s follow deadline).
Classify every terminal state. succeeded→SUCCEEDED (exit 0); failed/canceled/partiallySucceeded→FAILED/CANCELED/PARTIAL (exit 2); follow deadline hit→RUN_TIMEOUT (exit 5); auth/HTTP error→ERROR (exit 3). Exactly one machine-parseable PIPEWAIT_RESULT=… marker line is emitted to stdout.
Detach for durability. pipewait.sh runs the poller under setsid, reparented to init — a real OS process divorced from the harness session that survives every turn boundary regardless of how long the pipeline runs. It writes progress + the terminal marker to a deterministic log (/tmp/pipewait-<pipeline>-<short-sha>.log). That log is the source of truth: on any later turn the agent greps it for PIPEWAIT_RESULT= to learn the outcome, whether or not any notifier survived.
Notify best-effort. The agent arms the Monitor tool with until grep -m1 'PIPEWAIT_RESULT=' <log>; do sleep 15; done for a prompt push when the marker lands; --notify-fail additionally sends a Pushover alert on non-success. Neither is authoritative — the marker file is.

Alternatives considered

Keep polling in-session with az pipelines runs list. Rejected: dies at turn boundaries (failure mode 2) and depends on CLI filtering semantics that have drifted between versions.
Match “the latest run”. Rejected: wrong under concurrent multi-session pushes (failure mode 3).
Trigger our own run with the existing run op and follow that. Rejected for the push case: it queues a second, redundant run and still doesn’t observe the auto-triggered one; run remains correct only when we genuinely want to initiate a run.
Monitor tool alone (no detached poller). Rejected as the durability mechanism: Monitor caps at 60 min and is a harness construct; the OS-level detached poller is what actually guarantees the wait completes. Monitor is layered on top purely as the notification convenience.

Consequences

After any git push origin HEAD:main, the reliable wait is pipewait.sh --pipeline-id <id> --sha $(git rev-parse HEAD) then arm Monitor on the printed log — no missed triggers, no lost long waits, failures caught by exit code and marker.
The pattern generalises the “Pipelines Are Slow — Wait With Patience” and “Long waits: Monitor, not background pollers” operating principles into an executable, tested tool rather than remembered discipline.
run/show behaviour is unchanged, so the infisical-rotate role and rotate_arr.yml that consume ado_pipeline_run.py from the repo checkout are unaffected.
The stale “runs list hides in-progress” memory belief is superseded; the Build-API discovery path sidesteps the question entirely.
Lesson — the trigger-miss bug survived until a genuinely in-progress run tested it. Classification tests against historical completed runs passed while the discovery ordering was broken, because completed runs carry a finishTime and sorted into the window. A tool of this kind cannot be considered proven against completed runs alone — it must be exercised against a live, still-running pipeline. The rollout push provided that test for free.
The --notify-fail Pushover is catalogue-conforming — component=pipewait, event=run_failed, registered in notification_catalog.yml with a kb anchor into the runbook (ADR-0034/0062). The notification-catalog CI guard blocked the first bare-call version; that gate working as intended is the reason the tool ships conforming. Two CI gates (docs link-check, notification-catalog) caught two regressions during rollout — both were fixed before close, and both failures were themselves detected and reported by the new tool, an incidental live proof of its failure-detection path.
2026-07-20 hardening (Issue #1860) — four as-built extensions from ~49 real waits during the Encryption-in-Transit rollout. The pattern is unchanged; these extend it: (1) the biggest reliability win is guidance, not code — the Monitor must be armed persistent:true with no fixed timeout_ms (a fixed 15–30 min timeout fired against 40+ min runs and stranded the wait); the launcher hint and soul.md now mandate it. (2) --sha-only drops the --pipeline-id requirement and follows every run a SHA triggered across all pipelines (docker-stacks fans out to 6 path-scoped ones), killing the DISCOVERY_TIMEOUT that came from watching the wrong pipeline; it emits a PIPEWAIT_RUN= per run plus one aggregate PIPEWAIT_RESULT=. (3) queue time and run time are now separate budgets — --timeout (default raised to 3000 s) counts only from first inProgress, and a run stuck notStarted on a full pool burns the new --queue-timeout (3600 s) as QUEUE_TIMEOUT, so pool contention no longer masquerades as a slow run. (4) an approval-gated run (ManualValidation / environment-approval Checkpoint.Approval in state=inProgress) is detected from the build timeline and reported as NEEDS_APPROVAL (exit 6) instead of a false RUN_TIMEOUT. All four were proven against the live deployed binary (approval detection on a paused tf-cloudflare run; --sha-only fan-out 2/2-ok; the docs-fix wait itself rode out a 27-minute run across turn boundaries with a visible queue→run transition). Runbook updated with each; the state table gains NEEDS_APPROVAL and QUEUE_TIMEOUT.
2026-07-26 resilience hardening (Issue #2019) — a single transient API fault could kill the whole wait. Following run 5892, ADO served one non-JSON response body mid-follow; api()’s bare json.loads raised, the detached poller died rc=1, no PIPEWAIT_RESULT= marker ever landed, and the armed Monitor stranded silently — the exact durable-wait guarantee this ADR exists to provide, broken by one flaky response. Two-layer fix, each closing a distinct failure class: (1) api() now never raises — URLError/5xx/non-JSON retry with 2/4/8 s backoff, HTTP 401/203 (ADO’s HTML sign-in page, i.e. the ~1 h az bearer expiring under a long wait) triggers a mid-run token refresh, and exhaustion returns the soft (0, _apierror) shape poll loops already skip; non-GET calls are never retried once delivery is possible, because a duplicated POST queues a duplicate build (idempotency-aware retry policy). (2) pipewait.sh appends PIPEWAIT_RESULT=CRASHED whenever the poller exits without a marker, making the marker contract unconditional even against a future poller bug. Lesson: a detached waiter’s liveness contract must not depend on the waiter’s own health — the wrapper, not the poller, owns the promise that a terminal marker always lands. Proven by mocked-fault unit tests (all five classes), a stub-worker crash test, and a live wait with the deployed binary; state table gains CRASHED.
2026-08-04 batched-trigger support (Issue #2151) — exact-SHA matching is wrong for a coalescing pipeline. pitlab-docs (18) sets trigger: batch: true (ADR-0054): while a run is in flight ADO coalesces every further push into ONE follow-up run labelled with only the last SHA of the batch. An earlier push in that batch therefore has no run carrying its sourceVersion, so equality matching burns the full discovery window and reports DISCOVERY_TIMEOUT — which reads as “the trigger never fired” and invites exactly the redundant manual queue this ADR forbids. Three such timeouts occurred in one session before the cause was understood. --batched widens acceptance to three paths, cheapest first: exact (unchanged), ancestry (git merge-base --is-ancestor <my-sha> <run-sha> — the run’s commit is at-or-after mine, so the batch that produced it included my push; pure git, no clocks), and start-time (the run started after the wait began). The discovery log names which path matched, so a non-exact match is never silent. Start-time is sound only because these pipelines re-checkout the branch at run time — the docs build’s Pull latest from all repos step runs git fetch && git reset --hard origin/<branch> on every source repo, so a run that starts after your commit reached origin/main builds your commit regardless of its label; it also catches schedule- and pipeline-resource-triggered runs. Deliberately opt-in, never auto-detected (the rejected alternative): the docker-stacks-* and ansible-* deploy pipelines build from the triggering commit, so a later-starting run there does not contain the change and --batched would report a false green — the worst possible failure for a gate, and the reason the flag is scoped by the operator rather than inferred. --repo selects the worktree for the ancestry test (defaults to CWD, resolved in the launcher before detachment). Proven against live pipeline-18 runs: red — eb29278, a commit on main that no run carries, returned DISCOVERY_TIMEOUT; green — the same SHA under --batched matched run 6970 via ancestry in 0 s; regression — exact-SHA discovery unchanged; negative control — an unrelated SHA under --batched still returns DISCOVERY_TIMEOUT, so the flag is not blanket-accepting. A DISCOVERY_TIMEOUT in non-batched mode now also logs a hint pointing at --batched.
2026-08-04 ancestry-by-default (Epic #2166, ADR-0311) — the batched-trigger entry above is partially superseded. With batch: true extended estate-wide, exact-only matching would false-DISCOVERY_TIMEOUT on any pipeline, so exact + ancestry acceptance are now always on (no flag): ancestry is sound under any checkout semantics — a run labelled with a descendant of my commit checked out a tree containing it, including checkout: self at the batch head. The launcher always resolves --repo from the CWD for the merge-base test. --batched now adds only the start-time path (schedule-/resource-triggered runs). The entry above’s justification for keeping the flag opt-in contained a factual error, corrected in this pass: the docker-stacks-* and ansible-* pipelines do not build from the triggering commit — they git reset --hard origin/main at run time (verified 2026-08-04) — so start-time acceptance is currently sound everywhere; it stays opt-in purely as future-proofing against a pinned-commit pipeline. Proven against live ADO with the deployed binary: older SHA ancestry-matched covering run 7110 with no flag; bogus SHA still returns nothing (negative control holds).
2026-08-07 amended by ADR-0339 (Issue #2210). DISCOVERY_TIMEOUT was documented here as “trigger never registered — a real finding”, but it also fired for a commit carrying a CI-skip token and for a run merely deferred by batch: true. Those are now split: a subject-line skip token fails fast as CI_SKIPPED (exit 7) — measured at 1.5 s against a real skip commit, versus burning the full window — a body-only token is an ERROR, the discovery default is re-derived from data (600 s covered only 58.9% of observed run wall-clocks → 900 s, extending adaptively while a run is in flight), and every marker line now carries agent=<workerName>. The marker contract in this ADR gains agent= and the CI_SKIPPED state.
2026-08-20 — the ancestry matcher was present but DISARMED, and the fix is repo resolution, not a matcher (Epic #2581). The ancestry-by-default entry above records that exact + ancestry acceptance are always on and that “the launcher always resolves --repo from the CWD for the merge-base test”. That second clause was the defect: every cc-pool session’s working directory (~/sessions/cc-1..5) is not a git repository, so unless the agent had cd’d into a worktree, ancestry silently switched off and the wait burned its entire budget before reporting DISCOVERY_TIMEOUT — which this ADR defines as “the trigger never fired”. Measured over 324 real waits, 16–20 Aug 2026: 36 ran with ancestry disarmed (11%), and 6 of the 14 DISCOVERY_TIMEOUTs (43%) were disarmed waits. On 2026-08-20 commit aa37bc5 cost 2700 s that way while its change had already built green twice.

The defect was announced and unread, which is the part worth recording. Line 2 of both failing logs said ancestry (UNAVAILABLE: no git repo — coalesced batch runs will not match), and the closing hint repeated it. Announcing a degradation is not a control (FC-03): the tool diagnosed itself correctly, in writing, twice, and proceeded anyway because the headline result looked self-explanatory. The initial write-up then misdiagnosed it as equality-matching and proposed building the matcher that already shipped — every measurement in it correct, the one causal sentence never observed (FC-19), inherited thereafter by a lesson page and by the Epic itself (FC-09).

The decision: resolve the ancestry repo from THE COMMIT, and refuse rather than degrade. Preflight walks a ladder — explicit --repo (which must contain the commit, or it is refused rather than quietly overridden), the CWD’s toplevel, a probe of every checkout on the host with git cat-file -e <sha>, then a git fetch origin of the canonical checkouts and a re-probe. A SHA is unique to its repository, so this needs no repo-name mapping and no registry to keep current; the candidate set is derived (every directory carrying a .git under ~ and ~/sessions/*/) rather than listed (FC-19), and .git is tested for existence rather than for being a directory, because in a worktree it is a file (FC-12). Exhausting the ladder is permanent, so it refuses.

Knowing the commit’s repository makes the wrong-definition case free. --pipeline-id is now validated at arm time against the repository that definition actually builds: definition 28 (ansible-claude-config-docs) builds ansible and can never produce a run for a claude-config commit, which on the same day cost 900 s of DISCOVERY_TIMEOUT read as a trigger fault, and was the second recorded instance of a wrong-definition-id wait. It now refuses in ~2 s. A permanent contract error refuses; a transient one arms — an unreachable ADO definitions API is a blip, not a wrong id, and a guard that rejects a legitimate caller is its own failure class (FC-17). Both directions are covered by the self-test.

New terminal state REFUSED (exit 2), and the marker contract is otherwise unchanged. PIPEWAIT_RESULT=, PIPEWAIT_LOG= and PIPEWAIT_RUN= keep their exact tokens — Monitor arming across every skill greps them verbatim, so a rename would strand every wait in the estate; this pass adds a new value only. A refusal writes no marker log and prints no PIPEWAIT_LOG= line, mirroring waitfor, so there is no path a Monitor could be armed on by mistake. It does leave a durable pipewait-refusal journald record (label, pipeline, caller, reason) which control01’s Alloy already ships to Loki, so a false refusal — a valid wait wrongly rejected — cannot hide. DISCOVERY_TIMEOUT survives and now means what it says: it is reachable only with ancestry proven armed.

Arm-time refusal and the durable record are the Wait & Async-Verification Standard’s own clauses, which stated them for every wait and named pipewait.sh in its conformance checklist while binding only waitfor in every Enforcement row (FC-13). That standard is amended in the same change to bind pipewait.sh too — see ADR-0455.

Proven red before green (ADR-0141): a hermetic 10-case --self-test (throwaway git repos under a temp HOME, the ADO definitions lookup stubbed, no network) runs as a blocking ansible-ci step beside waitfor’s; gutting the checkout probe fails 3 cases. The first red proof also caught a defect in the test — an assertion matching resolved by probe was satisfied by resolved by probe-after-fetch, so it passed with the feature removed (FC-02) — and it now anchors on the closing parenthesis. Live acceptance against the two real failures, run from a non-repo CWD: aa37bc5 on definition 18 discovers a covering run via ancestry in 0 s (was 2700 s and a false DISCOVERY_TIMEOUT), and 0ce7dd4 on definition 28 refuses in 2 s (was 900 s).

--sha-only becomes the documented default form (Pit’s ruling, 2026-08-20): it needs no definition id, so the wrong-id class cannot occur at all, and it derives the expected-pipeline set (ADR-0367). --pipeline-id remains fully supported as the exception — gating one specific deploy, or --silence, which is per-pipeline and unavailable under --sha-only — and is now arm-time validated, so choosing it is safe rather than a coin flip.