0339 — pipewait distinguishes a skipped commit from a lost trigger, sizes discovery to measured queue latency, and names the agent

pipewait reported one opaque outcome — DISCOVERY_TIMEOUT, documented as “trigger never registered — a real finding” — for three situations that mean entirely different things. This ADR splits them, re-derives the discovery window from measured queue latency instead of a guess, and surfaces which agent ran the job. All three gaps were evidenced by incidents in the days before, not imagined.

Status

Accepted — 2026-08-07. Delivered under Epic #2187 (CI runner pool), Issue #2210. Amends ADR-0205 (pipewait); reuses the skip-token semantics from ADR-0329; the batching behaviour it accounts for is ADR-0054. Regression gate scripts/test_ado_pipeline_run.py runs blocking in ansible-control01-sync.

Context — one marker for three different worlds

DISCOVERY_TIMEOUT meant “no run appeared for this SHA within the window”. Three unrelated causes produce that, and only the first is the finding the marker claims:

1. A genuinely lost trigger. Real, rare, worth investigating.
2. A commit that told ADO not to build. Self-inflicted and instantly diagnosable — yet it burned the full 600-second window and reported the same opaque result. It cost ~10 minutes plus an investigation on 2026-08-06, when a commit describing skip-token handling carried the literal token in its body. Writing about the mechanism triggered the mechanism.
3. A run that was merely deferred. Every pipeline here sets batch: true, which means ADO does not queue a successor at all until the in-flight run finishes — the build record does not exist to be discovered. A push made behind a long run therefore exceeds the window and is reported as a trigger that never fired.

Cause 3 was quantified rather than assumed. Across 1000 completed runs (2026-07-26 → 08-07), the 600-second default covered only 58.9% of observed run wall-clocks — so a batch-deferred successor could exceed the discovery window roughly two pushes in five. pitlab-docs run 7363 sat notStarted for 60 minutes.

Separately, the pool became multi-agent at the runner-pool cutover and pipewait reported nothing about which agent ran the job, so every queue-contention question meant opening the ADO UI.

Decision

1. A subject-line skip token fails FAST as CI_SKIPPED; a body-only token is an ERROR

Before entering the discovery loop, the commit message for the SHA is read and matched. The matcher is imported from ado_ci_reconciler.py, not re-implemented — two copies of “what counts as a skip token” would drift, and the subject-versus-body distinction is the part that is easy to get subtly wrong.

The two placements mean opposite things and must not share a result:

• Subject → a deliberate no-run. New marker PIPEWAIT_RESULT=CI_SKIPPED, new exit code 7, emitted in seconds. The regression test asserts find_build is called zero times: a correct verdict reached after burning 900 seconds would still cost the ten minutes this exists to save.
• Body only → almost certainly accidental prose, so it is an ERROR, not an expected outcome — the push believed it was building and was not. This is not a judgement call: measured across the estate over 120 days, all 298 deliberate skips are subject-line and zero are body-only.

pipewait.sh does not page on exit 7. Paging for a deliberate skip trains the operator to ignore the channel.

2. The discovery window is re-derived from data, and extends while a run is in flight

The static default moves 600 → 900 seconds, which covers 100% of post-cutover run wall-clocks (p95 10.8 min) against the old default’s 58.9% coverage of the pre-cutover distribution.

A static number alone cannot solve this, because the tail is set by whatever is already running. So while a run for the watched definition is still in flight, the window extends rather than expiring, in --discovery-timeout increments up to --discovery-max (3600 s). Batching guarantees the successor cannot have queued yet, so an in-flight run is positive evidence to keep waiting — not a reason to declare the trigger lost. In --sha-only mode the definition is unknown up front, so the same reasoning applies one level wider: any run in flight on the branch.

The busy check fails closed. On an API fault it reports not busy, so a genuinely lost trigger is never granted a false extension and made slower to report.

Raising the default alone was rejected: a window wide enough for the worst observed deferral (2700 s for 95% pre-cutover coverage) would make every genuinely lost trigger take 45 minutes to report — optimising the rare case by punishing the one that matters.

3. The terminal marker names the agent

agent=<workerName> is added to every PIPEWAIT_RESULT= and PIPEWAIT_RUN= line, read from the Timeline API’s per-Job workerName and de-duplicated. Queue-time and concurrency debugging on a multi-agent pool becomes a grep instead of a click, and it feeds the queue-time SLI work.

This is diagnostic colour, never a result: the lookup is wrapped so any failure degrades to agent=-. A telemetry addition that can turn a green run red is a worse trade than no telemetry.

Alternatives considered

• Re-implement the skip-token matcher locally. Rejected: CI_SKIP_TOKENS has seven spellings and the subject-versus-body rule is behavioural, evidenced by a 120-day census. Two copies drift, and the copy that drifts is the one not under test.
• Treat a body-only token as a skip, matching ADO’s actual behaviour. Rejected. ADO honours it, which is why it is dangerous: the commit is silently not built while the author believes it was. Reporting the platform’s behaviour without flagging the mistake is what let the 2026-08-06 case through.
• Raise the static discovery default and stop there. Rejected — see decision 2.
• Make discovery unbounded while anything is in flight. Rejected: a wedged or stuck-queued definition would hang the wait forever, which is the failure mode pipewait’s markers exist to prevent. The --discovery-max cap keeps every path terminal.
• Leave agent attribution to the Grafana queue-time dashboard. Rejected as the wrong altitude: the question “which agent ran this run” is asked while reading the marker, and a dashboard round-trip is exactly the friction that stops people asking.

Consequences

• DISCOVERY_TIMEOUT now means what it always claimed to mean. The two impostors have their own outcomes: CI_SKIPPED (exit 7) and a window that extends when extending is provably correct.
• A deliberate skip is reported in seconds instead of 600, and never pages.
• A batch-deferred run is no longer misreported as a lost trigger — the defect that invites the redundant manual az pipelines run this estate has already been burned by.
• Every marker line carries agent=, so the runner-pool contention questions are answerable from the log a wait already writes.
• scripts/test_ado_pipeline_run.py ships as a blocking CI gate, proven red per defect: against the pre-change module all three groups fail individually (the harness isolates each group, because a bare loop would abort at the first missing symbol and report one crash instead of three unguarded defects). This is the evidence the Build Hook & Pipeline-Gate Test Standard (ADR-0141) requires.
• Residual, explicit: the pre-flight is an optimisation, not a gate. With no local repo, or a commit not present locally, it stays silent and discovery proceeds exactly as before — so a skip token in a commit the waiter cannot read still produces the old opaque timeout. Making it a hard requirement would break every wait launched from outside a worktree, which is a worse trade than an occasional missed shortcut.
• The SRE framing is MTTD on the tooling itself: the outcome was always eventually knowable, but the marker sent the operator down the wrong path first. A signal that is right on average and misleading in the specific case is worse than a slower signal that is always right.