ADR-0435: A wait concludes or refuses — it never hangs

Status: Accepted Date: 2026-08-18 Epic: #1802 — Waits that cannot silently fail Standard: Wait & Async-Verification Standard

Context

Every wait in this estate was hand-written at the moment it was needed, and a wait that cannot conclude is indistinguishable from one that is merely being patient. Both are silent. So a broken watch is discovered only by the thing it was supposed to catch, and the cost is measured in how long nobody looked.

Three documented instances, all one shape:

Date	Instance	Why it could never conclude	Cost
2026-08-04	Handoff script “still running”	pgrep -f <script> matched the monitor’s own command line, so liveness was true forever	6 hours
2026-08-04	Monitor never fired on a green pipeline	Marker path reconstructed from a 7-char SHA; the real file used 8	Result found by hand, much later
2026-08-18	Recovered-probe wait	Shell quoting sent service%3D%5C%22…%5C%22; Prometheus returned HTTP 400 forever	15 minutes

ADR-0200’s original case — a CI gate reading an asynchronously-ingested result once, immediately, and misreading queue lag as systemic loss — is the same class seen from the pipeline side.

The decisive observation is that doctrine already existed and did not work. The Token-Optimization Standard already published “immediately after arming, confirm the file or condition the watch names actually exists” before the 2026-08-18 failure, in the same corpus, addressed to the same reader. Writing it more emphatically was not an available fix.

Decision

A wait concludes or refuses. It never hangs. Three coupled commitments, delivered as one mechanism (waitfor) plus the Wait & Async-Verification Standard that declares it.

1. Refuse at arm time rather than wait on an unanswerable question. The predicate is evaluated once, synchronously, before anything is detached. A permanent contract error — unknown system, verb off the read surface, HTTP 4xx, bash -n failure, missing command, non-boolean exit — is refused with a reason. A transient error (5xx, connection failure) still arms, so “wait until the service recovers” remains expressible.

2. A terminal marker on every path — SATISFIED, TIMEOUT, REFUSED, ERROR, CRASHED — written to the log the waiter actually greps. This makes silence unambiguous: it means “still running” and nothing else. The CRASHED case is written by the launcher when the poller dies without reaching a terminal state, so even an OOM leaves a result.

3. Typed predicates, with a gated shell escape hatch. A system read through pitlab-access or a file/marker check removes the hand-composed shell string from the common path — in the typed path the quoting failure is impossible, not merely detected. --shell survives for container, queue and git waits, gated on a bash -n parse plus one real execution at arm time.

pipewait.sh is a sibling and is untouched. It owns the ADO-pipeline domain under ADR-0205, had 149 clean runs on the day this was designed, and shares the terminal-marker contract already.

Alternatives considered

Alternative	Why rejected
Standard only, no helper	This is today’s state with more words. The doctrine already existed and did not prevent the 2026-08-18 failure — that is the finding, not a reason to restate it.
Helper only, no standard	Leaves ADR-0200’s CI-gate half ungoverned, and nothing would make a future wait use the helper.
Typed predicates only, no --shell	Sends every container/queue/git wait back to hand-rolled loops — the escape hatch is what keeps the typed path from being routed around entirely.
--shell only, no typed forms	Detects the quoting failure rather than eliminating it. The dry-run cannot catch a predicate that runs cleanly and asks the wrong question.
Absorb pipewait.sh into waitfor	Re-implements SHA discovery, cross-pipeline fan-out and approval detection for a tidier diagram, against a tool with two ADRs of domain correctness and zero observed malfunctions.
Wrap pipewait.sh	Indirection for no correctness gain, and six soul.md citations would need rewording.
Enforcement class review	Auto-demotes to advisory without a checkpoint, and a post-session review cannot catch a mid-session strand — the failure happens and resolves inside one session.
Self-test only, no scheduled synthetic	Blind to out-of-band breakage: an expiring credential, a changed pitlab-access read contract, a deploy that dropped the script. Those are exactly the failures that leave every gate green while waits stop concluding.

Consequences

What improves. MTTD is the metric this moves: the failure class removed had a time-to-detect of 15 minutes in the best documented case and 6 hours in the worst, because the signal was silence. The 2026-08-18 failure becomes an instant named refusal — verified live, the malformed query that stalled for 15 minutes now exits 2 immediately citing HTTP 400 — the request is malformed, not pending.

Two layers of assurance, because they are blind to different things. The --self-test in ansible-ci proves correctness at a commit; the daily synthetic proves the deployed helper still concludes on the host. Neither subsumes the other, which is what earns the second layer under the Standard-Enforcement Standard’s two-layer rule.

The residual is honest and stated. A --shell predicate that parses, runs, and asks the wrong question is still a valid wait that will time out rather than refuse — the dry-run bounds structural breakage, not semantic error. The timeout marker is what keeps even that case from becoming silence. Separately, a wait armed without the helper is still unguarded; detecting that would require parsing session transcripts, so it is accepted rather than gated.

Frequency was never measured. The case rests on three documented failures, not a census of how often waits are hand-rolled. If the true rate is low this is over-built — recorded here rather than discovered later, and the two-week early-life soak in the Production Readiness Review (#2530) watches specifically for false refusals, since a valid predicate wrongly rejected is what would drive usage back to hand-rolled loops.

Cross-repo landing order is a real cost. The registry that registers a standard lives in ansible/ while the standard lives in pitlab-docs/, and the notification catalog resolves its kb anchors the other way, so this change could not land in fewer than four ordered pushes without reddening a shared pipeline for every concurrent session. The forthcoming: mechanism from Issue #2219 is what makes that ordering safe.

Amended 2026-08-19 at the Production Readiness Review (Issue #2530): a refusal must also be VISIBLE, not merely correct. The review found that the review’s own acceptance criterion could not be executed. It called for a two-week soak watching for a false refusal — a valid predicate wrongly rejected, the failure mode that would push usage back to hand-rolled loops — but a refusal happens in preflight, before any log path exists, so it wrote nothing anywhere. “No false refusals in two weeks” would therefore have been indistinguishable from “nobody could look”: the blind-watcher shape (FC-03) this ADR exists to remove, sitting inside its own sign-off criterion.

The fix is deliberately the smallest one that makes the soak real: every refusal emits a waitfor-refusal journald record carrying the label, predicate form, caller and reason, which control01’s existing Alloy pipeline already ships to Loki. No new file, no rotation to manage, no new infrastructure — and the record is estate-queryable, which matters because “has anything been wrongly refused lately?” is not a per-host question. The daily synthetic gained a fifth case asserting the record lands, keyed on a label carrying its own pid so a previous run’s line cannot satisfy it; it was proven red against a launcher without the record before being trusted green.

The general lesson is now a clause of the Wait & Async-Verification Standard: a terminal state that precedes the artifact still owes a durable record somewhere. A control whose evidence is only ever a transient stream is not a control — and this one was written by the same work that exists to abolish exactly that pattern, which is the more useful half of the finding.