Design: Waits that cannot silently fail

Status — the MECHANISM is delivered and LIVE; the Production Readiness Review is parked (19 Aug 2026)

status: proposed is stale in the dangerous direction and is left as-is deliberately: the front-matter is stamped as-delivered by /wrapup Check O at Epic close, and this Epic has not closed. Do not read it as “nothing is built”.

Built, deployed and proven — DO NOT REBUILD: waitfor (launcher + poller) at /usr/local/bin on control01; the Wait & Async-Verification Standard; ADR-0435; the --self-test wired blocking into ansible-ci; the daily 16:10 AEST synthetic; and the Zabbix dead-man (item waitfor.synthetic.age, trigger waitfor synthetic not running >36h). Delivered under Issue #2529 across four commits, all pipelines green.

Parked: Issue #2530, the Production Readiness Review. It is time-blocked, not blocked on work — the review’s first criterion is the synthetic proven green through its REAL cron trigger, which first fires 16:10 AEST 19 Aug 2026. Everything before that date was a manual invocation, so the cron path itself is unexercised.

Known thin spot, stated at approval and still true: the frequency of hand-rolled waits was never measured — the case rests on three documented failures, not a census. If the soak shows waitfor is simply not being reached for, that is a finding about adoption, not about the helper.

Every wait in this estate is hand-written at the moment it is needed, and a wait that cannot conclude is indistinguishable from one that is merely being patient. This design makes waiting a tool with a contract — typed predicates that remove the hand-composed shell string from the common path, a terminal marker on every path so silence is never ambiguous, and a helper that refuses to arm on a predicate it cannot evaluate. It widens Epic #1802 from its original CI-only framing to cover any wait, agent or pipeline.

The failure this exists to prevent — three instances, one shape

A watch that fires spuriously is obvious. A watch that can never fire looks exactly like patience, so it is discovered only by the thing it was supposed to catch.

Date	Instance	Why it could never conclude
2026-08-04	Six-hour stall on a handoff script	pgrep -f <script> matched the monitor’s own command line, so liveness was always true
2026-08-04	Monitor never fired on a green pipeline	The marker path was reconstructed from a 7-char SHA; the real file used 8
2026-08-18	Fifteen-minute stall on a recovered probe	Shell quoting sent service%3D%5C%22…%5C%22; Prometheus returned HTTP 400 forever

ADR-0200’s original case — a CI gate reading an asynchronously-ingested result once, immediately, and misreading queue lag as systemic loss — is the same class seen from the pipeline side. All four are a wait whose question was wrong.

Prior art

ADO board sweep, WIQL over System.Title and System.Description, across six mechanism-class vocabularies — pipewait, waiting, wait, Monitor, watcher, polling.

Candidate	Verdict
#1802 Async Verification Gate standard	This Epic. To Do, ungrilled, no children, created 2026-07-18. Its description already framed the decision — “no standard governs ‘verify async work by polling to a deadline with early-exit, only fail after the window’. Decide whether to write the standard + ADR, or close as not-worth-standardising.” Widened here rather than duplicated.
#2488 Agent system-access helpers	Partial overlap, deliberately not merged. Doing, tagged grilled, approved 2026-08-18. Covers the helper-by-default ruling and lists Azure DevOps under partial coverage (pipewait/snag only). Its scope is systems with a credentialed control plane; waiting is orchestration, so waitfor is a consumer of pitlab_access, not a member of it.
#1988 pipewait: cross-repo resource-triggered runs	A narrow pipewait trigger feature. Does not cover waiting generally.
#1831 pipewait hardening	Done. Historical hardening of the ADO waiter only.
#2506, #2503, #463, #2339	Incidental matches on the words “wait”/“monitor”. No coverage.

Docs corpus sweep, two phrasings over designs/, adr/, standards/:

• ADR-0200 — the single-instance precedent (poll Dependency-Track to a 240 s deadline, early-exit, proven both ways). Not generalised.
• ADR-0340 / the guard-must-observe-what-it-guards lesson — the same class stated for guards rather than waits.
• Token-Optimization Standard — already publishes the discipline: “immediately after arming, confirm the file or condition the watch names actually exists.” This is the key negative finding: the doctrine existed and did not prevent the 2026-08-18 failure, which is the whole argument for a mechanism over more words.
• No async-verification, wait or equivalent standard file exists.

Decision tree

Decisions

Decision	Choice	Rationale	Rejected
Where it belongs	Widen #1802	Its stated question is this decision; ungrilled, no children, open since 18 Jul	A new Epic (duplicates approved work); folding into #2488 (stretches its credentialed-access scope)
Deliverable	Standard and helper	The doctrine already existed in the Token-Optimization Standard and did not prevent the failure	Standard only (is today’s state with more words); helper only (leaves ADR-0200’s CI half ungoverned)
Predicate	Typed, with --shell fallback	Removes the hand-composed shell string from the common path, so the quoting failure is impossible rather than detected	Typed-only (sends container/queue/git waits back to hand-rolling); shell-only (detects rather than eliminates)
pipewait	Sibling, untouched	149 runs on 2026-08-18, all with terminal markers, zero malfunctions. Two ADRs of domain correctness (0205, 0367)	Absorbing it (re-implements SHA discovery, fan-out, approval detection for a tidier diagram); wrapping it (indirection for no correctness gain, and six soul.md citations would need rewording)
Home	ansible/scripts/ → /usr/local/bin	PD6 category 1; launcher + poller mirrors pipewait’s own shape	Inside pitlab-access (blurs the access contract #2488 just defined); its own repo (disproportionate)
Enforcement	Class machine, helper is the mechanism	Structural at the point of use, not inspected afterwards	review (auto-demotes to advisory without a checkpoint, and a post-session check cannot catch a mid-session strand)
Assurance	Self-test and scheduled synthetic	The typed path can break out-of-band — an expiring credential, a changed pitlab_access read contract — which is what earns the second layer under the meta-standard	Self-test only (blind to out-of-band); the observability ratchet alone (governs observability, not function)
Delivery	/go, PRR held back	A sub-agent that just built the helper is the least independent reviewer available	/go closing its own PRR (self-certification, in a design about watches that cannot self-certify)

NFR resolution

Requirement	How this design satisfies it
Pillar 1 — Monitoring	Zabbix item + trigger on the scheduled synthetic’s freshness dead-man.
Pillar 2 — Observability	Poller logs to stdout → journald → Loki. No dashboard is owed: this is a CLI helper, not a service with an RED profile.
Pillar 3 — Alerting	The dead-man alerts when the synthetic stops concluding. Severity warning — a broken waiter degrades autonomy, it does not break production.
Pillar 4 — SBOM/vuln	Stdlib-only Python plus pitlab_access, which carries its own coverage under #2488. No new image, no new dependency surface.
Pillar 5 — Docs as code	The standard, its ADR, and a runbook section land in the same change as the code.
Config as code	Script in ansible/scripts/, declared in control01.yml, deployed by playbook. Schedule declared in schedule_config.yml.
Software update & patch lifecycle	House pattern (1) host operational script — it has no independent version. It updates when the repo does, deployed by the sync_control01 pipeline on push. No pinned upstream, so no bump trigger and no staleness signal are owed; its dependency pitlab_access is versioned and pinned under #2488, which owns that refresh path. Staleness of function rather than version is covered by the scheduled synthetic.
Security	No new credentials — the typed path borrows pitlab_access’s existing resolution. No network exposure. No secret ever in argv.
Notifications	Dead-man routes via the existing Alertmanager path. Nothing new emails.
Tagging	Loki stream labels per the taxonomy; Zabbix item and trigger tags per RULE 10.
Minimal decomposition	2 issues, 1 /go run, PRR held back for a human.
DORA / SRE	Deployment frequency and lead time come from the existing sync_control01 pipeline data. Change failure rate is the self-test’s red rate. MTTD is the metric this design actually moves: the failure class it removes is one where time-to-detect was 15 minutes in the best documented case and 6 hours in the worst, because the signal was silence.

Compliance & control mapping

Control	Framework	Evidence	Residual risk
The helper refuses an unevaluable predicate	NIST CSF 2.0 Protect · internal Standard-Enforcement Standard (class machine)	--self-test proven red then green in ansible-ci	A wait armed without the helper is still unguarded — accepted; detecting it would require parsing session transcripts
Scheduled synthetic + freshness dead-man	CPS 234 control testing · NIST CSF Detect	Dead-man proven to fire when the synthetic is skipped	Detects that the helper stopped concluding, not that it concluded wrongly
Terminal marker on every path	NIST CSF Detect · ADR-0205’s existing pattern	Marker asserted present on success, failure and timeout	—
Runbook + cold-read walkthrough	ISO 27001 A.5.37 (documented operating procedures)	PRR checklist item: walked cold by a session with no context	—

Control testing is the live obligation here; CPS 230 tolerance-for-disruption and material-service-provider clauses do not apply — this introduces no third party and supports no critical operation directly.

Operational readiness & handover

• Service record. Not a service: agent tooling, like pipewait. No NetBox asset, no criticality tier. It owes a runbook section alongside the existing operations/waiting-for-ado-pipeline-runs page, which is where a reader already goes for waiting.
• Reliability contract. The SLI is does an armed wait reach a terminal state. The dead-man is the alert; there is no error budget ceremony for a CLI helper and inventing one would be theatre.
• Test strategy. Per sub-issue: the self-test proven red before trusted green (ADR-0141’s house pattern). No testplan.yaml is owed — the Service Test Plan Standard governs services with a golden path, and this has no runtime surface; the scheduled synthetic is its executable acceptance contract. Any future change to the helper runs the self-test first as a regression baseline and updates it in the same change.
• Day-2 BAU & toil. None recurring. The synthetic is automated and declared in schedule_config.yml. The handover does not increase manual load — it reduces it, by removing the per-wait composition step.
• Backup & restore. Nothing stateful. Recovery is a redeploy from Git.
• Standing owner. ansible-ci runs the self-test on every push; the script-observability ratchet inventories it; the schedule reconciler tracks the synthetic; the dead-man pages if it stops. No new human owner is created.
• Exit. Retiring it means deleting the script, its manifest entry and its schedule row, and demoting the standard’s clause to advisory. No third party, so no provider-failure fallback applies.

Delivery plan — 2 issues, 1 /go run

1. #2529 — waitfor helper, the standard, and its assurance, landed atomically. Everything in one delivery. Atomic because the standards meta-gate resolves declared mechanisms against the live repos, and a cross-repo declaration cycle reds the shared pipelines for every concurrent session. Begins by enumerating pitlab_access’s read verb names, which are unverified.
2. #2530 — Production Readiness Review. Held back from /go deliberately. Evidence-checked by Pit. Two-week early-life soak watching for false refusals — a valid predicate wrongly rejected — since that is the failure mode that would drive usage back to hand-rolled loops.

Known thin spots, stated at approval

• Frequency was never measured. The case rests on three documented failures, not a census of how often waits are hand-rolled. If the true rate is low, this is over-built.
• pitlab_access read verb names are UNVERIFIED. Both probes on 2026-08-18 were refused for wrong verb names. Child 1 must enumerate them first.
• No schedule slot has been checked for the synthetic.
• Layer-3 compliance was worked lightly — control testing is mapped; no full ISO 27001 Annex A traversal was done for what is an internal CLI helper.

Deviations (as-delivered)

Filled by wrapup Check O at Epic close.