A wait that cannot silently fail

The short version

Bottom line. A watch was armed on a file path built from memory, and a liveness check matched its own process. Both returned silence, and silence was read as still running — in one case for six hours.

What it cost — a job that had gone green long before, unnoticed; and a dead script reported as healthy for most of a working day.
What changed — a wait must now conclude or refuse, never hang: paths are captured from the tool rather than constructed, the artifact is proven to exist before the watch is armed, and every exit writes a terminal marker so silence can no longer mean a broken watch.
What it costs — waiting is more expensive to write. The full treatment applies only where a wait crosses a process or turn boundary, because those are the ones that fail invisibly.

The problem

Long-running work — a pipeline, a soak, a background job — is watched rather than blocked on. A watch produces one signal: an event when something happens, and silence when nothing has. The whole mechanism rests on silence meaning not yet.

It does not. Silence equally means the watch was pointed at the wrong thing, or the job it was watching died. Both are indistinguishable from patience, and both happened.

A wait was armed on a marker file whose path had been constructed from memory. The real file had a different name because of how the tool had been invoked. The watch never fired; the job had gone green much earlier and nobody knew.
A liveness check used a process-name match to decide whether a script was still running. The pattern matched the checking command’s own process, so it returned “alive” forever. A dead script was reported as still running for six hours.

Neither is a bug in the wait. Both are a watch that could not observe the thing it was watching — and a watch like that has no failure mode, because its output is identical whether it works or not.

The design

The remedy is stated as a property a wait must have, not as a longer checklist. A wait concludes or refuses; it never hangs (ADR-0435, generalised into the wait and async-verification standard).

Four commitments fall out of it:

Never construct the artifact you are about to watch. Capture the path the tool prints, or list it back from the filesystem. A path from memory and a path from the tool look identical right up to the moment one of them is empty.
Prove the artifact exists before arming the watch. If it is absent seconds after launch, the wait is not armed — that is a bug to fix now, not a thing to sit and wait on.
Test liveness by the artifact, never by a process name. The marker, the log’s modification time, a process id — anything that cannot match the observer itself.
Refuse an unevaluable predicate at arm time, and write a terminal marker on every path. A wait that cannot evaluate its own condition must say so immediately rather than waiting forever, and every exit — success, failure, timeout, refusal — must leave a marker, so silence can never mean a broken watch.

The related discipline is on the wait’s duration. Queue time and run time are separate budgets, because a full pool makes a queued run look like a slow one, and conflating them makes a capacity problem masquerade as a performance problem (ADR-0339).

The trade-off

This makes waiting more expensive to write. A one-line background poll becomes a tool invocation with a discovery phase, an arming assertion, two separate timeouts and a terminal marker. For a job that takes twenty seconds, that is absurd overhead.

The line drawn is that a wait crossing a turn boundary or a process boundary gets the full treatment, and an in-line wait does not. The reason is not duration but observability: a wait inside one process fails visibly, and a wait that outlives its caller fails silently.

The second cost is real and accepted: the discovery window has to be sized from measured run-time data, and the first attempt was too short. It covered under sixty per cent of observed run wall-clocks, so a slow run was misreported as a dead trigger. Re-sizing it from the actual distribution — rather than from a round number that felt generous — was part of the fix.

The evidence

The wait now writes a terminal marker on every path, and the marker’s absence is itself a detectable state rather than an ambiguous silence.
A run paused at a manual-approval gate used to read as in progress forever. It is now a distinct terminal signal, because “waiting for a human” and “hung” are different problems with different responses.
The discovery window was re-derived from measured run wall-clocks after the original value was shown to cover 58.9% of them.

Full reasoning: the design, ADR-0435, ADR-0205 and the wait and async-verification standard.