0382 — A gate aggregating over a discovered set must distinguish empty from satisfied

A check that discovers what to inspect, then aggregates the verdicts, has a third outcome besides pass and fail: it found nothing. Reported as a pass, that outcome is indistinguishable from success and is strictly worse than no check at all, because it is a green light with nothing behind it. ADR-0367 fixed this for one tool; this ADR makes it a rule that binds every caller and every gate of that shape.

Status

Accepted — 2026-08-10. Delivered under Issue #2283 (Epic #2259, the standards gap register). Generalises ADR-0367 from pipewait.sh to the whole class; ADR-0367 remains the record of that tool’s implementation.

Context

pipewait --sha-only aggregated over whatever runs it discovered for a SHA and finished when they did. It had no notion of which pipelines should have run, so a pipeline producing no run was not “missing” — it was simply absent from the aggregate, and the marker read SUCCEEDED.

With batch: true estate-wide (ADR-0054) that is not a race but the normal case: the fast always-triggering pipeline (docker-stacks-secret-scan, ~20 s) reliably completes before the batch-deferred deploy pipeline is even queued. On 2026-08-08 a dashboard change reported PIPEWAIT_RESULT=SUCCEEDED result=1/1-ok three seconds after launch; the pipeline that actually deploys the change queued eight minutes later. The wait was truthful about what it measured and useless as a deploy gate: it proved the runs that happened were green, never that the deploying pipeline ran.

ADR-0367 fixed the tool — it derives the expected pipeline set from the push’s changed paths and emits a NOT_TRIGGERED terminal state. What it did not do is write the rule down. It binds pipewait, and nothing else: a future wait built another way, or any other gate of the same shape, repeats the reasoning from scratch.

And the shape is common in this estate, because so many controls are discovery-driven — a gate that scans pipelines/*.yml for invocations, one that enumerates stacks from a compose tree, one that censuses exception registers, one that resolves mechanism names against indexed repo roots. Every one of them can return an empty set for a reason that has nothing to do with conformance: a renamed file, a moved directory, a regex that stopped matching, a checkout that was not synced. In each case the failure is silent and green, which is precisely the PD12 trap — an assertion satisfiable without the feature is not a test — with the added sting that the assertion is satisfiable without any subject.

Decision

Two clauses, one general and one specific, land in the CI/CD & Pipelines Standard.

The general rule. A gate that aggregates over a set it discovers must be able to tell empty from satisfied, and must not report success on empty. Concretely it owes three things:

1. An expectation independent of the discovery — derived from the change, an inventory, a registry or a declared minimum — so “nothing found” can be compared against “nothing expected”.
2. A distinct terminal outcome for the shortfall, named and non-green (NOT_TRIGGERED, EMPTY_SET, missing), never folded into the pass path. “I could not look” and “I looked and it was fine” must never share an exit code — the same principle the pool-parity standard already applies to a check that cannot run.
3. A self-test fixture in which the discovered set is empty, asserting the gate does not pass. This is the enforceable half: it sits inside the --selftest contract every gate already owes (ADR-0141), so it is checked in CI on every run rather than trusted to a reviewer’s memory.

The deploy-gate specific. A wait used as a deploy gate must establish which pipelines the change was expected to trigger and fail if any produced no run. A green covering only the runs that happened to exist is not evidence of deployment — the deploying pipeline is the one that must be seen, and the always-on guard finishing first proves only that the guard ran.

Enforcement is review, and that is stated rather than dressed up. pipewait.sh implements the rule and fails closed, but it cannot be named as a resolvable machine mechanism — the meta-gate tokenizes .py/.yml/.yaml only, and more fundamentally the population of callers is agents in conversation, not files in a tree, so there is nothing in-repo to enumerate. What is mechanically checked is the narrower, real thing: clause 3’s empty-set fixture, enforced by the --selftest-before-the-gate step every pipeline already runs.

Dogfooded on delivery. shift_left_coverage_gate.py (ADR-0381), written in the same change, discovers the pipeline’s Python invocations by regex and therefore has exactly this failure mode: a renamed pipeline file or a broken matcher would yield zero invocations and a clean bill of health. It carries EMPTY_SET as one of four failure classes and a RED fixture that proves it, so the first gate authored under this rule was built to it rather than retrofitted.

Alternatives considered

• Amend ADR-0367 to bind all callers. The lighter option, and rejected: ADR-0367 is a specific implementation record for one shell tool (path-filter parsing, marker states, queue-vs-run timeouts), and widening its scope in place would leave the general rule buried in a tool’s changelog where no future gate author would find it. A separate ADR that cites it keeps both readable.
• Class the rule machine and name pipewait.sh. Rejected on two counts: a .sh does not resolve in the meta-gate’s index, and — the real objection — naming the reference implementation as the mechanism would claim enforcement over callers nothing checks. Under-claiming and over-claiming are the same defect, and over-claiming here would inflate the adherence SLI for a control that binds one script (unknown is a finding).
• Write a gate that greps the estate for hand-rolled ADO polling loops as a proxy for “a wait not built on pipewait”. Attractive, and rejected: the legitimate cases (ado_ci_reconciler.py, ado_pipeline_metrics.py) poll the same API for entirely proper reasons, so the check would need an allowlist longer than its findings — and it would still miss the actual risk, which is an agent hand-rolling a wait in a conversation, not a script committed to a repo.
• Make an empty discovered set a WARN rather than a failure. Rejected: a warning in a green run is not read. The estate already has the correct softer instrument for a legitimately-empty case — a declared expectation of zero — which distinguishes “nothing expected” from “nothing found” instead of blurring both into a pass.
• Leave it to code review. The status quo, and the reason ADR-0367’s fix binds nothing: the defect is invisible in a diff, because the aggregating code is correct. It is the absent third outcome that is wrong, and absence does not appear in a diff.

Consequences

• Every new discovery-driven gate owes one more --selftest fixture. That is a handful of lines, and it is the only fixture that can catch a matcher that quietly stopped matching.
• Existing gates of this shape are not swept retroactively by this ADR. Each is audited when next touched; the two authored under it (shift_left_coverage_gate.py, pipewait.sh) are conformant now.
• The deploy-gate clause makes pipewait’s NOT_TRIGGERED a reportable outcome to act on, not a curiosity: a deploy is unproven until the deploying pipeline has been seen to run, which is a stricter bar than the estate held before ADR-0367.
• A residual, stated: nothing prevents an agent hand-rolling a wait that ignores this rule. The mitigation is that the rule is now in the standard the agent reads, and that pipewait is the sanctioned path — not a machine gate, and it is dishonest to imply otherwise.