ADR-0200 — The Trivy import-verify gate polls DT for async BOM ingestion before declaring systemic loss

Changes how the ansible-trivy-scan-docker-images pipeline’s import-verification step decides that SBOM uploads to Dependency-Track failed. The original gate read the DT project list once, immediately after uploading ~59 image SBOMs, and failed the play if too many had not yet imported. Because DT ingests BOMs asynchronously off a queue, that single early read saw the queue still draining and misread ingestion lag as systemic loss — a false-red that failed the pipeline 16–18 Jul 2026 while DT was healthy throughout. The gate now polls the project list until every import lands or a bounded deadline passes, so only imports still absent after the window count as loss (Epic #1799, Issue #1801).

Status

Implemented — 2026-07-18 (Epic #1799, Issue #1801). The change is in ansible/playbooks/scan_trivy_docker.yml (the “Verify docker image SBOM imports actually landed in Dependency-Track” task, plus two new vars dt_import_verify_deadline_s: 240 / dt_import_verify_poll_s: 20).

Generalised by ADR-0435 (2026-08-18). The single-instance pattern decided here — poll an async result to a bounded deadline rather than reading once and calling lag a failure — was widened estate-wide into the Wait & Async-Verification Standard, which covers any wait (agent or CI) and adds the two commitments this ADR did not need: refuse an unevaluable predicate at arm time, and write a terminal marker on every path so silence never means a broken watch. This decision stands as-is; nothing here is reversed.

Context

The import-verify gate (Issue #1733, Prime Directive 12) exists because a returned DT upload token only proves DT accepted the BOM, not that it imported it — stable stacks were re-scanned every run yet sat weeks stale in DT because their imports were silently lost (the #1732 docker01-host failure at fleet scale). The gate asserts each scanned image’s project lastBomImport advanced past the run start, and fails the play LOUD on systemic loss (> trivy_scan_fail_ratio, 34%).

The original implementation deliberately did one batched DT read (“one DT read, not a poll per image”) straight after the upload loop, with only a 30 s grace floor. But DT’s BOM import is a queued, asynchronous operation: uploading ~59 SBOMs in a burst and then reading lastBomImport seconds later sees a partially-drained queue. On 18 Jul run 4491 the gate reported landed=31/59 and failed as SYSTEMIC: 28/59 imports did not land … DT likely OOM'd/restarted mid-scan — yet dtrack was up, healthy, at 0.7/12 GiB with OOMKilled=false. The diagnosis was wrong: nothing was lost, the queue simply had not drained by read time. The gate had conflated two states that look identical in a single snapshot — ingestion lag (transient, self-resolving) and import loss (persistent, real) — and defaulted to the alarming one, turning a healthy scan into a persistent red pipeline and a Zabbix “ADO pipeline failed” alert.

Decision

Poll the DT project list until imports land or a bounded deadline elapses; only imports still absent after the full window count toward the systemic-loss ratio.

• The verify body re-reads the batched project list every dt_import_verify_poll_s (20 s) and recomputes the not-landed set, breaking early the instant every target has landed, and stopping at dt_import_verify_deadline_s (240 s — comfortably longer than a full-fleet burst takes to drain).
• Each poll is still one batched read of all projects (not a per-image request), preserving the original efficiency intent — polling adds a bounded number of whole-list reads, not N×attempts image lookups.
• The systemic-loss assertion (> 34% → raise SystemExit(2)) is unchanged in threshold and spirit; it now runs on the post-poll not-landed set, so a genuinely wedged/OOM’d DT still fails the play LOUD after the window, while transient lag resolves silently.

Proven both ways before closing (PD12), against the real verify body with a mocked DT queue: an ingestion-lag scenario (all imports land by the 3rd poll) → exit 0 (PASS — the exact 16–18 Jul false-red); a genuine-loss scenario (4/5 never land) → the poll window expires with 4/5 absent → SYSTEMIC → exit 2. The gate is red when it should be red and green when it should be green.

Alternatives considered

• Keep the single read, widen the 30 s grace floor — a fixed grace is a guess at queue-drain time; too short still false-reds under load, too long delays a real failure. A poll adapts to actual drain time and exits as soon as imports land.
• Add a fixed sleep before the single read — pays the worst-case wait every run even when the queue drained in seconds, and still false-reds if a burst runs long. Polling with early-exit is strictly better on both latency and correctness.
• Drop the import-verify gate, rely on the daily DTSBOMProjectsStale freshness alert — loses the in-pipeline, same-run proof that Prime Directive 12 requires; a silent import loss would only surface a day later.
• Poll per image instead of the whole list — N× the DT reads for no benefit; the batched list already contains every project’s lastBomImport.