ADR-0321 — Run-to-completion workloads are gated on their produced artifact, not their exit code

Status: Accepted Date: 2026-08-05

Context

pitlab has three post-deploy gates, and a whole class of workload falls through all of them.

A run-to-completion workload starts, produces or refreshes an artifact a consumer reads, and exits — a cron oneshot, a systemd-timer render, a batch indexer. For these, every existing signal answers the wrong question:

• smoke_gate.py (ADR-0067) asserts a container is running. A oneshot that correctly exited is indistinguishable from one that crashed, so the gate cannot be pointed at it at all.
• testplan_gate.py (ADR-0102) proves the service that serves the artifact works. vectormap’s nginx serves a meaningless map with exactly the same success as a good one — the plan passes either way.
• A freshness metric (vectormap_last_build_timestamp, PitMemoryIndexStale) detects a job that stopped running. It is structurally blind to the failure that matters here: a job that runs, exits 0, and writes a degraded artifact refreshes the timestamp, so freshness monitoring reads the bad outcome as healthy.

The result is a workload that can ship a wrong artifact behind a completely green estate. This was observed live, not theorised: in 2026-06 a vectormap render started while the pit-memory Qdrant collection was mid-re-index, scrolled only 1040 of ~5300 points, projected a fraction of the corpus, and wrote a fresh success timestamp over the last good map (ADR-0315). Nothing alerted. ADR-0315 fixed that one job with a bespoke in-render completeness check and named the general gap explicitly in its Consequences: “a weekly-cron render job is covered by neither the smoke gate nor the service test plan… raised separately rather than folded into this change.” This ADR is that separate item.

ADR-0315 also established, while proving its own fix, the three properties such a check needs — the obvious acceptance test (“the image builds and the job exits 0”) is worthless, because a projection can succeed mechanically and still encode no structure. The artifact has to be judged, the threshold has to be calibrated against the artifact already in production, and the check has to be proven able to fail.

Decision

Every run-to-completion workload that produces a consumed artifact owns an artifactplan.yaml, judged by the shared docker-stacks/scripts/artifact_gate.py on every run — per run, not per deploy, because the artifact is re-produced every run.

Four binding rules, generalising ADR-0315 from one job to the class:

1. Assert properties of the produced artifact — point count, byte size, row count, structural quality. Never the exit code, never a timestamp; both are satisfied by a job that produced garbage.
2. Calibrate against the last known-good artifact, never an a-priori number. Tolerances are fractions of a recorded baseline (at_least_fraction_of_baseline), so they remain correct as the corpus grows and need no re-tuning. A threshold the artifact already in production cannot clear is a broken threshold, not a failing run — the first vectormap purity threshold failed the live production map.
3. Prove the check can reject a degraded artifact before trusting it green (--control), per ADR-0141.
4. A rejected artifact must not be published, and must not read as a healthy run. The workload stages and promotes only on a pass, or retains and restores the previous artifact; the heartbeat is not stamped on a rejection, and the non-zero exit pages through the existing cron wrapper.

Failing closed is explicit: an unmeasurable artifact is a rejection, not a pass; a measurement whose command returns empty output raises “the measurement is broken, not the artifact” rather than scoring 0; and a measurement with no baseline or no declared tolerance fails rather than passing silently — the discipline of ADR-0317.

vectormap-render is the reference implementation, wired into vectormap_render_cron.sh after a successful render. The class inventory and each member’s status live in the CI/CD & Pipelines Standard: freescout-rag, the freescout incident distiller (ADR-0068), claude-pod episode render, the KB draft agent, and the pipeline-18 doc_gen generators are the remaining members. Backup jobs are deliberately out of scope — their artifact is already proven by a stronger mechanism, the monthly restore drill, which asserts recoverability rather than shape.

Alternatives considered

Extend testplan.yaml with artifact steps instead of a new plan type. Tempting for uniformity, and the two share a philosophy. Rejected because the trigger differs fundamentally: a service test plan runs post-deploy and gates a deploy, with auto-rollback of a commit as its failure action. A batch job’s artifact must be judged post-run, on every run, and its failure action is “do not publish this artifact” — there is usually no commit to revert, and the job may run for weeks without a deploy. Overloading one plan type would make both triggers ambiguous.

Make the freshness metric stricter / add an artifact-size alert. Rejected as the primary mechanism: it is detective, not preventive — it fires after the degraded artifact is already live and being consumed — and a static size threshold is exactly the a-priori calibration rule 2 exists to forbid. Freshness alerting is retained for the complementary question (did the job run at all).

Leave each workload to hand-roll its own check, as vectormap already did. Rejected: that is the status quo, and it produced exactly one guarded job out of six. The bespoke check also could not be tested independently of the job, so its correctness rested on a single careful author rather than a shared, self-tested runner.

Have the gate itself perform the promotion (stage → validate → publish). Attractive and genuinely stronger — it is the same shadow-then-swap shape used for the pit-memory index (ADR-0319) — but publication is workload-specific (an nginx volume, a Qdrant collection, an RSS feed, a docs tree). Making the gate own it would require a plugin per workload. The gate returns the verdict; the standard binds the workload to act on it. Revisit if a second workload wants the same publication shape.

Consequences

• A batch job can no longer ship a degraded artifact behind a green estate. The specific 2026-06 vectormap failure (1040 of ~5300 points) is covered by the gate’s regression test as a named case.
• vectormap-render now exits 4 and pages when its artifact is rejected, and deliberately does not stamp its heartbeat — so a repeatedly-rejected render is also caught by the existing freshness alerting.
• Each workload needs a one-time calibration (--seed-baseline) against a known-good artifact before the gate asserts anything. Until then its measurements fail loudly rather than passing silently, which is the intended direction of the error.
• The baseline is only advanced on an accepted run (--accept), so one bad run cannot lower the bar for the next.
• artifact_gate.py --self-test is hermetic (no live services) and proves the negative cases: a degraded artifact is rejected, a rejected run does not move the baseline, a missing artifact raises rather than measuring 0, a tolerance-free measurement fails, and --control itself fails a plan whose tolerance would accept anything.
• Five workloads remain unguarded pending their own plans; the standard now names them, so the gap is tracked rather than rediscovered.

Related

• ADR-0315 — the vectormap artifact acceptance this generalises, and the Consequences entry that raised this gap
• ADR-0102 — the service test plan this sits beside (services) rather than replaces
• ADR-0067 — the container-started gate that cannot cover a oneshot
• ADR-0317 — assert the real artifact, fail closed
• ADR-0141 — the proven-red requirement --control satisfies