ADR-0102: Per-service test plans as an enforced post-deploy / post-remediation gate

Every pitlab service gains a test plan as code — an executable, versioned acceptance test with a mandatory functional floor — run by one shared runner and wired into three gates (post-deploy, vulnscan post-remediation close, manual /verify), with auto-rollback on failure. This turns Prime Directive 12 (Prove It Works) from a per-service, remembered-or-forgotten habit into a structural, checkable standard. The enforceable detail lives in the Service Test Plan Standard; this ADR records why.

• Status: Implemented (2026-07-04) — foundation + all three gates built; fleet retrofit 23/31 done (see Consequences). Amended 2026-08-07 by ADR-0341: the deliberately rollout-safe skip is retired — a service with no plan now FAILS the gate unless it holds a tracked row in docker-stacks/scripts/testplan-exemptions.yml, the gate enforces its own pipeline coverage, and the third trigger (/verify) finally has an agent entry point.
• Date: 2026-07-03 (accepted) · 2026-07-04 (implemented)
• Epic: ADO #886 (motivator) → rollout Epic #1476 (Service Test Plan Standard — fleet rollout)

Context

The fleet is becoming heavily automated: ADR-0101 grants vulnscan authority to change and deploy production unattended, and ADR-0099 auto-restarts unhealthy containers. Both prove “the change applied” and “the port answers” — neither proves “the service still does its job.” ADR-0011 already named the gap: readiness ≠ functional health — a container can be Up and probe-green while functionally dead (the radarr dependency-dead and tandoor 5xx failure modes). The existing gates — container smoke (ADR-0067), Loki log-shipping (ADR-0086), blackbox readiness — are all shallow and per-service ad-hoc.

As automation deploys without a human watching, the shallow layer is no longer sufficient: an autonomous remediation that clears a CVE but silently breaks the service is worse than the CVE. The owner’s directive: every service needs a comprehensive test plan, run post-remediation and wired into every build, governed by a standard — and then every existing service brought into line.

Decision

Adopt the Service Test Plan Standard: one test-plan-as-code per service, a mandatory functional golden-path floor, one shared runner, three mandatory triggers, auto-rollback on failure. Key choices:

1. Hybrid authoring — a declarative YAML spec (http/exec/metric steps) for the common cases, with a committed script escape-hatch for complex stateful flows. One runner executes both. Chosen over code-per-service (too much boilerplate, no enforceable shape) and declarative-only (awkward for real acceptance flows).
2. Functional golden-path floor — every plan must exercise real behaviour end-to-end (ingest→OCR→searchable, canary workflow executes, round-trip write→read), not just a readiness/port check. Readiness/smoke may be included but do not satisfy the floor.
3. Three mandatory triggers — post-deploy pipeline gate, vulnscan post-remediation close-gate, and manual /verify. A scheduled synthetic run is recommended, not mandated (adopt per drift risk).
4. Auto-rollback + alert on failure — a failed gate reverts to the last-good deploy automatically (generalising vulnscan’s revert-on-regression to all deploys) then alerts. A broken change never stays live awaiting a human.

Alternatives considered

• Keep the shallow gates (smoke/Loki/readiness) only — rejected: they prove liveness/readiness, not function, which is exactly the risk automation introduces. ADR-0011 already flagged the gap; automation makes closing it non-optional.
• Code-per-service test suites (pytest everywhere) — rejected as the default: maximum flexibility but heavy boilerplate and no shared shape to audit/enforce. Retained as the escape-hatch (script step) where a flow genuinely needs it.
• Declarative-YAML-only — rejected: cleanest to enforce but can’t express multi-step stateful acceptance flows (file ingest, DB round-trip). The hybrid keeps uniformity while allowing depth.
• Hold-and-alert instead of auto-rollback — rejected as the default: leaving a proven-broken service live until a human acts contradicts the fail-safe posture; auto-rollback keeps prod working. Pre-promotion gating (canary) is preferred where a staging step exists, but most stacks don’t have one yet.
• Full behavioural suite as the floor — rejected: strongest guarantee but too heavy to author for every service; it would stall adoption. One golden-path test is the enforced minimum; deeper suites are encouraged.

Consequences

• PD12 becomes structural. “Done” and “safe to auto-close” now have a checkable, executable definition per service, not a per-change judgement call.

• vulnscan’s close-gate matures from “CVE cleared + probe green” to “CVE cleared + the service’s test plan passes” — the ADR-0101 autonomy loop gains real functional proof before it closes anything.

• Auto-rollback generalises across all deploys, not just vulnscan’s — the GitOps pinned-tag revert becomes the universal failure response.

• A fleet-wide retrofit is required. Every existing service must author its plan and wire the gate. This is tracked as a dedicated rollout Epic (created with this decision), with one Issue per service and the shared runner + a reference implementation built first.

• Shared-runner risk: one runner is a single dependency for every gate; it must itself be covered (its own test) and versioned. Building it is the first rollout task.

• Authoring cost is real — a golden-path test per service is non-trivial to write well. The floor is deliberately one test (not a full suite) to keep the retrofit achievable; depth grows over time.

• Learned in operation (2026-08-05): the ADR-0163 scope guard’s “shared paths affect all stacks” fallback is where an innocent commit still gets reverted. The guard works as designed — it narrows a revert to commits that touched the regressing stack — but deliberately treats scripts/, pipelines/, group_vars/, inventory/, vault.yml and ansible.cfg as capable of breaking any stack, so a commit touching those is not narrowed and is reverted for a failure in a stack it never touched. That fired on 2026-08-05: a commit adding scripts/renovate_config_gate.py and editing two pipelines/*.yml was reverted because pit-memory’s corpus-indexed step failed — a step it could not have affected. The conservative fallback is defensible in principle (a shared-path change genuinely can break anything) but its cost is real on a shared main with concurrent sessions, and it is paid every time an unrelated, transient failure coincides with a shared-path commit.

  Two lessons for anyone authoring a plan under this standard. (1) A metric step must be robust to the service’s own legitimate lifecycle. The failure here was not a regression at all: corpus-indexed read an instantaneous gauge that a delete-then-repopulate reindex drives to zero, so a healthy service failed its own gate (measured live mid-reindex: instant 0, max_over_time(...[1h]) 5338). Use a windowed aggregate for anything a reindex or restart can zero — never a bare instant read. This is the second time this same plan produced a false revert. (2) The shared-path fallback deserves a second condition, e.g. requiring the failure to persist across runs (the current 90 s debounce is far shorter than a ~7 min reindex) before a shared-path commit is reverted — tracked as Issue #2183. Until then, treat a red test-plan gate on a shared repo as capable of reverting a shared-path commit that had nothing to do with it.

As-built reconciliation (2026-07-04, Epic #1476)

• Runner + gate live in docker-stacks/scripts/, not Ansible. testplan_run.py (the runner) and testplan_gate.py (the deploy gate) sit beside their sibling gates smoke_gate.py/loki_log_gate.py and are invoked repo-relative from the control01 agent checkout — one source of truth, no /usr/local/bin copy to drift. This deviates from the standard’s earlier “deployed via Ansible” wording (since corrected) and is a carve-out from the all-scripts-in-Ansible rule for pipeline gates — now formally codified in ADR-0106 and the Pipeline-Gate Script Location Standard.
• Auto-rollback is mode-gated; set to live on 2026-07-04 (Arron-approved). The autonomous git-revert-and-push failure action is gated by /var/lib/pitlab/testplan-gate/mode (dryrun / live), mirroring vulnscan’s ADR-0101 autonomy gate, plus a debounce retry and a revert-loop guard. The mode file is Ansible-managed (sync_control01.yml) at live. Dryrun was proven end-to-end (clone + revert-applies-cleanly, no push); the final live git push in the pipeline agent context proves on the first real regression and is fail-safe — a failed rollback still fails the run and alerts, so enabling live only ever adds the auto-revert on top of detect-and-alert.
• Retrofit: 23 of 31 gate-wired docker-stacks services covered, all with auth-clean functional golden-paths (behavioural Prometheus metrics, unauth functional reads, real round-trips, read-only protocol probes). Tracked in service-test-plan-coverage.md.
• 8 services deferred — a standard gap surfaced, now closed. Services whose only functional API needs a key (paperless, tautulli, netbox, freescout, …) cannot get an auth-clean golden-path; extracting keys to the host is disallowed. This needed a new authenticated-golden-path secret-injection pattern (runner resolves an Infisical-backed secret ref into memory at run time), delivered under Issue #1483 as ADR-0104. Outcome: the deferral shrank 8→3 — paperless/tautulli/netbox/freescout are now authed-green and allm proved auth-clean; qbittorrent (WebUI infra fault), iptv-proxy (dormant/placeholder config), and paperless-ai (session-only, no token API) remain deferred on their own merits, not the secret gap. Coverage 23→28 of 31 (coverage doc). Host/LXC services remain a separate untouched tranche.

Debounce backoff on the re-run — hardening after a false rollback (2026-07-04, Issue #1516)

The debounce re-run (safety layer 1) originally fired immediately after a failed plan, with no delay. On 2026-07-04 this reverted a sound deploy (vectormap-search FastAPI bump, Issue #1494, pipeline 22 run 2631 → rollback commit 771e8a0): vectormap’s functional golden-path hits a live Ollama-embed + Qdrant query, a few-second dependency blip failed the initial run, and the back-to-back retry re-sampled the same transient window and failed too → the gate declared a confirmed regression and (in live mode) auto-reverted. The gate then passed 3/3 on a manual re-run — the deploy was never broken.

Fix (Issue #1516): a short backoff precedes the debounce re-run so the retry samples a fresh window rather than the same blip. A configurable --retry-delay (default 25 s, 0 = the old immediate behaviour) is slept once before re-running the failed plans; only failed plans pay the delay — a fully-passing gate never waits, so the happy path is not slowed. This tunes the debounce from “re-run instantly” to “re-run after the transient has had time to clear,” which is what a debounce is supposed to do.

• Why a fixed backoff, not exponential/multi-retry: the failure mode is a sub-minute dependency warm-up/blip; a single ~25 s wait clears it. More retries or exponential backoff would lengthen every genuine-regression pipeline (which must still fail fast to auto-rollback) for no additional transient coverage. One delayed retry keeps the two-sample debounce while de-correlating the samples.
• Genuine regressions still roll back. A plan that is actually broken fails both the initial run and the delayed retry, so the confirmed-regression → auto-rollback path is unchanged; the backoff only adds latency to the failed branch, never suppresses a real failure. Demonstrated with a stubbed failing-then-passing plan (no rollback, gate green, backoff waited) and a persistently-failing plan (rollback still attempted, exit 1).
• A per-service “golden-path warm-up” hint (pre-touching a lazily-warmed dependency such as an Ollama cold model load before the timed golden-path) was scoped as an optional follow-up; the backoff alone resolves the observed false-rollback class, so warm-up is deferred until a service is shown to need more than the debounce window provides.

The gate is a deploy control, not a test harness — a manual run inherits its side effects (2026-08-12, Issue #2385)

Two safety layers above assume the gate is being driven by CI on a commit CI just deployed. Nothing in the decision said what happens when a human runs the same command — and the answer, discovered by running it, is: everything CI’s copy does.

Proving the pit-memory cross-context canary could actually go red required deliberately breaking the lesson write path and running the gate exactly as the pipeline does — cd ~/docker-stacks && python3 scripts/testplan_gate.py pit-memory. It went red, correctly. It also paged Pushover at 17:44 AEST for a failure that had been manufactured on purpose, sat out the 90 s debounce and 600 s confirmation windows, and then ran auto-rollback [live] against a concurrent cc-pool session’s commit for an unrelated Epic — because the rollback target is whatever heads the canonical checkout, not the operator’s work. The revert failed only because the worktree it ran in sat on a local unpushed SHA (fatal: bad object). That is luck, not a control, and it is this ADR’s own innocent-commit revert class (Issue #1516, #2183) reached from the manual side.

Consequence, now a rule: any manual, diagnostic or deliberate-break invocation of testplan_gate.py must pass --no-rollback --no-notify (plus --no-retry when only the immediate verdict is wanted). The bare invocation belongs to CI. /verify — which reaches the runner through testplan_verify.sh and carries neither side effect — remains the sanctioned on-demand path, and that separation is now the reason it exists rather than a convenience. Recorded in the Service Test Plan Standard.

The generalisation worth carrying beyond this gate: before running anything “exactly the way CI runs it”, enumerate what CI’s copy is permitted to do to shared state — notify, revert, deploy, tag, publish. A manual run inherits every one of those powers, including over other people’s commits.