Design: falsifiable service test plans — a golden path that cannot fail is not a test

The Service Test Plan Standard (ADR-0102) mandates that every service owns a golden-path test, but says nothing about whether that test is capable of failing. At least eight plans assert on a version endpoint or a health ping, and tandoor’s was proven blind on 2026-08-14 — the production image serves both its readiness and golden-path assertions with no database reachable at all. That gate drives auto-rollback on a family-facing service. This design adds a falsifiability clause to the standard and two mechanisms that make an untestable test impossible to ship and prove the rest rather than assume them.

The problem: the standard governs whether a test exists, not whether it can fail

The conformance floor is “at least one functional (behavioural) test, beyond readiness/smoke”. That wording is satisfied by any assertion that is behavioural in form, regardless of whether it discriminates. The result, measured across the 34 docker-stacks plans on 2026-08-14:

Class	Count	Examples
Genuinely strong — asserts on real data or a real transformation	~20	qdrant (canary write→read round-trip), paperless, netbox, freescout, topcharts, vectormap
Blind — assertion survives a dead backend	8	tandoor, incident-webhook, ombi01, sab01, rustdesk01, dtrack, pingvin, portainer
Stateless — correct as authored, no persistence layer to touch	5	gotenberg (HTML→PDF), tika (text extraction), smtp (EHLO), docs, cloudflared

The tandoor proof is the worked example. Running vabene1111/recipes:2.6.13 with prod-faithful environment and --network none, so no database is reachable:

Plan step	Result with the database dead
readiness — GET /	HTTP 200 — passes
golden_path — GET /openapi/	HTTP 200, body carries Tandoor API Docs — passes

Both steps go green with Postgres gone. This is failure class FC-02 assertion-cannot-fail: a check that would still pass with the feature removed. It also invalidates the reasoning in ADR-0011, which assumed tandoor’s readiness probe catches its real failure mode — GET / likewise returns 200 with the database gone.

The discipline already existed one layer down, and was never inherited. ADR-0011’s own Consequences section requires that probe depth be proven: “depth must be verified (induce a dep failure, confirm probe_success→0) before treating a path as more than readiness”. That rule was applied to blackbox probes and never carried into the Service Test Plan Standard, so the same class of unverified-depth assertion reappeared at the acceptance-gate layer. This design is largely that existing rule, promoted to where it was missing.

Decision tree — the resolved architecture

Decisions

Decision	Choice made	Rationale	Rejected alternatives
Clause form	Falsifiability, class-agnostic: the assertion MUST fail when the service’s primary dependency is unavailable	Catches tandoor without making the five stateless services non-conformant	A literal persistence-layer clause (breaks gotenberg/tika/smtp/docs/cloudflared); a per-class floor matrix (heavier, needs every service classified first)
Enforcement	Push-time shape gate and periodic red-proof	Authoring review alone is precisely the regime that produced the eight blind plans	Review only; chaos red-proof on every deploy (doubles gate runtime, new flakiness in the deploy path)
Sever method	Ephemeral clone booted from the rendered compose with dependencies severed	The method that actually caught tandoor	Synthetic-null assertion replay — would have passed tandoor, because its assertion does discriminate against an empty body while the service serves a full one; severing the real dependency in a maintenance window — deliberately breaks production, including family-facing services
Harness shape	One generic harness deriving image and env from the compose template	No per-service fixtures to drift out of step — fixture drift is the same bug class being fixed	34 committed per-service fixtures; scaling real deps to zero (port and volume collisions with production on docker01)
Metric plans	Convert the 6 thin proxies; retain 4 and fix their *_over_time windows	Blanket conversion would destroy maintainerr’s seven temporal invariants, and count(up == 1) is the artifact for Prometheus itself	Convert all ten; keep all ten unchanged (stale-tolerant windows defeat the red-proof)
Secrets	Real rendered secrets, passed by env-file, --network none, --rm	Dummy secrets make the service fail to boot for the wrong reason, manufacturing a conformant verdict that was never earned	Dummy placeholders; non-secret env only; real secrets for authenticated plans only
Cadence	Red-proof on plan change in CI, plus a monthly full sweep	A newly authored blind plan is caught at birth; drift in an unchanged plan within 30 days	Weekly full sweep (a blind plan still ships for up to a week); weekly sample of 3–5 (a blind gate could drive auto-rollback for a quarter)
Findings routing	By service lifecycle: in delivery or hypercare → snag on that Epic; past hypercare → FreeScout incident	Arron’s house pattern — it makes “accepted into PROD” mean something precise rather than a judgement call	All findings as snags (invisible until an Epic closes); Pushover only (no record, no owner)
Lifecycle source	New ansible/scripts/doc_gen/service_lifecycle.yml overlay	Service pages are auto-generated from the Alloy probe inventory (ADR-0059) — a field written onto the page is destroyed at the next regen. doc_gen/ is the established home for declarative per-service metadata	A field on the generated page (overwritten); querying ADO for an Epic by description text (FC-11 guessed-identifier); a hand-maintained exception list
Script home	docker-stacks/scripts/	Follows the ADR-0105 synthetic-gate precedent — a routine-driven gate living beside the stacks it checks	ansible/scripts/ → /usr/local/bin (pins the gate to one provisioned agent)
Doctrine migration	Deferred to its own grill	Migrating /grillme’s embedded doctrine into a standard touches the skill, a new standard, /wrapup reconciliation and every future service build — it deserves its own gate	Folding it into this Epic (doubles an already three-workstream batch, and the migration itself would ship ungrilled)

The three-way verdict is what makes the red-proof attributable

A red-proof that only asks “did the assertion fail?” is satisfied by a clone that never started — which is the same self-deception one level up. Because the harness supplies the real rendered environment, the only thing missing from the clone is the dependency, so each outcome is attributable:

Clone outcome	Verdict	Meaning
Container will not boot	Conformant	The service hard-depends on its dependency; any assertion is necessarily falsifiable
Boots, golden path passes	BLIND — non-conformant	The assertion cannot distinguish a working service from a hollow one
Boots, golden path fails	Conformant	The assertion genuinely discriminates

This is why dummy secrets are rejected: they collapse row 1 and row 3 into an indistinguishable “failed for some reason”, and the harness would record a conformance it never established.

NFR resolution

Requirement	How this design satisfies it
Monitoring	The monthly sweep carries a freshness dead-man — a sweep that silently stops running is otherwise indistinguishable from a sweep finding nothing (FC-03 blind-watcher).
Observability	Sweep emits per-service verdicts and a run summary; findings carry the service, the assertion and the clone outcome. Logs ship to Loki per the Logging Standard.
Alerting	Dead-man alert on sweep staleness. Findings route by lifecycle rather than paging — a blind plan is a defect record, not an interrupt.
SBOM / vuln coverage	No new long-lived image. The harness runs the pinned images already tracked in Dependency-Track; clones are --rm and ephemeral.
Docs as code	This design doc, the standard amendment, the governing ADR, and an operations runbook for the sweep — all in the same change as the code.
Config as code	Gate and harness in docker-stacks/scripts/; lifecycle overlay in ansible/scripts/doc_gen/; sweep schedule declared in schedule_config.yml. Nothing manual-only.
Security	Real credentials are resolved as the deploy resolves them, passed by env-file — never argv (/proc/<pid>/cmdline is world-readable on a host five sessions share), into a container with --network none (no egress path) and --rm (seconds of lifetime). Harness output is scrubbed before any finding is recorded.
Update & patch lifecycle	Fits the GitOps pinned-tag house pattern: the harness scans whatever tag group_vars currently pins, so it follows the fleet automatically with no separate refresh path. The gate and harness scripts are ordinary repo content, updated by commit; their staleness signal is the dead-man plus the on-change trigger, which exercises the harness every time any plan changes. No pinned-forever component is introduced.
Planned maintenance	The design deliberately never touches production — the clone is isolated — so no silence is required to run it. The one-off fleet remediation redeploys services and follows the normal mute-first discipline.
Notifications	No new Pushover path. Findings become snags or incidents; the dead-man uses the existing alerting route at priority 0, within quiet hours.
Tagging	Findings carry service, domain and verdict labels conforming to the Tagging Standard.
Enterprise pattern	This is mutation testing applied to acceptance gates — deliberately break the system and require the test to notice. The industry equivalents are mutation-testing frameworks (Stryker, PIT) and chaos engineering’s steady-state hypothesis.
DORA / SRE	Directly improves the validity of Change Failure Rate: a blind post-deploy gate under-reports failed changes, so the metric currently flatters us. The gate is a detective control whose own effectiveness is now measured.
Minimal decomposition	4 sub-issues → 2 prompts.

Compliance & control mapping

Control / decision	Framework mapping	How it is tested / evidenced	Residual risk
Falsifiability clause in the standard	CPS 234 — control implementation commensurate with threat; ISO 27001 A.8.29 (security testing in development and acceptance)	Published standard clause with a conformance checklist item	A clause is only as good as its gate; hence the two mechanisms below
Push-time shape gate	NIST CSF 2.0 PR.PS (platform security, secure development); ISO 27001 A.8.28	Gate ships its own regression test, proven red against the real unfixed tree per ADR-0141	Shape matching catches known-blind forms, not novel ones — the red-proof covers the remainder
Periodic red-proof of every plan	CPS 234 — regular testing of control effectiveness (the closest direct mapping in the whole design: an acceptance gate that cannot fail is an untested control); NIST CSF 2.0 DE.CM / GV.OV; ISO 27001 Clause 9.1 (monitoring and evaluation of control effectiveness)	Monthly sweep run record, per-service verdicts, dead-man proving the sweep itself still runs	Coverage limited to services that boot standalone — unquantified, recorded below
Auto-rollback driven by a validated gate	CPS 230 — operational risk controls and tolerance for disruption on critical operations	Handover proves the gate blocks a real blind plan and permits a conformant one	A rollback triggered on a false red is a new (small) disruption source
Lifecycle-based findings routing	CPS 230 operational-risk record-keeping; NIST CSF 2.0 RS (response)	Routing proven both ways at handover — snag for in-delivery, incident for accepted	Depends on service_lifecycle.yml being current
Real credentials in the harness	CPS 234 information-asset protection; ISO 27001 A.8.12 (data leakage prevention), A.5.15	Env-file only, --network none, --rm, scrubbed output; reviewed at handover	Credentials transit a container image’s env for seconds; mitigated by having no network path out

Operational readiness & handover

Service record. This is estate tooling rather than a user-facing service, so it takes no NetBox asset record and no criticality tier of its own. It is a control over the test-plan gate, and its operating record is the standard plus this design. The service_lifecycle.yml overlay it introduces, however, becomes estate-wide service metadata and must carry a value for every service page.

Reliability contract. The SLI is proportion of test plans with a current, passing red-proof. The objective is 100% of red-provable plans proven within the last 35 days (the monthly cadence plus tolerance). The error budget is spent by any plan whose red-proof is stale or failing; the burn signal is the dead-man plus the sweep’s own finding count.

Runbook. docs/operations/ gains a page covering: how to run the harness against a single service by hand, how to read the three-way verdict, what to do when the harness cannot boot a service, how to route a finding by lifecycle, and how to clear a stale dead-man. It must be followable cold by a session with no context — proven by walking it at handover.

Test strategy. Per sub-issue: the shape gate ships a regression test proven red before it is trusted green (ADR-0141); the harness is proven by reproducing the tandoor result — the known blind plan must be reported BLIND, and a known-strong plan (qdrant’s canary round-trip) must be reported conformant. The mechanisms themselves are the deliverable, so “it ran without error” is explicitly not acceptance evidence for any of them.

Day-2 BAU and toil. One monthly automated sweep, declared in schedule_config.yml in a slot clear of the overnight backup and patching window (~22:00–07:00 AEST). No manual recurring work is introduced. The toil this removes is the per-service judgement about whether a test is any good, which currently has no mechanism at all.

Backup and restore. No state is created. The harness is stateless and its findings live in ADO and FreeScout, both already covered.

Standing owner. The monthly sweep is the standing mechanism; /health surfaces the dead-man so a stopped sweep is visible without anyone remembering to look. /vulnscan continues to consume test plans as its post-remediation close-gate and directly benefits from their strengthening.

Exit. If the red-proof proves not worth its cost, the sweep and its schedule entry are removed and the falsifiability clause degrades to the shape gate alone — a strictly better position than today.

Delivery plan — 4 sub-issues, 2 prompts

#	Issue	Batch
#2413	Falsifiability clause in the standard + push-time shape gate (with its own proven-red regression test)	1
#2414	Red-proof harness, CI on-change trigger, monthly sweep with dead-man, service_lifecycle.yml overlay	1
#2415	Fleet remediation — 8 blind plans, 6 metric conversions, 4 over_time windows	2
#2416	Handover / Production Readiness Review	2

Known unknowns, recorded deliberately

These were resolved by assumption during the grill and are named so that approval was not given blind:

• How many services boot standalone is unmeasured. Tandoor did; netbox almost certainly will not. This directly bounds red-proof coverage and could shrink it materially. Measuring it is the first task of #2414.
• The four retained metric plans rely on exporter-sever, a red-proof mechanism that has not been demonstrated to work.
• service_lifecycle.yml must cover 66 service pages, not just the 34 carrying test plans; the effort to populate and keep it current is not sized.
• Tightening *_over_time windows may introduce flapping in the four retained metric plans; no flap analysis has been done.

Deviations (as-delivered)

Filled by wrapup Check O at Epic close.