ADR-0277: Fleet-wide secret-scan coverage with a conformance gate — a declared control must be enforced everywhere, not wired once

Status: accepted (2026-07-29)

Context

ADR-0046 introduced gitleaks secret scanning — run in an ephemeral container on docker01 via secret_scan.sh — and described it as “reusable across every pitlab repo”. It was reusable, but nothing enforced that every repo actually reused it. In practice the gate was only ever wired into pitlab-docs (inline in the docs pipeline) and, later, the docker-stacks SecurityScan stage. The four Terraform repos and the ansible repo had a scanner available and no gate calling it — a control that was declared fleet-wide but deployed to a subset.

The cost of that gap became concrete: a live DT_API_KEY sat in plaintext across five dt_annotate_*.py scripts in the ansible repo and went undetected until a manual audit found it (ADO #1644). ansible had no secret-scan gate, so nothing ever looked. The scanner existed; the coverage didn’t. This is the classic decay mode of a control asserted in prose but not conformance-checked — the same failure shape ADR-0270 found in trigger paths and the Standard-Enforcement design (ADR-0176) exists to prevent: a gate that silently under-covers is indistinguishable from one that passes, until an incident proves otherwise.

Per Prime Directive 9 (flag a standards blind spot, don’t just patch the instance), the right fix is not “add a gate to ansible” — it is to make fleet-wide coverage a property the system enforces and cannot silently lose.

Decision

Every code repo carries its own blocking secret-scan gate, and a daily conformance sweep asserts that coverage never decays. Three coupled parts:

1. A per-repo pipelines/secret_scan.yml in every code repo — ansible, docker-stacks, pitlab-docs, tf-cloudflare, tf-proxmox, tf-azure, tf-backblaze. Each is a blocking CI gate (pool: pitlab, trigger: branches include main on all paths — a secret can land in any file, pr: none) that runs secret_scan.sh --source "$(Build.SourcesDirectory)" --label <repo> --alert. A committed credential fails the build (gitleaks exit 1) and fires a --alert Pushover before it can spread. pitlab-docs and docker-stacks keep their existing inline/stage scans; the standalone pipeline is the uniform fleet mechanism.

2. checkout: self (fresh, tracked-only), not a shared working tree. Each run gets a fresh git checkout that contains only tracked files. Gitignored local state — Terraform .tfstate, .tfvars, .venv — is absent, so it can’t produce false failures, while a force-added secret file is tracked and is caught. This was a deliberate correction: an earlier cd ~/repo && git reset --hard design would have left a real gitignored terraform.tfvars in the shared tree (reset does not remove ignored files) and failed the gate on a non-committed local secret.

3. A daily coverage-conformance gate — validate_secret_scan_coverage.py. Folded into standards_enforcement_reconciliation.yml (beside the systemd-selfheal and docker01-cac guards, sharing the standards.enforce.age dead-man). It asserts every code repo has BOTH a tracked .gitleaks.toml AND a tracked pipeline invoking secret_scan.sh on origin/main — the authoritative committed state, not a local copy. Any gap is a hard finding (exit 1 → reconcile_fail Pushover). This is the anti-decay control: a deleted, renamed, or never-wired gate is caught out of band, since no commit triggers its absence.

Each repo also carries a .gitleaks.toml ([extend] useDefault = true + a per-repo false-positive allowlist for --map doc examples, var. references, placeholder tokens, and Access-policy UUIDs), validated clean against a tracked-only tree before rollout.

The Secrets & Encryption Standard Enforcement table is amended: the machine gitleaks row now names the per-repo fleet gate, and a new scheduled row records the coverage-conformance obligation with its standards.enforce.age dead-man.

Alternatives considered

• Just add a gate to ansible (patch the instance): rejected — it fixes the one repo that leaked and leaves the same latent gap in the four tf repos and any future repo. The finding was the missing conformance control, not the missing ansible gate.
• One central pipeline scanning all repos in a loop: rejected — a single gate’s failure blocks unrelated repos, it doesn’t run on the push that introduced a secret (wrong trigger locality), and it re-creates a single silent point of decay. Per-repo gates fail independently and trigger on the offending push.
• Reuse the shared ~/<repo> checkout with git reset --hard: rejected — reset does not remove gitignored files, so real local .tfvars/state would leak into the scan and false-fail; checkout: self gives a clean tracked-only tree by construction.
• Rely on the auto-mode classifier alone (it blocks literal secrets at author time): kept as a complementary advisory layer, but rejected as the control — it guards the agent’s own writes, not human commits or pre-existing history, and #1644 proves committed secrets still reach main.
• History rewrite to purge the leaked key: out of scope here and rejected as the remediation per ADR-0030 — the key was rotated at source (the finding-closing action); the scanner covers the working tree, matching the pitlab-docs config.

Consequences

• Coverage is now a machine-enforced invariant: seven code repos each fail their own build on a committed secret, and the daily sweep reds if any repo loses its gate or a new repo joins without one. Adding a code repo means adding it to REPOS in the validator, which then demands its gate — decay surfaces as a finding, not an incident.
• The conformance validator was proven red before trust (Prime Directive 12): at authoring it reported all 11 pre-rollout gaps (only pitlab-docs conformant, docker-stacks correctly flagged config-present-but-no-standalone-pipeline). It flips to PASS once the six pipelines land.
• Baseline scans found no other committed secret anywhere in the fleet — the tf-repo findings were all false positives (Access-policy UUIDs, var. refs, --map examples, gitignored state), confirming #1644 was isolated and the allowlists are correct.
• Six new ADO pipelines require registration + pitlab-pool authorization. The tf-cloudflare secret_scan.yml does not match cloudflare.yml’s trigger paths (*.tf, scripts/*.py, *.json), so landing it is Cloudflare-inert (RULE 14 clear).
• Known limitation: like the pitlab-docs config, the scan targets the working tree, not git history — the now-rotated DT_API_KEY still in ansible history pre-6908eac is not flagged; rotation-at-source (ADR-0030) already invalidated it, so a history rewrite is deliberately not pursued.

Amended 2026-08-04 — the gate now also asserts the pipeline is registered and ENABLED in ADO

Checks (a) and (b) read git only, so they prove a gate is authored, never that it is wired up. A pipeline YAML that no ADO definition points at, and one whose definition is disabled, are both gates that cannot run — and from origin/main they are indistinguishable from a working gate. The sweep would have reported PASS for a repo whose scanning had been switched off entirely.

ADR-0312 made that blind spot materially sharper: the ansible repo’s four always-on guards now share one definition, so deleting or disabling ansible-ci switches off gitleaks and the notification-catalog, schedule-timezone and pipeline-path-coverage guards in a single action. A conformance control that cannot detect its own removal is not a control.

validate_secret_scan_coverage.py therefore gained check (c): every tracked gate YAML must map to a registered ADO build definition with queueStatus == enabled. It costs one API call — includeAllProperties=true returns process.yamlFilename and queueStatus inline, so there is no per-definition fan-out — authenticated by the Entra Bearer flow (az account get-access-token) already used by ado_pipeline_run.py. It fails loud, not open: an unreachable ADO API exits 2 (UNVERIFIED) rather than reporting a green sweep on unchecked state. Shipped proven-red via --self-test, which runs before the live check in the daily reconciliation (ADR-0141) and covers registered-and-enabled, no-definition, registered-but-disabled, one-enabled-among-several, and a wrong-repo definition negative control. Epic #2166.

Amends ADR-0046. Epic [Claude] fleet-wide secret-scan coverage — 2026-07-29.

• 2026-08-07 — the coverage gate counted a COMMENT as a gate (Issue #2191). validate_secret_scan_coverage.py ran a bare git grep -lI secret_scan.sh over tracked pipelines/, with no comment stripping. The retired docker-stacks.yml umbrella therefore matched on the strength of its own retirement-header prose, which explains that the sanctioned gate is secret_scan.sh elsewhere. It was harmless only because secret_scan.yml matched too and one enabled candidate is enough — but a repo whose only match was a comment would have reported fully covered with no secret gate at all. The pattern is now ^[^#]*secret_scan\.sh, with self-test cases pinning invocation-vs-mention. A conformance gate that greps prose certifies the claim, not the code.