0334 — A pool's agents must actually be interchangeable — composition, parity, and the sub-tables nobody was enforcing

A pool is a set of agents a job may land on interchangeably. Nothing in this estate required that they actually be interchangeable until there was more than one of them — and by then the assumption was load-bearing across ~120 pipeline invocations in nine repos. This ADR adds the Agent Pool Composition & Parity section to the CI/CD & Pipelines Standard, citing the parity gate built at ADR-0331 as its enforcement. Writing that section then exposed a second, larger gap: the standards meta-gate read only a standard’s top-level ## Enforcement table, so every per-section ### Enforcement sub-table introduced by domain-unit consolidation was silently unenforced — including the one this ADR was adding.

Status

Accepted — 2026-08-07. Delivered under Epic #2187 (CI runner pool), Issue #2207. Adds the Agent Pool Composition & Parity section to the CI/CD & Pipelines Standard and amends standards_enforcement_gate.py under the Standard-Enforcement Standard. Companion to ADR-0333; follows ADR-0330, ADR-0331 and ADR-0252.

Context — divergent agents fail nondeterministically, which is the worst way to fail

Two agents in one pool with different toolchains produce, from the same commit against the same state, a different result depending on which one is free. The failure appears and disappears at random, which makes it the hardest class to attribute and the easiest to “fix” by re-running.

This is observed, not theoretical. At Issue #2199 control01 held bpg/proxmox 0.99.0 while a freshly-initialised runner01 resolved 0.111.1 from the same ~> constraint, because .terraform.lock.hcl was gitignored — same commit, same state file, different plan, and which one you got depended on the queue. Lock files are now committed in all four Terraform roots, which fixes that instance. The gate exists to stop the class: the next divergence is an apt install typed on one runner at 2am, or a role applied to one host and not the other, and nothing else in the estate would notice.

The design named the rejected alternative precisely: “Role on both without a gate — cheaper by one gate, decays on the first manual apt install.” A declared control with no conformance check decays silently.

Decision

1. A pool owes four things, and the standard says so

• Composition is declared in inventory (ci_runners), not discovered from the platform. The gate sources composition from there deliberately: the runners have no azure-devops az extension and no az login session, so a gate sourcing composition from the ADO API could not run on the agents it gates. The reverse direction — registered in ADO, absent from inventory — is covered by the pool-agent-count metric in ado_pipeline_metrics.py, so neither direction is silently missed.
• Every agent is built from one role (roles/ci_agent) with the same variables. An agent the role cannot rebuild from scratch is a pet; the test is that a runner can be destroyed and re-provisioned from terraform.tfvars plus a role apply with no hand steps.
• Conformance AND parity are both gated, because they fail differently. Conformance catches an agent that drifted from the declaration; parity catches drift the declaration cannot see — an undeclared binary, an extra collection, a differing Python. An agent can conform and still be different, which is why the gate reports the full observed matrix rather than a bare pass/fail.
• Unverified is a failure, not a skip. An agent that cannot be reached exits 2, never 0. “I could not look” and “I looked and it was fine” must never share an exit code — a parity gate that passes when it could not inspect a runner converts an outage into a green tick.

2. Adding an agent is a pool-wide change, not an addition

A new agent changes what every pipeline may assume. Before it takes work it must be inventory-declared, role-built, parity-passing, and every deployed-path dependency any pipeline invokes must already reach it (the agent-plural rule, ADR-0333). Its patch ring is chosen so the pool never drops below its minimum useful concurrency mid-cycle.

3. The meta-gate reads ### Enforcement sub-tables, and resolves .sh mechanisms

This was found by trying to prove the new section’s enforcement rows red — they would not go red. extract_enforcement_table() parsed only the exact ## Enforcement heading, so a per-section ### Enforcement table was invisible: its rows could name a gate script that does not exist and the meta-gate stayed green. Domain-unit consolidation (ADR-0252) is what created the shape — a folded standard keeps its own enforcement table beside the rules it enforces — and the gate never learned about it. The CI/CD & Pipelines Standard is the only file affected, with two such sub-tables, one of them pre-existing since the trigger-economy section landed.

Three changes, each proven red before trusted green:

1. Union the sub-tables. Rows from ## Enforcement and every ### Enforcement table are checked together. The ## table’s header defines the schema; a sub-table with a different header is reported as ragged rather than skipped — skipping is exactly what made them invisible.

2. Normalise the sub-tables to the canonical schema. Both had drifted to an ad-hoc Requirement | Tier | Stage | Mechanism | Signal vocabulary instead of Obligation | Class | Layer | Mechanism | Dead-man. A table the gate cannot parse is a table the gate cannot enforce.

3. Index and match .sh and .j2 mechanisms. FILE_TOKEN_RE and build_file_index() both covered only .py/.yml/.yaml, so a row naming a real shell gate reported that it named nothing at all. Both halves had to change together — the regex decides whether a token is a mechanism, the index decides whether it resolves; extending one alone turns “shell gates are invisible” into “shell gates always fail”. Six real rows across five standards (secret_scan.sh, ado-pool-auth-reconcile.sh, pihole_pin_audit.sh, service_catalog_regen.sh, stalled_grab_watchdog_cron.sh, ci_shared_state_backup.sh) had been unresolvable and therefore unenforced.

   A third instance of the same class surfaced immediately afterwards, and is the best evidence the fixed gate works: ADR-0335’s enforcement row names pitlab-user-setup.sh.j2, and the regex stopped at .sh and looked for a file that does not exist — so a Jinja template that is itself the enforcing artifact could not resolve. The .j2 suffix is now an optional group outside the alternation, so the longer form wins. The index grew 469 → 567 → 601 files across the two extensions.

Alternatives considered

• Move the new rows into the top-level ## Enforcement table and leave the parser alone. Rejected: it fixes this instance and leaves the pre-existing trigger-economy sub-table — and every future folded section — silently unenforced. The whole point of this gate is that an enforcement declaration nobody resolves is a claim, not a control; tolerating that one level down is self-defeating.
• Ban ### Enforcement sub-tables outright, forcing all rows into the one table. Rejected: co-locating a folded standard’s obligations with its rules is genuinely more readable, and consolidation deliberately preserved it. Teach the gate rather than degrade the document.
• Source pool composition from the ADO API rather than inventory. Rejected on evidence: verified live that az devops invoke on runner01 fails with an extension prompt, and there is no az login session there. A gate that cannot run on the agents it gates is not a gate.
• Parity by convention — apply the role to both and trust it. Rejected as the design already named: it decays on the first manual install, and the failure is nondeterministic, so decay is invisible until it costs a debugging session.

Consequences

• A pool has a written definition of what it owes, with a machine gate behind every objective clause and a conformance checklist for a new agent.
• The residual risk is explicit and accepted: the declaration is a floor, not a ceiling. An undeclared binary installed by hand is invisible to conformance, and visible to parity only if it differs between agents. Identical hand-installs on both runners pass.
• Every ### Enforcement sub-table in the corpus is now resolved, so the CI/CD & Pipelines Standard’s trigger-economy obligations are genuinely enforced for the first time since consolidation.
• Six previously-unresolvable shell mechanisms across five standards now resolve. None was wrong; they were unchecked, which is the same risk with better optics.
• test_standards_enforcement_gate.py gains seven cases (sub-table resolves, bogus sub-table mechanism fails, wrong sub-table schema fails closed, shell mechanism resolves, bogus shell mechanism fails, template mechanism resolves, bogus template mechanism fails). All seven were confirmed to fail against the pre-change gate and pass after — the proven-red evidence the Build Hook & Pipeline-Gate Test Standard (ADR-0141) requires.
• A future runner03 is a terraform.tfvars line plus a role apply, gated by parity — the SRE framing is toil elimination: a runner becomes cattle rather than a hand-built pet.