Design: CI runner pool — control01 leaves CI, runner01/runner02 take it

A second self-hosted parallel job was purchased on 2026-08-06, but a licence delivers nothing without a second agent — one agent runs one job at a time regardless of entitlement. This design spends that entitlement by moving CI off control01 entirely onto two purpose-built, interchangeable runners (runner01, runner02), leaving control01 to own only the cc-pool Claude sessions and the singleton orchestrator crons. It also resolves the three hazards that block real concurrency — Terraform state that existed only on one agent’s disk, pipelines pinned to control01-local state, and un-serialised concurrent access to the same target host — and defines the queue-time SLO by which the Epic is judged.

The purchased entitlement reads total=2 and is the gate on all other work

az devops invoke --area distributedtask --resource resourcelimits --api-version 7.1-preview reports the self-hosted private row as totalCount=2, FreeCount=1, PurchasedCount=1. The free tier grants one self-hosted parallel job, so exactly one purchased job is required for concurrency 2; an initial purchase of two was reduced to one on 2026-08-06 after the API showed totalCount=3. Verified before any build work started, because two agents against one licence improves nothing.

Decision tree

Decisions

Decision	Choice made	Rationale	Rejected alternatives
Scale up or out	Scale out	67 of 68 ansible pipelines run cd ~/ansible && git reset --hard origin/main; two agents sharing one ~ race on ~95% of the estate. Separate hosts dissolve the race rather than mitigating it.	Scale up control01 — needs either ~74 pipelines rewritten to checkout: self, flock that serialises the very jobs that queue most, or a second OS user costing most of a new node’s onboarding without the isolation.
Where agents run	Two new runners; control01 runs no agent	Full separation of concerns: runners become interchangeable cattle rebuildable from a role, CI can no longer starve the five cc-pool sessions, and removing ~/myagent frees 3.7 GB — taking control01 from 73% to ~48% disk, which retires the “disk is a hard blocker” finding with no resize at all.	control01 + control02 — keeps control01 special, preserves the CI/session contention, and leaves the original 15G rootfs squeeze in place.
Host names	runner01 / runner02	The lifecycle-naming standard requires a name whose function is readable without a lookup. The estate already runs zabbix-agent, patchmon-agent, alloy, hermes-agent and Claude agents, so a bare agent01 fails the standard’s own conformance checklist on ambiguity. runner is the industry term for a CI execution node.	agent01/agent02 — ambiguous; ciagent01/ciagent02 — conformant but further from common platform vocabulary.
Pipelines pinned to one host	Make them host-agnostic; no demands:	A capability/demands: scheme adds a concept no pipeline uses today, and a constraint that is declared rather than derived is easy to omit on a new pipeline with nothing to catch it.	Agent capability + demands: on the affected pipelines; replicating control01-local state onto the runners, which splits each heartbeat across two hosts and makes both stale half the time.
Pipeline liveness heartbeats	A control01 cron polls the ADO API for each watched pipeline’s last successful run and stamps the existing <check>.last file	Host-agnostic by construction at any agent count, so it never breaks again when a runner03 appears. Generalises to every pipeline rather than the two that happen to exist, removes the stamping step from pipelines, and measures the outcome — did the run succeed — instead of a side-effect on one box.	Zabbix trapper + history.push from the pipeline, which keeps the pipeline→heartbeat coupling; a Loki log line with absence alerting, which moves two checks off the estate-standard <check>.age shape.
secret_scan.sh invocation	Run repo-relative from $(Build.SourcesDirectory)/scripts/	Removes the /usr/local/bin deploy dependency that pins it to a provisioned agent, and fixes a latent correctness bug: CI currently scans a new commit using the deployed copy, so a commit that improves the scanner is never tested by itself.	Deploying the script to both runners via the ci_agent role — preserves both the pinning and the stale-copy bug.
Toolchain management	ci_agent role over both runners plus a CI parity gate	Two agents in one pool with different toolchains fail nondeterministically depending on which picks up the job. The gate is what stops them drifting apart later; a declared control with no conformance check decays silently.	Role on runner02 only — guarantees divergence from day one. Role on both without a gate — cheaper by one gate, decays on the first manual apt install.
Terraform state	Shared ZFS dataset bind-mounted into both runners; local backend retained	The house pattern for shared filesystems between LXCs, per plex01’s mp0/mp10. Empirically proven: a concurrent terraform apply fails with Error acquiring the state lock: resource temporarily unavailable, and flock was shown to block across two containers sharing a bind mount on the same kernel — so the local backend’s lock genuinely serialises both runners. Zero new dependency, zero cost, no cloud round-trip on every plan.	Remote backend (azurerm or B2/s3) — correct for a multi-premises estate and it would close the off-site gap, but it drags a four-repo state migration into a concurrency project and over-solves the actual blocker. Self-hosted MinIO — circular dependency: if docker01 is down and Terraform is needed to fix it, the state is unreachable.
Terraform state durability	PBS host-level backup of the dataset plus ZFS snapshots	PVE cannot back up bind mounts — vzdump skips them, so moving state out of control01’s rootfs would silently drop it from the 01:30 nightly that currently covers it. Explicit backup coverage makes the end state better than today, where state is buried inside a guest image.	Relying on the existing guest backup — would silently lose coverage.
Same-target concurrency	flock on the shared dataset, keyed by target host	Reuses infrastructure this design already builds, and the mechanism is proven rather than assumed. Only same-target runs serialise, so the queue win is preserved for the majority of pairs.	ADO environment exclusive-lock — platform-native and visible in the UI, but clunky, per-pipeline, and declared rather than derived. Accept-and-measure — the failure mode is a silently wrong deploy.
Runner identity	Reuse control01’s SSH key, ADO PAT and Infisical identity	Arron’s decision, confirmed twice including after the premise changed to two brand-new hosts. Nothing new to provision or rotate, fleet authorized_keys untouched, runners stay interchangeable.	A shared CI identity distinct from control01’s; per-runner identities. Both were recommended for least privilege and are recorded below as accepted residual risk.
Runner sizing	8 cores / 12 GB / 4 GB swap / 60 GB rootfs each	Ansible and Terraform are not strongly parallel, so 8 cores is generous for a dedicated runner. The 60 GB rootfs is the load-bearing number: the agent _work directory alone is 3.6 GB on control01 and docker build contexts grow. pve01 has 28 CPUs at ~33% and 554 GB free on local-zfs.	12c/24GB — more headroom than a CI-only node needs; 4c/8GB — deliberately reproducing control01’s known-tight spec.
Patch lifecycle	Both runners in ring2, coordinator staggers within the ring	The reboot coordinator already patches serially within a ring, so at most one runner is down at a time and CI degrades to concurrency 1 — exactly today’s capacity, so nothing breaks.	Split across ring1/ring2 — maximum availability separation, but deliberately runs two differently-patched runners for part of each cycle, which is the toolchain divergence the parity gate exists to prevent.
Success measure	p90 pipeline queue time as an SLI with an SLO, plus DORA lead time for changes	Gives the Epic a measurable pass/fail and a standing signal that justifies a third parallel job later, instead of a one-off anecdote that drifts back unnoticed.	Re-baseline once and record the delta — proves delivery but leaves no standing signal.

The lxc.tf module cannot currently stand up a working container

Provisioning runner01/runner02 exposes a gap in tf-proxmox/lxc/lxc.tf: the resource declares no features block and no hook_script_file_id, and features sits in ignore_changes. A container created purely by Terraform therefore gets no nesting=1 and no hookscript, so it never runs /usr/local/sbin/pitlab-user-setup.sh and comes up with no pit/ansible users and no SSH keys — unreachable. Today the 5-minute pitlab-hookscript-deploy.sh cron on pve01 attaches the hookscript after the fact, but the hook only fires on post-start, so a freshly created container additionally needs one restart before it bootstraps. All seven existing LXCs carry both settings, applied outside Terraform. This design closes the gap in the module rather than working around it per-container.

NFR resolution

• Pillar 1 — Monitoring. Both runners get a Zabbix host with the standard LXC item set plus agent-specific items: vsts.agent service state, _work directory size against the 60 GB rootfs, and the pool-auth reconcile registration check.
• Pillar 2 — Observability. Alloy ships journald to Loki from both runners with device_type set to the CI functional role per the loki-labels taxonomy, verified against the live Loki API before close. One conforming per-runner-pool Grafana dashboard carries RED+USE+logs plus the queue-time SLI, and registers a probe_success{service,domain} blackbox target.
• Pillar 3 — Alerting. Zabbix triggers on agent-service-down and _work disk pressure; an Alertmanager rule on the queue-time SLO burn. The parity gate failing is itself an alertable condition — divergent runners are the failure this design most wants to catch early.
• Pillar 4 — SBOM / vulnerability coverage. Both runners get Dependency-Track projects and join the scheduled Trivy scan, on the same footing as every other host.
• Pillar 5 — Documentation as code. Covered by the corpus sweep below and by ADRs written at delivery.
• Config as code (Prime Directive 7). Containers declared in tf-proxmox/lxc/terraform.tfvars; toolchain, agent install, systemd hardening and repo checkouts in the new ci_agent role; the shared dataset and its bind mounts in Terraform; nothing left as a UI edit.
• Software update & patch lifecycle. Host OS rides the existing overnight security-patching window, patchmon-agent-tracked, with both runners in ring2 and the coordinator staggering them so CI never drops below concurrency 1. The ADO agent binary and the pinned toolchain (terraform, az, trivy, ansible collections) are versioned in the ci_agent role and bumped in Git — the same GitOps pinned-tag discipline the docker-stacks stacks use, with Git revert as the rollback handle. The bump trigger is the existing automated dependency-update tooling plus the ADO agent’s own upgrade signal; the pre-deploy gate is the parity gate plus a post-apply assertion that both runners still register in the pool; staleness is detected by the Trivy/Dependency-Track projects from Pillar 4 and by the parity gate flagging version drift. No component here is pinned-forever without a refresh path.
• Planned maintenance. The control01 agent decommission and each runner cutover are disruptive to CI, so an Alertmanager silence and a Zabbix maintenance window covering control01, runner01 and runner02 are created before the first action and removed only once metrics return to baseline.
• Notifications. Operational signals from the runners route to Pushover only. The Epic’s delivery report goes to the docs Blog with an email blurb from the Claude Engineer persona at wrap-up.
• Tagging (Prime Directive 10). NetBox records derive from the same containers_by_vmid map that provisions the containers; Zabbix host groups, Loki stream labels, Grafana dashboard tags, inventory alloy_device_type and ADO work-item fields all carry the CI functional role consistently.
• Enterprise pattern. A pool of interchangeable, role-built runners with a conformance gate is the standard platform-engineering shape — the same model as GitHub Actions runner groups or GitLab runner fleets. The SRE framing is toil elimination: a runner becomes a terraform.tfvars line plus a role apply rather than a hand-built pet.
• DORA / SRE delivery performance. Deployment frequency and lead time for changes come from ADO run data, commit to deploy-complete. Change failure rate comes from post-deploy gate red rate and rollback count. Time to restore is measured alert-fire to recovery. The SLI is p90 pipeline queue time for pool pitlab, with an error budget whose exhaustion is the evidence for buying a third parallel job. Baseline to beat: median queue 8.7 min, p90 24.1 min, 60% of wall-clock queueing on 2026-08-05.
• Minimal decomposition. Three sub-issues delivered in three prompts — see the delivery plan.

Compliance & control mapping

Control / decision	Framework mapping	How it is tested / evidenced	Residual risk
Runners built from ci_agent role, not by hand	CPS 234 control implementation; NIST CSF Protect; ISO 27001 A.8.9 configuration management; internal IaC & Deployment Standard	The role is the artifact; a --check run against both runners evidences conformance, and the parity gate fails closed on drift	Role initially reverse-engineered from control01’s accreted state, so an undocumented dependency may surface at first pipeline run
Parity gate asserting identical toolchain across the pool	CPS 234 regular testing of controls; NIST CSF Detect; ISO 27001 A.8.9	Gate proven red against a deliberately divergent runner before it is trusted green	Gate covers declared tools only; an undeclared binary installed by hand goes unseen
Terraform state on a shared, snapshotted, PBS-backed dataset	CPS 230 business continuity, RTO/RPO for critical operations; NIST CSF Recover; ISO 27001 A.8.13 information backup	A restore drill reads state back from PBS and runs terraform plan to an empty diff	State remains on-premises on pve01; a total-site loss still loses it between PBS syncs
flock-serialised concurrent applies and same-target playbook runs	CPS 230 operational risk controls; NIST CSF Protect; internal CI/CD & Pipelines Standard	Proven experimentally: concurrent apply returns Error acquiring the state lock, and cross-container flock blocks as designed	Lock is advisory — a pipeline that bypasses the wrapper is not serialised, so the wrapper needs its own conformance check
Alert dead-man liveness derived from the ADO API	CPS 234 detective controls; NIST CSF Detect; internal Standard-Enforcement Standard	The new poller carries its own <check>.age dead-man, so a silent poller failure is itself alerted	Adds one indirection: an ADO API outage reads as pipeline staleness rather than an API fault
Runners reuse control01’s SSH key, PAT and Infisical identity	CPS 234 access control; NIST CSF Protect / Govern; ISO 27001 A.5.15 access control, A.5.16 identity management	Identity inventory records the shared credential and its blast radius; rotation covers all three hosts together	Accepted risk, explicit owner decision. control01’s identity resides on two hosts that execute arbitrary pipeline code; no per-node revocation, and ADO/Infisical audit logs cannot attribute activity to a specific runner. Recommended alternative — a CI-plane identity distinct from control01’s — was declined in favour of operational simplicity
Runners in patching ring2 with staggered reboots	CPS 230 tolerance for disruption; NIST CSF Recover; internal Fleet Update Standard	A patch cycle is observed to take exactly one runner at a time, with CI continuing at concurrency 1	A coordinator defect taking both at once would fully stop CI; the runners’ Zabbix triggers are the detection
Queue-time SLO with error budget	CPS 230 operational risk appetite; NIST CSF Govern; internal Metrics & Dashboards Standard	The SLI renders on the runner-pool dashboard and drives an Alertmanager burn rule	SLO target is set from one baseline day and may need revision after a fuller sample
Corpus corrected so no doc asserts a false topology	ISO 27001 A.5.37 documented operating procedures; NIST CSF Govern; internal Documentation Standard	20 identified docs re-verified post-cutover; ADRs superseded rather than rewritten	286 docs mention control01 and are triaged rather than exhaustively re-read, so a stale minor reference may survive

Four standards gaps this design exposes

Each is a gap in a governing standard, not just an instance to patch — recorded here so the amendment lands with the work rather than being rediscovered.

• Pipeline-Gate Script Location Standard has a multi-agent blind spot. Its decision rule branches on “a docker-stacks pipeline gate runs repo-relative” versus “a host cron/timer script lives in ansible/scripts/ and deploys to /usr/local/bin”. An ansible-repo pipeline gate falls between the two and defaulted to /usr/local/bin, which silently pins it to a provisioned agent. Proposed amendment: a script invoked by a pipeline must either run repo-relative from the checkout, or be deployed to every agent in the pool — never to one.
• The <check>.age dead-man pattern encodes a single-agent assumption. The Standard-Enforcement Reconciliation doc states verbatim that “the pitlab ADO agent runs as pit on control01, which owns /var/lib/pitlab”. That is a documented estate-wide pattern built on an assumption a second agent invalidates. Proposed amendment: pipeline-fed liveness derives from the pipeline system, never from agent-local filesystem state.
• No standard governs agent-pool composition or parity. Nothing requires that agents in a pool be interchangeable, that their toolchain be declared, or that divergence be gated. Proposed: a short pool-parity clause in the CI/CD & Pipelines Standard, with the conformance gate as its enforcement.
• No standard requires shared or remote Terraform state. Local, gitignored state on a single agent was invisible until it nearly broke this work — and it is simultaneously a business-continuity gap. Proposed amendment to the IaC & Deployment Standard: Terraform state must be reachable by every runner permitted to apply it, and must carry explicit backup coverage.

The docs corpus sweep is a first-class workstream, not a closing checkbox

The topology change makes existing documentation factually wrong, so the corpus is corrected as part of delivery. Measured on 2026-08-06 across 776 docs:

Set	Count	Treatment
Docs asserting the ADO agent runs on control01	20	Must-fix — these become false on cutover
Docs encoding a single-agent assumption in different words	25	Must-fix — same assumption, phrased as “single serial worker” / “one agent”
Combined must-fix candidates	34	Triage for false positives, then correct or supersede
Docs referencing Terraform state or tfstate	17	Review against the shared-dataset model
Docs mentioning control01 at all	286	Triage — most remain correct, since control01 still exists and still runs the crons

Search on the assumption, not the phrasing. The first sweep looked only for “the agent runs on control01” wordings and found 20; widening it to the same assumption stated differently — “the agent pool is a single serial worker”, “one agent”, “single-agent” — found 25 more, 34 combined. A doc is in scope because of what it assumes, not because of the words it uses, so the sweep query must cover both.

The must-fix set splits by document class, and the classes get different treatment: operations, reference and standards pages describe current state and are corrected in place; ADRs are historical records and are never rewritten — a superseding or amending ADR is written and the original gains a forward-pointing status note; blog posts are a dated record and are left untouched.

The load-bearing cases are the ADRs this Epic itself produced. ADR-0311 (trigger economy) and ADR-0322 (job economy) both rest on “the agent pool is a single serial worker”, and ADR-0312 (ansible-ci guard consolidation) reasons from the same premise — that premise becomes false at two runners. Their conclusions still hold, because job count drives queue traversals independently of agent count, but the stated rationale must be amended rather than left asserting a topology that no longer exists. ADR-0235 (CI agent memory boundedness), ADR-0106, ADR-0234 and ADR-0270 are also in the affected set and take the supersede path.

Delivery plan — 3 sub-issues, 3 prompts

Sized so each issue is one substantial session rather than a piecemeal handoff.

1. Build and cutover (Issue #2199, repurposed from its original scale-up scope) — close the lxc.tf features/hookscript gap; provision CT 122/123 as runner01/runner02; create the shared ZFS dataset, bind mounts and PBS backup coverage; write and apply the ci_agent role to both; register both agents in pool pitlab carrying the existing KillMode=control-group clean-reap and OOMScoreAdjust hardening; migrate Terraform state; add the flock target-serialisation wrapper; make the three pinned pipelines host-agnostic; decommission agent id=9 from control01.
2. Pillars, proof and measurement (new sibling issue) — Zabbix hosts/items/triggers, Alloy→Loki shipping verified against the live API, the runner-pool Grafana dashboard and blackbox target, Dependency-Track projects and Trivy coverage; the parity gate proven red before trusted green; prove concurrency from the ADO API — two runs inProgress with overlapping start/finish windows on distinct agent ids; stand up the queue-time SLI/SLO and DORA lead-time panels; re-baseline against 2026-08-05 and record the delta on the weekly pipeline-performance report (#2197).
3. Corpus sweep and governance (new sibling issue) — correct the 20 must-fix docs, triage the 286, review the 17 state-related pages; write the delivery ADRs; land the four standards amendments above.

Deviations (as-delivered)

Status moved to as-delivered on 2026-08-07: the design’s three-issue delivery plan is complete — build and cutover (#2199), pillars and measurement (#2206), corpus and governance (#2207). Every decision in the table above was implemented as designed; the runner sizing, pool composition and shared-dataset layout were re-verified live at close (2 agents online, 8 cores / 12 GB each, rpool/data/subvol-12{1,2,3}-disk-0). The deviations below are the places where delivery went beyond or against the design, each with its governing ADR.

Designed	Delivered	Why	ADR
Four standards gaps would be closed	Eight were: the design’s four, plus side-effect re-homing, gate-in-the-right-repo, the <check>.age control01-only registry, and fleet UID allocation	Each was a class the build exposed, not an instance — a one-off fix would have left the next occurrence to repeat the reasoning (Prime Directive 9)	ADR-0331, ADR-0333, ADR-0334, ADR-0335
ansible-ci keeps its guards as separate jobs (upheld by ADR-0322)	Collapsed to one job	The exemption was measured against one run on a quiet pool; under real contention run 7373 lost the agent for 54 minutes between its third and fourth job. Job count is a multiplier on contention, not a fixed cost	ADR-0332
Prime Directive 1’s pvesh snapshot form is the pre-change safety step	A helper was required — the pvesh form fails silently on the bind-mounted guests this design created	Creating the ci-shared bind mounts made three of the estate’s most-changed guests un-snapshottable by the documented method, and the failure is indistinguishable from success	ADR-0336

Not deviations, recorded for completeness. Four post-design follow-ons were discovered during delivery and delivered as sibling children rather than design changes — all four closed 2026-08-07:

Follow-on	Outcome	ADR
HAOS deploy pipelines failed on the runners (#2216)	Peer, not pin. The coupling was dissolvable — the runners already held the key and checkouts; the real gap was a HAOS host key on non-default port 22222. Credential paths anchored to the controller’s HOME; a ninth standard section (Pipeline Agent-Portability) with a machine gate	ADR-0337
ADO definition 19, the docker-stacks umbrella (#2191)	Definition deleted, then the inert YAML removed — that order, because the reverse reds the next push. Secret-scan coverage re-verified PASS across all 7 repos, which exposed a comment-matching defect in the coverage gate itself	—
ansible-ci’s run count (#2215)	Accepted on the measured SLI: queue-time attainment moved from 71.1% to 100% against a 95% target, and ansible-ci’s queued-vs-work ratio fell from 1.89:1 to 1.02:1 while its share of traffic rose. The one trigger change made goes the other way, for correctness	ADR-0338
pipewait’s three conflated outcomes (#2210)	CI_SKIPPED split from DISCOVERY_TIMEOUT; the discovery window re-derived from 1000 runs and made adaptive to batch: true deferral; agent= added to every marker	ADR-0339

The queue-time SLO — this design’s own success measure — was NOT met. Corrected 2026-08-12 under Epic #2386.

The original close-out recorded the objective as met at 100% of runs inside the 10-minute objective against a 95% target, from 94 runs over 8 hours on cutover day. It called sample size “the residual” and argued the window carried above-average ansible-ci load, so the verdict held. It did not.

Window	Runs	Attainment	p90 queue
Cutover afternoon, as originally recorded	94	100%	5.6 min
2026-08-07 → 08-11, first full post-cutover days	1,017	80.9%	14.0 min
2026-08-03 → 08-12, whole store	1,541	75.0%	19.2 min

The baseline comparison stands — the pool is a large improvement on the 2026-08-05 single-agent figures (median 8.7 min, p90 24.1 min, 60% queueing), and the daily attainment table shows real days above 90%. What does not stand is the claim that the 95% objective was achieved. It was not, on any representative window.

Two failures compounded. The sample was unqualified, which ADR-0396 now forbids. And the burn alerts that existed to catch the drift could not fire — their windows were shorter than the interval at which their source counter advances, so the SLO read as met and the alerts stayed silent, agreeing with each other for the life of the Epic (FreeScout incident #1758). Neither the SLI nor the instrumentation was wrong; the claim made from them was.

The row above on ansible-ci’s run count (#2215) inherits the same correction: its “attainment moved from 71.1% to 100%” is the same 94-run sample. The direction of that finding is unaffected — the queued-vs-work ratio improvement is measured independently — but the attainment figure in it should be read as 80.9% over 1,017 runs, not 100%.