ADR-0330: Terraform state and CI cross-host artifacts live on a bind-mounted ZFS dataset, and same-target pipeline runs serialise on it

Status: Accepted Date: 2026-08-07

Context

ADR-0329 and the CI runner pool design move CI off control01 onto two purpose-built runners, runner01 and runner02, so the second purchased parallel job actually buys concurrency. The agents were built, registered and proven — ansible-ci run 7454 executed entirely on them with control01 disabled, two jobs overlapping on distinct agent ids — and then deliberately left disabled in the pool, because concurrency was only half the problem.

The other half is that a second agent invalidates an assumption three separate mechanisms had quietly baked in: that there is exactly one machine, and its local disk is a safe place to keep shared truth.

Terraform state existed on one agent’s disk, and was gitignored

All six tf-* roots — tf-proxmox/lxc, tf-proxmox/vms, tf-azure/entra, tf-azure/storage, tf-backblaze and tf-cloudflare — used the implicit local backend beside their .tf files (or, for tf-cloudflare, an absolute path under control01’s home, ADR-0266). Every one of those paths is gitignored, so a runner’s fresh clone contains no state at all.

That is not a degraded mode, it is a destructive one. terraform plan against an empty state does not fail; it proposes to create everything the root already owns. And tf-proxmox’s lxc pipeline gates on HAS_DESTROY=true: a plan that is pure creation carries no destroys, so it would have skipped the ManualValidation stage and auto-applied all nine containers with no human in the loop. Verified before any change: runner01 had no tf-proxmox/lxc/terraform.tfstate; control01 did.

Two more mechanisms encoded the same single-agent assumption in different words

• Pipeline liveness heartbeats. eit_reconciliation.yml and standards_enforcement_reconciliation.yml each ended by writing /var/lib/pitlab/<check>.last on the agent’s own disk, watched by a Zabbix <check>.age dead-man. With two runners the stamp lands on whichever host won the job, so each host’s copy is stale half the time and both dead-men flap.
• Prometheus textfile metrics. Three pipeline steps — standards_enforcement_reconcile.prom, standards_enforcement_ci.prom, script_observability_ci.prom — write into /var/lib/pitlab/restore-drill/metrics, which is control01’s Alloy textfile-collector directory and the only exporter of them. On a runner those files would land on the runner and control01 would export nothing. This one fails silently: no red run, no alert — the metrics simply stop advancing, and ADR-0180’s synthetic probe_success recording rule goes with them.

Concurrency introduces a hazard that concurrency 1 made impossible

At one agent it was structurally impossible for two pipelines to touch one host at the same time. At two it is routine: 16 pipelines target docker01 and 14 target pve01. Two runs can interleave apt/dpkg, write the same config file with last-writer-wins (both runs green, the estate silently wrong), or restart a service out from under the other’s health wait. Terraform’s own state lock covers the tf-* roots; nothing covered the Ansible ones.

The fleet does not allocate pit a consistent uid

Measured across the estate: pit is uid 1000 on control01 and urbackup01, 1001 on runner01/runner02/zabbix/plex01/dispatcharr01/xt035/iventoy01, and 1002 on docker01 — and on the runners ansible holds 1000, i.e. the two accounts are swapped relative to control01. These are unprivileged LXCs, so container uid N is host uid 100000+N. Any shared filesystem between control01 and a runner therefore sees the same logical account under two different host uids, and a file written by one is owned by the other account’s uid as seen from the other.

Decision

State and cross-host CI artifacts live on one ZFS dataset, bind-mounted into every CI guest

rpool/data/ci-shared on pve01, with three bind mounts into CT 121 (control01), CT 122 (runner01) and CT 123 (runner02):

Mount	Contents	Mode
/srv/tfstate	the six tf-* roots’ state, each pinned by an explicit backend "local" { path }	2770
/srv/ci-locks	the target_lock serialisation lockfiles	2770
/var/lib/pitlab/restore-drill/metrics	control01’s Alloy textfile-collector directory	2775

This is the house pattern for shared filesystems between LXCs — plex01/CT 115’s mp0 + mp10. mount_point sits in tf-proxmox/lxc/lxc.tf’s ignore_changes, so the mounts are Ansible-applied, not Terraform-managed (ADR-0047); playbooks/configure_ci_shared_state.yml owns them and is what a container recreate re-runs. pct set hotplugs a bind mount into a running container, so no guest is restarted — which matters because control01 hosts the cc-pool Claude sessions.

control01 keeps all three mounts even though it runs no agent: it retains the interactive and manual Terraform path, and its Alloy is the single exporter of the textfile metrics.

Access is granted by a pinned gid and a default POSIX ACL, not by file ownership

A group citf with the fixed gid 3000 exists on all three guests with pit as a member, and the directories are setgid. That alone is not enough — a process with the usual umask 022 creates 0644, which is not group-writable, and Terraform rewrites its state file in place, so the second host to run would fail. acltype=posixacl on the dataset plus a default:group:103000:rwx ACL fixes it at the filesystem layer: a file created under umask 022 lands 0660 owned by group citf. acl=1 on the pct mountpoint is load-bearing — without it the container mounts without ACL support and the default ACLs are inert.

metrics is 2775/other-readable rather than 2770 because Alloy runs as the alloy user, not as pit. tfstate stays 2770: state carries resource attributes that are effectively secrets.

The local backend is retained

Not for want of alternatives — because its lock genuinely works here. flock was proven to block across two containers sharing this bind mount on the same kernel, so a concurrent terraform apply from two runners fails with Error acquiring the state lock rather than corrupting state. A remote backend remains the tracked ADR-0080 target for off-premises durability.

The dataset carries its own backup, because vzdump cannot back up bind mounts

vzdump skips bind mounts. Moving state off control01’s rootfs therefore removed it from the 01:30 nightly guest backup that covered it — which would have made this cutover a business-continuity regression dressed as an improvement. Two layers replace it:

• Local, fast — com.sun:auto-snapshot=true puts the dataset in the estate’s existing zfs-auto-snapshot schedule (frequent every 15 min keep 4, daily keep 31), for “Terraform ate the state five minutes ago”.
• Off-guest, durable — ci_shared_state_backup.sh on pve01, 12:10 AEST daily, snapshots the dataset and ships a client-encrypted host/ci-shared-state to pbs-data-local, riding the datastore’s existing prune (keep-daily 7 / weekly 4 / monthly 3, no group filter) and the 06:00 B2 offsite sync.

The job restore-verifies every run: it pulls its own snapshot back and asserts each restored terraform.tfstate parses with a lineage. It also fails loudly when it finds no state file, because a valid backup of an empty directory is the “absence of errors is not proof of function” trap in its purest form. Dead-man: the pve01 Zabbix staleness trigger on /var/lib/pitlab/ci-shared-state-backup.last (>28 h), stamped only on a full PASS. It uses the host-playbook vfs.file.time shape rather than a <check>.age key because zabbix_cron_heartbeats_monitoring.py — the only registry the standards meta-gate could read — creates its items on the control01 Zabbix host and so is structurally incapable of describing a dead-man for a job on any other host. That blind spot made every scheduled machine control outside control01 undeclarable; the gate now indexes host-playbook heartbeats too, proven red against a heartbeat no playbook declares.

Same-target pipeline runs serialise through target_lock, and a gate watches the wrapper

target_lock (/usr/local/bin/target_lock, deployed to every pool agent by roles/ci_agent and to control01 by sync_control01.yml) wraps a command, derives the target hosts from the playbook’s own hosts: expanded against the static inventory, and holds one flock per host on /srv/ci-locks for the command’s lifetime. Same target serialises; different targets stay fully concurrent, which is what preserves the capacity this Epic bought.

Three properties are deliberate:

• Targets are derived, not declared. A declared constraint is one a new pipeline can silently omit; a derived one cannot go stale against the playbook it guards. --limit narrows the set exactly as Ansible does.
• Locks are acquired in sorted order. Two runs with partially overlapping target sets would deadlock if each grabbed its locks in its own order.
• It fails open, loudly. An unparseable playbook runs unlocked with a warning rather than failing, because breaking the estate’s CI over a parser bug in a serialisation helper is worse than the interleaving it prevents.

That fail-open branch is a hole, so validate_target_lock_coverage.py closes it statically and runs blocking in ansible-ci. It asserts three things, each a distinct way the control can silently die: every runnable ansible-playbook invocation across all nine pipeline-bearing repos is wrapped; every wrapped playbook resolves to at least one target; and the wrapper is declared for deployment to every agent. It exits 2 UNVERIFIED — never a clean pass — if a repo it must scan is unreadable.

Heartbeats and metrics stop being agent-local

The two reconciliation pipelines no longer stamp /var/lib/pitlab/<check>.last; ado_ci_reconciler.py derives both from each pipeline’s last succeeded run finishTime via the ADO API (ADR-0329). The .prom files need no pipeline change at all — their directory is now shared, so whichever agent writes, control01 exports.

Alternatives considered

Alternative	Why not
Remote backend (azurerm / B2 / S3)	The right long-term answer for off-premises durability, and it would close the off-site gap this ADR only mitigates. Rejected for now because it drags a six-root state migration into a concurrency project and over-solves the actual blocker. Stays tracked under ADR-0080.
Self-hosted MinIO for state	Circular dependency: if docker01 is down and Terraform is needed to fix it, the state is unreachable.
Replicate state onto each runner	Two divergent copies of the one artifact whose whole value is being singular.
Align the fleet’s uids so ownership works	The deeper fix, and a real finding — but it is a fleet-wide migration touching every host and every home directory, and emphatically not something to do inside a CI cutover. Raised separately; the pinned-gid + ACL scheme is uid-agnostic and unblocks this work today.
Leave state on control01 and pin the tf pipelines to it via demands:	Re-creates the single-agent bottleneck for the pipelines most likely to need the second slot, and a declared demand is easy to omit on a new pipeline with nothing to catch it.
umask 002 in every pipeline that touches the dataset	Ninety-odd call sites, each of which must remember. The default ACL enforces the same outcome once, at the filesystem, and cannot be forgotten.
ADO environment exclusive-lock for same-target serialisation	Platform-native and visible in the UI, but per-pipeline, declared rather than derived, and blind to interactive runs from control01.
Accept the same-target hazard and measure it	The failure mode is a silently wrong deploy with two green runs — the least detectable class of defect in the estate.
A PATH shim shadowing ansible-playbook	Would make bypass structurally impossible, which is genuinely attractive. Rejected because it changes what a universally-understood command means, invisibly, in a way no reader of the pipeline YAML could see — and the coverage gate gets the same guarantee while staying legible.

Consequences

• A runner can no longer plan against an empty state. Proven: after migration all six roots plan to No changes, both locally and from CI — tf-proxmox lxc/vms, tf-azure entra/storage, tf-backblaze and tf-cloudflare runs all report it in their published plan artifacts.
• Terraform state gains explicit, restore-tested backup coverage it never had while it was buried inside a guest image, and the restore is exercised daily rather than during an incident.
• Same-target runs serialise across containers. Proven experimentally: with control01 holding the docker01 lock, runner01 blocked and named the holder; a different target ran concurrently and returned 0.
• The state remains on-premises on pve01. A total-site loss still loses it between PBS B2 syncs. This is the residual risk the remote backend would close.
• The lock is advisory. A pipeline that bypasses target_lock is not serialised — which is exactly why the coverage gate ships with it rather than after it.
• target_lock adds a lock acquisition to ~119 pipeline invocations. Where targets overlap, wall-clock goes up by the wait; that is the intended trade and it is visible in the queue-time SLI the Epic already stands up.
• The citf gid is now load-bearing estate state. A host that allocated its own gid for that name would silently lose access to everything the others write; configure_ci_shared_state.yml pins it and proves it with a live write probe on every run.

Four more single-agent assumptions the cutover surfaced, each found live

The three instances in the Context above were the ones design analysis predicted. Enabling the runners and watching real pipelines land on them found four more, and the pattern is worth stating plainly: every one of them had been correct for years and became wrong the instant a second agent existed. None was visible to static review; each needed a real run on a host that was not control01.

What broke	Why it had always worked	How it presented
Provider versions diverged. .terraform.lock.hcl was gitignored in tf-proxmox, against this estate’s own IaC Standard tracked-artifact clause.	One agent resolved the providers once and kept them.	Same commit, same state, same ~> 0.78 constraint: control01 held bpg/proxmox 0.99.0 + netbox 5.6.2, a freshly-init’d runner01 resolved 0.111.1 + 5.7.0. control01 planned No changes; the runner planned to update cpu.units in place on CT 125 and 126. Which agent won the job decided what CI proposed to do to production. Fixed by committing the lock in every tf-* repo.
The saved plan travelled on the agent’s disk. Plan wrote <root>/tfplan; Apply read the same absolute path.	Both jobs always landed on the only agent.	Run 7488 planned on runner01, applied on control01, read control01’s leftover plan and died with Saved plan is stale. It failed safe only because the state serial had moved on — with a matching serial it would have applied a plan computed by a different run, silently. Fixed by publishing/downloading the plan as a pipeline artifact in all five pipelines (the shape tf-cloudflare has used since ADR-0266), plus a terraform init in Apply, which never existed because Apply always inherited the planning agent’s .terraform/.
The runners had no ansible SSH key. The role deployed control01’s pit key and proved reachability with docker version.	docker version goes over the pit key via ~/.ssh/config; every playbook goes over id_ed25519_ansible as remote_user=ansible.	The role’s own PD12 probe was green on a runner that could not execute any of the ~120 playbook invocations in the estate — an assertion satisfiable without the feature it claimed to test. The first real playbook to land on runner01 died on no such identity. Fixed, and the probe replaced with one that runs an actual play (ansible … docker01 -m ping).
The runners did not trust the internal CA.	configure_ca_trust.yml is fleet-scoped, but only runs when its own paths change — so a host created between runs has no root until something unrelated triggers it.	Every Infisical secret fetch failed with unable to get local issuer certificate, which is most pipelines. The deeper finding is in onboarding: the tf-proxmox “Run Base Playbooks” stage never installed the CA root at all, and was still invoking configure_unattended_upgrades.yml — deleted in July by ADR-0186 — so the first real container creation would have failed that stage outright.

Two smaller ones round it out. .prom textfile metrics written by pipelines were agent-local (covered above, and the reason the metrics directory is a shared mount). And a pool enabled: false park is not durable: re-applying roles/ci_agent restarted both agents and they came back enabled, which was noticed only because the queue began draining to them. Treat the pool disable as a courtesy brake — the real control is that a runner is safe to receive work.

The generalisable lesson is narrower than “test on both agents”. It is: an assumption that a single machine holds shared state does not announce itself, because for as long as the assumption holds every check passes. The only thing that surfaced these was running the real work on a host that had never run it.

Standards gaps this exposed

• The <check>.age dead-man pattern encoded a single-agent assumption, documented verbatim as “the pitlab ADO agent runs as pit on control01, which owns /var/lib/pitlab”. Pipeline-fed liveness must derive from the pipeline system, never from agent-local filesystem state. Closed here and in ADR-0329.

• The same assumption had a third instance nobody had named — pipeline-written Prometheus textfile metrics read by one host’s collector. The gap is not “heartbeats”; it is any pipeline artifact consumed off-agent. The IaC & Deployment Standard should state it that way.

• No standard required Terraform state to be reachable by every runner permitted to apply it, or to carry explicit backup coverage. Local, gitignored state on a single agent was invisible until it nearly broke this work, and it was simultaneously a business-continuity gap.

• The fleet has no uid-allocation standard. pit holds three different uids across the estate and is swapped with ansible on the runners. Nothing depends on it today beyond this dataset, but any future shared filesystem will hit the same wall.

• 2026-08-07 — target_lock derived NO lock for a whole class of play (Issue #2216, ADR-0337). Targets are derived from the playbook’s hosts:, and localhost is a non-target because it names a different machine on each agent. So a play that is hosts: localhost and reaches its real target over raw ssh — every HAOS playbook — was wrapped in target_lock -- and serialised nothing, while looking correctly covered. Not hypothetical: the two HAOS deploy pipelines started 7 seconds apart on runner01 and runner02 on 2026-08-07. Such a play must now name its target explicitly (target_lock -t homeassistant --). The coverage gate additionally scanned only <repo>/pipelines/*.yml, so hass-config’s repo-root azure-pipelines.yml was never opened — it reported 121/121 wrapped while the estate’s one genuinely unwrapped invocation sat in a file it never read. Both layouts are now enumerated (122/122).