0333 — The CI pool is plural — gate-script location and pipeline-fed liveness stop assuming one agent
For years the pitlab pool was one self-hosted ADO agent on control01, and a great deal of the estate’s reasoning was built on that without ever naming it as an assumption. ADR-0330 made the pool runner01 + runner02 and left control01 running no agent at all. This ADR amends the two standards whose rules were silently premised on the old topology — the Pipeline-Gate Script Location Standard (where a pipeline-invoked script lives, and which agents must be able to run it) and the Standard-Enforcement Standard (where a scheduled check’s liveness comes from) — and records the corpus treatment: what was corrected in place, what was amended rather than rewritten, and what was deliberately left alone.
Status
Section titled “Status”Accepted — 2026-08-07. Delivered under Epic #2187 (CI runner pool), Issue #2207. Follows ADR-0329 and ADR-0330. Amends the Pipeline-Gate Script Location Standard and the Standard-Enforcement Standard. Companion to ADR-0334, which adds the pool-composition and parity clause.
Context — a premise nobody wrote down is a premise nobody re-checks
Section titled “Context — a premise nobody wrote down is a premise nobody re-checks”The costly property of the single-agent assumption was not that it was wrong. It was load-bearing and unstated. Documents did not say “assuming one agent, therefore X”; they said “X”, because at the time X simply was the case. So when the pool grew, nothing flagged. A measured sweep on 2026-08-06 over 776 docs found:
| Set | Count |
|---|---|
Docs asserting the ADO agent runs on control01 | 20 |
| Docs encoding the same assumption in different words (“single serial worker”, “one agent”) | 25 |
| Combined must-fix | 34 |
Docs mentioning control01 at all | 286 |
The methodological finding matters more than the counts. The first sweep searched the phrasing and found 20; searching the assumption found 25 more. A document is in scope because of what it assumes, not the words it uses — so a corpus sweep after a topology change must be run against the belief, not the string.
Decision
Section titled “Decision”1. A pipeline-invoked script runs repo-relative, or is deployed to EVERY agent — never to one
Section titled “1. A pipeline-invoked script runs repo-relative, or is deployed to EVERY agent — never to one”The Pipeline-Gate Script Location Standard branched on two cases: a docker-stacks pipeline gate runs repo-relative versus a host cron/timer script lives in ansible/scripts/ and deploys to /usr/local/bin. An ansible-repo pipeline gate matches neither. It fell through to the /usr/local/bin branch and was thereby pinned to one provisioned agent — invisibly, because with one agent “deployed to control01” and “present wherever the job lands” were the same sentence.
The amendment generalises the first branch to any pipeline-invoked script and adds the binding rule: run repo-relative from the run’s own checkout, or deploy to every agent in the pool. Never to one. secret_scan.sh was the live instance and is now invoked as "$(Build.SourcesDirectory)/scripts/secret_scan.sh".
Repo-relative is preferred for a second reason independent of agent count: a deployed copy makes CI test new commits with the old script. checkout: self gives a fresh tree at the run’s sourceVersion, but the deployed script is whatever was last synced to a host — so a commit that improves a gate was never exercised by itself, and a commit that broke one passed CI and bit at the next sync. That was a live correctness bug in secret_scan.sh, not a hypothetical.
2. Pipeline-fed liveness derives from the pipeline system, never from agent-local filesystem state
Section titled “2. Pipeline-fed liveness derives from the pipeline system, never from agent-local filesystem state”The <check>.age dead-man pattern says: the check stamps /var/lib/pitlab/<check>.last, and Zabbix watches that file’s mtime. Correct for a host cron. Wrong for anything a pipeline runs, because the stamp lands on whichever agent won the job — so each host’s copy is stale roughly half the time and the dead-man flaps between them, firing not because the check stopped running but because it ran somewhere else.
The Standard-Enforcement Reconciliation doc stated the assumption verbatim: “the pitlab ADO agent runs as pit on control01, which owns /var/lib/pitlab, so the stamp needs no privilege.” True for years; false the moment there were two runners.
The amendment: a pipeline-run check derives liveness from the pipeline system — ado_ci_reconciler.py on control01 polls the ADO API for the last succeeded run and stamps from its finishTime (ADR-0329). A host cron/timer check still stamps locally, because it genuinely runs on one host. Both shapes are legitimate; naming the wrong one is the failure.
This is host-agnostic by construction at any agent count, so it does not break again when a runner03 appears — and it measures the outcome (did the run succeed) rather than a side effect on one box.
3. A dead-man may be registered on any host, not only control01
Section titled “3. A dead-man may be registered on any host, not only control01”zabbix_cron_heartbeats_monitoring.py creates its items on the control01 Zabbix host, so a scheduled control running anywhere else structurally could not declare a dead-man — and therefore could not be declared conforming at all. That is a limit of one registry, not a property of the pattern. Two registration shapes are now first-class and standards_enforcement_gate.py resolves both: the control01 JOBS table <check>.age key, and a host playbook’s vfs.file.time[/var/lib/pitlab/<check>.last,modify] item.
4. Corpus treatment differs by document class, deliberately
Section titled “4. Corpus treatment differs by document class, deliberately”- Operations, reference and standards pages describe current state and are corrected in place. A page that says something false about today is a defect, not a record.
- ADRs are historical records and are NEVER rewritten. An ADR captures what was decided, on what evidence, at a point in time — rewriting it destroys the only artefact that explains why. Affected ADRs instead gain a forward-pointing status admonition naming this ADR, exactly as ADR-0312 and ADR-0322 were handled at Issue #2202.
- Blog posts are a dated record and are left untouched. A post is a snapshot of a day; correcting it would be falsifying a diary.
5. The load-bearing ADRs keep their conclusions and lose their stated premise
Section titled “5. The load-bearing ADRs keep their conclusions and lose their stated premise”ADR-0311 (trigger economy), ADR-0312 (guard consolidation) and ADR-0322 (job economy) all reason from “the pool is a single serial worker”. Their conclusions still hold, because job and run count drive queue traversals independently of agent count — two agents drain the queue faster but do not remove a single boundary. Only the stated rationale needed amending, and ADR-0332 already made the stronger statement: job count is not a fixed cost, it is a multiplier on contention.
ADR-0235 is the one whose conclusion gets stronger. Its argument was “one agent on a RAM-constrained node, so a fleet play must be memory-bounded”. At two agents the implicit serialisation is gone: two heavy runs can now land on different runners concurrently, so a play sized against “the whole box” is sized against a box it no longer owns. Boundedness is more necessary, not less.
ADR-0231 and ADR-0234 hardened the agent unit on control01. That agent is gone; the hardening was carried forward, not lost — roles/ci_agent templates the same KillMode=control-group + OOMScoreAdjust=500 drop-in onto both runners, so the clean-reap standard is now applied twice over.
Alternatives considered
Section titled “Alternatives considered”- Rewrite the affected ADRs in place. Rejected: it destroys the historical record, which is the entire reason ADRs exist separately from standards. A reader six months from now needs to know the decision was made on a single-agent premise.
- Write one superseding ADR per affected original (ten-plus ADRs). Rejected as ceremony without information: the amendment is one change of premise with per-ADR consequences, so one ADR plus ten specific forward-pointers carries the same content and reads in one sitting.
- Leave the standards alone and just fix the instances (move
secret_scan.sh, re-home the one heartbeat). Rejected under Prime Directive 9: a one-off fix means the next occurrence repeats the reasoning from scratch. Both gaps were classes — the next ansible-repo gate, the next pipeline-fed dead-man — and a standard that silently under-covers a real idiom is the same failure as a missing one. - Add a
demands:capability scheme so pipelines could require the agent carrying a given script. Rejected in the design: it introduces a concept no pipeline uses today, and a constraint that is declared rather than derived is easy to omit on a new pipeline with nothing to catch the omission.
Consequences
Section titled “Consequences”- Two standards now state their agent-count assumption explicitly rather than encoding it, so the next topology change has something to check against.
- A pipeline-invoked script has one correct answer at any pool size; “it works today because we only have one agent” is no longer reachable by accident.
<check>.ageliveness for pipeline-run checks no longer depends on which agent won the job, and survives arunner03with no change.- Ten ADRs carry a forward-pointing note; none was rewritten, so the historical record is intact and a reader is routed to the current position in one hop.
- The 286
control01mentions are triaged, not exhaustively re-read. Most remain correct — control01 still exists and still runs the singleton orchestrator crons. A stale minor reference may survive; that is the accepted residual risk of triage over exhaustive review. - The sweep methodology itself is the durable lesson: search the assumption, not the phrasing. Recorded here because the next estate-wide premise change will need it.