0337 — The HAOS deploy pipelines are made peers of the pool, not pinned to control01 — and pipelines gain a portability class

Two Home Assistant deploy pipelines — hass-config (definition 20) and ansible-hass-config (56) — passed on control01 and failed on runner01/runner02 from the moment the runner pool took over CI. Whether a Home Assistant config change reached the house depended on which agent won the queue. This ADR records the decision to dissolve the coupling rather than pin around it, and adds the Pipeline Agent-Portability section to the CI/CD & Pipelines Standard so the next pipeline of this shape is caught by a gate rather than by an incident.

Status

Accepted — 2026-08-07. Delivered under Epic #2187 (CI runner pool), Issue #2216. Adds the Pipeline Agent-Portability section to the CI/CD & Pipelines Standard, enforced by scripts/validate_pipeline_agent_portability.py. Extends ADR-0333 (the agent-plural rule, which governs scripts) and ADR-0334 (pool parity, which governs agents); this ADR governs pipelines. Resolves incidents #1578 (ansible-hass-config failing) and #1579 (hass-config failing).

Context — the failure, and the two things everyone got wrong about it

playbooks/hass_config.yml reaches HAOS by raw ssh on port 22222 with StrictHostKeyChecking=yes, using haos_key: "~/.ssh/id_ed25519_ansible". Run 7451 on control01 succeeded; runs 7576 (runner01) and 7577 (runner02) failed:

Warning: Identity file /root/.ssh/id_ed25519_ansible not accessible: No such file or directory.
No ED25519 host key is known for [homeassistant]:22222 and you have requested strict checking.
Host key verification failed.

Investigation contradicted the two assumptions in the issue as raised, and both corrections changed the decision:

1. The runners were not missing the key, and were not running as root by configuration. The ADO agent runs as pit on both runners (User=pit in the unit), roles/ci_agent had already deployed id_ed25519_ansible there, and the key is byte-identical across control01, runner01 and runner02 (md5 f9d72571…). The runners already had the checkouts too — hass-config is in ci_agent_repos.

2. The /root in the error is Ansible escalating, not the agent running as root. ~ in an ssh -i argument is expanded by ssh itself from the effective uid. inventory/group_vars/all/vars.yml sets ansible_become: true, and a connection variable overrides a play keyword — so hass_config.yml’s become: false is ignored and every task runs as root. Verified directly on runner01: a play with become: false reports uid=0 HOME=/root.

So the deploy had silently depended on control01’s root profile, hand-accreted years earlier: a HAOS entry in /root/.ssh/known_hosts, and a default identity /root/.ssh/id_ed25519 that HAOS happened to accept. The declared key was never used on control01 either — -i pointed at a file root could not read, which is only a warning, and ssh fell back. The pipeline that “worked” had been broken in a way that produced green ticks.

The genuine gap on the runners was exactly one thing: no HAOS host key. ci_agent_known_hosts listed ten fleet hosts and scanned them in a single ssh-keyscan with no -p, so a host on 22222 could never be recorded — and a keyscan that records nothing still exits 0.

Decision

1. Peer, not pin — the coupling is dissolvable, so dissolve it

Option 1 was to pin both pipelines to control01 with a pool demands:. Rejected, on three grounds:

• The premise was false. Pinning was argued as “honest about inherent coupling”, but the coupling was not inherent: the runners already had the key, the checkouts and the identity. The residue was one missing host key on one non-default port — a 12-line role change, not an architecture.
• It would enshrine a latent bug. Pinning makes “the deploy works because one host’s /root/.ssh was populated by hand in 2026” load-bearing forever, and keeps the silent -i-fallback that meant the declared credential was never exercised. A pin freezes the accident.
• The design rejected demands: estate-wide, and the reasoning still holds here. The runner-pool design rejected a capability/demands: scheme because “a constraint that is declared rather than derived is easy to omit on a new pipeline with nothing to catch it”. That objection is about unenforced declarations. It would not have been fatal here — this ADR adds the gate that makes a demands: block enforceable — but with the coupling dissolvable in a dozen lines, the question of whether to pin well never arises. Pinning stays available as a declared class for a coupling that genuinely cannot be dissolved; this was not one.

Pinning also costs capacity permanently: hass-config is one of the estate’s more frequently-triggered pipelines, and binding it to a single agent hands back part of what the second parallel job was bought for.

2. Anchor credential paths to the controller’s home, never to ~

~ is unusable in this estate’s playbooks because escalation makes it /root. Every HAOS-touching playbook now resolves paths from {{ lookup('env', 'HOME') }} — evaluated on the controller, before any escalation, so it is the agent user’s home on whichever agent took the job, and readable by root as well as by pit. IdentitiesOnly=yes and an explicit UserKnownHostsFile are part of the form: without them an unreadable key is a warning and ssh silently falls back, which is precisely how the defect stayed invisible.

Applied to all four HAOS playbooks — hass_config.yml, deploy_haos_loki.yml, update_homeassistant.yml, configure_ha_nginx_proxy.yml — not just the two that failed. Two of the four have no pipeline today; fixing only the failing pair would leave the same defect waiting for the day they get one.

3. ci_agent_known_hosts entries carry a port, and HAOS reachability is probed

Entries become {host, port} with port defaulting to 22, and the scan runs one ssh-keyscan -p per host. [homeassistant]:22222 joins ci_agent_known_hosts_required, asserted with ssh-keygen -F after the scan. A new role probe performs the actual deploy hop as the agent user — the same absolute paths, port, strict checking and IdentitiesOnly the playbook uses. The existing probes could not cover it: the docker01 probe goes over the pit key via ~/.ssh/config, and the ansible -m ping probe goes over the ansible key to an inventory host on port 22. HAOS is none of those.

4. Every pipeline declares a portability class, enforced by a gate

The standard’s rule: a pipeline is portable (no host-local credential, checkout or path dependency) or pinned (demands: plus a # PINNED: <why> comment). No third class. scripts/validate_pipeline_agent_portability.py enforces it in ansible-ci, reading ci_agent_repos and ci_agent_user from the role defaults so the gate and the role cannot drift.

The gate resolves variable indirection to a fixed point, and that is the load-bearing part. The first implementation checked lines, passed the self-tests, and reported the estate clean — against the actual pre-fix tree. The real defect was split across two lines and one hop:

haos_key: "~/.ssh/id_ed25519_ansible"
haos_ssh: "ssh -i {{ haos_key }} -p 22222 ... root@homeassistant"

Neither line is a finding alone: the first has no ssh, the second has no ~. A gate that cannot see the failure it was written for is not a gate. After taint-propagation it flags all three broken pipeline→playbook pairs by name.

5. Two adjacent defects the same delivery exposed, fixed here

• validate_target_lock_coverage.py scanned only <repo>/pipelines/*.yml. hass-config’s pipeline is a repo-root azure-pipelines.yml — ADO’s own default layout — so the gate reported “121/121 wrapped” while the estate’s one genuinely unwrapped ansible-playbook invocation sat in a file it never opened. Both gates now enumerate both layouts; the count is 122/122.
• target_lock derived no lock for these runs at all. It reads the playbook’s hosts:, and every HAOS playbook is hosts: localhost reaching HAOS over raw ssh. localhost is a non-target — it names a different machine on each agent — so target_lock -- here locked nothing. Both HAOS pipelines now name the target explicitly (target_lock -t homeassistant --). This mattered on the day: runs 7576 and 7577 started 7 seconds apart on the two runners, both deploying to the same HAOS.

6. The hass-config pipeline’s docs commit-back step is removed

It regenerated ansible docs from a hass-config pipeline using the git stash --include-untracked / git pull --rebase / git stash pop || true reconcile the Host-Playbook CI Standard explicitly bans on a shared mirror. It was also pure duplication: every push to hass-config also triggers ansible-hass-config through its resources.repositories trigger, and that pipeline already regenerates the same docs through the conforming shared template (ADR-0237). Two agents running git push against the same ~/ansible mirror seconds apart bought nothing. The same pipeline’s git pull mirror refreshes become fetch + reset --hard, per the shared-mirror convergence rule.

Alternatives considered

• Pin both pipelines to control01 with a pool demands:. Rejected — see decision 1. Available as a declared class for a coupling that cannot be dissolved.
• Copy the ansible key and HAOS known_hosts into /root/.ssh on both runners. Rejected: it makes the escalation accident permanent, puts a deploy credential into the root profile of two hosts that execute arbitrary pipeline code, and leaves ~ meaning different things on different code paths. Anchoring the path costs the same and removes the ambiguity.
• Force the play to run unescalated by setting ansible_become: false in play vars. Rejected as the primary fix: it changes the effective user of a working production deploy for a benefit that absolute paths deliver without the blast radius, and it would leave the same trap for the next play that forgets. Absolute paths are correct under either user — root can read the agent user’s .ssh, and so can the agent user.
• Blanket StrictHostKeyChecking=no for HAOS. Rejected: a permanent accept-anything setting on hosts that execute arbitrary pipeline code, to avoid declaring one host key. The role already does one-time trust-on-first-use inside the private VLAN, which is the same trade the fleet keyscan makes.
• Keep the portability rule as review-only. Rejected on this estate’s own evidence: “a declared control with no conformance check decays silently” (ADR-0334). A review-only portability rule would have caught none of the four playbooks, because every one of them looked normal.

Consequences

• Both HAOS deploy pipelines are portable across the whole pool; the capacity the second parallel job bought is preserved, and a runner03 inherits the property with no per-pipeline work.
• The estate has a name for a class of pipeline it previously had no name for. Portable-versus-pinned is now a decision someone makes and records, not a property that emerges from whichever host happened to run CI first.
• The gate is proven red two ways — ten synthetic cases via --self-test, and a recorded red run against the real pre-fix tree naming all three broken pairs — as the Build Hook & Pipeline-Gate Test Standard (ADR-0141) requires.
• A previously invisible pipeline layout is now scanned by two gates, and the estate’s one unwrapped playbook invocation is wrapped.
• Two pipelines that deploy to the same Home Assistant on the same trigger are genuinely serialised for the first time. They remain duplicates, which is a run-economy question rather than a correctness one, and is recorded against Issue #2215 rather than resolved here.
• Residual risk, explicit: the gate reasons about ssh invocations and cd targets it can see in a pipeline YAML or in a playbook that pipeline names. A dependency reached through a deployed script the pipeline calls is governed by the agent-plural rule instead, and a dependency constructed at runtime from a fact is invisible to both.
• The SRE framing is eliminating a class of nondeterminism, not fixing an outage: the defect’s cost was never a red run, it was that the red run and the green run came from the same commit.