ADR-0291 — Scheduled all-host deploys tolerate transient single-host unreachability

Status: Accepted Date: 2026-08-01 Epic / Issue: Epic #2107 (Alert false-page triage & hardening) / Issue #2115

Context

On 2026-08-01 the ansible-exim4 scheduled deploy (07:30 AEST) failed and raised a Zabbix High ADO pipeline failed trigger + a FreeScout incident (#1405). Root cause: xt035 (PBS) was SSH connection-refused for ~90s during an ad-hoc reboot. The deploy succeeded on all 13 other hosts — only xt035 was unreachable — but Ansible exits 4 whenever any host is unreachable, so the whole run went red.

This is not a one-off. ansible.cfg carried no connection-resilience posture (no retries, no wait_for_connection, no SSH keepalives, timeout at the 10s default), so a single host down for the length of a reboot (any host, any cause) hard-fails an entire scheduled all-host deploy. 17 scheduled always-run deploy pipelines shared this brittleness — it is the root of the recurring “ADO pipeline failed/stuck” FreeScout backlog.

Two facts reframe the fix:

1. A per-pipeline failure does not page. The Zabbix ADO Pipeline Failures → Pushover action is deliberately disabled (Epic #1564); a failed run surfaces as a red High trigger + a FreeScout incident (triage toil), not a Pushover page. The harm is noise, not a 07:30 wake-up.
2. Genuine host-down is already detected by the right instrument. Every host carries the Zabbix Linux: Zabbix agent is not available trigger (Average). A scheduled deploy is a poor outage detector — it runs only on a schedule and reports a confusing “pipeline failed” — so it must not be relied on for that. Its job is to converge reachable hosts; a host briefly away will converge on the next scheduled run.

The design tension named in the issue: tolerate-transient vs detect-genuine-outage. The fix must stop a transient reboot reddening the run, without silently masking a host that is genuinely gone.

Decision

Two complementary layers, plus an explicit non-goal.

Layer 1 — estate-wide connection resilience in ansible.cfg (every playbook, zero churn). [ssh_connection] retries = 3, ssh_args adds ServerAliveInterval=15 ServerAliveCountMax=3 (keepalives that survive a brief mid-task network drop) while preserving ControlMaster/Persist multiplexing, and [defaults] timeout = 30 (up from 10). This absorbs brief SSH blips and half-open connections across the whole estate. It does not by itself span a full reboot: retries re-attempt an SSH error (rc 255) back-to-back within seconds, and a connection-refused fails instantly — too fast to cover a ~90s reboot.

Layer 2 — a wait_for_connection pre-task on the all-host deploys (spans a full reboot). The shared include playbooks/tasks/wait_for_reachable.yml polls each host’s SSH transport every 5s up to 120s before the play’s real work, so a host rebooting mid-run is waited-for and the run passes. It is imported as the first pre_task on the multi-host scheduled deploys — configure_exim4, configure_timezone, configure_users, configure_dns_pinning, deploy_alloy, deploy_zabbix_agent — each of which sets gather_facts: false so the implicit fact-gather (which would fail-first on the absent host, before the wait) is deferred; playbooks that need facts run an explicit ansible.builtin.setup: immediately after the wait.

Detection is preserved, not masked. A host still unreachable after 120s fails the play exactly as before (exit 4 → red run) — that is now a genuine outage, and it is independently paged by the Zabbix agent-availability trigger. Tolerate-transient, still-detect-outage.

Scope — multi-host deploys only. Single-target deploys (sync_pve01, sync_docker01, enforce_vm_rtc_utc, configure_pve_patching, configure_pihole_tls) get the Layer-1 baseline but not the wait_for_connection pre-task: for a sole-target deploy, that target being unreachable is the deploy being un-runnable — close to a genuine unavailability rather than the “one of many” transient the pre-task exists to absorb. sync_control01 runs on the controller itself (a reboot kills the run, not a task). Localhost validators (validate_*, *_reconciliation) have no remote-SSH surface.

Alternatives considered

• ansible.cfg retries only. Cheapest and truly zero-churn, but does not span a full reboot (the exact incident). Kept as Layer 1, insufficient alone — hence Layer 2.
• Reclassify the alert to page only after N consecutive failures. Rejected. Per-pipeline failures already don’t page; the noise is the red trigger + FreeScout incident, which this would not remove (the run still reds). It also needs a new run-granular Zabbix item (the current ado.pipeline.result item stores the last completed run’s text result and is re-polled every 5m, so count() would count poll samples, not distinct runs — it cannot express “N consecutive runs”). More build, treats the symptom not the cause.
• ignore_unreachable estate-wide. Rejected — it masks a genuine outage entirely from the deploy path, and there is no ansible.cfg global for it anyway.

Consequences

• A transient single-host reboot during a scheduled all-host deploy no longer reds the pipeline or raises a FreeScout incident; the recurring “ADO pipeline failed” backlog loses its root cause.
• A genuinely-down host still fails the run (after the 120s wait) and is paged by Zabbix agent-availability — the outage is detected by the correct instrument, not lost.
• Cost: a genuinely-down host adds up to ~120s (plus the retry budget) to that play before it fails — an acceptable trade for killing the false-red class. Healthy hosts connect on the first poll, so steady-state deploys are unaffected (proven: wait_for_connection returns immediately for a reachable host).
• This ADR is the standard-of-record for the pattern: a multi-host scheduled deploy imports tasks/wait_for_reachable.yml as its first pre_task with gather_facts: false. New scheduled all-host deploys follow the same shape.

Proof (PD12)

• ansible-lint passes at the production profile on the include + all six edited playbooks; --syntax-check passes on all six. (Quoting the pre_task name that contained #2115 also fixed a latent YAML-comment truncation of the task name.)
• The live ssh command line for a real host carries -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o ControlPersist=60s -o ConnectTimeout=30, and -m ping returns pong — the new ssh_args apply and connectivity (with multiplexing) is intact.
• wait_for_connection tolerance was demonstrated against a real SSH endpoint made to start connection-refused and then become reachable: with the endpoint down it polled the full window then failed unreachable (genuine-outage branch); with the endpoint reachable it connected, the deferred setup gathered facts, and the play succeeded (ok, hostname resolved) — the exact gather_facts:false → wait → setup shape used by the fact-dependent playbooks.