ADR-0054: Coalesce the pitlab-docs publish pipeline with a batched CI trigger, and shift cheap validation left to a pre-push hook

This ADR records why the pitlab-docs CI build (pipeline 18) now uses trigger: batch: true to collapse a burst of pushes into the minimum number of runs, and why the cheap structural checks moved left to an opt-in pre-push git hook. It exists because under cc-pool’s five concurrent Claude Code sessions, a docs sweep fired one full end-to-end pipeline run per push, drained serially on a single self-hosted ADO agent — so wall-clock scaled linearly with the number of agents, even though every run republishes the same final state.

Status	Accepted
Date	2026-06-25
Epic	ADO #835 (Split the PLs further — pipeline latency)

Context

The pitlab-docs site is built by a single monolithic pipeline: pull every source repo → secret-scan → four generate_* steps → assemble the cross-repo tree → publish-completeness guard → Mermaid validation → mkdocs build → rsync deploy to docker01 → pit-memory reindex. The CI trigger was a plain branches: include: main — every push starts a full run.

cc-pool runs five concurrent sessions (cc-1…cc-5), each on its own worktree off shared main, each pushing to main to deploy. The ADO agent pool is self-hosted and effectively serial for this pipeline, and the individualCI trigger carries a ~3–4 minute registration latency before a run even starts. So a coordinated docs change across five sessions produced five full runs queued back-to-back — five registration taxes, five npm-install-mermaid spins, five full mkdocs builds, five pit-memory reindexes — and the last agent waited behind all of them.

The critical observation that unlocks the fix: the build is idempotent and last-write-wins. Step one git pulls the latest main of every source repo and the whole pipeline republishes the entire assembled tree — it is not a per-commit incremental build. So five queued runs each produce essentially the same /srv/docs as a single run executed last. Four of the five runs are redundant work, not independent units.

This is the classic CI anti-pattern that build coalescing (a.k.a. a merge train) exists to solve: when N triggers all converge on the same idempotent output, collapse them into one build. ADO supports this natively on the trigger.

Decision

• Coalesce with batch: true on the CI trigger. While a run is in progress, ADO holds every further push to main and starts one follow-up run covering all of them when the current run finishes — instead of queueing a run per push. A five-session burst collapses to ~2 runs (the in-flight one plus one batched follow-up), paying the registration tax and the build/reindex tail twice rather than five times. This is safe because the build is last-write-wins over all repos: a batched run sees every batched commit and can never drop one. The schedules: weekly rebuild and the source-pipeline resource triggers are unchanged.

• Shift the cheap checks left to an opt-in pre-push hook. Coalescing fixes throughput but widens the feedback gap — an agent would otherwise learn its diagram broke the build only when the next batched run finishes. The new .githooks/pre-push runs the two checks that are both fast and local: the gitleaks secret scan (defence in depth) and Mermaid syntax validation (the high-value catch — mkdocs build does not parse Mermaid, so a broken diagram deploys clean and only fails in the browser). Each session gets sub-minute local feedback; the coalesced run becomes a publish mechanism, not a feedback loop.

• The publish-completeness guard stays CI-only. validate_docs_published.py (ADR-0038) compares committed docs against the cross-repo assembled tree, which only exists in CI after pulling every source repo. It cannot run from a single worktree, so it is deliberately excluded from the pre-push hook and remains an enforced CI gate.

• The hook is best-effort, not a gate. Mirroring the existing pre-commit, the pre-push hook warns-and-allows when a tool (gitleaks, docker, the validator) is absent, and is opt-in per clone via git config core.hooksPath .githooks. The pipeline re-runs both checks and is the authoritative, unbypassable gate.

Alternatives considered

• Cron-coalesce — drop the CI trigger, publish on a fixed schedule (e.g. every 10 min, always: false). Coalesces all pushes in the window, but adds up to a full interval of latency even for a lone quiet-hour push, and makes “I pushed, is it live?” non-deterministic. batch: true is strictly better for this event-driven workload: it coalesces the burst without taxing the common single-push case. Rejected.

• Add more self-hosted ADO agents to run pushes in parallel. Removes the queue but does the opposite of what’s wanted — it runs N redundant full builds concurrently (more load on docker01, N reindexes racing the pit-memory volume) instead of collapsing them. Parallelism is the wrong tool when the outputs are identical. Rejected as the primary lever (a second agent is still worth having so a fast validate and a publish can overlap, but that’s incidental).

• A merge-queue / staging-branch aggregation. Agents push to a staging branch; a serialized queue batches and promotes to a published branch. More control, but materially more moving parts (a second branch, promotion logic, conflict handling) for a homelab docs site where batch: true delivers the same coalescing for one line. Rejected as over-engineered for the need.

• Split the monolith into a fast per-push “validate” pipeline and a coalesced “publish” pipeline. Conceptually clean (CI-vs-CD separation), but the expensive work is the build, and validation needs most of the assemble step — so a separate validate pipeline would duplicate most of the cost rather than removing it. The pre-push hook achieves the fast-feedback half locally and for free; coalescing achieves the publish-throughput half. Rejected in favour of those two cheaper moves.

Consequences

• A five-session docs sweep collapses from five serial full runs to ~two, cutting the dominant costs (registration latency, mkdocs build, pit-memory reindex) by roughly 60%. No commit is ever lost — the build’s last-write-wins design is what makes batching correct.
• Agents that opt into the hooks get sub-minute local feedback on secrets and Mermaid before pushing, so coalesced runs are almost always green.
• Publish latency under load increases slightly by design: a push that lands mid-build now waits for the current run to finish before its batched run starts, rather than immediately queueing its own. For a docs site this is the right trade — freshness within a couple of minutes, far less redundant work.
• One residual per-run cost is logged as follow-up tech debt: the Mermaid validation step npm installs mermaid@11 jsdom inside the ephemeral container every run (and now every pre-push). Prebaking a pinned validator image would remove that repeated overhead — raised as an Issue under Epic #835. (Resolved — see Resolution, Issue #1223.)
• The Five Pillars for this change: Docs = this ADR plus the in-file comments on the trigger and the hook; Monitoring/Alerting are unchanged — pipeline-failure surfacing already routes through Zabbix (ADR-0036), and a batched run that fails still fails visibly. Observability/SBOM are not engaged: no new service, endpoint, or image is introduced (the prebaked validator image, if built later, will get a Trivy scan at that point).

Applicability across the pipeline fleet

batch: true is a fleet pattern, not a docs one-off, but it is only correct where runs are idempotent (last-write-wins) AND burst-triggered by cc-pool’s concurrent sessions. The fleet was assessed against those two tests (Epic #835); the path-filter half of the optimisation was already in place on nearly every pipeline, so this rollout adds only the coalescing half where it fits.

• docker-stacks ×6 (the 5 domain pipelines + the legacy catch-all): batched. Each run git pulls latest and re-applies declared state via ansible/compose — idempotent — and these are the pipelines five sessions hammer during stack work. Same profile as docs, so they get the same fix. They were already path-filtered (a media-stack edit triggers only docker-stacks-media), so batching completes the optimisation.
• terraform ×5 (lxc, vms, azure-storage, backblaze, cloudflare): deliberately NOT batched. terraform apply is stateful and low-frequency, and the value of a clean 1:1 commit→apply audit trail (“which commit changed this infra”) outweighs coalescing. There is no burst problem to solve — TF changes are deliberate and rare — and path filters already scope them. Batching here would trade an audit property for a throughput gain that does not exist.
• ansible ×13: left as-is (optional, low value). Idempotent, so batching would be safe, but these are mostly low-frequency config pipelines (sshd, exim4, timezone) with little burst potential. Not worth the churn now; revisit only if one becomes a hot path.
• renovate / hass-config: out of scope. Scheduled or low-frequency, not cc-pool burst-driven.

The general rule for any future pipeline: batch when the run is idempotent and pushed in bursts; keep 1:1 when the run is stateful or its per-commit audit trail matters.

Resolution — shift-left hardening tech debt closed (2026-06-25)

Three follow-up Issues under Epic #835 hardened the shift-left model this ADR introduced. All three are implemented; the original decisions above are unchanged, these refine them.

• Issue #1226 — tag-vocabulary validation added to the pre-push hook. Two sessions failed the CI build’s tag gate within one hour (off-vocabulary tag ci-cd; a missing kind tag), each learning only after the coalesced run. The controlled-vocabulary check the wikilinks.py build hook runs (require one kind tag; every other tag in the mkdocs.yml extra.tag_vocabulary registry) is now extracted into a standalone validator, ansible/scripts/doc_gen/validate_doc_tags.py — pure Python + PyYAML, no mkdocs build. The pre-push hook runs it over the changed docs/**/*.md as a third check, same warn-and-allow-if-missing posture. The build hook over the full assembled tree stays the authoritative gate; the rules are kept byte-for-byte faithful between the two.

• Issue #1225 — core.hooksPath enforced across cc-pool worktrees. The hooks were opt-in per clone (git config core.hooksPath .githooks) and so silently never ran in freshly-provisioned session worktrees. The cc-pool provisioning playbook (ansible/playbooks/sync_control01.yml) now sets core.hooksPath for each pitlab-docs session worktree. Non-obvious decision: it is scoped per-worktree via extensions.worktreeConfig, not the shared repo config. Git worktrees share one config by default, so a plain git config core.hooksPath in the common config would also enable the hooks on the canonical /home/pit/pitlab-docs checkout — and the daily claude-config docs cron (claude_config_docs_regen.sh) commits and pushes from that checkout. A shared setting would silently start gating that automation (a gitleaks/Mermaid run on every refresh, with a blocking risk). Worktree-scoping enforces hooks where agents work while leaving the automation’s checkout untouched. Rejected alternative: shared core.hooksPath — simpler, but regresses the regen cron.

• Issue #1223 — prebaked the pinned Mermaid validator image. The residual per-run npm install -s mermaid@11 jsdom (CI and pre-push) is removed by baking mermaid@11 + jsdom + validate_mermaid.mjs into a local image, pitlab/mermaid-validate:11, built on docker01 by ansible/playbooks/build_mermaid_validate.yml (the vectormap-render local-build pattern — no registry in this homelab). Both the docs pipeline step and the pre-push hook now docker run --pull never the image instead of installing at runtime. (Wiring the hook up exposed a latent bug it had shipped with: docker runs on docker01 via the SSH context, so the hook’s -v ${REPO_ROOT}/docs bind mount resolved against docker01’s filesystem — an empty path — and the Mermaid check validated zero files. Fixed by staging docs/ to docker01 first, mirroring what the CI step already does.) mermaid is pinned to major 11 to track the version the live site loads from unpkg; the validator script is baked, so the build playbook is the single source — rerun it to bump deps or the script. Pillar 4: the image is added to scan_trivy_docker.yml extra_images (it never appears in docker ps), auto-creating the docker-mermaid-validate Dependency-Track project.

Resolution — extended estate-wide (2026-08-04, ADR-0311)

The “ansible: left as-is (optional, low value)” assessment above aged out: the ansible fleet grew from ~13 pipelines at assessment time to 72, several of them all-paths guards that run on every push — precisely the burst profile this ADR batches. The revisit clause fired: ADR-0311 (Epic #2166) extends batch: true to every ansible CI trigger and promotes the pattern from a per-repo optimisation to a standard clause (CI/CD Standard §Trigger Economy). The terraform ×5 carve-out above (1:1 commit→apply audit trail) stands unchanged — those pipelines remain deliberately unbatched, now with the standard’s why-not-comment escape hatch. Lesson recorded in ADR-0311: a fleet assessment without a standard clause decays silently as the fleet grows — the 0-of-72 gap accrued invisibly because nothing re-ran the assessment.