Queue time is a CI pool's latency

The problem

CI felt slow. That is not a measurement, so the first job was to get one.

Over four days, 387 pipeline runs: queue time median 8.7 minutes, p90 24.1 minutes, and 60% of all pipeline wall-clock spent queueing — while the single agent was idle 79% of the time. The pool was not saturated. It was serialised: one agent, so any two pushes within a few minutes of each other queued behind one another regardless of how little work either was doing.

There was also nothing recording it. The estate had a point-in-time wedge detector — an alert for a run stuck beyond a threshold — and that alert had genuinely fired on a pathological run that sat in progress for 65 minutes. It was ignored, because at the prevailing baseline a 20-minute run was routine and the alert could not distinguish slow from wedged.

So the real finding was not a missing alert. It was a missing series: nothing recorded how long runs took over time, so a gradual regression was undetectable and there was no data to set an objective against. Adding a second duration alert would only have deepened the fatigue.

The design

Three parts, and the ordering matters — the measurement came before the capacity change, so the capacity change could be evaluated.

One collector, two consumers. A single poller against the build API writes a metrics exposition (pool-wide queue and duration histograms, per-pipeline aggregates, agent gauges) and a rolling run store. Percentiles and a rate-based burn rate need buckets, so those live in the metrics store; per-pipeline percentiles come from the run store, because bucketing ninety-nine pipelines would cost thousands of series for numbers read once a week.

The rule attached to it is stated bluntly: do not build a second collector. Two pollers against one API drift apart, and the disagreement is discovered during an incident. The public delivery performance panel on this site obeys that rule literally — it does not restate the deploy classification, it imports it from that collector, so a change to what counts as a deployment moves the public figure with it.

An SLO with an error budget, and burn-rate alerting. The indicator is the share of runs that start within ten minutes of being queued; the objective is 95% over 28 days. Alerting is multi-window burn rate rather than a threshold on a percentile — a “p90 over 20 minutes” rule fires on any busy afternoon and is muted within a week, which is precisely how the previous stuck alert became ignorable. Burn rate asks whether the budget is being spent faster than it can be afforded: a normal burst is silent, a sustained regression is not.

A second runner, and a parity gate to keep them interchangeable. Two agents built from one role. The parity gate asserts two properties that fail differently — conformance (each runner matches its declaration) and parity (each runner matches the other, including on things the declaration does not pin) — and catches a third shape both are blind to: two places in the repository pinning the same tool, where the runners agree with each other and are simply alternately wrong.

The trade-off

Doubling the pool halves the serialisation and doubles the surface that has to stay identical. A pool of interchangeable agents is only interchangeable while nothing has drifted, and the moment one agent has a tool the other lacks, every pipeline that uses it becomes a coin flip — a red that reproduces half the time and is close to undiagnosable from the run log.

That cost is paid with the parity gate and with a portability rule: a pipeline may not be pinned to a named agent, and a gate script may not be deployed to one (ADR-0337, ADR-0333). Both are enforced rather than documented, because “remember the pool is plural” is not a control.

The second trade-off is the SLO’s own scope. Queue time says nothing about whether pipelines do the right thing — a pool can hit 100% of its start-time objective while every run fails. It was chosen anyway because it is the number the capacity decision turns on, and an objective that cannot drive a decision is decoration.

Unreachable is not a pass: a runner that cannot be inspected exits unverified, never zero.

The evidence

6.3 minmedian commit to live
81.3deployments per day
2,377deployments in 28 days
97CI/CD pipelines, 76 of them deploying

Those figures are derived at build time from the same run history, over a 28-day window, and are published with their derivations on the delivery performance page. The baseline they should be read against is the one above: queue median 8.7 minutes, p90 24.1, 60% of wall-clock queueing.

Full reasoning: the CI runner pool design, ADR-0334 and ADR-0329.