0371 — servicemap freshness SLO and error-budget policy

Status

Accepted — 2026-08-09, at the servicemap Production Readiness Review (Epic #2260, Issue #2262). Refines the reliability contract stated in the servicemap design and implemented by ADR-0359. Supersedes the design’s stated 24h / 99% over 30 days objective — the threshold moves to 72h for the arithmetic reason below; the 99%/30d target and the choice of freshness-as-SLI are unchanged.

Context

servicemap’s SLI is freshness, not availability. That decision was made in the design and is not revisited here: a perfectly available map showing last month’s estate scores 100% on uptime and is worthless, so uptime measures the least interesting failure this service has.

What the design did not settle is the two things an SLI is useless without: a threshold whose breach is meaningful, and a pre-agreed answer to “what happens when the budget burns”. This ADR settles both, deliberately now — while it is cheap — rather than during an incident, when the temptation is to widen the window until the red goes away.

The design’s 24h threshold does not survive contact with the render cadence. The render runs daily at 15:00 AEST, so in steady state the map’s age sweeps from 0 to just under 24h and back, every day. An objective set at 24h therefore sits on the boundary the design touches every single day, and ordinary jitter spends the budget while nothing is wrong: a render finishing ten minutes late each day burns 5 hours of a 7.2-hour monthly allowance. Worse, a single genuinely missed render burns 24h — 333% of the entire month’s budget in one day. A budget that is either untouched or catastrophically blown is not an error budget; it is an alert with extra steps, and it can never function as the graded signal error budgets exist to be.

Decision

The objective

SLI	Share of time the published map reflects an estate state no older than 72 hours, measured from servicemap_last_build_timestamp
SLO	99% over a rolling 30 days
Error budget	7.2 hours per 30 days of map age beyond 72h
Target currency (not the SLO)	24h — what the daily timer delivers, shown as map age on the dashboard and ambered past 36h on the map header itself

72h is three render cycles. Two consecutive missed renders are absorbed by design — systemd Persistent=true catches a run missed while control01 was down — so only a third sustained failure spends budget. That separates “the estate hiccuped” from “servicemap is unreliable”, which is exactly the distinction the objective must carry.

The three thresholds escalate, and the page arrives a full day before the budget starts burning:

Threshold	Signal	Class
> 40h	Zabbix servicemap.render.age dead-man on control01	cross-surface: the job is not running at all
> 48h	ServicemapStale (Prometheus → Pushover)	page
> 72h	error budget begins to spend	budget

Budget burn therefore always means “we were told and did not act”, never “we were surprised”. That ordering is the point: an SLO threshold tighter than its own alert threshold would spend budget before anyone could know, and an SLO measured at the alert threshold would make the budget a duplicate of the page.

The error-budget policy — what happens when it burns

The response is graded by budget remaining, reported on /d/servicemap. servicemap is T3 and has no downstream consumers, so no burn state ever justifies interrupting work on a higher-tier service; this policy governs servicemap’s own change budget, not estate priority.

Budget remaining	Response
≥ 50%	Normal. Derivation and frontend changes ship freely.
< 50%	Warning state. The next change to servicemap must be a reliability change, not a feature.
0% (exhausted)	Feature freeze on servicemap. The only permitted changes are ones that reduce render-failure rate or shorten recovery. ServicemapErrorBudgetExhausted raises a FreeScout incident (ticket-class, severity: info, deliberately not a page).

The two reliability changes already identified, in preference order — so the freeze has somewhere to go rather than becoming a stall:

1. Make the render self-retry instead of waiting for the next daily slot. Recovery time currently rounds up to 24h for any failure nobody manually re-runs, and that single fact dominates the burn.
2. Decouple the derivation sources. --require-sources aborts the whole render when any source fails, so reliability is bounded by the least reliable source (today, the docker01 PostgreSQL step). Degrading one edge class instead of the whole map is strictly better — but only if the calibrated artifact gate can still detect the thinning, which is the reason it is second and not first.

Explicitly NOT permitted as a response to burn:

• Widening the freshness window or lowering the SLO target.
• Re-baselining the artifact gate to accept a thinner map (--seed-baseline) to clear a completeness-driven burn.

Both convert a reliability problem into a measurement problem. This is the same rule the runbook already states for the artifact gate — never widen a tolerance to make a red go away — applied one level up, to the objective itself.

Who decides: Arron. Nothing is automated on budget state; the budget informs a decision, it does not take one.

How it is measured

A recording rule samples the SLI every 60s, so the 30-day budget maths stays cheap on both the alert and the dashboard:

servicemap:freshness_violated = (time() - servicemap_last_build_timestamp) > bool 259200

avg_over_time(servicemap:freshness_violated[30d]) is then the bad-time ratio directly. /d/servicemap carries SLO attainment (30d) and error budget remaining (30d) as stat panels beside the availability tile; ServicemapErrorBudgetExhausted fires at ratio > 0.01.

Alternatives considered

• Keep the design’s 24h / 99%. Rejected on the arithmetic above: a healthy system violates it routinely and one bad day exceeds the month’s allowance threefold.
• 48h, aligning the SLO threshold with ServicemapStale. Rejected — the budget would begin burning at the same instant the page fires, so it would carry no information the page does not already carry, and there would be no window in which acting promptly costs nothing.
• A multi-window burn-rate alert pair (the design’s proposal, and the house pattern used by the CI queue-time SLO). Rejected for the fast-burn half only. With a time-based 0/1 SLI, a 14.4× fast burn trips after ~4 minutes of staleness — i.e. immediately after ServicemapStale, which already paged. It would be a duplicate page with worse wording. The slow-burn half is retained in substance as the budget-exhaustion ticket, which is the signal that genuinely differs from the page: recurrent staleness across the month rather than staleness right now.
• Measure the SLI in render cycles (“share of days the map refreshed”) rather than seconds. Conceptually cleaner for a batch workload, rejected as harder to express honestly in PromQL and harder to reconcile with the second-denominated alert thresholds it must sit beside.
• No SLO at all — T3, so let the alerts carry it. Rejected: alerts answer “is it broken now”, and the question this service actually raises over time is “is the derivation reliable enough to keep trusting”. Only a budget answers that, and Epic #2260 exists precisely because a map that is quietly wrong is worse than one that is loudly down.

Consequences

• The budget is now a graded signal with headroom, so “spend a few hours recovering” and “servicemap has been unreliable all month” are finally distinguishable — which was the whole point and was not true under the 24h objective.
• The response to burn is decided in advance and written down, including the two things that are forbidden. The failure mode this prevents is the honest-looking one: quietly widening the window during an incident and recording a green month.
• An operator now has three ordered signals rather than one, and the ordering itself is documented — a dead-man at 40h, a page at 48h, budget at 72h.
• The stated SLO in the design doc is superseded, and the design’s Deviations table records it. Anything quoting 24h / 99% for servicemap is stale and should cite this ADR.
• The SLI series is new, so the 30-day window only becomes fully meaningful 30 days after deploy. Until then avg_over_time averages a shorter history — it under-reports burn rather than over-reporting it, which is the safe direction, and the absent-series case is separately covered by ServicemapBuildMetricAbsent.