0389 — Hypercare is an enforced delivery stage with an evidence-based exit

Status

Accepted — 2026-08-11. Realises Epic #2369 (Hypercare as an enforced delivery stage). Generalises the cc-pool early-life window from ADR-0377 / Epic #2292.

Amended by ADR-0399 (2026-08-13) on the tag half only, and on nothing else: the dated hypercare-until-<date> tag this ADR required is retired — it minted one single-use tag string per delivery — and is replaced by a flat hypercare tag (amber on the board) plus a trailing (HC DD-MM-YY) review date on the Epic title. The old dated tag stays readable by hypercare_countdown.py permanently. Everything this ADR decides about the stage — evidence-based exit, no automation may close a window, Doing not Done, the separate untagged signal, and querying by COLUMN rather than by tag (now doubly right, since the tag no longer carries the date at all) — stands unchanged.

Context

pitlab had no stage between delivered and done. An Epic whose service was live and proven moved straight to Done, and everything learned in the first week of real operation — a threshold that turns out to be noisy, a runbook nobody has walked, an SLI whose window has not yet filled — landed on nobody’s board.

The word “hypercare” was already in use as a convention. Epic #2339 (Domestic hot water — monitoring, alerting & availability SLO) was accepted on 2026-08-10 with a stated one-week hypercare window to 2026-08-17, and the exit criteria written onto its service page. Nothing anywhere told anyone that window existed, when it came due, or that it had passed. A convention with no mechanism behind it lapses silently, and the lapse looks exactly like success.

The mechanism already existed for exactly one capability. cc-pool session recovery ran a working early-life-support countdown: Zabbix items cc.pool.els_active / cc.pool.els_age, a trigger firing only if the window was still open after 30 days, and a Pushover on exit. Two of its design properties are the reason it works, and both had to survive generalisation:

The exit is evidence-based, never time-based. The cc-pool window closes when a real coordinator reboot resumes every session unaided — not when a timer expires.
A window that never closes is the failure mode to watch. The trigger keys on the window’s age, so a window that closed and a window that never opened both stay quiet. Only a lingering one pages.

The question this ADR answers is how to make that one-off mechanism a general stage of delivery without losing either property.

Decision

Hypercare is a board column mapped to `Doing`, not to `Done`

A delivered Epic is parked in the Hypercare column of the Epics board (between Blocked and Done) and tagged hypercare; hypercare-until-YYYY-MM-DD. The column maps to ADO state Doing.

Doing is the correct state because delivered is not handed over. Work in hypercare still carries an owner, still has open acceptance criteria, and can still fail back. Mapping it to Done would make the board assert completion that has not been earned, and — because a Done Epic must have no open children — would force the residual work off the board entirely, which is precisely the disappearance this stage exists to prevent.

No ADO process change was required: the board already maps multiple columns to one state (Up Next and Planned both map to To Do).

The exit is a human close-out review; the date only summons it

The hypercare-until- date is a review-due date, not an expiry. Reaching it does not end hypercare — it makes the close-out review owed. The window closes only when a human re-runs the Epic’s acceptance criteria against the live service, clears the residuals, and then moves the Epic to Done.

Nothing automated may close a window. A timer-based exit would let an unreviewed service graduate by the mere passage of time, which is the failure this stage exists to prevent. The countdown’s only power is to make a due window impossible to ignore.

The alert keys on lingering windows, so a healthy board is silent

hypercare_countdown.py reads the Hypercare column daily and publishes, per window, its age and its days-remaining. The paging signal is hypercare.overdue — a count of windows at or past their review date.

This preserves the cc-pool property exactly: an estate with nothing in hypercare and an estate whose windows are all in date both report 0. Only a window that has outlived its review can raise anything. Silence is the natural state of a healthy board, so the alert must be the exception rather than the presence.

hypercare.untagged is the generalised form of the same hole. An Epic parked in the column with no hypercare-until- tag can never become overdue, so it would linger invisibly while the countdown reported all-clear. Under the cc-pool mechanism that case could not arise — there was one window, opened by writing a file. Generalised to “any Epic anyone parks on a board”, it is the most likely way the stage decays, so it is a first-class trigger rather than an edge case. It is deliberately not folded into overdue: the two need different actions (stamp a date vs run the review), and blending them would tell whoever is paged the wrong thing to do.

Window age is read from ADO’s own revision history

The age of a window is the time since the Epic entered the Hypercare column, which ADO records exactly in the work item’s revision stream. The collector walks that stream backwards and stops at the first non-Hypercare revision, so an Epic moved out and back in dates from its latest entry — a window reopened after a failed close-out review is a new window.

A local first-seen state file was rejected as the primary source because it is wrong in the one case that matters: on first deployment every existing window would read as just opened, resetting the very clock the signal exists to watch. That is the same day-zero reset that forced the cc-pool dead-man to key on window age rather than min() over history. The state file survives only as a fallback for when the revisions API is unreachable, and the output names which source it used.

Machine identifiers keep their names; only human-facing text is renamed

The estate-wide rename of early-life support → hypercare covers prose, item names, trigger descriptions and notification text. It deliberately does not touch machine identifiers: the Zabbix item keys cc.pool.els_active / cc.pool.els_age, the Prometheus series cc_pool_early_life_support_active, the marker file /var/lib/pitlab/cc-pool/els-active, the Pushover event name els_exit, or the docs anchor #early-life-support.

Renaming them buys nothing and costs something real:

Renaming a live Zabbix item key orphans its history. The graph and the trend data start again from zero.
Renaming the marker file resets the window’s measured age. cc.pool.els_age is computed from the mtime of els-active. A rename gives it a fresh mtime, so the 30-day dead-man — the whole point of the mechanism — silently restarts its clock. This is not a cosmetic loss; it destroys the exact signal being generalised.
Renaming the event name breaks every historical Loki and Pushover record keyed on it.

An identifier is a name for a machine, not a description for a human. A rename that would have required an Alertmanager silence and a Zabbix maintenance window to perform safely, in exchange for no operator-visible benefit, is not worth performing. The new generalised mechanism takes fresh hypercare.* keys, so nothing is migrated and no mute window is needed.

The one exception is trigger descriptions, which are human-facing (they appear in Zabbix and in the Pushover body) and are also the upsert key this estate’s provisioners match on. Renaming one therefore creates a second trigger beside the first, both evaluating the same item and both firing. The rename is applied with an explicit retirement of the superseded description, following the precedent already set by the 36h→96h reboot-pending rename.

Relationship to the Production Readiness Review

The two are sequential halves of one handover, and neither substitutes for the other:

	Production Readiness Review	Hypercare
When	Before the service carries real load	The first period after it does
Question	Is this ready to be accepted?	Was that acceptance correct?
Evidence	Designed and tested behaviour	Observed behaviour under real conditions
Outcome	Accepted into PROD	Epic moved to `Done`

A PRR is a prediction; hypercare is where the prediction is checked. This is why exit criteria are written at PRR time onto the service page and re-run at close-out — the criteria that justify acceptance are the ones the window exists to confirm.

Once an Epic leaves hypercare, a defect found in that service is no longer project work: it is an operational defect in something accepted into PROD, and routes to a FreeScout incident per ADR-0372. Hypercare is precisely the window in which that routing does not yet apply.

Alternatives considered

A time-based auto-close. Move the Epic to Done when its date passes. Rejected outright: it converts the stage into a delay and guarantees that a service which failed its window graduates anyway. The whole value is the review, not the wait.

A new ADO state rather than a board column. Rejected as disproportionate. It requires an inherited-process change, affects every board in the project, and buys nothing the column does not already give — the board already maps multiple columns to one state.

Tag-only, no column. Query Epics by the hypercare tag instead of by column. Rejected because the tag and the truth would drift: an Epic could be tagged and not parked, or parked and untagged. Querying by column makes the board the authority and, critically, makes the untagged case visible instead of invisible — a tag-driven query would silently exclude precisely the misconfiguration worth alerting on.

Keeping the countdown inside cc_drain.sh. Rejected: the cc-pool window is opened and closed by evidence local to that mechanism (a real reboot), while the general case is opened and closed by a human decision recorded on a board. Reusing the script would have coupled every future hypercare window to the reboot coordinator’s lifecycle.

Renaming the Zabbix keys to cc.pool.hypercare_*. Rejected — see above. History loss and a reset window age for no operator-visible gain.

Consequences

An Epic can no longer reach Done by being finished. It reaches Done by being reviewed after being finished, which is a real added step in every delivery.
/wrapup no longer closes an Epic at delivery. It parks it in Hypercare with an expiry tag, and a separate close-out entry point performs the review and the move to Done. Without that change the next Epic would bypass the stage entirely.
A window that lingers past its review date pages at Zabbix Average, routed by the existing Pushover action — one alerting path, not a second notifier.
The collector’s healthy output is a set of zeroes, so a dead collector reads exactly like a quiet board. hypercare.countdown.age (26h dead-man) is what distinguishes them, and is not optional.
The cc-pool window keeps its own evidence-based exit and its 30-day dead-man unchanged. It is now described in the shared vocabulary rather than a private one, but it is not managed by the board countdown — its window is opened by a file and closed by a reboot, not by a column.
Adding a hypercare window to an Epic costs one tag. Failing to add the tag while parking it in the column is now itself an alert, so the cheap mistake is caught rather than silently tolerated.
A window can now regress to Doing, and that is an ordinary transition (ADR-0448, 2026-08-19). This ADR left the stage one-way: a failed criterion re-dated the (HC …) suffix and the Epic stayed parked, which is right when the fix is a threshold to tune or a runbook to walk, and wrong when the window catches a real defect needing new child Issues — the card then reads soaking under supervision while delivery actively runs out of it. /wrapup Check S2 now splits those two cases: clearable-in-place re-dates in Hypercare, needs-new-children moves the Epic back to the Doing column, keeping the hypercare tag and date, and S1 re-parks it with a fresh date once proven. The stage is unchanged; what is added is the honest leftward move it was missing.