ADR-0458 — Historical duplicate incident records are archived to a second mailbox, not deleted

Draining 744 duplicate incident conversations out of the FreeScout Incidents mailbox looked like a deletion job until the rows were actually read. They are not empty forks: they carry 3,095 notes, of which 309 are operator [fix: remediations and 202 are distilled {symptom → root cause → fix} records — the corpus incident recall retrieves from. FreeScout’s API offers no soft delete, so the drain moves the forks to a second mailbox instead, which clears the Incidents mailbox to one conversation per condition while destroying nothing and staying reversible.

Status	Accepted — 2026-08-21
Date	2026-08-21
Extends	ADR-0457 (oldest-wins collapse), ADR-0068 (the record model and recall corpus)
ADO	Epic #2458 (One incident per condition); Issue #2460

Context

ADR-0457 made live duplicates self-heal, and explicitly left the historical backlog out of scope: the in-pass collapse acts only on tokens holding more than one open conversation, and the backlog is closed. Issue #2460 is that drain.

Three facts, each measured against the live mailbox on 2026-08-21 rather than carried from the card that specified the work:

• The backlog is 185 duplicated tokens holding 744 excess rows in a 1,915-row / 1,169-token mailbox — the receiver’s namespace 146 tokens / 598 rows, the poller’s 39 tokens / 146 rows.
• 11 of those 744 excess rows are ACTIVE, not closed. Both the work item and its resume brief asserted that every excess row was closed. Each of the 11 is a token whose only open row is a non-oldest fork, which the in-pass collapse cannot reach because that path requires two or more open conversations. All 11 underlying conditions were verified resolved at measurement time — eight Zabbix events carrying an r_eventid and absent from problem.get, three Alertmanager fingerprints absent from a cluster reporting ready.
• The excess rows carry irreplaceable content. 3,095 notes across the 744; 309 carry a genuine operator [fix: note and 202 an [evt:distilled] record. A first count returned 722 fix notes and was wrong — the resolved-note boilerplate ends with “Add a note starting `[fix:`”, so a substring test matched every resolved record whether or not an operator had ever written one. That is an assertion satisfiable without the thing under test (FC-02); the corrected predicate requires the note to begin with the marker.

FreeScout’s API DELETE /api/conversations/{id} calls $conversation->deleteForever(). There is no soft delete on the API surface — the deleted state seen on one row in this mailbox comes from the web UI, and such rows remain in the API listing, so a UI-style soft delete would not clear the census either.

Decision

1. The drain MOVES duplicates to an “Incidents Archive” mailbox; it never deletes. Each fork is moved with PUT /api/conversations/{id} {mailboxId}, FreeScout’s native moveToMailbox. Every note survives, the move is reversible by moving the row back, and the Incidents mailbox reads one conversation per condition — which is the goal the Epic actually states. Deleting would have destroyed 309 operator remediations and 202 distilled records to achieve the same visible outcome.

Scoping the census to the Incidents mailbox is not a redefinition of success: the running index (rebuild_conversation_index) reads exactly mailboxId=2, so a moved fork leaves the index as well as the human view.

2. The survivor is the oldest by createdAt, and this is asserted against a pre-drain snapshot. The drain applies ADR-0457’s policy unchanged, so the record it leaves is the one the running containers would keep. Verification asserts the survivor by conversation id against a snapshot of the token’s original row set captured before the first mutation — never against post-drain state, where a token holding one row satisfies “the survivor is the oldest” no matter which row was kept.

3. Stuck-open forks are resolved→closed only against a run-time liveness proof, and the check demands a positive control. Before any mutation the drain re-derives liveness live — Zabbix problem.get for incident-zbx-<eventid>, the Alertmanager active set for incident-<fingerprint[:8]> — and refuses the entire run if any condition is live. It equally refuses if either monitor fails its positive control (Alertmanager not ready, Zabbix not returning a list), because an unreachable monitor and a quiet estate both answer empty (FC-01). A refusal changes nothing.

4. The surviving record gets a correcting note. An earlier drain on 2026-08-18 collapsed onto the newest row and stamped “superseded by conversation #N” onto the oldest — the behaviour ADR-0457 reversed. Every survivor therefore receives an [evt:collapsed] note stating that it is the surviving record and that the older claim is void. [evt:collapsed] is inert to current_state(), which reads only [evt:firing] / [evt:refired] / [evt:resolved], so annotating a record cannot change what the poller believes about it.

Alternatives considered

Hard-delete the forks. What the work item specified, and the simplest thing that satisfies the acceptance criterion. Rejected once the rows were read: it destroys 309 operator fix notes and 202 distilled records irreversibly, and those are precisely what ADR-0068 recall retrieves when a similar alert next fires.

Merge each fork’s notes onto the survivor, then delete. The tidiest end state — one record carrying the whole history. Rejected on blast radius: it is several thousand additional writes against a production record store, and it is still irreversible if it goes wrong halfway.

Rewrite the fork subjects so their tokens fall outside the incident- namespace. Removes them from the index without moving or deleting anything. Rejected because it leaves all 744 rows in the mailbox a human reads, which is the actual complaint.

Leave the 11 stuck-open forks untouched, honouring the work item’s “closed rows only” wording literally. Rejected on Pit’s ruling: it keeps the active count at exactly 47 at the cost of leaving 11 tokens permanently duplicated, so the census can never reach zero and the Epic’s contract is met everywhere except the rows most visible to a human — the open ones.

Consequences

• A second mailbox, Incidents Archive, now exists and holds the collapsed history. It is not a working queue and nothing writes to it; it is the recovery handle that makes the drain reversible.
• The acceptance criterion “active count identical before and after” is not met as written, and is reported as such. The active count moves 47 → 36 because 11 stuck-open forks are closed. The invariant that actually holds — and the one the criterion was reaching for — is that no live condition lost its open record, proven per row at run time.
• The drain ships as a /tmp handoff script under the blocked-action convention, not as deployed tooling: it is a one-off backfill, and RULE 6’s homes are for code that runs again.
• It is idempotent and carries a hard-guarded dry run — but only after the amendment below; as first delivered it was not. In dry-run any non-GET raises rather than being skipped, so the mode cannot silently withhold only its loudest side effect (FC-15). The fail-closed precondition was proven red three ways — a fabricated live Zabbix condition, a fabricated live Alertmanager fingerprint, and a failed positive control — each refusing with exit 2 before any write (ADR-0141).
• The duplicate gauges are unaffected. Both legitimately read 0 throughout, because they count open forks the collapse could not heal and this backlog was closed — which is why a drain was needed at all and why its completion is invisible to the alerts.

Amendment 2026-08-21 — the re-run path was never exercised, and criterion (d) is what walked it

The drain’s first execution passed every assertion: 185 duplicated tokens → 0, 744 forks archived with zero errors, the survivor asserted by conversation id across all 185 tokens, and the active count moving 49 → 38 for exactly the 11 verified-resolved stuck-open forks. The second execution failed at exit 126 without mutating anything, and the defect was in the script rather than the drain.

The script captured its pre-drain snapshot to a file and, on any subsequent run, re-fed it to its own container as a base64 environment variable. execve counts argv and environ against ARG_MAX, so a ~181 KB variable exhausted it and every external command after the export failed with Argument list too long — docker run, grep, and the am/zapi helpers alike, which is also why that run’s alert-surface readings came back ERR.

It hid because the export is conditional on the snapshot file already existing, making the fault structurally unreachable on a first run. The first-run path had been exercised three times — a dry run, the red-test suite, and the real execution — and the re-run path exactly zero times: every green run took the branch where the risky code is disabled. That is FC-17 unreachable-control, whose tell is a documented invocation that has never been executed end to end. Acceptance criterion (d) — run it twice, the second run reports 0 collapsed rather than erroring — exists to walk that branch, and caught it on the first attempt. A criterion that looks like a formality is the one worth keeping.

The fix removes the carry rather than enlarging it. The pre-drain row set is now reconstructed from live state — each token’s surviving record in the Incidents mailbox plus its forks in the archive — so there is no file to go stale, nothing crossing a boundary that has a size ceiling, and the assertion is derived rather than remembered, which makes it stronger than the snapshot it replaces. Generalised as an agent lesson under unreachable-control: never carry a verification payload through the environment.