ADR-0457 — Duplicate incident records collapse onto the oldest, and each container detects only its own namespace

The FreeScout incident bridge counted duplicate records but never healed them, so the reliability contract “one ticket per condition” rested on a detector with no remediation behind it. Worse, the detector covered only one of the two token namespaces writing to the mailbox — and it was the smaller one. This ADR records three coupled decisions: duplicates self-heal onto the oldest record, each container heals and measures only its own namespace, and the collapse is fail-closed on the reopen.

Status	Accepted — implemented 2026-08-21
Date	2026-08-21
Extends	ADR-0068 (the incident record model), ADR-0141 (proven-red gate tests)
ADO	Epic #2458 (One incident per condition); Issue #2459

Context

Two containers write incident records into the FreeScout Incidents mailbox: the Zabbix poller (zabbix_incident_poller.py, minting incident-zbx-<eventid>) and the Alertmanager receiver (incident_webhook.py, minting incident-<8-hex-fingerprint>). Both looked an existing record up by subject token, and both were blinded by FreeScout silently ignoring ?query=/?search= — so every re-fire forked a new conversation instead of reopening the original. The lookup was repaired on 2026-08-18; the wreckage it left was not.

Three facts drove this decision, each measured against the live mailbox on 2026-08-21 rather than recalled:

• 185 tokens held 744 excess rows across a 1912-row, 1166-token mailbox.
• The receiver’s namespace held 598 of those 744 excess rows — 80% — across 146 of the 185 duplicated tokens. The poller’s held 146 rows across 39 tokens.
• Only the poller published a duplicate gauge. zabbix_incident_poller_duplicate_tokens read a healthy 0, and it was telling the truth about the fifth of the mailbox it could see.

The detector was therefore present and green while most of the corruption sat structurally outside what it measured — the present-but-incomplete failure class, which is harder to catch than an empty result because a populated, healthy-looking signal invites no second look.

Decision

1. Duplicates collapse onto the OLDEST record, sorted explicitly by createdAt. The surviving conversation is the oldest for that token; younger forks are closed against it, each carrying an [evt:collapsed] note naming the survivor so a human landing on a dead row can follow it to the live one. The original is the record that carries the condition’s history — every [evt:firing], [evt:refired] and [evt:resolved] note written before the lookup went blind — so keeping the newest preserves an empty record and discards the useful one.

The sort is on createdAt explicitly, never on listing position. The previous index kept the first row the listing returned, on a comment asserting FreeScout answers newest-first; the survivor it actually produced in test was neither the oldest nor the newest but simply first in the page. Listing order is an API behaviour, not a contract.

2. Each container collapses and measures ONLY its own token namespace. The poller mints incident-zbx-<eventid>; the receiver mints incident- followed by eight hex characters, and hex cannot produce zbx-. The namespaces are therefore disjoint by construction rather than by convention, which is precisely what makes a per-token in-process lock a complete guard: there is no cross-container contention to exclude. Collapsing across the boundary would reintroduce the race the lock is claimed to prevent, with no shared lock able to arbitrate it.

The gauges partition the same way — zabbix_incident_poller_duplicate_tokens and incident_webhook_duplicate_tokens — so together they cover the mailbox without overlap, and each alert’s runbook names the container that owns the fix. Each gauge counts what its container failed to heal, not merely what forked, so a positive reading now means two things went wrong rather than one.

3. The collapse is fail-closed on the reopen. If the surviving record is closed while its forks are open, the condition is live and the survivor is reopened first. If that reopen fails, nothing is closed. Leaving duplicates is visible, alertable and recoverable; closing the open forks onto a survivor that could not be reopened would leave a firing condition with no open record at all — trading a noisy fault for a silent one.

Alternatives considered

Newest-wins. Rejected: the newest fork is by definition the one carrying no history. It is marginally simpler (it matches the old listing-order behaviour) and it destroys the thing worth keeping.

Collapse the whole mailbox from one container. Rejected: it is less code and it breaks the locking argument. Whichever container ran it would be mutating rows the other container is concurrently creating and updating, with no shared lock, which is the cross-process race an in-process lock cannot address.

One combined gauge over both namespaces. Rejected: a single expression cannot tell an operator which container to inspect, so the alert’s runbook would send whoever is paged to a metrics endpoint that may be perfectly healthy. Two disjoint gauges cost one extra alert rule and keep the diagnosis in the alert.

Detector only, no self-heal — drain by hand when it fires. This was the pre-existing state and it is what the Epic exists to end: a reliability contract of “no duplicate record persists beyond one lookup cycle” cannot be met by a counter that waits for a human.

Consequences

• Duplicate records now self-heal within one index rebuild, and the gauges have been redefined from “something forked” to “something forked and I could not fix it” — a stronger signal from the same number.

• The receiver’s namespace is observable for the first time. IncidentRecordDuplicationReceiver is a new alert with promtool unit tests proven red against the rule’s own absence.

• The historical backlog is not touched by this change. The in-pass collapse acts on tokens holding more than one open conversation, so the closed backlog is invisible to it by design. Draining it is Issue #2460, which applies the same oldest-wins policy so the survivor the drain leaves is the one the running system would have kept.

  ⚠️ Correction, 2026-08-21 — “all 744 excess rows are closed” was wrong. This consequence originally asserted that every excess row was closed. Re-measuring the mailbox at drain time found 11 were ACTIVE: tokens whose only open row is a non-oldest fork. That shape is unreachable by the collapse for a different reason than the closed backlog — not because nothing is open, but because the collapse requires two or more open conversations before it acts, so a single stuck-open fork beside closed siblings is skipped and the index quietly points at the closed oldest instead. Both duplicate gauges read 0 throughout, correctly, since each counts open forks the collapse failed to heal. The drain resolved→closed all 11 against a run-time liveness proof; see ADR-0458.

• incident_webhook_image is digest-pinned as part of the same change, giving the incident record path a rollback handle it did not have and giving Renovate something to bump.

• The collapse mutates production incident records from an automated path. The fail-closed reopen bounds the damage of a partial failure, and the [evt:collapsed] note keeps every closure attributable.