0353 — Corpus retrieval technique is a governed standard, not agent habit

The agent’s knowledge of this estate is whatever it can retrieve from the docs corpus, yet how to query that corpus was governed by nothing. Retrieval technique lived only as one agent’s habit, and the habit was tuned for half the engine.

Status

Accepted — 2026-08-08. Establishes the Corpus Retrieval Standard. Complements the Documentation Standard (authoring) and the Documentation Architecture Standard (structure), which govern the write side only.

Context — a retrieval engine used at half capacity, with no way to notice

pit-memory has been a hybrid retriever since it was built: docs_search fuses SQLite FTS5 with Qdrant/nomic-embed-text via Reciprocal Rank Fusion, over heading-scoped chunks. Nothing recorded what that implies for a caller, so the agent queried it as though it were a keyword index — keyword-salad strings that make a good sparse query and a weak embedding, one shot per question, answers drawn from the returned fragment without retrieving the surrounding page, and empty results read as proof of absence.

None of that fails loudly. A half-tuned query returns plausible chunks and the agent proceeds, so the deficit is invisible in exactly the way a missing index is not. It surfaced only because Arron asked directly whether the agent knew the search was hybrid.

The same session exposed a second, sharper failure: the agent had repeatedly reported the corpus as “not connected” on the evidence of a ToolSearch miss — a fact about its own exposed tools, escalated into an untested claim about a remote service that was healthy throughout. Retrieval discipline and availability diagnosis are the same subject, because both decide whether the agent searches or guesses.

Fixing this in soul.md alone would have been sufficient for one agent and useless for the pool. The agent fleet (five concurrent cc-pool sessions today, autonomous agents planned) all read this corpus, and a technique that lives in one persona file is not inherited by anything else.

Decision

Retrieval technique is governed by the Corpus Retrieval Standard, an estate standard covering: the hybrid mechanism and what it implies for callers; queries written for both halves (natural-language intent plus exact tokens); re-querying a thin result along a different axis; doc_get for load-bearing chunks; empty-as-hypothesis; incidents_search before first principles; the tool-surface-versus-server-health distinction; the MCP-over-HTTP fallback; deliberate limit tuning; and writing learned facts back.

The agent operating principles (soul.md, Searching the Corpus) mirror the standard rather than replace it — the standard is the source of truth, the persona file is its per-agent expression.

Alternatives considered

Leave it in soul.md only. Rejected: it fixes one agent. The corpus is shared infrastructure and its access technique is a property of the estate, not of a persona.

Fold it into the Documentation Standard. Rejected: that standard is the authoring contract, already dense, and read by humans writing pages. Retrieval is a different audience (agents) and a different lifecycle — it changes when the retriever changes, not when the house style does.

Machine-gate conformance in CI. Rejected as infeasible: no lint can observe how an agent phrased a query. Enforcement is behavioural, through the operating principles and the wrap-up Check M harvest.

Consequences

Every agent that reads the corpus now has a citable technique, and a new agent inherits it by reading one standard instead of re-deriving it. The standard must be updated whenever the retriever changes — a new embedding model, a change to the RRF constant, or a third retrieval half would each make parts of it stale, and it names those specifics deliberately so drift is visible rather than silent.

The availability clauses carry a durable operational fact: ~/.claude/mcp-servers.json is a template holding an unexpanded ${PITMEM_MCP_BASIC} placeholder and returns 401 if used directly; the live credential is in ~/.claude.json, expanded there by claude_mcp_reconcile.py. That distinction cost a wrong diagnosis once and is now written down.