ADR-0018: Agent knowledge retrieval — RAG over docs-as-code via pit-memory MCP, not a parallel memory store

Status	Accepted — implemented (Epic #988, 2026-06-21); pit-memory live on docker01, ~71 docs / ~704 chunks indexed
Date	2026-06-21
Deciders	Arron + Claude (design pressure-tested via /grillme, 5 branches)
Supersedes	Extends ADR-0016 (commits to its “Tier 2 — external MCP retrieval” path)
Context	ADR-0016 kept agent memory in the native flat-index design and deferred external retrieval “only on evidence”. The MEMORY.md cleanup (Epic #937) supplied the evidence: the native 200-line/25 KB index ceiling is fundamentally too small for a growing fleet knowledge base, and 84 of 154 memory files are system documentation squatting in agent memory (a RULE 9 violation). The index didn’t just overflow — it had merged two different kinds of knowledge into one capped store.

Context

Claude Code’s native auto-memory loads MEMORY.md wholesale at session start, hard-capped at the first 200 lines or 25 KB (ADR-0016, spike #958) — no sharding, no @import, no pluggable retrieval hook. Topic files are recalled on-demand by an opaque implicit mechanism. The binding constraint is entry count (the 200-line ceiling, ~195 entries), not bytes.

Auditing the corpus during Epic #937 exposed the deeper issue. The store holds two genuinely different kinds of knowledge that were merged into one capped index:

	Descriptive / reference	Corrective / behavioural
Prefix	reference_* (31), system project_*	feedback_* (84)
Nature	How the system works — API patterns, ports, IDs	“Don’t do the obvious thing, it breaks Y”
Growth	Fast — every service ever documented	Slow — a new gotcha every few sessions
Authoritative home (RULE 9)	pitlab-docs	native memory
Safe to make pull/cold?	Yes — fetched when already working in-domain	No — fires when the agent thinks the naive path is fine; it won’t self-trigger a search

The fix is not a bigger index or a parallel memory database. It is to (1) put each kind of knowledge in its correct home, and (2) give the agent a retrieval tool over the authoritative, unbounded one (the docs).

Decision

Build pit-memory, an MCP server that is a hybrid retrieval layer over the docs-as-code corpus (pitlab-docs) — not a new database of record. Knowledge stays markdown-as-code in git; the server indexes it and serves on-demand search to the agent as MCP tools.

The design was pressure-tested through five decision branches (/grillme); each resolved decision is load-bearing:

B1 — Push/pull split is by KIND, not importance

• PUSH (native MEMORY.md, always loaded): operating rules + corrective feedback_* gotchas. Grows slowly (~100 lines), fits under the ceiling for years.
• PULL (pit-memory, on-demand, unbounded): reference_* + system project_* + the doc corpus. Going cold is safe because the agent retrieves it when already working in-domain.
• Rationale: gotchas are the corrective layer; they only work because they are passively present. The agent will not self-trigger a search for a fact it does not know it needs, so corrective knowledge must stay pushed.

B2 — Write path is a smart memory_write(content, kind) tool

Kills the friction asymmetry that created the junk drawer (a memory file is one cheap Write; a proper doc is heavy). One tool call routes by kind: feedback/pref → native memory file; reference/system → scaffolds a structured doc stub. In-session cost is one call either way.

B2a — Quality gate is a DRAFT TIER (capture ≠ publish)

pit-memory indexes the repo markdown source, not the built MkDocs site — so “retrievable by the agent” and “published to the public site” are separate gates. memory_write commits system knowledge to a draft (draft: true / _drafts/): excluded from the MkDocs nav (the public site stays curated) but indexed by pit-memory (retrievable the same session). A curation pass promotes draft → published.

B3 — Retrieval quality via structure-aware chunking

Chunk on markdown headings; each chunk is one dense, self-contained fact + breadcrumb context. This requires a doc-structure standard — one dense fact per descriptive heading, tight body — enforced on RULE 9 and the migration (reference facts kept dense, not padded into prose). Ranking is hybrid: keyword (exact tokens — hostnames, IDs, env vars, where dense vectors are weak) + semantic.

B4 — Freshness via two paths

• memory_write → write-through synchronous index before returning → guarantees the agent can retrieve what it just learned (no false “it was lost” → duplicate).
• Bulk git pushes to pitlab-docs → batch reindex via a pipeline-18 webhook (minutes-eventual; nobody blocks on these).

B5 — Scope: full hybrid on day one

Keyword + qdrant + nomic-embed (pit-mini) from the start. The exact-token majority is served by keyword, but the vector layer is cheap to add (infra already idle) and the complete enterprise pattern is the point — the homelab is the vehicle, fluency in the pattern is the goal. The service satisfies the Five Pillars from day one and runs as a container on docker01.

Architecture

Retrieves from: pitlab-docs markdown + _drafts + residual native-memory files. The blog/ subtree is the one deliberate exclusion — narrative wrap-up reports are session logs, not reference knowledge, so they are filtered out of the index (ADR-0032); _drafts by contrast stays indexed. Writes to: nothing authoritative of its own — authoring always lands as a reviewable markdown file in git; the server only (re)indexes. It is a read/retrieval brain, not a store of record.

Alternatives considered

Option	Reason rejected
“Pull covers everything” (original ADR draft)	84/154 memories are corrective gotchas that fire exactly when the agent feels certain it is right — it won’t self-trigger a search for them. Making them cold trades away the behavioural safety layer. Resolved by B1: split by kind, gotchas stay pushed.
Parallel memory vector store (server owns its own authored corpus)	Duplicates the source of truth and creates a two-store sync problem. The insight is that “agent memory” of system facts is docs-as-code — index it, don’t fork it.
Auto-publish doc stubs	Relocates the junk drawer into the published doc site — worse, because that site is the curated source of truth. Resolved by B2a: draft tier separates capture from publish.
Naive fixed-window chunking	Splits/merges facts across boundaries → noisy, low-actionability retrieval. Resolved by B3: structure-aware chunking + doc standard.
Periodic-only reindex	Same-session writes invisible until the next cycle → agent thinks captures were lost and duplicates them. Resolved by B4: write-through for tool writes.
Consolidation only / keyword grep, no service	Defers the ceiling rather than removing it, leaves system knowledge in the wrong home, and forgoes the enterprise-pattern learning that is the actual objective.
Raise / shard the native index	Impossible — closed harness constraint (ADR-0016).

Consequences

• Single knowledge base, two consumers: humans via the MkDocs site, the agent via MCP retrieval. What the agent can recall equals what is documented — a forcing function that improves doc quality (RULE 9 / pillar-5 intent).
• Migration debt is paid down: ~70 reference_* and system project_* memories migrate to structured docs (one-time, phased). The native index shrinks to rules + feedback_* (~100 lines) and the ceiling stops being a live constraint.
• pit-memory is a new Five-Pillars service on docker01, reusing qdrant + pit-mini (no new heavy infra). It must monitor its own retrieval health and index freshness.
• Pull reliability is mitigated by design (gotchas stay pushed; standing CLAUDE.md search-first instruction) but remains a watch-item to validate in practice.
• Index integrity is a hard property, learned the hard way. The reindexer reads a docs volume that is bind-mounted into the container and rebuilt on each sync. Two safeguards are load-bearing: (a) the sync clears the volume’s contents (find -mindepth 1 -delete), never the directory inode — deleting the inode leaves the running container mounted on the old, empty inode so the indexer reads 0 files; and (b) a reindex that finds 0 chunks keeps the existing index rather than wiping it. Without both, a transient empty source silently destroys retrieval. See the pit-memory service doc.
• Implemented under Epic #988 (2026-06-21): pit-memory runs on docker01; the four open parameters below are all resolved.
• Semantic retrieval can fail silently while indexing stays green (2026-06-22, Epic #1066). docs_search was found returning keyword-only results: qdrant-client floated to 1.18.0 (unpinned) which removed QdrantClient.search(), so search_semantic() raised, swallowed the error, and returned []. FTS5 also breaks on hyphenated terms, so pit-memory returned nothing. The write-path alerts (PitMemoryIndexEmpty, PitMemoryIndexStale) missed it — qdrant held 727 valid vectors the read path never queried. Fix: .query_points() + pin qdrant-client==1.18.0 (ADR-0005); added Loki alert PitMemorySemanticSearchErrors on the read path. A docs_search CI smoke test would catch the class earlier.
• Retrieval surface extended beyond docs to the incident corpus (2026-07-02, Issue #1430). pit-memory gained an incidents_search MCP tool that queries the ADR-0068 distilled incident corpus (Qdrant freescout-incidents, kind=incident) — a second retrieval collection beside the pit-memory docs collection, reusing the same nomic-embed-text/pit-mini embed path. This generalises the ADR’s original scope (“RAG over docs”) to “RAG over both the documented design and lived incident resolutions”: docs_search answers “how is this meant to work”, while incidents_search answers “how was this actually fixed last time”. It is the interactive “Claude triage” consumer that ADR-0068 named but never built. A new pit_memory_incident_searches_total counter covers it (Pillar 1).

Open parameters — resolved during implementation (Epic #988, 2026-06-21)

All four were settled while building pit-memory; recorded here so the decisions aren’t lost.

1. Curation cadence — RESOLVED. Drafts are promoted at session wrap-up. /wrapup Check E enumerates pending _drafts/, proposes a publish list, takes Arron’s explicit approval, then publishes the approved to docs/reference/ (strip draft tag → commit → pipeline-18 publishes to the site + reindexes pit-memory). Cadence = every wrap-up; ownership = Arron’s approval at that gate. Rejected: a standing scheduled curation agent — heavier to run, and the wrap-up is already a natural human checkpoint. The point of the draft tier is that capture ≠ publish; this gate supplies the missing publish step so _drafts/ can’t silently become the junk drawer this ADR set out to avoid.
2. Hot-set composition + domain map — RESOLVED. Native memory is trimmed to operating rules + corrective feedback_* gotchas, plus one always-loaded reference-pit-memory-search entry: a search-first instruction and a terse domain map of what’s retrievable, so the agent forms good queries instead of missing docs it doesn’t know exist. The standing search-first instruction also lives in CLAUDE.md RULE 9.
3. Migration mechanics — RESOLVED. ~31 reference_* memories were migrated to _drafts/, standardised to the Documentation Standard, then promoted to the published docs/reference/ section. Faithfulness was verified against a pre-migration filesystem snapshot (one doc had lost detail in the original migration; restored).
4. Five Pillars wiring — RESOLVED. Monitoring/observability via /health + the pit_memory_chunks_indexed / pit_memory_index_age_seconds metrics; alerting via PitMemoryIndexEmpty (chunks == 0) and PitMemoryIndexStale (index_age > 8d); SBOM via the image’s Dependency-Track project; docs via this ADR and the pit-memory service page.