Skip to content
Arron Pitman — Systems Engineering

Failure classes

The taxonomy exists because of one sentence: the agent never observes the system, only its evidence — exit codes, dashboards, API responses, its own account of what it just did. Every class below is a place where the evidence and the system quietly diverge, and the divergence is invisible exactly when it matters, because the signal that would warn you is the one that is wrong.

Failure classes — 402 recorded lessons in 22 categories

Snapshot taken — build-time, not live
  • 402lessons recorded
  • 20/21failure classes hit
  • 294are a named failure
  • 55in the largest class

Pick a class to read what it actually cost

Categories that are not failure classes

402 of 402
ClassWhat went wrongEvidenceDated
FC-16An alert threshold chosen without checking WHEN the source publishes fires for months on a healthy state.confirmed2026-08-24
FC-10Running the service-catalog generator with --no-llm STRIPS the blurb from every existing page — 62 pages, 708 deletions.confirmed2026-08-23
FC-19On a battery-powered display, measure the DOMINANT load before fixing the one you can see — the screen is watts, the software is milliwatts.verified2026-08-23
FC-06In prototype mode, a CI deploy of an EARLIER commit silently reverts your later live driftverified2026-08-23
FC-05A multi-edit patch script that throws mid-way writes NOTHING, and the earlier edits are lostverified2026-08-23
FC-21A prometheus.scrape block is NOT a blackbox probe — only the probe drives the rollup, the catalog page and the start-page tile.confirmed2026-08-23
FC-21A mechanical rewrite is only as complete as the enumeration behind it — and each successive widening that finds more is proof the enumeration was never done.verified2026-08-23
FC-09A Home Assistant integration's poll interval is a constant in its source, not something the entity's behaviour reveals — read it before trusting a feed to be fresh.verified2026-08-23
FC-05A failed command substitution turns a targeted update into a silent deletion, and the tool still reports successverified2026-08-23
FC-04A custom Lovelace card can be installed, registered, configured and rendering NOTHING — and every server-side gate stays green.verified2026-08-23
FC-19A control described in present tense in a comment ships as a claim that it existsverified2026-08-23
FC-12Two pushes in quick succession race: a docker-stacks run git-pulls at start, so run N can grade commit N+1's contract against commit N's deployment.confirmed2026-08-22
FC-01A substring selector silently defines the population a monitor watches — enumerate what it matched, not what you meant.verified2026-08-22
FC-19Read the config's own comments before reporting a defect in it — the answer is often already written there, with proofverified2026-08-22
FC-12A 127.0.0.1 URL inside `docker exec` reds the EiT gate across EVERY docker-stacks pipeline — address the container by name.confirmed2026-08-22
FC-11Claim the ADR number with the helper — deriving it by hand collides, and the collision arrives lateverified2026-08-22
FC-05`az boards work-item update --fields System.Tags=` MERGES tags — it can never remove oneverified2026-08-22
FC-01A substring selector silently DEFINES the population a monitor watches — enumerate what it actually matched, never what you meant it to match.verified2026-08-22
FC-02A scripted string replacement that matches nothing exits 0 and changes nothingverified2026-08-22
FC-18A resource-release metric read INSIDE its own detection window looks like a leakverified2026-08-22
FC-15`zabbix_cron_heartbeats_monitoring.py --self-test` MUTATES live Zabbix — it is not a dry runverified2026-08-21
FC-10Normalising a display NAME breaks every cross-store join keyed on that nameverified2026-08-21
FC-01Never suppress stderr on `az boards query` — it converts a rejected query into a fake empty boardverified2026-08-21
FC-17Never carry a verification payload through the environment — at ~181 KB it pushed past ARG_MAX and every later exec died E2BIG, on a path only the second run reaches.reproduced2026-08-21
FC-04In Home Assistant the entity registry `name` is a SUFFIX, not the name — assert on state `friendly_name`verified2026-08-21
FC-12`git push origin HEAD:main` pushes the whole BRANCH, not your commit — re-audit `origin/main..HEAD` immediately before EVERY push, not once earlier in the session.verified2026-08-21
FC-12`git add -A` after a conflicted `git stash pop` commits somebody else's conflict markers — auditing `origin/main..HEAD` checks COMMITS, not the FILES inside them.verified2026-08-21
FC-02Asserting that an overlay/modal/viewer 'shows' something must measure its OVERLAP with the visible region — never the element's existence or its dimensions.verified2026-08-21
FC-02Asserting a rendered size from an element's INLINE STYLE STRING is not an assertion about rendering — measure getBoundingClientRect, because a CSS property that does not apply to…verified2026-08-21
FC-14An ADO run's sourceVersion is its TRIGGER, not what it tested — ansible-ci resets to origin/mainverified2026-08-21
FC-03A sub-agent's Monitor dies with the sub-agent — and the polling-block hook proves REGISTRATION, not livenessverified2026-08-21
FC-10A published claim is shared state — correct it with a corpus sweep, not by editing the occurrence you foundverified2026-08-21
FC-07A helper that does not parse your flag ignores it in silence — identical values across 'different' windows is the tell, not a flat trend.verified2026-08-21
FC-12A commit in a shared /home/pit/<repo> checkout is destroyed within 15 minutes by the repo-sync cron — push the commit OBJECT by SHA, never rely on the working tree surviving.verified2026-08-21
FC-08A CI gate can key on CONTENT, not path — enumerating gates by changed file misses itverified2026-08-21
FC-10A behavioural rule I add to a shared skill has consumers I did not enumerate — and one of them has no human in itverified2026-08-21
FC-08The pitlab-docs pre-push hook validates links and tags but NOT anchors — it prints 'scope: links + tags (ANCHORS NOT CHECKED)' and CI checks anchors as well, so a green hook…verified2026-08-20
FC-13The control01 `/usr/local/bin` deploy copies script names VERBATIM — there is no `.py`-stripping or rename step anywhere.verified2026-08-20
FC-21Sampling eight pages and finding the fault on all eight proved the fault was universal — it wasn't, and the working page was the diagnosisverified2026-08-20
FC-12Resuming a subagent with SendMessage and then spawning a fresh one puts TWO agents on the same Issue — /go's serial-mutation rule is broken by the orchestrator, not the agentsverified2026-08-20
FC-10Publishing a port on docker01 changes SHARED estate state — check the listener registers BEFORE the first run, not after it reddens the fleet.verified2026-08-20
FC-19A tool's own log said why it failed — read that before diagnosing it, and never mistake a card's evidence for its causeverified2026-08-20
FC-12Before pushing to a SHARED single working tree, check that origin/main's CI is already green — your commit inherits someone else's red, and the failed run lands under YOUR SHA.verified2026-08-20
FC-12A stale shared canonical checkout silently breaks the host crons that read itverified2026-08-20
FC-12A prompt handed over in a fenced code block MUST name its destination outside the block — otherwise it reads as a shell command.verified2026-08-20
FC-19A brand-new Cloudflare Pages custom domain 522s while it validates — read the attachment's own status before you diagnose the DNS underneath itverified2026-08-20
FC-21A 'flaky' pipeline can be a half-broken resolver pair — query BOTH Pi-holes, never just oneverified2026-08-20
FC-09A BLOCKED verdict is only true at the moment it was measured.verified2026-08-20
FC-05Writing an ADO Epic's state does not move its board column, and on some columns the write is a silent no-op.verified2026-08-19
FC-13The nearest precedent can be the broken one — a rule written as a comment on ONE task does not govern the other twelve that violate it.verified2026-08-19
FC-05Reducing an ADO tag set needs ONE `replace` op — the CLI path exits 0 and removes nothingverified2026-08-19
FC-04Prove you can STORE a credential before you MINT one — resolving an identity is not permission to write with itverified2026-08-19
FC-12Pitlab-access's pre-push hook reads the CANONICAL ~/ansible, not your worktree — pushing ansible to main is not enough, you must sync the canonical checkout tooverified2026-08-19
—Grepping a config file by key prefix pulls the secret in alongside the setting — filter the value out, not just the linesverified2026-08-19
FC-12`git stash` is SHARED across all worktrees of a repo — in cc-pool, a stray `git stash pop` destroys another session's work.verified2026-08-19
FC-13Derived data committed in a different repo from its source will drift, and 'I just fixed this' is when you are least likely to re-checkverified2026-08-19
FC-01Claim an ADR number with `claim_adr_number.sh` — deriving it by hand costs a 10 kB read AND still collides.verified2026-08-19
FC-12Backticks in a double-quoted `git commit -m` silently delete words from the permanent recordverified2026-08-19
FC-12Another session's dirty working tree blocks YOUR push — the pre-push hook scans the whole tree, not your commitverified2026-08-19
FC-09An aggregate gate's remediation hint is generic boilerplate, not a diagnosis — open the failing guard's own logverified2026-08-19
FC-09A snag's own text is a diagnosis you wrote, not a finding — re-derive every load-bearing claim before adjudicating itverified2026-08-19
FC-09A red ansible-ci run is NOT evidence that your commit is bad — the same SHA can go green three minutes later, because the guards reset sibling repos to origin/main mid-runverified2026-08-19
FC-02A gate piped to `tail` inside a single agent tool call launders red into green — the harness reports the PIPELINE's exit status, not the gate'sverified2026-08-19
FC-14A FreeScout incident number and an ADO work-item id are different namespaces, and they collideverified2026-08-19
FC-19A derived threshold and the quantity the code actually measures can diverge silently — check the implementation computes the SAME quantity you measuredverified2026-08-19
FC-09A decision record ages into a lie without changing a byte — write paths are cheap, re-assertion paths are rareverified2026-08-19
FC-09A CORRECTION to a wrong diagnosis is itself a diagnosis — it inherits the same doubt, and reads as more trustworthy for having corrected somethingverified2026-08-19
FC-01A `cmd 2>&1 > file` redirect captures NOTHING when the tool writes to stderr — and the resulting empty file reads exactly like a clean result.verified2026-08-19
FC-07A bisect cannot attribute a check that reads state git does not version — and the same sweep can produce a false green and a false red in opposite directionsverified2026-08-19
FC-13When a machine-enforced gate refuses nearly everything, suspect its PRODUCER was never coupled — not that everyone is non-compliant.verified2026-08-18
FC-03Run an until-loop's condition ONCE by hand before arming a Monitor on it — a malformed predicate and an unmet condition are both just silenceverified2026-08-18
FC-21In a pipewait fan-out the PIPEWAIT_RUN lines are members, not the verdict — only PIPEWAIT_RESULT answersreproduced2026-08-18
FC-06A doc page can be a build artifact — read its header before editing it, because a generator will silently delete your change within the hourproven2026-08-18
FC-13Compose designs and Epics for /go fan-out by default, not for the fewest human prompts.verified2026-08-18
FC-13Compose designs and Epics for /go fan-out by default, not for the fewest human promptsverified2026-08-18
FC-03Collapsing independent checks into one sequential job inherits its abort semantics — the first red check silently blanks every check behind it, and a skip is indistinguishable from a passproven2026-08-18
FC-15A gate that proves a notifier by sending the notification pages you on success — and the page is indistinguishable from the real faultreproduced2026-08-18
FC-02A credential PAIR is only proven by an authenticated call — never by a shape check on either halfverified2026-08-18
FC-17A control whose healthy state is ZERO must be tested against zero — every green run it has ever had took the non-zero path.verified2026-08-18
FC-21A backlog you are about to drain is EVIDENCE before it is noise — read the members, and search the incident corpus, before calling any defect newly foundverified2026-08-18
FC-19Verify the CONSTRAINT that makes a design hard before you build around it — if it dissolves, the design disappearsverified2026-08-17
FC-04Verify on the USER'S path, not a convenient stand-in — a container-direct render, an API probe and an origin fetch each proved a page worked while the user saw nothing, three times in one session.verified2026-08-17
FC-15Title: The probe you use to TEST a guard must be inert, because if the guard has failed the probe is the damage.verified2026-08-17
FC-15The probe you use to TEST a guard must be inert, because if the guard has failed the probe is the damage.verified2026-08-17
FC-10`git push origin HEAD:main` from a cc-pool session branch pushes EVERY commit on that branch, not the one just made — and a correct cross-repo push order still loses to an in-flight pipeline.verified2026-08-17
FC-01No fix available' cannot be read off the artifact we run — check the index that ships the fixverified2026-08-17
FC-12NEVER `git stash` IN A CC-POOL WORKTREE — THE STASH STACK IS REPO-WIDE, SO A POP CAN TAKE ANOTHER SESSION'S WORKverified2026-08-17
FC-04Verify gethomepage config changes via its /api/* endpoints, not SSR grep \u2014 the SSR shell embeds a stale SWR fallback; also curl with a Host override to a Caddy vhost returns an\u2026verified2026-08-17
FC-01Dependency-Track's component API silently caps at 100 rows — `pageSize` is ignored, and the truncated page reads as the whole project.verified2026-08-17
FC-19Correcting an authored cause with another authored cause is the same mistake twice — say 'the data is ambiguous' instead.verified2026-08-17
FC-12`~/.claude` IS ONE WORKING TREE SHARED BY ALL FIVE CC-POOL SESSIONS — NEVER `git add -A` THEREverified2026-08-17
FC-03An unauthenticated blackbox probe on a Cloudflare-Access-gated host proves the EDGE answers, never that the host is reachableverified2026-08-17
FC-05A two-step ADO operation described in PROSE loses a fifth of its writes; only a script that reads the result back is reliable.verified2026-08-17
FC-02A suffix-shaped attribution rule launders an object to the wrong owner, and the substantiation gate that exists to catch exactly that passes it cleanverified2026-08-17
FC-04A scanner that filters its OWN findings can delete a whole ecosystem from an SBOM while every downstream control stays greenverified2026-08-17
FC-01A DOCS ENUMERATION IS A SNAPSHOT, NOT AN AUTHORITY — 'not in the list' and 'should not exist' are byte-identical.verified2026-08-17
FC-03A dashboard band is not a monitoring control — a reactive signal behind it can stay silent through the entire failure.verified2026-08-17
FC-10A cosmetic rename is still a shared-identifier change — lowercasing the start-page group headings silently broke the calendar widget, one hour after warning the operator about exactly this.verified2026-08-17
FC-06A comment-only edit to a declarative card is still a content edit — `--apply` is the last step before `git push`verified2026-08-17
FC-01Zabbix `item.get` returns `lastvalue: '0'` for an unsigned item that has never been populated, and that is byte-identical to a genuine computed zero.verified2026-08-16
FC-13Setting `mode:` on an ACL-bearing directory silently rewrites its ACL maskverified2026-08-16
FC-04Host uptime is not application uptime — a container/core restart leaves the host counter untouchedverified2026-08-16
FC-02A file-list gate given no files exits 0 having checked nothing — argparse prefix-abbreviation makes that easy to cause by accident.cited2026-08-16
FC-01The ADO board is living memory — search it as well as the corpus, or you will re-derive a design that already exists.cited2026-08-16
FC-19Before designing a control, measure how often the thing it guards has actually happenedverified2026-08-16
FC-01An absence that is a recorded DECISION looks identical to an absence that is a gap — check before offering Arron options, or he answers on a false frameverified2026-08-16
FC-01A page-1 filter over a fuzzy search reads exactly like 'no match existsverified2026-08-16
FC-13A new soul.md/CLAUDE.md rule must carry its rules/inventory.yml entry in the SAME commit — and the claude-config gate reds the build for all five cc-pool sessions when it doesn'tverified2026-08-16
FC-04A Home Assistant markdown card whose template iterates a whole entity domain renders BLANK on the kitchen iPadverified2026-08-16
FC-05A green 'Trigger pit-memory reindex' pipeline step does not mean `docs_search` is serving the new content — and an MCP client timeout on `reindex_docs` does not mean the reindex failedverified2026-08-16
FC-01Zabbix `trigger.get` ignores `search` when combined with a `host` filter — it silently returns [] rather than erroring, so 'no such trigger exists' is a query artifact, not a…verified2026-08-15
FC-01Zabbix `history.get` returns an empty array — no error — when `history:` names the wrong value type, and that is byte-identical to 'this item has no history'.verified2026-08-15
FC-07When a probe's framing is itself an input to the thing being measured, hold the framing constant across arms or the comparison is void.verified2026-08-15
FC-19Title: An unverified claim inside a multiple-choice option is a laundered premiseverified2026-08-15
FC-19The corpus is only a control if I SEARCH it before acting — on 2026-08-15 I walked into a trap that was already documented twice, then wrote a third copy of it.verified2026-08-15
FC-04Terraform 'update in-place' is a state-graph claim, not a runtime-disruption claim — verify the provider's apply MECHANISM before touching live workloads.verified2026-08-15
FC-02Probing the Claude Code permission layer with `echo` or `dig` gives a false positive for every hypothesis, because both are auto-approved as inherently-safe read-only commands…verified2026-08-15
FC-19A per-request error rate is not a finding until you multiply it out over the chain — 0.34% is nothing, and 0.34% × 811 never finishes.measured2026-08-15
FC-01An immediate read-back after a remote config write can report the OLD state — retry before concluding the write failedverified2026-08-15
FC-01An approved design sitting in `To Do` is invisible to the queries you would naturally run to check for overlapping work.verified2026-08-15
FC-09A reported cause for alert noise is a hypothesis, not a finding — triage the signal before acting on itverified2026-08-15
FC-03A rebuild that abandons its shadow on failure turns ONE failure into a compounding outage — and a discarded return value makes it invisible while it compounds.verified2026-08-15
FC-01Search for the MECHANISM by its own name before claiming none exists — a symptom-shaped query returns a true negative about the wrong subject.verified2026-08-14
FC-07LXC guests on pve01 report the HOST's /proc/vmstat and /proc/pressure verbatim — a fleet-wide alert on them fires ~10x for one eventverified2026-08-14
FC-07LXC guests on pve01 report the HOST's /proc/vmstat and /proc/pressure verbatim — a fleet-wide alert on them fires ~10x for one eventverified2026-08-14
FC-19An inventory I author from recall is a hypothesis, not a census — derive the population before counting itverified2026-08-14
FC-11A watcher keyed on a SHA you extended from the short form fires instantly and reports the unchanged state as a changeverified2026-08-14
FC-08A stale sibling worktree makes a local cross-repo gate report a false GREEN, not just a false red — and the green is the dangerous directionverified2026-08-14
FC-02A service test plan's golden path must assert on something the service can only produce with its dependency alive — a generated API-schema or docs page cannot fail.verified2026-08-14
FC-09A remediation brief's claim that a finding is 'patchable' is a hypothesis — prove the fix exists, on both sides, before planning work around itverified2026-08-14
FC-19A multiple-choice question launders an unverified assertion into a decided fact.verified2026-08-14
FC-02A gitleaks allowlist regex matches the SECRET, not the line — so a pattern containing the key name can never fire.verified2026-08-14
FC-01`zapi item.get` (and the other Zabbix `*.get` methods) with `'searchWildcardsEnabled': true` requires the pattern to CONTAIN explicit `*` wildcards.verified2026-08-13
FC-01Parsing a value live instead of hardcoding it is only robust to the value CHANGING — not to it DISAPPEARINGverified2026-08-13
FC-14A log glob that matches rotated files replays history into Loki at the current ingest time, so Loki alerts fire on old events.reproduced2026-08-13
FC-03Launch pipewait from INSIDE the repo worktree — from a non-git cwd it silently loses ancestry matching, so a batch-coalesced pipeline run can never match your SHA.verified2026-08-13
FC-01Iterating API results with `for row in $(jq -r ...)` word-splits on ANY field containing a space, silently forging phantom rows and risking skipped records — never build a sweep…verified2026-08-13
FC-03Diagnosing a guard as 'blind to bursty events' does not stop you rebuilding the same blindness into its replacement — check the new predicate against the real event before writing…verified2026-08-13
FC-11Dependency-Track files a finding under the vulnerability ID of whichever SOURCE supplied it — OSV (`GO-…`), GitHub (`GHSA-…`), NVD (`CVE-…`) — and carries the other identifiers…verified2026-08-13
FC-14An alert built on an exporter-derived metric inherits the REPORTING host's label, so its host names the messenger, not the fault.reproduced2026-08-13
FC-03A heartbeat stamped on a no-op run is a blind watcher — and `|| true` can turn an error message into both your data and your cursorverified2026-08-13
FC-09A convention can change UNDER you mid-session — re-verify a board/config state you set hours earlier against the live mechanism, not against your memory of it.verified2026-08-13
FC-01`zapi item.get` with `search: {key_: 'x.y'}` returns EMPTY unless the value carries explicit `*` wildcards \u2014 a substring guess reads as a real absenceverified2026-08-12
—A new Zabbix UserParameter item shows state=1 'Unknown metric' until zabbix-agent2 restarts \u2014 a transient, not a broken item.partial2026-08-12
—Long-standing Zabbix trigger won't recover after a macro/threshold change \u2014 toggle status to force re-evaluationcited2026-08-12
—LLD-discovered trigger tags can't be updated via API; use action filter exclusion or DB update insteadcited2026-08-12
FC-10Disabling a Zabbix host (status=1) silently freezes ALL its items, including server-side HTTP-agent/Script items monitoring unrelated infrastructure \u2014 disable the specific\u2026cited2026-08-12
—HA firmware item prototype needs unknown\u21920 preprocessing; hermes Zabbix hostname mismatch patternpartial2026-08-12
—Zabbix discovery action silently reverts manually-added templates on host down\u2192up transitionscited2026-08-12
FC-02When remediating a defect class, write the conformance gate BEFORE fixing anything and run it against the unfixed tree \u2014 my sweep searches for the symptom I just saw, the gate\u2026cited2026-08-12
—Workflow tool \u2014 embed list/array inputs in the script, don't rely on the args field arriving as an arraycited2026-08-12
—Writing to /var/lib/pitlab/vulnscan/ (baseline.json, mode) is harness-gated for the agent \u2014 use a /tmp handoff script Arron runs.cited2026-08-12
—virtiofs dirs created by privileged LXC root appear as UID 0/root on pve01 and block Docker containers running as PUID != 0cited2026-08-12
FC-04Prove a credential/config path works using the exact tool, host AND repo the real consumer uses \u2014 a sibling tool succeeding is a proxy, not evidence; vars scoping differs per repopartial2026-08-12
FC-09Before writing a client limitation into a standard/ADR as universal, enumerate the clients the estate actually has \u2014 the constraint may not apply at all.cited2026-08-12
FC-07A failing health/diagnostic probe can be the probe's own fault \u2014 verify the instrument path before declaring the system brokencited2026-08-12
FC-04HTTP 200 / a green probe is NOT acceptance \u2014 prove the actual feature renders real data/does the real action, especially authenticated integrations.cited2026-08-12
FC-08docker-stacks push has TWO pipelines \u2014 the umbrella (scan-only) and the per-stack deploy; a green umbrella does NOT mean the deploy ran, and batch:true can swallow the deploy\u2026partial2026-08-12
FC-04An autonomous run's digest is a self-report, not evidence \u2014 verify its material claims against the live source before relayingcited2026-08-12
FC-09verify a derived/aggregated signal (a tool's summary flag, count, status) against 2-3 raw source samples BEFORE building a fix on it \u2014 it can be a false positivepartial2026-08-12
FC-05A filtered `git push` can hide a rejection \u2014 confirm with merge-base --is-ancestor, never report 'pushed $(git rev-parse HEAD)'.partial2026-08-12
FC-05When you claim to have written something to an external system (ADO Epic/issue, board, remote store), verify the write actually landed \u2014 don't state it as done from the API\u2026cited2026-08-12
FC-08Before pushing pitlab-docs, verify md-link/wikilink targets exist,partial2026-08-12
FC-08For a docs change, locally validate that every .md link TARGET file exists (incl. ADR links), not just that anchors resolve \u2014 the wikilinks hook checks bothcited2026-08-12
—On any service deploy/change, reconcile the home.pitbun.com launcher + preserve seamless SSO \u2014 per the exposure & SSO standards; full standards list is the index rolluppartial2026-08-12
FC-06An emergency fix applied live but not committed is not 'temporarily diverged' \u2014 it is armed: the next deploy of any unrelated change silently re-applies the thing you reverted.cited2026-08-12
—Never make changes to network config (UDM/UniFi) without explicit permission or approved plan \u2014 no exceptionscited2026-08-12
FC-10UDM local_dns_record replaces DHCP hostname \u2014 only supports one value; breaks existing hostname resolutioncited2026-08-12
—UDM rest/device (AP radio) writes are classifier-blocked; rest/user (client pin/alias) writes pass \u2014 hand device changes to a /tmp script.cited2026-08-12
—In user-facing time output, show the timezone abbreviation (AEST/AEDT), never a place name like 'Melbourne time'.cited2026-08-12
—When troubleshooting an alert/error/failure, query pit-memory incidents_search early for how a similar incident was actually resolved before reasoning from scratch.cited2026-08-12
—Always convert UTC times to Melbourne local time (AEST/AEDT) when presenting to Arroncited2026-08-12
—When a change makes names/docs/thresholds stale, fix them in the same pass \u2014 don't just flag and wait to be askedcited2026-08-12
—tf-proxmox/vms/vms.tf has one lifecycle.ignore_changes shared across all VMs via for_each - pre-existing drift on 3 VMs blocks un-ignoring disk/memorycited2026-08-12
—Always use Terraform (tf-proxmox on control01) to provision CTs and VMs on pve01 \u2014 never create them manuallycited2026-08-12
FC-05A red 'Saved plan is stale' on a tf-cloudflare apply usually still applied \u2014 verify live before re-runningcited2026-08-12
—The post-deploy testplan gate auto-reverts the deploy's triggering commit on failure \u2014 including your own fix; fix forward, don't assume your push is intact.partial2026-08-12
FC-12Test through the real invoker, not through your own shellverified2026-08-12
FC-14A journald/Loki line from a `tail -n N` of a cumulative log may be a STALE replay \u2014 check it belongs to this run before diagnosingcited2026-08-12
—Always conform to docs-as-code tagging standard; proactively suggest better or more tag usage \u2014 context is kingcited2026-08-12
FC-11Never hand-type a SHA, ID, or token into a command \u2014 substitute it from command output, or you silently watch a thing that doesn't exist.partial2026-08-12
FC-09A user-cron that lost itspartial2026-08-12
FC-07A queue item's status field read at one instant says what it is NOW, not that it is stuck \u2014 re-sample before calling anything 'permanently blockedcited2026-08-12
—Authoring a conformingpartial2026-08-12
FC-05git add followed by more edits then `git commit -m` (no -a) silently ships the pre-edit version; CI goes green because the commit is internally consistent.cited2026-08-12
—Keep fragile/drifting command syntax OUT of soul.md/CLAUDE.md \u2014 carry the rule + a docs_search hook, let docs hold the exact syntax.partial2026-08-12
FC-09Don't infer the cause of an all-vCPU soft-lockup from the kernel's stuck-task list \u2014 correlate timestamps against real workload history first.cited2026-08-12
—Always snapshot ALL VMs/LXCs before any config change, then delete snapshot after work confirmed completecited2026-08-12
FC-05Never write a literal CI-skip token ([skip ci], [ci skip], skip-checks: true, ***NO_CI***) anywhere in a commit message \u2014 including prose describing the feature. ADO honours it\u2026partial2026-08-12
FC-02Dry-run a destructive change by importing/exercising the REAL code tables, never a regex re-implementation of them \u2014 a re-implemented harness has its own bugs and gives both\u2026partial2026-08-12
—Before firing a long ad-hoc/force podcast render, silence the SERVICE render alerts and expect sibling starvation on the shared pit-mini \u2014 pipeline --silence is not enoughpartial2026-08-12
FC-11An unanchored grep for a short label/identifier matches it as a SUBSTRING of the longer, correct one \u2014 reproduce a finding's premise live before scoping remediationverified2026-08-12
FC-12In cc-pool, /tmp, the git stash stack and my worktree's cleanliness are all shared or volatile \u2014 re-verify each before trusting earlier-in-session state, and prefer committing\u2026cited2026-08-12
—Agent cannot docker-exec-write shared prod containers (n8n import etc.) nor self-add a settings permission to bypass \u2014 both harness-blocked; hand to the user.cited2026-08-12
FC-12Editing /home/pit/<repo> (the shared non-worktree checkout) silently reverts your changes; always use the session worktree.partial2026-08-12
—Editing ~/.claude/settings.json (hooks) is harness-gated as self-modification \u2014 needs Arron's explicit in-conversation OK; the file is also gitignored in claude-config.cited2026-08-12
—When user says 'we're done' or similar, that's the signal to wrap up \u2014 close ADO Epic, save knowledge, clean upcited2026-08-12
FC-12Re-derive the next ADR/sequential-artifact number from origin/main immediately before pushing \u2014 five concurrent sessions race for it, and a collision does not conflict in git.verified2026-08-12
—Before proposing a new standard clause for a problem you just hit, search the standard \u2014 the rule usually already exists and your failure was not reading it.cited2026-08-12
—Before grilling/designing a generator or automation, search the ansible repo for an existing house pattern to followpartial2026-08-12
—Permanent scripts must be in ansible/scripts/ + deployed via playbook; one-off/temporary scripts are exemptcited2026-08-12
FC-02A gate that greps repo content matches its own subject inside comments \u2014 strip comment lines and anchor on the invocation form, or a comment can certify compliancepartial2026-08-12
FC-07A metric collected through the limit you are sizing measures the LIMIT, not demand \u2014 size clear of saturation and re-measure, never scale off a pinned peakcited2026-08-12
FC-10Running testplan_gate.py by hand is running PRODUCTION machinery — it pages Pushover and attempts a live auto-revert of whatever commit heads the canonical checkout, which is very…verified2026-08-12
FC-08Before pushing to a repo, run every gate in its CI job, not the subset that looks related to the diff \u2014 a new dependency can red a gate that has nothing to do with the files\u2026partial2026-08-12
FC-08Never hand-roll a check the repo already has a gate for \u2014 my ad-hoc link regex passed a broken link and blocked the shared docs publish queue.partial2026-08-12
—Arron dislikes deferring \u2014 resolve everything on the plate this session, fully; do it once, do it right.cited2026-08-12
FC-07Re-permission a self-managing token (e.g. a CF token with Account API Tokens Write) only from an EXTERNAL credential, never via mutating self-edits \u2014 a bisection dropped the\u2026cited2026-08-12
—Renovate Docker self-hosted gotchas \u2014 RENOVATE_CONFIG_FILE required, pipeline run step needs sshpartial2026-08-12
FC-12A blocked-action handoff script that mutates a REMOTE host (pve01) must run ON that host \u2014 either ssh internally or hand off `ssh <host> 'sudo bash /tmp/x.sh'`, never 'paste on\u2026cited2026-08-12
FC-10Title withheld — it contains an internal identifierpartial2026-08-12
FC-08Read the pipeline's own command for a gate before running it locally \u2014 guessing the layout/entrypoint/port costs several failed runs and can fake a redcited2026-08-12
FC-11Resolve an ADO pipeline's definition id from a real run, never from assumption \u2014 two definitions can share a plausible-looking historyverified2026-08-12
FC-09An ADO issue's stated root cause is a HYPOTHESIS, not a plan \u2014 re-derive it from the live system before implementing, especially an issue I wrote myself at wrap-upcited2026-08-12
FC-05qm reboot soft-reboots the guest in the same QEMU process \u2014 a non-hotplug memory/HW change needs qm stop+startcited2026-08-12
—Enabling the PVE datacenter/guest firewall on pve01 drops ALL bridged VM traffic (estate-wide outage) \u2014 it forces bridge-nf-call-iptables=1 into pve01's Docker-owned FORWARD\u2026partial2026-08-12
FC-02Proving a new test RED by deleting the symbol it imports is a STRUCTURAL red, not a behavioural one — it proves nothing about whether the test detects the bug.verified2026-08-12
FC-01An empty/zero result is only evidence of absence once the query itself is proven to work \u2014 run a control that MUST return data before concluding something isn't there.cited2026-08-12
FC-08Invoking a gate's script directly is not proof the gate passes \u2014 run it through the scheduler, wrapper or pipeline step that will actually execute it, from the checkout it will\u2026partial2026-08-12
FC-08Editing homelab.rules.yml requires running promtool TEST rules (unit tests assert exact annotations/labels), not just promtool CHECK rules (syntax) \u2014 adding an annotation\u2026partial2026-08-12
FC-02Promtool `promql_expr_test` evaluates an expression written INSIDE the test file, so a suite full of them can pass while never loading `rule_files:` at all — and it cannot see…verified2026-08-12
—Always format handoff/agent prompts as markdown code blocks, not blockquotes or inline prosecited2026-08-12
—A custom Prometheus collector's collect() runs in the scrape thread \u2014 keep it cheap; never do file I/O or corpus-scale work therecited2026-08-12
FC-04Probe the user-facing UI root, not an internal API health endpoint \u2014 API can be 200 while UI is 404cited2026-08-12
—Arron prefers fixing root cause + unwinding unnecessary scaffolding over adding defensive layers; let him close proof-pending issues himselfcited2026-08-12
FC-02Running .githooks/pre-push by hand gives a FALSE PASS \u2014 it reads the ref list from stdin, so with no stdin it validates nothing and prints 'local checks passed'.partial2026-08-12
FC-04A post-deploy testplan/HTTP gate on an ansible docker-recreate deploy must retry \u2014 the deploy step returns a not-yet-listening container because notify handlers recreate it\u2026partial2026-08-12
—Epic closure = operational acceptance \u2014 findings on a Done Epic's capability are FreeScout incidents, never new ADO items; new Epics only for genuinely new goals with explicit\u2026partial2026-08-12
—DELETE /api/stacks/{id} can remove a live container even when the stack's ProjectPath directory is empty/missing \u2014 matches on compose project name, not on-disk filespartial2026-08-12
FC-10Unpublishing a docker01 host port \u2014 the consumer sweep MUST include host.docker.internal scrapers, deploy.yml self-gates, and the testplan runner's own reach path, or the\u2026partial2026-08-12
—Playwright base image dist-info for pip packages (setuptools/wheel) persists in base layer even after pip-install layer overwrites them; must upgrade base image to clear Trivy\u2026cited2026-08-12
—RESOLVED 2026-08-12 — pit-memory docs_search used to return ~600k-char oversized results because one catalog page was a single huge chunk; the cause is fixed, the limit 3-4 workaround is retiredverified2026-08-12
FC-05Piping `git rebase` into `tail` turns a rebase conflict into a silent success, and the following push is a no-opverified2026-08-12
FC-03Monitor the deterministic /tmp/pipewait-<id>-<sha>.log, not the launch wrapper log \u2014 the wrapper echoes the literal 'PIPEWAIT_RESULT=' string and false-matches.partial2026-08-12
FC-09pipewait DISCOVERY_TIMEOUT \u2260 broken trigger \u2014 a backed-up ADO pool registers the batchedCI run late (~14 min)partial2026-08-12
—Never manually trigger ADO pipelines that already have CI triggers \u2014 check first, pipelines are slowcited2026-08-12
FC-03Piping grep into sed/awk masks grep's exit status \u2014 a watch loop built that way fires 'found' instantly and exits without ever matching.verified2026-08-12
FC-12Don't stack concurrent pipeline pushes \u2014 control01 is memory-constrained (4GB) and OOMs the shared CI agentcited2026-08-12
—After moving /var/lib/docker to a new block device via rsync, all containers need compose down+up \u2014 overlay2 stale mount breaks container filesystemscited2026-08-12
FC-02Grepping the journal for 'oom' false-matches Zabbix mem-headroom trigger DEFINITIONS (text contains 'pre-OOM'); use the precise kernel-kill signaturecited2026-08-12
FC-12validate_notification_catalog.py is a cross-repo gate; a stale docker-stacks/pitlab-docs worktree yields false-positive kb failures \u2014 sync all siblings before trusting it.partial2026-08-12
—Never put a secret VALUE in argv — not `docker run -e`, not a script argument, not `curl -u`. Pass it by env or stdin; argv is world-readable in the process table.verified2026-08-12
FC-01Before calling a pipeline broken because nothing is arriving, prove the SOURCE is producing \u2014 force an event and watch it travel end to end.cited2026-08-12
—Never install packages on pve01 (hypervisor). File server is xt030.cited2026-08-12
—Never hardcode IPs or edit /etc/hosts \u2014 always use DNS shortnames; troubleshoot DNS if it failscited2026-08-12
—Never use email as a notification destination \u2014 always Pushoverpartial2026-08-12
FC-06nginx proxy_pass with a literal container name resolves ONCE at startup \u2014 a redeployed sidecar gets a new IP and every proxied request 502s until nginx restartscited2026-08-12
FC-13adding a new --event to any pushover_notify.py call site requires a notification_catalog.yml entry \u2014 the blocking ansible-ci gate reds MAIN for every session, not just minepartial2026-08-12
—Adding a blackbox probe target before its DNS record exists negative-caches NXDOMAIN at TWO layers (Pi-hole + dockerd embedded DNS)cited2026-08-12
FC-06A newly-catalogued Pushover event sends kb=MISSING until control01-sync deploys /etc/pitlab/notification_catalog.yml \u2014 verify the live send AFTER the sync completes, not right\u2026partial2026-08-12
FC-13Any new ansible/scripts/ script that writes /var/lib/pitlab/*.last or PITLAB_RUN_RESULT must be declared in script_observability_baseline.yml in the SAME commit, or the\u2026partial2026-08-12
FC-02A new gate's first draft nearly always over-fires; run it against the whole corpus, and pin whatever it wrongly flagged as a GREEN negative-control fixture.cited2026-08-12
FC-13A new top-level pitlab-docs section needs a cp -r line in pipelines/docs.yml assemble step, else validate_docs_published.py fails after a clean local buildpartial2026-08-12
FC-12control01 is shared by cc-1..cc-5 \u2014 pkill -f on a shared tool name kills other sessions' processes, and matches your own command line toopartial2026-08-12
FC-06Editing a shell script in place while a copy is running reroutes the live run into the new code \u2014 bash reads scripts lazily by byte offset.partial2026-08-12
—Hard-won lessons about qemu-nbd + qcow2 + VHD conversion for disk imaging workflowscited2026-08-12
—n8n operational gotchas \u2014 Merge node version traps, manual-trigger dead ends, DB-location confusion, credential PATCH no-partial-merge, Error Trigger workflow as alert signalcited2026-08-12
—New Zabbix triggers evaluate against EXISTING history and fire immediately on import \u2014 mute Pushover first or they page en masse.cited2026-08-12
—RULE: every service and every change must implement monitoring, observability, SBOM/vuln tracking (DT), and docs-as-code \u2014 not optional, not deferredpartial2026-08-12
FC-13Adding/renaming/repointing a Zabbix item or trigger in a playbook requires updating scripts/doc_gen/monitoring_config.yml in the SAME commit, else the MaC-drift reconciler\u2026partial2026-08-12
FC-05pit-memory memory_write does NOT create the draft file \u2014 it returns a scaffold for ME to Write to the returned path, then commit+pushcited2026-08-12
—MEMORY.md index has a hard load cap \u2014 200 lines OR 25KB, whichever firstpartial2026-08-12
FC-09Measure the cause before it becomes scopeverified2026-08-12
FC-07A reading taken while the fault is active describes the fault, not the system \u2014 re-measure after it clears before generalisingcited2026-08-12
FC-01A ToolSearch miss says nothing about MCP server health \u2014 run `claude mcp list` before claiming anything is down.cited2026-08-12
—Arron \u2014 don't over-engineer; match the solution's weight to the problem's actual size, especially for cleanups/governance.cited2026-08-12
FC-09When a reconcile/audit job suddenly flags a large batch as broken, suspect the audit's model drifted from an evolved reality before believing mass breakage.partial2026-08-12
—M365 MCP read_resource on a mail attachment returns extracted TEXT, never the bytes \u2014 a binary attachment cannot be saved via MCP, ask Arron to upload itcited2026-08-12
FC-07LXC containers read the pve01 host kernel loadavg, not their own \u2014 Zabbix load triggers false-positive unless threshold scaled by host/vCPU ratiopartial2026-08-12
FC-03For long-running remote jobs, harness-managed background Bash pollers get culled at turn boundaries; use detached nohup on the remote host + the Monitor tool to get notified.cited2026-08-12
FC-09An ADO apply step that fails with NO task log + abandoned post-job means the CI agent itself was OOM-killed on control01 \u2014 check the vsts unit journalcited2026-08-12
FC-07A synthetic load test using a hostname per request floods Pi-hole and rate-limits the whole client \u2014 resolve once, reuse with curl --resolvepartial2026-08-12
FC-13Alert rules citing a docs page in their kb/runbook annotation must be pushed AFTER that page lands on pitlab-docs main, or the shared ansible-ci guard reds for every other session.partial2026-08-12
—Ansible/Jinja dot-access on a dict key named `update` (or any dict-method name) returns the builtin method, silently breaking comparisons \u2014 name keys `desired` or use bracket\u2026partial2026-08-12
FC-07A `|| echo '{}'` default in a metric sampler turns a failed query into a recorded zero \u2014 the measurement can no longer tell 'no load' from 'no datacited2026-08-12
—For inline Python that contains quotes or parses JSON, use a quoted heredoc, not python3 -c with nested escaped quotescited2026-08-12
—When planning infra changes, default to minimal/proven moves \u2014 don't introduce new datasets or untested mechanisms to 'do it properlycited2026-08-12
FC-05An idempotent tool reporting success proves the end state holds, not that MY run caused it \u2014 read the log for which run did the work before claiming itpartial2026-08-12
—Never restart Home Assistant without explicit user approval \u2014 it's a production family systemcited2026-08-12
FC-08When making a control fail-closed, check WHAT IT IS ASKED ABOUT before hardening what it decides \u2014 a subject outside the input set is invisible either waypartial2026-08-12
—HAOS SSH access pattern and HA 2026.x custom component gotchas for YAML-only integrationspartial2026-08-12
FC-12A blocked-action handoff script runs in Arron's LOGIN shell, not mine \u2014 pin absolute tool paths; control01's ~/.local/bin ansible (no collections) shadows /usr/bincited2026-08-12
—Title withheld — it contains an internal identifiercited2026-08-12
FC-07A quiet HA /api/history window is not evidence a device is stale \u2014 corroborate against the entity's live last_changed before calling it a fault.cited2026-08-12
—HA iOS companion app remove+re-add preserves the widget/webhook registration; use it to fix a dead-token (invalid_grant) retry loop without losing custom configcited2026-08-12
FC-03A guard's silence proves nothing until you prove its trigger can actually observe the condition; a watcher wired to the wrong signal looks identical to a working one.cited2026-08-12
—During /grillme sessions, always present questions as multiple-choice via AskUserQuestion, not free-text prosecited2026-08-12
FC-12In a git worktree `.git` is a FILE not a directory \u2014 `os.path.isdir(root/'.git')` silently returns False and any git-history feature degrades to zeroverified2026-08-12
FC-05git add with multiple pathspecs aborts the whole add if ANY pathspec doesn't match \u2014 can silently commit only previously-staged changesverified2026-08-12
FC-06Service pages under pitlab-docs/docs/services/ are GENERATED \u2014 hand-edits are silently wiped by the next auto-refresh and take any kb anchor with them; edit alert_runbooks.yml\u2026partial2026-08-12
FC-08A gate-only change may not re-trigger its pipeline (path scope) and may break its own invocation \u2014 verify both before trusting the auto-runpartial2026-08-12
FC-02Prove a new conformance gate red against the REAL pre-change tree, not only synthetic fixtures \u2014 fixtures passed while the gate was blind to the actual defectpartial2026-08-12
FC-09A gate's red is not a finding until the invocation matches CI's and main reproduces clean \u2014 wrong --docs-src produced a wall of false failurespartial2026-08-12
FC-05freescout_incident_fix.sh --set silently CLOSES the incident despite reporting only 'fix note written' \u2014 verify status after, and reopen via the raw API with byUser.partial2026-08-12
—FreeScout HTTP 500 on every path = poisoned compiled bootstrap cache; a docker restart can't fix the baked variant (overlay layer + nginx-user clears wedged by root-owned\u2026partial2026-08-12
FC-04Forks/subagents may ignore explicit 'don't push / don't touch ADO' scope limits \u2014 verify shared state independently, never trust their self-report.cited2026-08-12
FC-12A fleet self-gating playbook validated on ONE host profile silently fails on others \u2014 test against the fleet's profile diversity (netplan vs PVE networkd, static vs DHCP,\u2026partial2026-08-12
FC-13A declaration guarded by BOTH a CI validator and a runtime reconciler needs the same fix in both \u2014 patching one leaves the other silently wrong and still pagingpartial2026-08-12
FC-10renaming or deleting shared state (a qdrant collection, a table, a topic, a path) requires grepping the whole estate for OTHER consumers first \u2014 a thorough acceptance test of\u2026cited2026-08-12
—ffmpeg and ffprobe are not interchangeable for stream-validation flags \u2014 -f null - is ffmpeg-only and ffprobe silently can't emit demux warningscited2026-08-12
—FastMCP rejects non-localhost Host headers by default \u2014 disable DNS rebinding protection for internal LAN servicescited2026-08-12
FC-05A `git rebase && git push origin HEAD:main` chain reports success while pushing NOTHING when the rebase conflicts \u2014 verify the SHA is an ancestor of origin/main, never trust\u2026cited2026-08-12
FC-05A failed Ansible play silently drops notified handlers, and convergence can NEVER re-trigger them \u2014 the config is already correct on disk, only the process is stalecited2026-08-12
FC-08A docker-stacks-deployed host cron must be declared in tz_guard_exempt, not as a schedule manifest job, or CI reds.partial2026-08-12
—exim4's dns_dnssec_ok=1 breaks lookups against internal-only DNS zones; fix and how it was diagnosedpartial2026-08-12
—cat-ing a .env/secrets file into the transcript gets blocked by the auto-mode classifier (Production Reads violation); transfer via host-to-host pipe insteadcited2026-08-12
FC-01A loop/probe that silently visits nothing returns empty, which reads identically to a real clean result \u2014 print a terminating line and prove the probe saw its targetspartial2026-08-12
—Arron only ever uses Apple Hide-My-Email proxy addresses on the internet, never his real address \u2014 ask him for a proxy whenever a service/config needs an email.cited2026-08-12
—Dynamic secret injection means source not copycited2026-08-12
—DT is the authoritative source for vuln status \u2014 not memory files or ADO epics. Always write quality data into DT.cited2026-08-12
FC-09DT MAL-* (malware) findings can be name+version collisions between local source trees and unrelated npm-published malicious packages \u2014 verify code content before assuming\u2026cited2026-08-12
FC-01DT component API paginates at 100 \u2014 always paginate all pages before concluding a component is absentcited2026-08-12
—DT audit status must be strictly maintained \u2014 per-CVE commentary required, DT is source of truthcited2026-08-12
—Validate every API/config call against the live system with zapi/curl BEFORE committing it to a slow pipeline \u2014 don't discover schema errors via failed deploys.partial2026-08-12
—When context gets tight mid-delivery, don't quietly switch from finishing the work to packaging a tidy handoff \u2014 finish, or say plainly that you're stopping and why.cited2026-08-12
—Once a Monitor is armed on a marker file, stop hand-polling that file \u2014 the monitor is the notification channel, and each extra poll is a wasted turn that tells me nothing new.cited2026-08-12
—Don't ask Arron to approve a hypercare park.verified2026-08-12
FC-10A new pitlab-docs section reusing existing service/page basenames breaks the build via the wikilinks basename index \u2014 exclude it from the slug target indexpartial2026-08-12
FC-08A broken cross-doc .md link fails the pitlab-docs CI link-check but PASSES local pre-push (which only checks tags+mermaid). Two flavors: (1) links to root docs the pipeline\u2026cited2026-08-12
FC-08pitlab-docs pre-push tag/lint hook validates the WORKING TREE, not the committed content \u2014 a working-tree-only fix passes the hook but pushes the unfixed commitcited2026-08-12
FC-08Test pitlab-docs build gates against the FULL assembled multi-repo tree, not pitlab-docs/docs alonepartial2026-08-12
—Docs-as-code is preferred over Claude memory as source of truth; docs live alongside code and are updated atomically with it; always review and refine for qualitycited2026-08-12
FC-08docker-stacks deploy pipelines trigger on stacks/group_vars/inventory paths only \u2014 NOT pipelines/** or scripts/**, so a gate/pipeline-only change won't auto-deploy; queue one\u2026partial2026-08-12
—Reusable gotchas from the docker-stacks Phase 0.5 (gotenberg/pingvin) e2e port - apply to every future stackpartial2026-08-12
—docker image prune -af removes ALL unused images, not just dangling ones \u2014 exceeds narrower approval scope even if no running container is affectedcited2026-08-12
FC-12Never kill containers selected by `docker ps --filter ancestor=<image>` \u2014 production services share the images I run ephemerally; kill by the exact container name I createdcited2026-08-12
—Use docker01 for library-heavy work; never install pip packages on control01; never invoke interpreters directly on docker01's host eithercited2026-08-12
—Patterns that worked/failed when building pipeline-committed doc stepspartial2026-08-12
—Auto-mode classifier blocks irreversible infra changes (apt autoremove, lvremove) even when pre-authorized in the same task message - but unlike the credential-leakage\u2026cited2026-08-12
FC-11A get-or-create keyed on a DESCRIPTION orphans the old object whenever anything in that description changes \u2014 including a threshold embedded in it, so the fix you shipped runs\u2026partial2026-08-12
FC-13Deleting/renaming a docs page that a Zabbix-trigger or alert-rule kb tag points at reds the notification-catalog guard (pipeline 33) \u2014 grep kb refs and repoint BEFORE pushingpartial2026-08-12
—Decom must never destroy PBS backups \u2014 they are the recovery safety netpartial2026-08-12
FC-13Deleting a playbook/script/file during a decom must include grepping the standards-enforcement registry + docs that name it as a 'mechanism'; a dangling reference reds the\u2026partial2026-08-12
FC-13A control declared fleet-wide but enforced in one place decays silently \u2014 build the conformance gate that watches it, not just the control.partial2026-08-12
FC-07A cumulative counter divided by uptime is a lifetime average, never a current rate \u2014 two samples and a subtraction, or don't claim a rate.verified2026-08-12
—Auto-Mode Bypass / Credential Leakage classifier blocks any command/file/playbook that would surface a literal stored credential, even via indirection \u2014 confirmed 6x; clean\u2026partial2026-08-12
FC-05A create-if-missing config-as-code run reporting ok does NOT mean the live object matches the code \u2014 for anything that already exists, the corrected value in Git is silently\u2026partial2026-08-12
—Rule to audit control01 for newly installed packages at session end; all runtime execution must go to docker01 ephemeral containerscited2026-08-12
—When asking the user to make a non-obvious decision, first summarize the investigation that leads into it \u2014 don't ask coldcited2026-08-12
—Container metrics come from Prometheus/cadvisor only, not Zabbix (ADR-0003)cited2026-08-12
FC-07ContainerHighMemory alert: two distinct false-positive patterns \u2014 cache-inclusive metric and div/0 for containers with no mem_limitcited2026-08-12
—All services must be configured as code \u2014 config management is everything; no manual config that isn't reflected in Ansible/Terraformcited2026-08-12
FC-12When a large consolidation deletes/absorbs many shared-repo files, another cc-pool session may edit those same files mid-flight \u2014 rebase surfaces modify/delete conflicts; port\u2026partial2026-08-12
FC-12Before fixing a defect in shared repos, check origin/main for a concurrent autonomous agent's commits \u2014 it may have already landed a better fix mid-runcited2026-08-12
—Always set explicit container_name in every docker-compose service \u2014 Docker auto-generated names (e.g. loving_driscoll) appear verbatim in Prometheus/Zabbix alerts and are\u2026partial2026-08-12
—Community scripts: check resource requirements and major version compatibility before runningcited2026-08-12
—When repeatedly using the same tool/API pattern in a session, save it to memory before the session endscited2026-08-12
—Title withheld — it contains an internal identifiercited2026-08-12
FC-10Editing the claude CF API token REPLACES its permission set \u2014 appending new groups strips DNS/Tunnel and breaks tf-cloudflare; token now self-edits; RULE 14 gates changescited2026-08-12
FC-07An EXIT trap must restore only what it actually changed \u2014 an unconditional 'put it back how I found it' silently undoes legitimate work a later phase didcited2026-08-12
—CLAUDE.md is sacred \u2014 never edit it as part of project work; only on Arron's explicit instructioncited2026-08-12
—Before any change that disrupts the Cloudflare edge/tunnel or Plex path, check for active Plex streams firstcited2026-08-12
FC-12Before concluding another cc-pool session did something, match the commit SHAs against what THIS session pushed \u2014 I misread my own work as a collision and told Arron to stand downcited2026-08-12
FC-12RESOLVED 2026-06-21 (Epicverified2026-08-12
FC-07cadvisor emits one series per container INSTANCE \u2014 profiling by `name` mixes stale pre-change containers; and enumerating from container_spec_cpu_quota hides every uncapped\u2026partial2026-08-12
FC-06A single-FILE docker bind mount pins the inode; ansible copy (atomic rename) leaves the container serving stale content \u2014 mount the directorycited2026-08-12
FC-13A blocking gate pushed to main ahead of the corpus repair it demands reds the shared pipeline for every concurrent pusher; cross-repo gates cannot land atomically, so sequence\u2026partial2026-08-12
FC-06deleting a bind-mounted dir inode breaks the running container's mount; clear contents (find -mindepth 1 -delete), never the dircited2026-08-12
FC-06Docker compose cannot see bind-mount CONTENT, so a config-only change deploys green while the container keeps running the old config \u2014 stamp the config checksum as a label to\u2026cited2026-08-12
FC-03A background waiter that polls `pgrep -f <pattern>` matches its own command line and never exits \u2014 wait on the PID with kill -0 insteadverified2026-08-12
FC-01pitlab-docs uses trigger batch:true, so there is never a run labelled with my SHA \u2014 verify by ancestry (git merge-base --is-ancestor my-sha run-sha), not by SHA match; pipewait\u2026partial2026-08-12
FC-07Reading a background task's .output file can catch it mid-write and return a misleading partial result \u2014 verify against logs/live state before concluding success or failurecited2026-08-12
FC-01Waiting on an ADO pipeline run after a push? Use pipewait.sh (ADR-0205), not hand-rolled polling. az pipelines runs list can hide the in-progress auto-trigger (FinishTime\u2026partial2026-08-12
—AWX SCM credential refresh \u2014 must set BOTH username and password, not just passwordcited2026-08-12
—Auto-mode permission classifier can block actions even after the user explicitly confirms in-chat; don't loop retrying \u2014 have the user run it once via `!` prefixcited2026-08-12
—Always set icon + category on new HA automation entities after creation \u2014 lighting automations use desk-lamp + lighting categorycited2026-08-12
FC-02Before hardening a lenient gate (skip\u2192fail-closed), audit live data for valid inputs the naive strict rule would false-positive on, and make the parser precise firstpartial2026-08-12
FC-08A green enforcement gate only proves its OWN scope \u2014 audit estate-wide invariants with an independent inventory (ss per host), not the gate's lens.cited2026-08-12
FC-03waiting on an async result (handoff-script log, pipeline run) \u2014 one long background watcher keyed on the terminal signal; never chain short re-armed polls, and don't watch at\u2026cited2026-08-12
FC-02A gate that greps for a substring can match incidental text and pass while the feature is broken \u2014 assert on parsed/structured fields, and prove the gate RED before trusting green.cited2026-08-12
FC-12An ansible assertion runs as root by default; git refuses another user's repo, so a check about pit's crons must set become_usercited2026-08-12
FC-10Ansible play vars: block overrides host_vars \u2014 put fleet defaults in group_vars/all insteadcited2026-08-12
FC-12ansible.builtin.shell runs /bin/sh (dash) \u2014 `set -o pipefail` and other bashisms die at runtime and ansible-lint does NOT catch itverified2026-08-12
FC-12Running `docker` from an Ansible shell/command task needs DOCKER_HOST=ssh://docker01 set explicitly; the docker01 context works from interactive bash but not Ansible's\u2026cited2026-08-12
FC-05Ansible cron jobs with date +%s silently fail to write \u2014 % must be escaped %cited2026-08-12
FC-04Ansible docker compose deploys need flush_handlers + health gate; recreate:always silently succeeds even on crashloop.partial2026-08-12
FC-12ansible command/shell module runs without HOME set \u2014 a script relying on ~/-relative fallback (infisical_env deploy-creds, ~/.config files) fails; set environment HOME=/home/pitpartial2026-08-12
—ansible `command: cmd:<repo-script>` runs the file directly, so it needs the +x bit \u2014 a 0644 script fails permission-denied every retry.partial2026-08-12
—ansible.builtin.command word-splits a JSON arg on spaces; use shell + single-quoted JSON, and $N in an ssh command expands locallypartial2026-08-12
FC-14When querying a past window (promq offset, log ranges), compute the anchor from `date -u`, never by inferring 'now' from an epoch in a tool response.partial2026-08-12
FC-12An external `git reset --hard` destroys uncommitted edits in a shared checkout, and git status then reports cleanverified2026-08-12
FC-03Alertmanager inhibition source must match how Prometheus gets the metric \u2014 InstanceDown only works for Prometheus-native scrapes, not Alloy remote_write targetspartial2026-08-12
—Alloy River config gotchas for syslog pipelines \u2014 stage.match selector limitations and RFC format handlingcited2026-08-12
FC-11Editing a host's alloy_extra_log_paths re-keys every later tail (components are named by LIST INDEX) so it re-reads the whole file into Loki \u2014 a replay storm that looks like a\u2026cited2026-08-12
—Bash allowlist patterns must use the full binary path, not bare command name \u2014 bare name won't match when the skill uses /usr/sbin/...cited2026-08-12
—mute_time_intervals does NOT propagate from parent to child routes in Alertmanager \u2014 each leaf route must declare it independentlycited2026-08-12
—Every new Prometheus alert rule must include runbook and dashboard annotations (ADR-0019).partial2026-08-12
FC-11Never write an ADR cross-reference filename from memory \u2014 the number is right and the slug is wrong; ls docs/adr/NNNN-* firstpartial2026-08-12
FC-01az boards flat WIQL query silently caps at ~1000 rows \u2014 never build child-count/rollup maps from one bulk query; use scoped per-parent queries.cited2026-08-12
FC-05az boards work-item create --fields 'System.Parent=N' is silently ignored \u2014 the item is created orphaned; use relation add and read the link back.cited2026-08-12
—One Epic per goal, not per session \u2014 resume existing Epics; create early once direction is clearcited2026-08-12
—Never mark an Epic Done if it has open child Issues or Taskscited2026-08-12
—az boards / az devops CLI has sharp edges that cost a retry each time \u2014 the top gotchas inline, full command reference in the docs corpuscited2026-08-12
—ADO pipelines have CI trigger enabled by default; schedules: block alone does NOT disable it \u2014 must add trigger: none explicitlycited2026-08-12
—Approving an ADO deployment/environment gate is classifier-blocked; cancel-and-requeue or hand off a /tmp script instead of retrying.cited2026-08-12
—If I'd label an option '(Recommended)', just do it \u2014 don't spend a question on it.cited2026-08-12
FC-02Never verify published docs by fetching docs.pitbun.com \u2014 Access returns 200/302 for any path, so absence-checks pass vacuously; read the origin container instead.verified2026-08-12
FC-03absent_over_time(x[Nh]) is TRUE from a service's first minute, so widening the window never fixes a warm-up false positive \u2014 gate on a precondition proving the series could\u2026cited2026-08-12
FC-11A test that reconstructs an identifier the service derives will pass against a service that never produced itverified2026-08-12
FC-02A synthetic fixture that is missing the real content's shape produces a green test over a broken fix — after a corpus/fleet-wide change, re-measure against the REAL population…verified2026-08-12
FC-17A guard that only ever runs in the mode nobody exercises has been tested for strictness, never for reachability.verified2026-08-12
FC-09A design doc's own inventory table is a prior measurement,verified2026-08-12

20 of the 21 named failure classes have been hit at least once. The 1 that have not —unmultiplied-rate — are listed rather than omitted, because an empty class is a class this estate has not yet been caught by, not a class it is immune to. Every lesson carries the tier of evidence behind it; a corpus that did not grade itself would be a pile of confident claims, which is the exact failure it exists to prevent. 3 titles carried an internal identifier and are withheld by the same gate that decides which pages may ship; the record stays in the log. The lesson pages themselves are not published — they carry the raw detail of real incidents on this estate — so what ships is the title, the class, the evidence tier and the date. Corpus at commit f1121ff.

Why two groups and not one number

Section titled “Why two groups and not one number”

Only some of these categories are failure classes. The rest are a different kind of record — a settled ruling about how the work should be done, or a fact about this specific environment that was expensive to learn. Rolling them into one count would be the same category error the taxonomy itself warns about: a populated list where the members that matter are not the ones being counted. So they are shown apart, and the split is derived from the estate’s own declaration of which names are failure classes rather than from a judgement made here.

The classes with nothing behind them

Section titled “The classes with nothing behind them”

Several named classes have no lesson attached. They are listed anyway. An empty class is not evidence of virtue — it is a class this estate has not yet been caught by, or has not yet recognised when it was. Hiding the empty ones would turn a taxonomy into a scoreboard, and a scoreboard is exactly the artifact that stops being read.

The evidence tier is the part that keeps it honest

Section titled “The evidence tier is the part that keeps it honest”

Each lesson carries how well it is actually evidenced: verified live against the system it concerns, reproduced, partially checked, or merely cited. Without that grading the corpus would be a pile of confident claims — which is precisely the failure mode it exists to prevent, applied to itself.

The uncomfortable entries are the useful ones. More than one lesson here records a weak assertion written in the same session as a warning about weak assertions. Knowing the class does not stop you instantiating it; what catches it is looking at the artifact with your own eyes.