Engineering notes from a working homelab

The short version

Bottom line. One hypervisor, ninety-nine pipelines and roughly forty services, run to the discipline a production estate is held to — and documented well enough that the reasoning survives the person who wrote it.

• What it demonstrates — platform, delivery, observability, security and AI-assisted operations as one operating model, not five tools. Every service owes monitoring, observability, alerting, vulnerability coverage and documentation before it counts as delivered.
• What is unusual — an AI agent does the day-to-day engineering under a written authority envelope, and the ways it has fooled itself are a named taxonomy loaded before the work rather than a postmortem written after it.
• What is honest about it — the numbers below are derived from the estate at build time and stamped with the moment they were taken. The gaps are stated on the page that owns them, not omitted.

This site is written for a peer — a systems engineer or an architect who will want to know why a thing is shaped the way it is, and who will notice if the answer is thin. It is not a tour of what software is installed.

The estate behind it is a single Proxmox host and the services on it, run deliberately against production discipline: everything declared in a repository, every change applied by a pipeline, every service owing monitoring, observability, alerting, vulnerability coverage and documentation before it counts as delivered. An AI agent does the day-to-day engineering under a written authority envelope, and the ways it has fooled itself are a named taxonomy it carries into every session.

• 471architecture decisions recorded
• 97CI/CD pipelines, 76 of them deploying
• 1,503pages in the documentation corpus
• 402recorded agent lessons

Every number on this site is derived at build time from the estate itself and stamped with the moment it was taken. None of it is hand-typed into prose, and none of it is presented as live.

The five capability pillars

• Platform and compute

  Three declaration paths, one privileged path to the hypervisor, and a snapshot helper that refuses to pretend a rollback point exists when it does not.

• Delivery and GitOps

  Ninety-nine pipelines on a two-agent pool, trigger economy as capacity design, and why a green pre-push hook is the most convincing counterfeit in the estate.

• Observability and SRE

  RED and USE with an honest fallback chain, log shipping proven rather than assumed, and alerting designed around a rule that nothing may ever wake the operator.

• Security and compliance

  Zero-trust at the edge with an approval register behind it, fleet SBOMs, a secret-scan conformance gate, and two prudential standards run properly rather than claimed.

• AI-assisted operations

  An authority envelope that has narrowed as well as widened, knowledge retrieved rather than remembered, and a failure taxonomy loaded before the work instead of written up after it.

Six things that went wrong, and what came out of them

Each of these is a real incident with a real remedy, written as problem, design, trade-off and evidence. They are the shortest route into how this estate actually thinks.

• The map that could not be trusted

  A generated dependency map reported “no consumers” for a database six stacks were using — and served it with a healthy 200.

• A wait that cannot silently fail

  A monitor watched a file that was never going to exist, and its silence was read as “still running” for six hours.

• The gate that had never been seen red

  A control that is always green is indistinguishable from a stub. Every gate here ships a poisoned fixture and must reject it.

• Twelve hours of samples proved a port unused

  It wasn’t. A sampler of standing state cannot see a consumer that connects for under a second, once a minute.

• Queue time is a CI pool’s latency

  Sixty per cent of pipeline wall-clock spent queueing while the agent sat idle 79% of the time — and nothing recording it.

• The alert that paged as an emergency

  An off-taxonomy severity label does not error. It mis-routes — and a low-importance signal paged as an emergency.

Explore the estate directly

• Estate topology

  The derived service-dependency graph, published with every identifier stripped and the stripping asserted.

• Decision log

  Every architecture decision in the corpus, filterable by domain, status and year — including the gaps.

• Delivery performance

  The four DORA keys over twenty-eight days of real pipeline history, each with its derivation and sample count.

• Agent operations

  Decision cadence, the failure-class taxonomy, and how well each recorded lesson is evidenced.

How to read the rest

The standards and designs sections are the estate’s own internal documents, published verbatim where they can be. They are written for an operator, not for a visitor, so they are dense and they assume context. That is deliberate: a showcase that only contains material written to be read by a stranger is a brochure, and the point of this one is that you can walk into the source material and cross-examine it.

Everything on this site is a snapshot of a running system, and it says so on every panel. If you want to talk about any of it, the colophon has the contact details and explains how the site is built.