https://arronpitman.com/ Engineering notes from a production-grade homelab: platform, delivery, observability, security and AI-assisted operations. en-au https://arronpitman.com/case-studies/a-wait-that-cannot-silently-fail/ https://arronpitman.com/case-studies/a-wait-that-cannot-silently-fail/ A monitor watched a file that was never going to exist, and its silence was read as "still running" for six hours. The remedy was a rule about what a wait is allowed to be. https://arronpitman.com/case-studies/queue-time-is-a-ci-pools-latency/ https://arronpitman.com/case-studies/queue-time-is-a-ci-pools-latency/ Sixty per cent of pipeline wall-clock was spent queueing while the agent sat idle 79% of the time. The fix was a second runner — and an SLO that stops the same regression returning invisibly. https://arronpitman.com/case-studies/the-alert-that-paged-as-an-emergency/ https://arronpitman.com/case-studies/the-alert-that-paged-as-an-emergency/ Three alerting surfaces, three severity vocabularies, nothing reconciling them. A low-importance signal carrying an off-taxonomy label did not error — it mis-routed, and paged as an emergency. https://arronpitman.com/case-studies/the-gate-that-had-never-been-seen-red/ https://arronpitman.com/case-studies/the-gate-that-had-never-been-seen-red/ Every gate in this estate ships a poisoned fixture and is required to reject it. The rule exists because a gate proven only on clean input is indistinguishable from no gate at all. https://arronpitman.com/case-studies/the-map-that-could-not-be-trusted/ https://arronpitman.com/case-studies/the-map-that-could-not-be-trusted/ A generated dependency map reported "no consumers" for a database six stacks were using, and served it with a healthy 200. The fix was not a better derivation — it was making the render assert its own output before publishing. https://arronpitman.com/case-studies/twelve-hours-of-samples-proved-a-port-unused/ https://arronpitman.com/case-studies/twelve-hours-of-samples-proved-a-port-unused/ A census sampled established connections every five minutes and concluded nothing used a port. Closing it instantly broke a fleet of devices that connect for under a second, once a minute. https://arronpitman.com/pillars/ai-assisted-operations/ https://arronpitman.com/pillars/ai-assisted-operations/ An agent operates this estate under a written authority envelope, retrieves what it knows from the same corpus this site publishes, and carries a named taxonomy of the ways it has fooled itself — loaded before the work, not written up after it. https://arronpitman.com/pillars/delivery-and-gitops/ https://arronpitman.com/pillars/delivery-and-gitops/ Ninety-nine pipelines on a self-hosted pool, a change that reaches production in six minutes, and the reason a green pre-push hook is the most convincing counterfeit in the estate. https://arronpitman.com/pillars/observability-and-sre/ https://arronpitman.com/pillars/observability-and-sre/ Metrics, logs and blackbox probes on a RED+USE baseline, one dashboard per service generated as code, and alerting designed around a constraint most estates do not have — nothing may ever wake the operator. https://arronpitman.com/pillars/platform-and-compute/ https://arronpitman.com/pillars/platform-and-compute/ One hypervisor, seventeen guests and a rule that nothing is provisioned by hand — how the substrate is declared, snapshotted before every change, and kept honest by a helper that refuses to lie about whether a rollback point exists. https://arronpitman.com/pillars/security-and-compliance/ https://arronpitman.com/pillars/security-and-compliance/ Zero-trust at the edge with an approval register behind it, SBOMs for the whole fleet, secret scanning with a conformance gate — and two prudential standards run properly rather than claimed.