Platform and compute

The short version

Bottom line. Compute is declared in three repositories and provisioned by nothing else. The credential that reads the hypervisor is deliberately unable to change it, and a snapshot is not treated as a rollback point until it has been read back from the system that holds it.

Risk managed — an unverified rollback point is worse than none, because it is spent as confidence. A guest was once patched unprotected and “rolled back” to a snapshot that had never existed; the helper that replaced that path refuses rather than reporting success.
The decision that mattered — keeping the API token audit-only, and routing every privileged action through one explicit path, instead of widening the token when it got in the way.
What it costs — no fast manual fix at the hypervisor. Every change is a code change first.

The substrate is a single Proxmox host running a mix of LXC containers and VMs, with storage on ZFS and backups to a separate Proxmox Backup Server that replicates off-site. None of that is unusual. What is worth reading is the set of constraints layered on top of it, because each one was bought with an incident.

87nodes in the derived estate graph
133derived dependency edges
8service domains
7kinds of derived relationship

The compute plane is three tiers, and each tier has one provisioning path

Three declaration paths, three runtime tiers, and the rule that decides which is which is not taste: what executes this code? A script run by cron on a managed host is Ansible’s. A gate run by a pipeline lives beside the thing it gates. Application code mounted into a stock image belongs with the compose file that mounts it. Code that must run in two deployment contexts is a versioned package in its own repository, installed by version at each binding — never a synced copy, because staleness in a copy is silent.

That last case only became a rule after it was got wrong once, which is why it carries its own decision record: a fourth script home for shared library code.

Nothing is provisioned by hand, and the interesting part is what that costs

Every guest is declared in Terraform. The temptation to click one out of the hypervisor UI is real — it takes ninety seconds and the declarative path takes twenty minutes — and it is refused every time, because the second guest created by hand is the one nobody can rebuild.

The trade-off that is not usually admitted: a fully declared estate is slower to change and much harder to experiment in. The mitigation here is that experimentation happens in ephemeral containers on the container host, which are --rm, memory-capped and never persisted, so the cost of the declarative path is paid only by things that survive the afternoon.

The hypervisor’s API credential is deliberately unable to change anything

The agent that operates this estate holds a hypervisor API token whose entire permission set is audit. It cannot start, stop, snapshot or reconfigure anything. Every privileged operation goes through one path — sudo over SSH — and that is a decision with its own record, not an accident of setup.

The argument for widening the token is obvious and was rejected: a single credential that can both read the estate for diagnosis and mutate it means every read-only investigation carries the blast radius of a write. Keeping them apart means an investigation cannot become an outage by typo. The cost is that one privileged path has to keep working, and when it breaks — as it does, loudly, with an access-control error — the correct response is to fix the path, never to widen the token.

A snapshot that was not verified is not a rollback point

The rule is that no configuration change touches a guest until a snapshot exists. The subtlety, and the reason this has its own tooling rather than a line in a runbook, is that the hypervisor’s snapshot API is asynchronous: it accepts the request, returns a task id, exits zero, and then fails. A guest with directory bind mounts cannot be snapshotted at all, and the refusal arrives in the task, not the submit.

So a guest was once patched with no rollback point, and “rolled back” to nothing. The fix was not more discipline; it was a helper that picks the mechanism, falls back to a ZFS dataset snapshot where the guest-level one is impossible, reads the real snapshot list back, and exits non-zero when it cannot produce a verified rollback point. A guest whose storage supports neither mechanism is correctly refused rather than quietly proceeding.

This is the shape of most of the controls on this site: the failure was not that a check was missing, but that a check reported success for something it had not observed.

What runs where is derived, never drawn

The estate’s dependency graph is generated daily from four sources that are each maintained for another reason — the metrics agent’s probe inventory, the configuration inventory, live database connections, and one declared file for a dependency no API can enumerate. Nothing on the map is hand-drawn, so the map cannot drift without something else breaking first.

It also refuses to publish a confidently-wrong version of itself. If a derivation source dies, the render aborts before staging; if a source merely answers thinly, a calibrated gate measures the staged graph against the last known-good one, per source, and keeps the previous map. A stale map announces its age; a half-empty map announces nothing at all. The reasoning is in ADR-0359, and the freshness contract it operates under — an SLO with an actual error budget — is in ADR-0371.

You can walk the sanitised version of that graph on the estate topology page.