Observability and SRE

The short version

Bottom line. Metrics, logs and probes on a RED+USE baseline, one dashboard per service generated as code, and an alerting design built around a constraint most estates do not accept: nothing may ever wake the operator.

Risk managed — a dashboard with a silently missing panel looks like a healthy service. Where a service exports no request metrics the gap is written onto the dashboard itself rather than left as an empty chart.
The decision that mattered — quiet hours with no carve-out, which forces every overnight failure mode to fail safe instead of failing loudly. A louder page is never the answer here; an automated safe state is.
What it costs — an emergency arising at 02:00 is not seen until 07:00. That is accepted knowingly, and it is what makes the rest of the design non-negotiable.

Every service in this estate owes five things before it is considered delivered: monitoring, observability, alerting, vulnerability coverage and documentation. They are not a checklist run at the end; a service missing any one of them is not done. This chapter is the middle three.

87nodes in the derived estate graph
8service domains
1,503pages in the documentation corpus
334diagrams across 280 pages

The signal path

RED and USE, with an honest fallback chain

The dashboard baseline is rate–errors–duration for anything that serves requests, plus utilisation–saturation–errors for the resources underneath it, plus logs, plus a blackbox availability probe. Most services in a homelab do not export request metrics, and pretending otherwise produces beautiful empty panels.

So the standard defines an explicit fallback chain: a service with no RED metrics runs on USE plus logs plus blackbox, and the RED gap is recorded on the dashboard itself. That recording is the load-bearing part. A dashboard with a silently missing panel looks like a healthy service; a dashboard that says this service exports no request metrics, so availability is carried by the probe is a dashboard you can reason from. The standard is ADR-0045.

Dashboards are generated as code and provisioned from files, one per service, and each is registered in a rollup driven by its own blackbox probe target. The rollup is dynamic: a service that gains a probe appears in it without anyone editing a dashboard. The service catalogue pages are generated from the same probe inventory (ADR-0059), so the inventory has exactly one definition and three consumers.

“Logs land in the platform” is proven, not assumed

Log shipping is where observability quietly fails, because the failure mode is partial. A container ships its startup banner and nothing else. A framework writes diagnostics to a file inside the container that no agent tails. Every dashboard is green and the one line that would have explained the incident is on a disk nobody reads.

The logging standard closes that with three requirements, and the third is the one that does the work:

One mandatory log platform. Not two, not “wherever the service already writes”.
File-loggers are redirected to standard output, or agent-tailed, so diagnostic lines reach the platform — a banner is not evidence.
Shipping is verified against the live log API before the work closes. Assumed shipping is not shipping.

Stream labels conform to a declared taxonomy rather than being invented per service, because a label vocabulary that grows per author cannot be queried across services — the same argument that governs every other metadata surface here, in the tagging standard.

Alerting is designed around a constraint: nothing may wake the operator

This estate has a hard rule that most production estates do not: between 21:00 and 07:00 local time, nothing pages. Not by convention — by a blanket route-level mute on both alerting surfaces, and a hard ceiling of the lowest notification priority, clamped at the chokepoint so a rule author cannot opt out. Both decisions are recorded: ADR-0264 and ADR-0326.

That constraint is far more interesting than it first looks, because it forces the design work upstream. If a real emergency at 02:00 will not be seen until 07:00, then everything that can fail overnight has to fail safe — halt, roll back, or refuse to promote — and not degrade further while unattended. A new failure mode that gets worse unattended breaks the premise, and the answer is never a louder page: it is an automated remediation or a safe halt state that holds until morning.

What that buys, beyond a night’s sleep, is that alert fatigue has nowhere to hide. An estate that can page at 3am can tolerate a noisy rule for months. This one cannot, so the rules got fixed.

The severity taxonomy exists for the mirror-image reason. Three alerting surfaces each had their own severity vocabulary and nothing reconciled them; a rule carrying an off-taxonomy severity did not error, it mis-routed — falling through to a default receiver and paging as an emergency for a low-importance condition. ADR-0078 reconciled the three vocabularies into one contract, and the alerting standard defines the shape a conforming rule must have, including the mandatory link from an alert to the runbook step that resolves it.

Dependency chains, because a single fault should raise a single alert

A host restart once produced forty-one notifications from one root cause: the restart, seven service-unreachable alerts for services on it, four pipeline failures and a memory alert. Every trigger fired independently because nothing expressed that they were the same event.

The fix was native trigger dependencies plus inhibition rules on the metrics side, so a parent problem suppresses its children rather than each one paging on its own — ADR-0001. It is the first decision record in the corpus, and it is still the one most often cited, because notification volume is the thing that decides whether alerting gets read at all.

SLOs exist where there is a decision to make with them

Not everything here has an SLO, deliberately. An objective nobody would act on is decoration. Two that are real:

CI queue time. The share of pipeline runs that start within ten minutes of being queued, targeted at 95% over 28 days. Alerting on it is multi-window burn rate, not a threshold on a percentile — a p90 threshold fires on any busy afternoon and is muted within a week, which is exactly how a previous stuck-run trigger became ignorable. Burn rate asks a different question: is the budget being spent faster than it can be afforded?
Map freshness. The derived estate graph has a freshness SLO with an error budget and a written policy for what happens when the budget is spent — ADR-0371.