ADR-0045: Per-service dashboards as code with a dynamic blackbox-driven rollup, on a RED+USE+logs baseline

Status: Implemented 2026-06-24 (Epic #1182 complete: prereqs + governance + rollup + dedicated section + full dashboard backfill — 45 dashboards across all tranches. Only #1189 RED-gap acceptance carries forward as standalone tech debt.)

Status	Accepted (implementation in progress)
Date	2026-06-24
Epic	ADO #1182

Implementation reconciliation (2026-06-24)

• Both prerequisite gaps are RESOLVED. compose_project ships as a Loki stream label (Alloy relabel + loki-labels.yml v1.3, Issue #1185); all 50 blackbox targets carry service+domain (Issue #1186) — 34 services / 7 domains live in probe_success.
• Folder mechanism resolved. The decision deferred “folder by domain” to the standard; as built, per-service dashboards live in a dedicated “Service Dashboards” Grafana section (separate file provider, sibling path) kept apart from the legacy “Homelab” dashboards, at Arron’s request. The rollup sits at the section root.
• Amendment (2026-06-24) — sub-folder nesting dropped. The section was originally provisioned with foldersFromFilesStructure: true to nest dashboards under “Service Dashboards / <domain>”. Grafana 11.6.15 (a renovate patch bump) tightened the provisioner to reject folder/folderUID alongside foldersFromFilesStructure ('folder' and 'folderUID' should be empty using 'foldersFromFilesStructure'), which crash-looped obs-grafana on the next deploy (the error fires at provisioner init, before any dashboard is read — unrelated to the dashboards). Resolved by keeping the named “Service Dashboards” section (folder/folderUid) and setting foldersFromFilesStructure: false: all per-service dashboards now collapse into the single section folder, with <domain> carried as a dashboard tag for filtering rather than as a sub-folder. The <domain>/ subdir tree is retained on disk (scanned recursively) and remains the basis for the rollup.
• Shipped (complete): the dynamic rollup (Issue #1188) and 44 conforming per-service dashboards (Issue #1187) — 20 Docker (radarr gold + 19 fan-out), 13 non-Docker (host node_* USE, {host=} logs), and 11 Docker edge cases (cloudflared/infisical RED-metric’d; the rest USE+logs+blackbox). 8 blackbox probes added across the tranches incl. a tcp_connect module for non-HTTP services (samba/smtp). The postgresql target was a ghost host and was removed (Issue #1190). ~21 services run on a documented RED-gap (Issue #1189).
• Lesson: probe_success identity stayed on the target label (ADR-0023) — service/domain were added alongside, not as replacements, so existing CF-tunnel alerts and dashboards were unaffected (additive, non-breaking).

Context

Grafana dashboards in pitlab grew organically — one per problem as it arose (plex-ipad-sync, cloudflare-tunnel-health, docker-containers, pihole-dns, …). There was no standard for what a dashboard per service must contain, no consistent way to find “the dashboard for service X”, and no single place that answers “is everything up right now?”. Prime Directive 8 (the Five Pillars — every service needs monitoring, observability, alerting, SBOM, and docs) named “a Grafana dashboard where applicable” under Pillar 2 (Observability), but “where applicable” is unenforceable and produced uneven coverage: some services have rich dashboards, others none.

Two platform changes made this the right time to standardise. First, kuma01 (Uptime Kuma) was retired — the homelab lost its single-pane synthetic-monitoring / status view, and the per-target HTTP up/down signal it provided. Second, the observability stack already grew a blackbox exporter (obs-blackbox-exporter:9115, ADR-0011, ADR-0023) probing both the Cloudflare edge (cf_tunnel_health) and docker01 workloads directly (internal_http_health). That exporter is the natural replacement for kuma01’s synthetic probe — defined as code, already feeding Prometheus — but its output was not yet shaped into either per-service health panels or a rollup.

The need: a single, enforceable definition of a per-service dashboard, a uniform health signal that feeds both those dashboards and an auto-discovering status rollup, and a governance hook so the standard is met at deploy time rather than retrofitted.

Decision

Adopt a per-service dashboard standard, authored as code, with one dynamic query-driven rollup, all keyed off a uniform blackbox health signal. Full authoring detail lives in the Dashboard Standard; this ADR records the load-bearing choices and their rationale.

1. The unit is a service/workload, joined by a stable key. For Docker workloads the unit is the compose stack, joined on the compose_project label — available on the metrics side from cAdvisor (container_label_com_docker_compose_project) and, once the prerequisite below lands, on the Loki side as a stream label. For non-Docker services (Plex on plex01, mesh01, zabbix) the unit is service-on-host, joined on the instance label plus the exporter job. One dashboard per unit, no more, no fewer.

2. Dashboards are code, loaded by Grafana file provisioning. Dashboard JSON is committed to docker-stacks/stacks/observability/templates/grafana/provisioning/dashboards/ beside the observability stack, deployed by deploy.yml, and picked up by the file provider (updateIntervalSeconds: 30) — hot-reload, no Grafana restart. No dashboard is created or edited in the Grafana UI as the source of truth.

3. A fixed baseline panel set per dashboard. RED (Rate, Errors, Duration) where the app exposes request metrics; else an exporter’s metrics if one exists; else proxy-RED derived from edge logs / blackbox; if none of those is feasible, fall back to USE + logs and raise an ADO gap item. Plus container/host USE (Utilisation, Saturation, Errors), Loki error logs for the unit, and a blackbox availability panel. This is the floor, not the ceiling.

4. One uniform health signal: probe_success{service,domain}. The blackbox exporter’s probe_success is the single health primitive. It drives each dashboard’s up-panel and the rollup. Probe targets are defined as code in the Alloy config; service and domain are required labels on every target.

5. The rollup is one dynamic, query-driven status matrix. A single dashboard groups services by domain, auto-discovers them from probe_success (no hardcoded service list), and drill-down-links each cell to that service’s dashboard. Adding a service to the probe targets makes it appear in the rollup automatically.

6. Grafana organisation and tagging are fixed. Folder by domain; required dashboard tags service, stack-or-host, domain, service-dashboard; dashboard UID = service name (stable, link-addressable).

7. Every alertable panel is coupled to its alert. Each panel that represents an alertable condition either links its Alertmanager/Zabbix alert or carries a documented “no alert — why” note. No silent gap between “shown on a dashboard” and “paged on”.

Prerequisites this decision exposes (build-phase work)

Verification against the live stack (Epic #1182, session 1) found the design’s join keys are not fully present today:

• Loki does not ship compose_project. The Alloy discovery.relabel "docker_labels" block maps container_name/image/host/env/job/device_type only; the Loki taxonomy (ansible/standards/loki-labels.yml) does not list it. → Add a __meta_docker_container_label_com_docker_compose_project → compose_project relabel rule and bump the taxonomy to v1.3 (ADO #1185). Cardinality is bounded (~34 live values), within the taxonomy’s low-cardinality rule.
• probe_success carries no service/domain labels. Today it carries {device_type, env, group, host, instance, job, target}; identity is the target label (ADR-0023). → Extend every blackbox target block with service + domain (ADO #1186).

cAdvisor’s compose_project label and Grafana file provisioning were verified present and working.

Alternatives considered

• Keep “a dashboard where applicable” (status quo). Rejected — unenforceable and demonstrably uneven. The whole point of Epic #1182 is to make Pillar 2 checkable.
• Re-deploy a synthetic-monitoring product (Uptime Kuma successor, Gatus, etc.) for the rollup. Rejected — the blackbox exporter already provides probe_success as code, feeds Prometheus, and is alert-integrated (ADR-0011/0023). A second uptime tool would duplicate the probe surface, add a stack to run/patch/SBOM, and split the health signal across two systems. Blackbox is the synthetic-monitoring layer now that kuma01 is gone.
• A static, hand-maintained rollup (one row per service, edited on each add). Rejected — it rots. A query-driven matrix that discovers services from probe_success labels cannot drift out of sync with reality, and “add a probe target” is already the required step to onboard a service.
• Golden Signals only (latency, traffic, errors, saturation) as the baseline. Rejected as the sole model. Golden Signals presume request-rate telemetry that many homelab services do not expose. RED+USE+logs degrades gracefully: RED where metrics exist, USE always (cAdvisor/host), logs always (Loki), blackbox always — so a dashboard is meaningful even for a service with zero app metrics, while still mandating an ADO gap when only USE+logs is achievable.
• Per-host dashboards instead of per-service. Rejected — a host runs many services; “is Plex healthy?” should not require reading a docker01 host dashboard. Host USE is a panel on the service dashboard, not the organising unit. (Host-level views remain valid as separate infra dashboards, out of scope here.)
• foldersFromFilesStructure left flat (single “Homelab” folder). Rejected for the rollout — folder-by-domain is required for navigability at scale. The mechanism (subdirs + recursive deploy copy, or per-domain providers) is settled in the Dashboard Standard.

Consequences

• Pillar 2 becomes enforceable. A conforming per-service dashboard + a registered blackbox probe target is now the bar; the /wrapup checklist gains a gate that fails if a service deployed/changed this session lacks them.
• Two prerequisite build tasks gate the backfill (ADO #1185 compose_project Loki label; #1186 service+domain on probes). Dashboards that consume those labels cannot be authored correctly until they land.
• The blackbox exporter is itself a service owing the Five Pillars — its newly load-bearing role (it underpins every dashboard’s health panel and the rollup) means its own monitoring/alerting must be airtight. obs-blackbox-exporter being down blinds the entire health view; this is covered by ContainerDown and Alloy self-scrape liveness (ADR-0037), and is called out explicitly in the standard.
• Onboarding a service is a fixed, code-only recipe: add a blackbox probe target (with service+domain), commit a dashboard JSON conforming to the baseline, deploy. The rollup updates itself.
• A one-time backfill sweep (ADO #1187) brings every existing compose stack plus Plex/plex01, mesh01, and zabbix up to the standard.
• Stale/dirty join data surfaced: cAdvisor compose_project values include a retired kuma01 and a typo gotenburg — to be cleaned as part of the prerequisite work so the rollup does not auto-discover ghosts.
• Dashboard sprawl is bounded by the one-per-unit rule and UID=service-name; duplicate or UI-authored dashboards are non-conforming by definition. Refined by ADR-0055 (2026-06-26): a service may additionally carry one linked operational drill-down dashboard — explicitly subordinate (tagged operational-drilldown, not service-dashboard; no probe target of its own; linked both ways from the canonical dashboard). The canonical dashboard remains the sole rollup cell; anything beyond this single subordinate drill-down is still a non-conforming duplicate.
• Log-panel join refinement (2026-06-30, Issue #1342): the unit join key is compose_project for metrics (cAdvisor), but a Docker workload that ships its diagnostics via an agent-tail (the FreeScout/sab01/qbittorrent content-gap class, see the Logging Standard) carries those lines on the job=file stream, where compose_project is absent and service is present — compose_project on such a service is only the stdout banner. So the Error logs and Recent logs panels for a file-tailed workload must query {service="X", job="file"}, not {compose_project="X"}. This was a live gap: the freescout and sab01 dashboards queried the empty banner stream until corrected. The rule is now in the Dashboard Standard Error-logs row.

Amendment (2026-06-27) — mandatory doc backlink + a deploy-time conformance gate (ADR-0059)

The service catalog (ADR-0059) makes each per-service dashboard one half of a bidirectional link with its docs home page. This amendment adds two requirements to the standard:

• Every canonical per-service dashboard MUST carry a doc backlink — a dashboard-level links entry to https://docs.pitbun.com/services/<domain>/<service>/. The convention is enforced by stacks/observability/scripts/inject_doc_backlinks.py (idempotent; catalog-aware — it never adds a link for a service that has no Alloy probe target, which would dangle). New dashboards inherit it from the scaffold.
• A conformance gate runs in the observability CI before deploy (check_dashboard_conformance.py, a Validate stage in docker-stacks-observability.yml). It fails the deploy if (A) any Alloy probe service has no dashboard (the Pillar-2 gap, now enforced — previously only checked at /wrapup), or (B) any catalog dashboard is missing its doc backlink. This is policy-as-code: the pipeline is the enforcement point for the standard, as OPA/conftest is for IaC.

Baseline, not retro-break. Pre-existing Pillar-2 gaps are recorded in dashboard_conformance_exemptions.yml (each with an ADO reference; Issue #1258) so the gate passes today and the baseline only shrinks. The inverse gap (a dashboard with no probe target, so absent from the rollup and catalog) is reported as a non-blocking warning and tracked separately (Issue #1260).

Backfill complete (2026-06-27, Epic #1267). The exemptions baseline is now empty — the four remaining gaps (grafana, prometheus, portainer, udm) were authored on the RED-gap → USE+logs+blackbox fallback and dashboard_conformance_exemptions.yml reduced to {}, so the gate enforces Pillar-2 for every Alloy service with no exceptions (#1258 Done). The inverse gap (#1260) was closed for the three services with an honest endpoint (cloudflared, infisical, iptv-proxy); the remaining three are deliberately not given a blackbox probe — qbittorrent’s WebUI is unreachable behind gluetun’s firewall (#1268) and renovate/ado-agent are batch/agent jobs whose health is a last-run freshness signal, not blackbox availability (#1269, per ADR-0011’s depth model). Forcing a probe on those would read permanently DOWN and corrupt the rollup, so they remain documented availability-gaps until their follow-ups land.