ADR-0075: Logging / Log-Capture Standard — Loki is mandatory, file-loggers must reach stdout, arrival is proven not assumed

Status	Accepted
Date	2026-06-30
Context	The FreeScout licence incident (Issue #1338) exposed an estate-wide log-capture gap: a third of running containers ship nothing to Loki, and even shipping services lose file-based logs. Pillar 2’s “Loki log shipping” was unmeasurable, so it was unevenly met. This ADR records the decision to make it a checkable standard.
Standard	Logging Standard (the enforceable spec this ADR ratifies)
ADO	Epic #1339 (Observability log-capture gap analysis & remediation); Issue #1340 (author the standard + ADR — this)

Context

The FreeScout incident gave a concrete, expensive instance of an abstract gap. The decisive line — Module API and Webhooks has been deactivated due to invalid license: site_inactive — sat in a container file (storage/logs/laravel-*.log) for three days, unsearchable, because FreeScout’s Laravel logs to a file, not stdout, so the Docker→Loki path never saw it. What alerted instead was a downstream heartbeat-staleness trigger on control01 — symptom-level, two systems removed from the cause. MTTD was fine; time-to-root-cause was the failure.

Verified evidence (2026-06-30) showed this was not a one-off:

• Coverage: docker01 runs 72 containers but Loki holds only 48 distinct container_name streams — 24 containers (a third) ship nothing. Examples: freescout, n8n, qbittorrent, sab01, portainer, gluetun, tika, several exporters — even obs-syslog-relay itself.
• Content: even wired services (FreeScout-class) write key logs to files the stdout path never reads.
• Detection vs root-cause: the monitoring-vs-observability line — we knew something broke, but the why was off-platform.

Pillar 2 said “Loki log shipping” but defined no checkable shape, so — exactly as “a dashboard where applicable” was unmeasurable before the Dashboard Standard (ADR-0045) — it was met unevenly and silently. Arron’s explicit ask was a durable standard-as-code so “this never happens again”, not a one-off remediation.

Decision

Author a Logging / Log-Capture Standard as code (docs/standards/logging-standard.md) and reference it from Pillar 2 in the operating instructions, exactly as the Dashboard Standard is. Its load-bearing decisions:

1. Loki is the single, mandatory log platform. Every service/container/host/device ships to obs-loki:3100. A log not in Loki is operationally invisible. No second store, no host-only exception.
2. Name the three capture gaps explicitly — Coverage (ships nothing), Content (ships a banner but files the diagnostics — FreeScout-class), Host (non-Docker journald/syslog missing) — so a review checks all three, not just the obvious Coverage one.
3. File-loggers must redirect to stdout/stderr (preferred) or be agent-tailed. LOG_CHANNEL=stderr is the exact FreeScout fix. Redirect beats tail because a tail adds a mount, an Alloy component, and a rotation failure mode.
4. Stream labels conform to the authoritative taxonomy (ansible/standards/loki-labels.yml, v1.3), low-cardinality, RULE 10 applied to logs.
5. “Logs land in Loki” is a proven Pillar 2 definition-of-done. Arrival is verified against the live Loki API (a LogQL query returning recent lines; for a Content fix, a query for a known diagnostic line, since a banner passes a naive presence check). Remediation closes only on shown evidence.
6. Log-content alerting (Loki ruler rules on root-cause signatures) so causes page, not downstream symptoms — Pillar 3 applied to log content.

The standard owns items 1–4 and 6 as the steady-state spec; item 5’s enforcement is being codified as the structural RULE 12 post-deploy gate (Epic #1339), building on the pre-deploy smoke gate.

Alternatives considered

• A bare soul.md / CLAUDE.md edit (“ship all logs to Loki”). Rejected — a prose rule in the operating instructions is not retrievable as a spec, has no conformance checklist, and drifts. The house pattern is a standard-as-code in the docs corpus (Dashboard, Documentation, Notification, Timezone standards all live there), referenced from Pillar 2. A bare edit repeats the “where applicable” mistake that caused this.
• Make Loki optional / allow a host-local log store. Rejected — optionality is exactly what produced the 24 dark containers. A log off-platform is unsearchable, uncorrelatable, un-alertable, and invisible to agent retrieval; the whole incident is an argument against it.
• Mandate agent-tail for every file-logger (skip the stdout redirect). Rejected as the default — tails are real config with a rotation failure mode and a per-file Alloy component. Redirect-to-stdout is zero-mount and zero-Alloy-change where the app supports it; tail is the documented fallback for apps that genuinely cannot.
• Treat “log path wired” as done (presence of an Alloy source). Rejected — that is the assumption that failed. Wiring ≠ arrival; a stdout-shipping FreeScout still loses its errors. Done means proven arrival of diagnostic content against the live API.
• Per-service log stores (ELK-per-app, etc.). Rejected — pitlab already runs one Loki; fragmenting the corpus breaks cross-service correlation and the single-pane Grafana/agent retrieval model.

Consequences

• Pillar 2 gains a measurable log-capture bar with a conformance checklist the /wrapup gate and reviewers apply, mirroring the Dashboard Standard.
• The remediation child issues of Epic #1339 (dark-container coverage, FreeScout-class content audit, host/VM/LXC coverage, post-deploy verification gates as RULE 12, log-content alerting) now have an authoritative spec to close against, each on shown Loki-arrival evidence.
• New-service work carries a log-capture cost: redirect-or-tail file logs, conform labels, and prove arrival before close. For stdout-logging Docker workloads this is near-zero (auto-discovery); for file-loggers it is one config line or one tail.
• The operating instructions (Pillar 2 in CLAUDE.md/soul.md) now cite both the Dashboard Standard and the Logging Standard, so observability is “metrics + dashboard + logs proven in Loki”, not “metrics + dashboard + logs (assumed)”.
• Residual: until RULE 12’s automated gate ships, the “logs reach Loki” assertion is a manual query step — a known, tracked interim (Epic #1339 issue 5), not a silent gap.
• Reconciliation (2026-06-30, Issue #1342 — the FreeScout content-gap exemplar): the motivating instance was closed by the agent-tail path, not the LOG_CHANNEL=stderr redirect that Decision point 4 calls “the exact FreeScout fix”. Redirect remains the preferred fix in general, but it proved not achievable for FreeScout specifically: the tiredofit/s6 image regenerates .env on every start and does not template LOG_CHANNEL (ADR-0072), so a stdout redirect needs an image fork. FreeScout’s /www/logs (Laravel app log + nginx/php-fpm) is therefore tailed via a named volume → obs-alloy loki.source.file (the sab01/qbittorrent precedent), proven against the live Loki API. The estate-wide content audit confirmed FreeScout was the only true content gap on docker01 — every other file-logger duplicates its content to stdout. Lesson: “redirect beats tail” holds as a default, but image-templating constraints can force the tail; check the image before assuming stderr is reachable.