ADR-0001: Monitoring dependency chains
| Status | Accepted |
| Date | 2026-06-18 |
| Epic | ADO #812 |
Context
Section titled “Context”On 2026-06-18 at 17:21, docker01 restarted. Zabbix fired 41 notifications: a restart alert, 7 API-not-responding alerts for services on docker01, 4 pipeline failure alerts, and a paperless memory alert — all from a single root cause. Every trigger fired independently because no dependency relationships were configured.
The operator received a phone notification burst with no clear indication of root cause. This is unsustainable at scale — any infrastructure failure cascades into a wall of noise that obscures the signal.
Separately, Alertmanager had no inhibit_rules. If a Prometheus scrape target (cadvisor, node_exporter) goes down, stale metric values can fire ContainerDown, ContainerHighMemory, DiskSpaceLow etc. as false positives.
Decision
Section titled “Decision”Zabbix: Configure native trigger dependency chains (the live chain is documented in Alerting & Dependency Architecture). When a parent trigger is in PROBLEM state, all dependent child triggers are suppressed — Zabbix does not send notifications for them.
Dependency tree implemented:
pve01 agent (39210)→ parent ofdocker01 agent (39151),HAOS agent (33282),urbackup01 agent (39122)docker01 agent (39151)→ parent of 24 service/API health triggers on docker01
Alertmanager: Add inhibit_rules scoped by host label:
InstanceDown{job="cadvisor"}inhibitsContainerDown,ContainerHighMemory,ContainerRestartedInstanceDown{job="node_exporter"}inhibitsDiskSpaceLow,DiskSpaceCritical,HighMemoryUsage
Alternatives considered
Section titled “Alternatives considered”Zabbix maintenance windows on action: Suppress notifications during a maintenance period when a host goes down. Rejected — maintenance windows require manual triggering or complex automation. Trigger dependencies are evaluated in real-time with no operator involvement.
Alertmanager silences: API-triggered silences when InstanceDown fires. Rejected — requires an external automation layer (n8n or similar) to create/expire silences. inhibit_rules are evaluated natively by Alertmanager with no external dependency.
Single “host down” mega-trigger: A composite trigger that fires when any service on docker01 fails. Rejected — loses granularity for single-service failures when the host is healthy.
Consequences
Section titled “Consequences”- A docker01 restart now produces 1 Zabbix notification instead of 8+.
- Service-level triggers still fire normally when docker01 is healthy (parent not in PROBLEM state).
- The 122 container memory triggers on docker01 are deliberately excluded from the dep chain — they have
for: 5mdelays that prevent them firing during a normal restart anyway. Re-evaluate if they prove problematic. - Trigger dependency IDs are hardcoded in
ansible/scripts/zabbix_configure_trigger_deps.py. If a trigger is replaced (e.g. via LLD rediscovery with a new ID), the script must be updated and re-run. - Alertmanager inhibit_rules are defensive — Alertmanager fired 0 alerts on 2026-06-18. The Zabbix dep chain is the primary signal reduction mechanism.