CONTROL-SYSTEM MAP Control/Mechanism Observation Source Authority Trigger Action Verification Test suite Pipeline logs Gate merge→canary Code merge Pass/fail signal Pass recorded 14:06 Canary deploy mechanism Pipeline logs Deploy to 5% Test pass Deploys new version to subset Deployed 14:07 Health-check/monitoring service Health-check logs Measures error rate, feeds rollback decision Continuous polling Emits error-rate metric Stopped emitting at 14:08 Auto-rollback controller Inferred (no separate logs cited) Trigger rollback if error rate >1% Health-check metric exceeding threshold Roll back canary No rollback fired Auto-promote controller Pipeline logs Promote canary→100% if error rate stays <1% for 10 min Absence of rollback signal / health-check "still healthy" state Promoted to 100% Promoted 14:17 Config-reload service Excluded from available logs (per constraint) Manages health-check's alerting config Unknown (reload event) Reset thresholds to default Reload occurred ~14:08 (inferred timing) Human on-call / manual rollback Incident timeline Final authority, human judgment User-reported outage Manual rollback Rolled back 14:41 User/customer feedback loop Incident timeline External detection Full outage Reports to on-call Reported 14:22 Shared dependency flag: The auto-rollback control and the auto-promote control both depend on the same single upstream signal source — the health-check service's metric stream. This is not two independent safeguards; it is one signal feeding two decisions. When that signal died, both controls were blinded simultaneously. This is a single point of failure disguised as two layers of protection. CONTROL CLASSIFICATION Test suite — WORKING. Confidence: VERIFIED. Passed and gated as designed at 14:06. Canary deploy mechanism — WORKING. Confidence: VERIFIED. Deployed to 5% at 14:07 as designed. Health-check/monitoring service — INCORRECT then MISSING. Confidence: VERIFIED (stopped emitting, per logs) / STRONGLY SUPPORTED (that the config reload reset thresholds to exclude this error type). It did not fail by being late — it failed by silently changing its own detection criteria and then going dark. Auto-rollback controller — INEFFECTIVE / DEPENDENT. Confidence: STRONGLY SUPPORTED. The controller itself may have been logically sound, but it is entirely dependent on the health-check feed; with no qualifying signal arriving, it had nothing to act on. Cannot confirm from evidence whether the rollback logic itself is functional — only that it never received a triggering input. Auto-promote controller — INCORRECT. Confidence: STRONGLY SUPPORTED. It appears to have treated "no error-rate breach reported" as equivalent to "error rate confirmed under 1%." Those are not the same state, and the controller does not appear to distinguish "healthy" from "no data." This is a design flaw, not a timing issue. Config-reload service — UNKNOWN in behavior/intent, but its effect is VERIFIED (health-check logs show it caused the reset). No visibility into why it reloaded, whether the reset was intended, or whether it logged the change elsewhere. Human/on-call escalation — WORKING but LATE relative to the automated safeguards' intended timescale (10-minute health-check window). Confidence: VERIFIED. It performed its function once alerted by an independent channel (users), not by internal monitoring. User feedback loop — WORKING, but never intended as the primary safety net. Confidence: VERIFIED. Notably, this was the only channel that actually detected the failure. EXPECTED VS ACTUAL FLOW Stage Expected Actual Occurred? On time? Correct? Detection Health-check continuously monitors error rate Health-check stopped emitting metrics at 14:08 No (after 14:08) N/A No Decision If >1%, decide rollback No decision made — no qualifying input No N/A No Authority Rollback controller has authority to act unilaterally Never invoked No N/A No Action Rollback executed None No — — Verification Confirm rollback succeeded N/A — — — Feedback Promote only if verified healthy Promoted anyway at 14:17 Yes On the pipeline's own clock, "on time" No — promoted on absence of data, not presence of health First divergence: 14:08 UTC — health-check service stops emitting metrics due to a config reload that reset its alerting thresholds. This is the earliest evidence-supported deviation from expected behavior, occurring just one minute after error rate had already begun climbing (4.8% observed by 14:09, per logs — implying the underlying rise likely began before the health-check went dark, though the exact rate at 14:08 is not directly evidenced). Significance: This single event severed both downstream safeguards (rollback and the promotion gate's health condition) at once, before either had a chance to act on the emerging error rate. Everything after 14:08 is propagation, not new causal input. FAILURE PATH Origin: Config reload at ~14:08 UTC silently resets health-check alerting thresholds to a default profile that excludes this error type. (STRONGLY SUPPORTED — evidenced by the health-check logs' own account of the reset and its timing; the reload service's internal logs are unavailable per constraint, so the cause of the reload itself is UNKNOWN.) Propagation: Health-check stops emitting usable metrics (14:08) — VERIFIED. Error rate continues rising unmonitored, reaching 4.8% by 14:09 — VERIFIED, but note: 4.8% was observed in logs, meaning some measurement existed at 14:09 despite the health-check "stopping" at 14:08. This is a timing tension in the evidence — either the 4.8% figure came from a different measurement source (e.g., raw metrics/logs, not the health-check's alerting layer) or the health-check's alerting function died while its raw data collection continued. This distinction matters and is currently UNKNOWN from the evidence given. No rollback fires because the (now-blind or now-permissive) health-check produced no breach signal — STRONGLY SUPPORTED. Auto-promote controller reads "no breach reported" as "healthy," and promotes to 100% at 14:17 — VERIFIED as an event; the controller's interpretation logic is INFERRED from the outcome. Full outage at 14:22 — VERIFIED, and this is the point where impact became externally visible, not the point of origin. Recovery trigger: Manual rollback by on-call engineer at 14:41, initiated in response to user-reported outage (14:22), not in response to any internal automated signal. Recovery ≠ proof any automated control worked — none did in this incident; recovery was entirely human-driven, using an external signal (user reports) as the effective detection mechanism. Distinct events, not to be conflated: (a) the config reload/threshold reset, (b) the health-check going silent, (c) the error rate actually rising, (d) the promotion decision, (e) the outage becoming visible to users, (f) the manual rollback. Each is a separate fact in the evidence; only (a)/(b) are causally upstream of (d). KEY CONTROL GAP The rollback and promotion safety net is not two independent controls but one signal with two consumers. There is no independent, out-of-band confirmation of canary health (e.g., a second monitoring path, a heartbeat/dead-man's-switch on the health-check itself, or a "no data = do not promote" default). The system's design implicitly assumes the health-check will always either report a breach or report health — it has no defined behavior for reporting nothing, and that undefined state was interpreted as "safe to promote." This is the single most consequential gap: a monitoring outage was functionally equivalent to a clean bill of health. CONFIDENCE SUMMARY VERIFIED: Test pass (14:06); canary deploy (14:07); error rate at 4.8% by 14:09; health-check stopped emitting at 14:08; promotion to 100% at 14:17; outage reported 14:22; manual rollback 14:41; no rollback was triggered by the automated system. STRONGLY SUPPORTED: The config reload's threshold reset caused the health-check's alerting blindness; the auto-promote controller treated absent/non-breaching signal as confirmation of health; the rollback and promotion safeguards share a single point of failure. INFERRED: The auto-rollback controller's own logic is otherwise sound and would have fired had it received a qualifying signal; the config reload was unintentional/unrelated to the deploy rather than a deliberate or triggered action. UNKNOWN: Why the config reload occurred, what triggered it, and whether it was related to the deploy event or coincidental; why raw error-rate data (4.8%) was still visible in logs after the health-check "stopped emitting"; whether the health-check's silence was total or partial (some metrics vs. all); whether any internal alert fired on the health-check's own silence (a meta-monitoring gap) — no evidence either way. KEY INVESTIGATION GAPS Config-reload service logs/trigger source — establishes whether the reload was deploy-triggered, scheduled, human-initiated, or a bug, and whether it's a recurring risk. Definition of the 4.8% measurement's source system — establishes whether raw error data collection and the alerting/threshold layer are the same component or separate, which changes where the actual break occurred. Auto-promote controller's decision logic/source code or config — establishes whether "no breach signal" was explicitly coded as "healthy," confirming or refuting the INFERRED classification above. Any meta-alerting on health-check service uptime/heartbeat — establishes whether a dead-man's-switch existed at all, or whether this is a net-new gap. History of prior config reloads on this service — establishes whether this is a known recurring failure mode or a first occurrence. IMPACT Demonstrated: Full production outage from 14:17 (promotion) to 14:41 (manual rollback) — approximately 24 minutes of customer-facing impact, with user reports starting at 14:22. Potential: Had users not reported the outage, there is no evidence of any other detection mechanism that would have caught it — the outage could plausibly have persisted longer, since the automated safety net was blind and no other alerting path is evidenced. CORRECTIVE ACTION (smallest fix, mapped to mechanism) Target the actual mechanism — the promotion controller's fail-open behavior on missing data — not the config-reload service or thresholds generally (those are downstream/unclear). Smallest reliable fix: change the auto-promote condition from "no breach reported" to "positive confirmation of health-check liveness AND error rate under threshold within the monitoring window." Absence of data must block promotion (fail-closed), not permit it. PREVENTION Add a heartbeat/liveness check on the health-check service itself, independent of its own alerting config, so its silence is itself an alertable event. Make canary promotion depend on an explicit "healthy" assertion with a timestamp/freshness check, not the absence of a "breach" assertion. Add a safeguard so that config reloads affecting alerting thresholds require explicit validation or a diff/approval step before taking effect, especially for a service gating production promotion. Add an independent secondary error-rate signal (e.g., from load balancer or infra-level metrics) that does not share the health-check's config path, restoring genuine redundancy rather than a single signal with two consumers. ENGINEERING CONCLUSION Established: The proximate cause of the failed rollback is that the health-check service's alerting stopped functioning at 14:08 due to a config reload resetting its thresholds, and the promotion controller advanced the deploy based on the absence of a breach signal rather than confirmed health. Unresolved: The root trigger of the config reload itself, and whether the promotion controller's "no signal = healthy" behavior is an explicit design choice or an unintended side effect, remain undetermined from the evidence provided. Next step: Obtain the config-reload service's logs/trigger history and the promotion controller's decision-logic source to close these two gaps before finalizing root cause in the postmortem.