# Incident Post-Mortem: Checkout Service 500 Errors **Incident Summary:** Checkout service returned 500 errors for 40 minutes, blocking all purchases. --- ## BLOCK A — Incident Classification & Impact Scope ### Incident Classification | Dimension | Classification | Rationale | Evidence Type | |-----------|---------------|-----------|---------------| | Incident severity | SEV1 (all users) | Incident summary states the failure was "blocking all purchases" — i.e., 100% of checkout attempts, the primary revenue path | INFERRED | | Incident type | Outage | Checkout API returned 500 errors, a hard failure rather than degradation | STATED | | Onset type | Sudden | Alert fired at a discrete timestamp (14:02) with no gradual-degradation signal in the timeline provided | INFERRED | | Detection method | Automated alert | Timeline begins with "alert fired" at 14:02, not a user or internal report | STATED | | MTTD (Mean Time to Detect) | ~0 min (assumed) | No separate "first symptom" timestamp was provided; the alert time is the earliest signal available, so MTTD is assumed near-zero pending log confirmation | INFERRED | | MTTR (Mean Time to Resolve) | 40 minutes (14:02 → 14:42) | Matches the duration stated in the incident summary and is consistent with the timeline deltas | STATED | ### Impact Assessment | Impact Dimension | Magnitude | Evidence Type | |-----------------|-----------|---------------| | Users affected | All users attempting checkout (100% of purchase traffic) | STATED | | Duration of impact | 40 minutes | STATED | | Revenue / SLA impact | Not quantified in inputs; likely material given all purchases were blocked for 40 minutes on a revenue-critical path | INFERRED | | Data integrity impact | None — root cause was resource exhaustion (connection pool), not data corruption or write failure | INFERRED | | Reputational exposure | Medium — checkout is a highly visible, revenue-critical user journey | INFERRED | ### Failure Mode Priority Map | Failure Mode | Priority | Evidence Type | Conf | |-------------|----------|---------------|------| | Detection gap — alert fired late or not at all | LOW | INFERRED | 55 | | Runbook missing or outdated — response time extended | MED | INFERRED | 50 | | Cascading failure not anticipated in architecture | HIGH | INFERRED | 75 | | Change review process insufficient | HIGH | INFERRED | 70 | | Human factors under pressure — wrong decision in fog of war | LOW | INFERRED | 40 | *Rationale:* The alert appears to have fired promptly (no evidence of a missed or delayed detection), so detection gap is scored LOW. The root cause traces back to an unbounded query shipped in a deploy, which strongly implicates the change review process (HIGH) and reveals that the architecture had no safeguard against a single query cascading into full connection pool exhaustion and a total checkout outage (HIGH). --- ## BLOCK B — Full Timeline Reconstruction | Time | Event | Source | Phase | Evidence Type | |------|-------|--------|-------|---------------| | 14:02 | Alert fired for Checkout API errors | Monitoring/alerting system | Trigger | STATED | | 14:02–14:15 | 13-minute gap before human response began | Paging system | Escalation | INFERRED (gap is a calculation from stated timestamps) | | 14:15 | On-call engineer paged and began investigation | Paging system / on-call process | Escalation | STATED | | 14:15–14:30 | Investigation of Checkout API, PostgreSQL primary, and PgBouncer to isolate cause | On-call engineer | Response | INFERRED (specific investigative steps not detailed in inputs) | | 14:30 | Root cause identified: PostgreSQL connection pool exhaustion | On-call engineer | Response | STATED | | 14:30–14:42 | Remediation executed: PgBouncer restarted, max connection limit increased, recent deploy (unbounded query) rolled back | On-call engineer / response team | Response → Resolution | STATED | | 14:42 | Checkout service restored to normal operation | Monitoring system / on-call confirmation | Resolution | STATED | **MTTD annotation:** First symptom timestamp is not separately available from the alert timestamp; MTTD is treated as ~0 minutes on the assumption that the 14:02 alert is coincident with onset. This should be verified against raw error-rate logs to confirm no earlier degradation window existed. **MTTR annotation:** 14:02 (detection) → 14:42 (resolution) = 40 minutes, matching the incident summary's stated duration. --- ## BLOCK C — Root Cause Analysis: 5-Why Causal Chain | Level | Why? | Finding | Evidence Type | Conf | |-------|------|---------|---------------|------| | Symptom | What did users experience? | Checkout returned 500 errors for 40 minutes; no purchases could complete | STATED | — | | Why 1 | What caused the symptom? | The Checkout API could not obtain database connections to serve requests, because the PostgreSQL connection pool (via PgBouncer) was exhausted | STATED | 90 | | Why 2 | What caused Why 1? | A recently deployed change introduced an unbounded query, which consumed and held an excessive share of available connections | INFERRED | 80 | | Why 3 | What caused Why 2? | The change did not undergo review or testing sufficient to flag a query without bounds (e.g., missing LIMIT/pagination) before reaching production | INFERRED | 55 | | Why 4 | What caused Why 3? | No automated safeguard (query cost linting, load testing against production-scale data, mandatory review checklist for DB-touching changes) existed to catch this class of risk pre-deploy | INFERRED | 50 | | Root cause | What systemic condition made this chain possible? | The deployment pipeline and architecture lacked systemic safeguards — at the code-review, query-execution, and connection-pool layers — to prevent a single unbounded query from cascading into a full checkout outage | INFERRED | 60 | ### Assumption Ledger | Assumption | Where It Enters | If False — RCA Effect | |------------|----------------|----------------------| | The unbounded query in the recent deploy directly caused the connection pool exhaustion (not a coincidental correlation with the rollback) | Why 2 | Root cause chain would need to shift toward another explanation (e.g., traffic spike, a connection leak elsewhere), pending log evidence | | The 14:02 alert time is coincident with the actual onset of user-facing impact | MTTD calculation | MTTD would be understated; actual detection latency could be longer than assumed | | No other concurrent deploys or infrastructure changes contributed to the pool exhaustion | Why 2 / Why 3 | Another concurrent change could be a contributing or root cause, changing prioritization of the prevention plan | | The PgBouncer max connection limit was reasonably sized for normal (non-degraded) load before this incident | Contributing Factors (Block D) | If undersized generally, capacity planning becomes a root-cause-level issue rather than a secondary contributing factor | ### ⚠ Silent Corruption Flags - **"DB connection pool exhaustion" reads like a root cause but is a proximate cause.** It is the mechanism of failure, not the systemic reason the mechanism was triggered — the deeper issue is the absence of safeguards that let an unbounded query reach production and run unchecked. - **"Unbounded query in a deploy" also reads like a root cause but is itself a symptom** of a change-review and testing process that does not systematically catch this failure class. --- ## BLOCK D — Contributing Factors Register | Factor | Category | How It Contributed | Systemic or One-Off? | Evidence Type | |--------|----------|-------------------|---------------------|---------------| | Unbounded query shipped in a recent deploy | Technical | Directly consumed excessive database connections, triggering pool exhaustion | Systemic (process gap, not a one-time fluke) | STATED | | 13-minute gap between alert and page (14:02 → 14:15) | Process | Delayed the start of human investigation, extending overall time-to-resolution | Systemic | STATED (timestamps); impact assessment INFERRED | | PgBouncer max connection limit | Technical | Limit was low enough that a single query pattern could exhaust it; was increased as part of remediation | Systemic | INFERRED | | No automated query-cost or timeout safeguard in production | Technical / Process | Allowed the unbounded query to run and hold connections indefinitely rather than being cut off | Systemic | INFERRED | | Change review / pre-deploy testing process | Process | Did not catch the unbounded query before it reached production | Systemic | INFERRED | | Checkout API's tight coupling to a single DB connection pool with no fallback | Architectural | When the pool was exhausted, Checkout API failed hard (500s) rather than degrading gracefully | Systemic | INFERRED | --- ## BLOCK E — Systemic Prevention Action Plan | Action | Type | Addresses | Owner | Priority | Completion Signal | Evidence Type | |--------|------|-----------|-------|----------|------------------|---------------| | Add automated query-cost/lint checks to the CI pipeline that flag queries without bounds (e.g., missing LIMIT/pagination) | Prevention | Why 3 / Why 4 | Backend Engineering Lead | 🔴 P1 | Linter is active in CI and blocks merges on unbounded-query patterns | INFERRED | | Set explicit per-query timeout limits at the PostgreSQL and PgBouncer layers | Prevention | Root cause | Database Platform Team | 🔴 P1 | Timeout configuration deployed to production and verified under load test | INFERRED | | Add proactive connection-pool utilization alerting (e.g., alert at 80% pool usage, before full exhaustion) | Detection | Failure mode: cascading failure | SRE / Platform Team | 🔴 P1 | Alert configured, tested against a simulated saturation event | INFERRED | | Reduce alert-to-page escalation time via tighter paging automation | Process | 13-minute escalation gap | Incident Response / SRE | 🔴 P1 | Median time from alert to page is under 2 minutes in the next SEV1 drill | STATED-derived | | Implement circuit-breaker / graceful-degradation behavior in Checkout API when the DB connection pool is saturated | Architecture | Cascading failure not anticipated | Checkout API Team | 🟡 P2 | Circuit breaker validated in staging and confirmed via chaos test in production | INFERRED | | Require mandatory load/performance testing against production-scale data for any DB-touching change before deploy | Prevention | Change review gap | QA / Release Engineering | 🟡 P2 | Perf test suite covers checkout's primary query paths and is a required CI gate | INFERRED | | Publish/update a runbook specifically for "DB connection pool exhaustion" with diagnostic and remediation steps | Response | Runbook gap | On-call / SRE Lead | 🟡 P2 | Runbook published, linked directly from the pool-exhaustion alert | INFERRED | | Add a pre-deploy checklist item requiring explicit sign-off on query resource impact for DB-touching changes | Process | Change review gap | Engineering Management | 🟡 P2 | Checklist item integrated into the standard deploy template | INFERRED | | Right-size PgBouncer max connection limits based on a formal capacity-planning review | Prevention | Connection pool sizing | Database Platform Team | 🟡 P2 | Capacity plan reviewed and limits updated with documented headroom targets | INFERRED | | Add one-click / automated rollback tooling for DB-schema or query-impacting deploys | Response | Remediation speed | Release Engineering | 🟡 P2 | Rollback tooling tested and available for the next DB-touching deploy | INFERRED | | Conduct an architecture review of Checkout's database dependency for resiliency patterns (e.g., read replicas, pool isolation per service) | Architecture | Systemic capacity/architecture risk | Platform Architecture | 🟢 P3 | Review document with recommendations delivered to engineering leadership | INFERRED | --- ## BLOCK F — What Went Well: Behaviors to Reinforce | What Worked | Why It Helped | Should Be Formalized? | |-------------|---------------|----------------------| | Automated monitoring detected the issue and fired an alert without waiting on user reports | Enabled early visibility into a revenue-critical failure | YES — keep automated alerting as the primary detection path for checkout | | On-call engineer isolated the root cause (connection pool exhaustion) within 15 minutes of being paged | Fast diagnosis directly reduced total downtime | YES — capture this diagnostic path in the new runbook as a reference case | | Response combined an immediate symptom fix (restart pooler, raise connection limit) with a root-cause fix (rollback of the offending deploy) in the same remediation window | Addressed both the immediate outage and its underlying trigger, avoiding a recurrence within the incident | YES — formalize "treat symptom and cause together" as the standard pattern for resource-exhaustion incidents | | Rollback of the suspect deploy was readily available and executed successfully | Enabled a durable resolution rather than a temporary workaround | YES — ensure rollback tooling remains fast and reliable, and extend it per the P2 action above | --- ## BLOCK G — Post-Mortem Integrity Audit | Check | Status | Note | |-------|--------|------| | Timeline is blameless — no individual blame, only system/process | ✅ | All findings are framed as system/process gaps; the on-call engineer is referenced only by role | | 5-Why chain reaches a systemic root cause (not "human error") | ✅ | Root cause lands on missing pipeline/architecture safeguards, not an individual decision | | Prevention plan has ≥ 8 specific actions with owners and completion signals | ✅ | 11 actions listed, each with a role-based owner and a concrete completion signal | | MTTD and MTTR are explicitly annotated | ✅ | Annotated in Block B; MTTD is flagged as an assumption pending log verification | | "What went well" section present | ✅ | Block F includes 4 reinforced behaviors | | All INFERRED findings are labeled | ✅ | Every table row carries an explicit STATED/INFERRED tag | **RCA CONFIDENCE: 62/100** — The proximate cause (connection pool exhaustion tied to an unbounded query) is well-supported by the stated timeline and remediation actions, but the deeper "why" (gaps in review/testing process) is inferred without direct evidence from code review records or CI logs. **Biggest unknown:** The query execution logs and connection-pool metrics for the 14:02–14:30 window, plus the code review/PR history for the rolled-back deploy, would most directly confirm (or overturn) the causal chain from unbounded query → pool exhaustion → outage. **STATED vs. VERIFY breakdown:** *STATED (directly supported by inputs):* - Checkout returned 500 errors for 40 minutes, blocking all purchases - Timeline: 14:02 alert, 14:15 paged, 14:30 root cause identified, 14:42 restored - Systems involved: Checkout API, PostgreSQL primary, PgBouncer - Immediate actions: PgBouncer restarted, max connection limit increased, recent deploy with unbounded query rolled back *VERIFY (would upgrade INFERRED findings to STATED):* - Exact first-symptom timestamp vs. alert timestamp, to confirm true MTTD - Whether the unbounded query was the sole driver of pool exhaustion or one of several contributing factors - Code review / PR history for the rolled-back deploy, to confirm whether a review gap actually existed - Revenue/SLA dollar impact and precise order-volume loss - PgBouncer connection limit values before and after remediation - Whether a runbook for this failure mode already existed and, if so, why it wasn't sufficient