🔎 PHASE 1 — PERFORMANCE INCIDENT RECONSTRUCTION Incident Timeline T-2 hours New report template deployed. The template introduces: - Complex charts - Embedded SVG graphics - Larger rendered documents No worker capacity increase occurred. T-90 minutes Average generated PDF size increased: 2.8 MB → 7.4 MB This indicates a substantial increase in output complexity. T-60 minutes Large reports begin consuming more worker CPU and memory. T-40 minutes Worker utilization approaches saturation. Worker concurrency reaches: 20/20 T-30 minutes Large jobs begin exceeding expected processing duration. T-20 minutes Chromium render timeouts appear. T-15 minutes Failed jobs begin retrying. T RabbitMQ queue reaches approximately: 1,850 jobs Report P95: 118s Report P99: 191s Incident Pattern Start: After new template deployment. Affected workload: Large report generation. Pattern: Strongly correlated with report complexity. Primary component: Report workers. Database correlation: Weak. Network correlation: None observed. --- 📊 PHASE 2 — BASELINE VS DEGRADATION Metric| Normal| Degraded P50 job duration| 18s| 42s P95 job duration| 31s| 118s P99 job duration| 45s| 191s Job failure rate| 1.1%| 8.7% Worker CPU| 62%| 97% Worker memory| 68%| 94% Worker concurrency| Unknown| 20/20 Queue depth| 35| 1,850 API CPU| 41%| 41% DB CPU| 38%| 38% DB query latency| 40-90ms| Similar Network| Normal| Normal Largest Deviations 1. Queue depth increased dramatically. 2. Worker CPU approached saturation. 3. Worker memory approached pod limits. 4. All worker concurrency slots became occupied. 5. P95/P99 generation latency increased substantially. 6. Job failures and Chromium timeouts increased. This strongly points toward the worker/rendering layer rather than API or database processing. --- 🧭 PHASE 3 — END-TO-END REQUEST PATH The workflow is: User ↓ Vue Frontend ↓ Go API ↓ RabbitMQ ↓ Report Worker ↓ PostgreSQL ↓ Data Processing ↓ Chromium/Puppeteer ↓ PDF Generation ↓ Object Storage ↓ Job Completion Latency Accumulation The API itself is not the major bottleneck. The critical path is: RabbitMQ ↓ Worker Scheduling ↓ Data Preparation ↓ Chromium Rendering ↓ PDF Generation The strongest evidence points specifically toward: Chromium rendering + worker resource contention --- 🌐 PHASE 4 — API & SERVICE LATENCY AUDIT API The API's primary responsibility is to enqueue the report job. API CPU: 41% API memory: 55% Therefore, the API does not appear saturated. Important Architecture Issue The stated requirement is: API should acknowledge job acceptance within 2 seconds. The architecture already uses asynchronous RabbitMQ processing. Therefore, user-facing API latency should not depend on PDF generation completion. The bigger problem is job completion latency, not API request execution. Finding API bottleneck: No strong evidence. Worker bottleneck: Strong evidence. --- 🗄️ PHASE 5 — DATABASE BOTTLENECK INVESTIGATION Database Query Report queries take approximately: 40-90ms Database CPU: 38% Connections: 110/400 Lock contention: None reported Assessment PostgreSQL is not currently the primary bottleneck. However, large reports retrieve: 80,000-150,000 records This means report workers may still spend significant time processing and serializing large datasets. But the available evidence does not indicate database saturation. Finding Primary database bottleneck: No. Potential secondary workload: Large result processing. Confidence: Low-Medium. Validation Measure: - Query duration. - Result-transfer time. - Worker-side parsing time. - Data transformation time. - Memory consumed before Chromium starts. --- ⚡ PHASE 6 — CACHE PERFORMANCE AUDIT No cache is involved in this workflow. Therefore: Cache bottleneck: Not applicable. No cache optimization should be recommended for this incident. --- 🔄 PHASE 7 — BACKGROUND JOB & QUEUE ANALYSIS This is the most important phase for this incident. Queue Depth Normal: 35 Degraded: 1,850 This is a major signal. Worker Concurrency Configured: 20 tasks/pod 10 pods: 200 theoretical concurrent Chromium tasks Each task is resource-intensive. Worker CPU 62% → 97% Worker Memory 68% → 94% Job Failures 1.1% → 8.7% Retry Behavior Chromium timeout: → Job retry → Additional worker utilization → Queue pressure → More concurrency → Resource contention This creates a positive feedback loop. Failure Amplification Complex Template ↓ More Chromium CPU/Memory ↓ Worker Saturation ↓ Rendering Slows ↓ Timeouts ↓ Retries ↓ More Jobs ↓ Queue Growth ↓ More Worker Contention This is a strong candidate for the central failure mechanism. --- 🖥️ PHASE 8 — INFRASTRUCTURE BOTTLENECK AUDIT Worker CPU 97% Strong evidence of CPU pressure. Worker Memory 94% Strong evidence of memory pressure. Kubernetes Limits Per pod: - 4 CPU cores - 4 GB memory Chromium tasks are memory-intensive. Running 20 concurrent Chromium tasks per pod may create substantial resource contention. Autoscaling Current autoscaling uses: CPU only But queue depth is the actual user-visible workload backlog. This creates an architectural mismatch. A worker may already be CPU saturated while autoscaling reacts too slowly or cannot appropriately account for queued jobs. Finding Infrastructure bottleneck: Worker resource saturation. Underlying issue: Workload concurrency is too aggressive for the available worker resources. --- 🧩 PHASE 9 — CONCURRENCY & RESOURCE CONTENTION This is a major root-cause candidate. Current Configuration 20 concurrent rendering tasks per pod. Memory limit: 4 GB. Worker memory: 94%. This indicates that the system is operating very close to its memory boundary. CPU: 97%. Therefore, the workers are constrained by both CPU and memory. Why Concurrency Is Important Higher concurrency does not necessarily mean higher throughput. If each Chromium process consumes substantial CPU and memory, excessive concurrency can result in: - Context switching. - Memory pressure. - Garbage collection. - Chromium contention. - Slower individual jobs. - Increased timeout probability. Therefore: 20 concurrent tasks may be above the efficient concurrency point. The exact optimal concurrency cannot be determined without controlled benchmarking. --- 🌍 PHASE 10 — EXTERNAL DEPENDENCY ANALYSIS Object storage is involved after PDF generation. Network telemetry is normal. No object-storage timeout is reported. Therefore: External dependency is currently classified as unrelated. The available evidence does not justify blaming object storage. --- 🚨 PHASE 11 — ROOT-CAUSE HYPOTHESIS ENGINE Rank 1 — Complex Chromium Rendering Hypothesis The new template substantially increased CPU and memory requirements for PDF rendering. Evidence For - Issue started after template deployment. - New complex charts and SVG graphics. - PDF size increased from 2.8 MB to 7.4 MB. - Worker CPU reached 97%. - Worker memory reached 94%. - Chromium timeout logs increased. Evidence Against Exact Chromium render-time telemetry is unavailable. Confidence High Expected Impact Potentially explains most of the increase in report processing time. Validation Render identical report data using: Old template vs New template Measure: - Chromium render duration. - CPU. - Memory. - PDF generation time. - Output size. --- Rank 2 — Excessive Worker Concurrency Hypothesis 20 concurrent Chromium tasks per pod creates CPU/memory contention. Evidence For - Concurrency reaches 20/20. - CPU reaches 97%. - Memory reaches 94%. - Chromium is resource-intensive. - Job latency increases under load. Evidence Against Optimal concurrency has not been benchmarked. Confidence High Validation Benchmark: - 4 concurrent tasks. - 8. - 12. - 16. - 20. Measure throughput and per-job latency. The best configuration should maximize completed jobs per minute without pushing resource utilization into unstable territory. --- Rank 3 — Retry Amplification Hypothesis Chromium timeouts create additional jobs and increase queue pressure. Evidence For - Render timeout logs exist. - Retry is scheduled after timeout. - Failure rate increased. - Queue depth reached 1,850. Evidence Against Exact retry count is unknown. Confidence Medium-High Validation Compare queue growth with: - Retry disabled in staging. - Current retry policy. Measure queue depth and worker utilization. --- Rank 4 — CPU-Only Autoscaling Hypothesis Autoscaling does not respond directly to queue backlog. Evidence For - Queue depth increased dramatically. - Autoscaling is based only on CPU. - Worker workload is asynchronous. Evidence Against Worker CPU already reaches 97%, so CPU scaling may eventually trigger. Confidence Medium Impact Primarily affects recovery time and queue growth rather than causing the initial rendering slowdown. Validation Simulate identical queue workload using queue-aware autoscaling and compare: - Queue drain time. - Job wait time. - Worker utilization. --- Rank 5 — Large PostgreSQL Result Sets Hypothesis Large datasets increase worker-side processing cost. Evidence For - 80,000-150,000 records per large report. Evidence Against - DB query duration remains 40-90ms. - DB CPU is only 38%. - No query regression observed. Confidence Low-Medium Validation Measure database time separately from: Data transfer → transformation → rendering --- 🧪 PHASE 12 — BOTTLENECK REPRODUCTION PLAN Test A — Template Comparison Generate the same report data using: Old template vs New template Capture: - Data-fetch time. - Transformation time. - Chromium startup time. - Rendering time. - PDF generation time. - Memory peak. - CPU peak. - PDF size. Expected Result If the new template is the primary driver, Chromium CPU/memory and render time should increase materially. --- Test B — Concurrency Benchmark Use identical report workloads. Test: - 4 concurrent jobs. - 8 concurrent jobs. - 12 concurrent jobs. - 16 concurrent jobs. - 20 concurrent jobs. Measure: - Jobs/minute. - P50. - P95. - P99. - CPU. - Memory. - Timeout rate. The optimal concurrency should be selected based on throughput and stability rather than simply maximizing concurrency. --- Test C — Retry Failure Injection Artificially cause Chromium rendering to timeout. Compare: Current retry behavior vs Bounded retry behavior Measure: - Queue growth. - Worker utilization. - Failure rate. - Recovery time. --- Test D — Queue Stress Test Inject a representative queue burst. Capture: - Queue depth. - Queue age. - Worker utilization. - Job wait time. - Job execution time. - Retry rate. - Autoscaling response time. --- 🛠️ PHASE 13 — REMEDIATION DESIGN Fix 1 — Optimize the New Template Current State Complex charts and SVG graphics increase rendering cost. Root Cause Higher Chromium rendering workload. Recommended Fix Profile the template and identify expensive elements. Evaluate: - Chart complexity. - SVG size. - Repeated DOM elements. - Unnecessary visual elements. - Large embedded assets. Optimize only components shown by profiling to consume meaningful rendering time. Expected Effect Lower CPU/memory consumption per report. Risk Visual differences in generated PDFs. Rollback Keep the previous template available behind a feature flag and revert if rendering correctness or performance worsens. --- Fix 2 — Reduce Chromium Concurrency Current State 20 concurrent tasks per pod. Root Cause Potential resource contention. Recommended Fix Benchmark lower concurrency and select the highest stable throughput configuration. Do not assume that reducing concurrency automatically improves total throughput. Expected Effect Potentially: - Lower memory pressure. - Fewer Chromium timeouts. - More predictable job latency. Risk Too much reduction may decrease overall throughput. Rollback Restore previous concurrency if queue drain time becomes unacceptable. --- Fix 3 — Make Autoscaling Queue-Aware Current State Autoscaling is CPU-based only. Recommended Fix Scale workers using queue-related signals such as: - Queue depth. - Oldest message age. - Jobs waiting. - Worker utilization. CPU should remain an additional safety signal. Expected Effect Faster response to queue buildup. Risk Aggressive scaling could increase infrastructure cost. Rollback Return to the previous scaling configuration if worker cost or stability becomes problematic. --- Fix 4 — Bound Retry Amplification Current State Chromium timeout triggers retry. Recommended Fix Use bounded retries with appropriate backoff. Classify failures into: - Retryable. - Non-retryable. - Resource-related. - Template-related. Do not repeatedly retry deterministic rendering failures indefinitely. Expected Effect Reduced queue amplification. Risk Too few retries may increase permanent job failures. Rollback Restore the previous retry policy if successful jobs decline unexpectedly. --- Fix 5 — Separate Large Reports From Normal Reports Current State Large and normal reports share the same worker pool. Recommended Fix Route large reports to a dedicated queue/pool with controlled concurrency. Example: Normal Reports ↓ Normal Queue ↓ Normal Workers Large Reports ↓ Heavy Report Queue ↓ Resource-Controlled Workers Expected Effect Large jobs stop starving normal report workloads. Risk More infrastructure complexity. Rollback Route workloads back to the unified queue if operational complexity outweighs the benefit. --- 📈 PHASE 14 — PERFORMANCE OPTIMIZATION ROADMAP 🚨 P0 — Control Worker Concurrency - Owner: Backend/SRE - Complexity: Low-Medium - Benefit: High - Dependency: Concurrency benchmark - Verification: P95/P99, CPU, memory, timeout rate 🚨 P0 — Investigate New Template Rendering Cost - Owner: Frontend/PDF Engineer - Complexity: Medium - Benefit: High - Dependency: Chromium profiling - Verification: Old vs new template 🔥 P1 — Bound Rendering Retries - Owner: Backend Engineer - Complexity: Medium - Benefit: High - Dependency: Failure classification - Verification: Queue growth under injected failures 🔥 P1 — Queue-Aware Autoscaling - Owner: SRE - Complexity: Medium - Benefit: High - Dependency: Reliable queue metrics - Verification: Queue drain time and oldest-job age ⚙️ P2 — Dedicated Heavy-Report Queue - Owner: Platform Architect - Complexity: Medium-High - Benefit: Medium-High - Dependency: Workload classification - Verification: Normal-report latency during large-report bursts 💡 P3 — Rendering Architecture Optimization - Owner: Principal Engineer - Complexity: High - Benefit: Potentially high - Dependency: Profiling and workload analysis - Verification: Sustained throughput benchmark --- 🧪 PHASE 15 — FIX VALIDATION Before Current: - P95: 118s - P99: 191s - Failure rate: 8.7% - CPU: 97% - Memory: 94% - Queue: 1,850 Target The stated API acceptance requirement is: <2 seconds For background report completion, no explicit SLO was supplied. Therefore, a new report-generation completion SLO should be established rather than inventing one. Required Measurements Compare before vs after: - Job acceptance latency. - Queue wait time. - Processing time. - End-to-end completion time. - P50. - P95. - P99. - Failure rate. - Retry rate. - CPU. - Memory. - Queue depth. - Oldest queued-job age. - Jobs completed/minute. - Cost per completed report. --- 🔄 REGRESSION PREVENTION Template Performance Benchmark Every new PDF template should be tested against representative datasets. Test: - Small report. - Medium report. - Large report. - Maximum expected report. Measure: - Render time. - Memory. - CPU. - Output size. Worker Load Test Before production deployment: Template + Representative Data + Expected Concurrency + Expected Queue Burst must be tested together. Monitoring Create alerts for: - Worker P95/P99 processing time. - Queue depth. - Oldest job age. - Chromium timeout rate. - Retry rate. - Worker CPU. - Worker memory. - Job failure rate. - Jobs completed per minute. Deployment Guardrail A new template should not be released if benchmark results show unacceptable increases in: - Rendering time. - Memory consumption. - CPU consumption. - Timeout rate. --- 🔎 BOTTLENECK MAP POST /api/reports/generate ↓ Go API ↓ RabbitMQ Queue ↓ Queue Backlog ↓ Report Worker ↓ PostgreSQL ↓ Data Transformation ↓ Chromium/Puppeteer ↓ CPU/Memory Contention ↓ Rendering Timeout ↓ Retry ↓ Queue Growth ↓ Longer Completion Time Primary Bottleneck Chromium rendering workload Major Amplifier Excessive per-worker concurrency Secondary Amplifier Retry behavior Recovery Limitation CPU-only autoscaling Unlikely Primary Causes - PostgreSQL - API server - Network - Object storage --- 🚨 ROOT-CAUSE RANKING Rank| Root Cause| Evidence| Confidence| Impact| Priority 1| New template increased Chromium workload| New deployment, larger PDFs, high worker CPU/memory, Chromium timeouts| High| High| P0 2| Worker concurrency too aggressive| 20/20 slots, CPU 97%, memory 94%| High| High| P0 3| Rendering retries amplify backlog| Timeout → retry, failure rate 8.7%, queue 1,850| Medium-High| High| P0 4| CPU-only autoscaling misses queue signal| Queue grew dramatically; autoscaling CPU-only| Medium| Medium-High| P1 5| Large DB result processing| 80k-150k records/report| Low-Medium| Medium| P2 --- 📈 EXPECTED PERFORMANCE IMPROVEMENT No exact percentage should be promised from the supplied evidence. The highest-confidence expectation is that: Template optimization + appropriate worker concurrency should reduce per-job resource consumption and timeout frequency. Queue-aware scaling + bounded retries should improve queue recovery and reduce backlog amplification. The actual improvement must be established through controlled benchmarking. --- 🎯 FINAL VERDICT 🟡 Diagnosed With Validation Pending The evidence strongly points to the new PDF rendering workload as the primary performance problem. The most likely causal chain is: New complex template → higher Chromium CPU/memory consumption → worker resource saturation → rendering slowdown → Chromium timeouts → job retries → RabbitMQ queue buildup → longer report completion times The database, API, network, and object-storage layers do not currently have sufficient evidence to classify them as primary bottlenecks. The diagnosis should become Clearly Diagnosed only after controlled template and concurrency experiments confirm that the new template materially increases rendering cost and that reducing concurrency/optimizing the template reverses the degradation. --- ✅ FINAL BEFORE-VS-AFTER VERIFICATION PLAN Run the same representative report workload before and after remediation. Measure: Request Layer - API acceptance P50/P95/P99 Queue Layer - Queue depth - Queue wait time - Oldest message age Worker Layer - Job processing P50/P95/P99 - CPU - Memory - Concurrency - Jobs/minute Rendering Layer - Chromium render duration - Chromium timeout rate - Retry rate Reliability - Job failure rate - Successful completion rate Cost - Worker utilization - Compute cost per completed report Pass Condition The fix passes only when the system demonstrates: Stable worker resource utilization + materially reduced rendering latency + sharply reduced timeout/retry behavior + controlled queue growth + improved P95/P99 completion time, while maintaining successful report generation. The final operating principle remains: MEASURE → TRACE → ISOLATE → DIAGNOSE → FIX → VERIFY → MONITOR