## 1. Current Coverage Assessment What the confirmed tooling provides today, mapped to the architecture. "Not confirmed" means the inputs do not say, and this plan does not assume it exists. | Component | Logs | Metrics | Traces | |---|---|---|---| | API gateway, BFF layer | In ELK; format and `trace_id` presence unknown per service `<NEEDS-INPUT: trace-id-coverage-list>` | CPU, memory, QPS at most; no dependency latency or error dimensions | None | | Core services (product, cart, pricing, inventory, order, coupon) | In ELK; formats differ; `trace_id` on roughly half of all ~35 services, which half is unknown | CPU, memory, QPS dashboards; no dependency-call latency or error rate; thread/connection pool metrics not among confirmed metrics | None | | Payment, risk control (sync from order) | In ELK; same caveats | Same as above | None | | Fulfillment, notification, loyalty (Kafka consumers) | In ELK; same caveats | Same as above; consumer lag not alerted on | None | | Kafka | Not confirmed | Not confirmed `<NEEDS-INPUT: kafka-lag-metric-source>` | None | | Sharded MySQL, Redis, Elasticsearch search cluster | Not confirmed | Not confirmed `<NEEDS-INPUT: datastore-metrics-status>` | None | | Kubernetes / hosts | Not confirmed | Host CPU/memory thresholds drive most alerts | None | Alerting today is dominated by host resource thresholds. These produce many false positives, and on-call staff have habitually started ignoring some of them. Business teams run separate on-call rotations with no unified process. Consequences for scope: - **Tracing is greenfield.** No tracing system exists, so Sections 3 and 6 must include a backend, context propagation across HTTP, gRPC and Kafka, and instrumentation of services. - **Logs and metrics are extension work, not greenfield.** ELK and Prometheus/Grafana exist, so the plan standardizes and extends them instead of replacing them. - **Datastore and Kafka coverage is unknown.** Anything depending on it is flagged with `<NEEDS-INPUT: ...>` rather than assumed. ## 2. Blind Spot Identification Four blind spots, each tied to a named pain point. Sub-gaps are lettered so later sections can cite them. **Cascade-origin blindness.** *Source: the promotion incident last quarter, where slow pricing database queries slowed cart and order, thread pools exhausted, timeouts cascaded, and locating the root-cause service took 3 hours.* - **(a) No cross-service causal view.** There were only per-service logs. On-call had to infer which service degraded first by comparing timestamps across services, which is why the 3 hours went to locating the origin and not to missing data. - **(b) No dependency-call latency or error metrics.** Grafana showed CPU, memory and QPS. QPS does not reveal slowness, and CPU often barely moves when threads block on slow I/O, so the degradation of the pricing-to-database and cart-to-pricing calls had no metric to show it. - **(c) No pool saturation metrics.** Thread pool exhaustion was the mechanism of the cascade, yet pool utilization and queue depth are not among the confirmed metrics. The trend toward exhaustion was therefore invisible until timeouts appeared. - **(d) Slow query not attributable.** Which query or shard was slow is a forensic question. Whether slow query logs or datastore metrics exist is unconfirmed `<NEEDS-INPUT: datastore-metrics-status>`. Boundary note: better observability shortens diagnosis, but it does not stop a cascade from forming. Timeouts, bulkheads and circuit breakers are resilience work outside this plan. Whether such factors contributed is asked in Section 8 `<NEEDS-INPUT: incident-contributing-factors>`. **Order-trace gap.** *Source: customer support complaints that "a certain user's order keeps spinning," with engineers unable to trace the complete request by user or order number.* - **(a) No distributed trace.** Nothing represents one request across gateway, BFF, order, payment and risk as a single artifact. - **(b) Broken log correlation.** `trace_id` appears in only about half of services and log formats differ, so even manual log-based correlation fails partway down the call chain. It is also not confirmed that `order_id` or `user_id` exist as consistent, searchable fields. - **(c) Async hop is a dead end.** The order service publishes events to Kafka for fulfillment, notification and loyalty. With no tracing there is no context carried across that hop, so a request that stalls after publish cannot be followed. The actual causes of past "spinning" cases are not stated `<NEEDS-INPUT: spinning-order-root-causes>`, so this plan covers both the synchronous and asynchronous explanations. **Silent consumer lag.** *Source: Kafka consumer lag is only discovered after business complaints.* Nothing watches the asynchronous side that notification, fulfillment and loyalty depend on. This is an aggregate trend question (is the backlog growing?) with no signal or alert behind it today. **Alert distrust.** *Source: host-threshold alerts with many false positives that on-call has started ignoring, plus no unified on-call process.* The alerts measure host causes, not user-facing symptoms, and noise has trained responders to ignore pages. Every new alert proposed below inherits that distrust unless the existing noise is dealt with. This is a precondition for the other three blind spots to be closable in practice, not only a nuisance. ## 3. Signal Strategy: Logs, Metrics, Traces Signal type is chosen per sub-gap based on the question it must answer. Tags: *Directly justified* (traceable to a named blind spot) or *Inferred general best practice* (lower priority). ### Cascade-origin blindness - **(a) Cross-service causal question → traces.** Adopt OpenTelemetry instrumentation with W3C trace context propagated over HTTP and gRPC, with the trace started at the API gateway/BFF. More logging would not fix this: the missing element was correlation, not log volume. Sampling: keep all error and slow traces and a small baseline fraction (starting at roughly 5–10%), using tail-based sampling in a collector if capacity allows, otherwise head sampling at a fixed rate `<NEEDS-INPUT: peak-traffic-volume>`. *Directly justified: Cascade-origin blindness (a).* - **(b) Trend/threshold question → metrics.** Per-edge client-side RED metrics (request rate, error rate, duration histogram) labeled by caller, callee, route template and status class, for every call on the critical path. Do not label with `user_id` or `order_id`, since that explodes cardinality; those belong in traces and logs. *Directly justified: Cascade-origin blindness (b).* - **(c) Trend question → metrics.** Thread pool and connection pool gauges (active, max, queue depth, wait time) for Java services on the critical path, plus the Go equivalents (in-flight request and connection pool stats) `<NEEDS-INPUT: service-language-by-component>`. *Directly justified: Cascade-origin blindness (c).* - **(d) Forensic question → logs, with a metric assist.** Client-side database call duration histograms per service make a slow query visible as a trend; the specific statement is a forensic question answered by slow-query logging that carries `trace_id` where feasible. *Directly justified: Cascade-origin blindness (d).* ### Order-trace gap - **(a) and (c) Cross-service and async causal question → traces.** The order service tags its span with `order_id`, and `user_id` where policy allows, so a trace is findable by business key. Propagate trace context in Kafka message headers (W3C `traceparent`) so producer and consumer spans join the same trace or are linked. *Directly justified: Order-trace gap (a), (c).* - **(b) Correlation fault → structured logs.** One JSON log schema with mandatory fields `timestamp` (UTC), `level`, `service`, `trace_id`, `span_id`, `message`, and conditional fields `order_id`, `user_id` (internal ID only, no email or phone), `error.type`, `duration_ms`. For Java, OpenTelemetry's agent can inject `trace_id` into logging context with little code change. For Go, a small shared logging helper reading the trace context does the same. *Directly justified: Order-trace gap (b).* The `order_id` and `user_id` fields are subject to `<NEEDS-INPUT: pii-logging-policy>`. ### Silent consumer lag - **Trend/threshold question → metrics.** Consumer group lag measured in time (seconds behind), with record count secondary, per group for fulfillment, notification and loyalty, plus a freshness signal for the metric source itself. Traces and logs are not the fix here: the failure is "is the backlog growing," not "what happened to one message." *Directly justified: Silent consumer lag.* Source of lag metrics depends on `<NEEDS-INPUT: kafka-lag-metric-source>`. ### Alert distrust - No new signal type is needed. This gap is closed by moving paging alerts from host causes to symptom metrics already proposed above (edge latency, edge errors, pool saturation, consumer lag), and by an audit of the existing alerts. Detailed designs are in Section 4. *Directly justified: Alert distrust.* ### Inferred general best practice (lower priority, not scheduled in Phase 1) - MySQL, Redis and Elasticsearch exporter dashboards beyond what Cascade-origin blindness (d) needs. *Inferred general best practice.* - Kafka consumer error and dead-letter counts. *Inferred general best practice.* - A "stuck order" gauge (orders in a non-terminal state longer than a threshold). It would catch spinning orders proactively but is not required to trace them. *Inferred general best practice.* - Profiling, real-user monitoring, synthetic checks. No named incident requires them. *Inferred general best practice.* ## 4. Dashboard & Alert Design Priorities Ranked by cost if the blind spot recurs. All thresholds are provisional starting points, because latency and error baselines do not exist today (only QPS is collected) `<NEEDS-INPUT: baseline-latency-error-data>`; plan to tune them after two weeks of data. ### Priority 1: Cascade-origin blindness **D1. Critical-path dependency map (Grafana).** Per-edge rate, error ratio and p99 latency for gateway/BFF to the six core services and order to payment and risk, with pool utilization panels per service. *Directly justified: Cascade-origin blindness (a)–(c).* **A1. Dependency latency vs. caller timeout.** Page when an edge's p99 latency stays at or above 50% of the caller's configured timeout for 5 consecutive minutes and the edge has at least 30 requests per minute. *Directly justified: Cascade-origin blindness (b).* - Why not too noisy: anchoring to the caller's timeout and requiring 5 sustained minutes plus a volume gate avoids firing on short spikes and on idle edges. A baseline-relative rule would be a guess today because no latency baseline exists. Timeout values are needed `<NEEDS-INPUT: caller-timeout-config>`. - Why not too quiet: by Little's law, in-flight requests equal arrival rate times latency. Purely as an illustration, an edge at 1,000 req/s whose latency rises from 50 ms to 2.5 s goes from about 50 to 2,500 concurrent requests, so pools fill well before calls hit their timeouts. Firing at half the timeout is meant to land before that point. **A2. Pool saturation.** Page when any critical-path thread or connection pool stays above 80% utilization for 3 minutes. *Directly justified: Cascade-origin blindness (c).* - Why not too noisy: the 3-minute window ignores short bursts, and pools that legitimately run hot get a per-pool override. - Why not too quiet: at 100% requests are already queueing and timing out; 80% gives lead time before the incident's actual failure mode. **A3. Edge error ratio.** Page when an edge's error ratio exceeds 2% over 5 minutes with at least 100 requests in the window; use 1% for order to payment and order to risk, where each failure is likely a lost order. *Directly justified: Cascade-origin blindness (a)–(b).* - Why not too noisy: the volume gate and window ignore isolated retries. - Why not too quiet: a few percent errors on a core edge is already user-visible. The accepted gap is low-traffic edges at night, which are gated out and judged lower impact. ### Priority 2: Order-trace gap **D2. Order lookup view.** A Kibana saved search and trace search keyed on `order_id` and `trace_id`, linking from any log line to its trace and back. This is a lookup tool, not a monitoring panel. *Directly justified: Order-trace gap (a)–(c).* **D3. Order-flow stage latency.** Panels for order to payment, order to risk and order to Kafka publish, built from the same edge metrics as D1. *Directly justified: Order-trace gap (a).* No new alert is proposed for this blind spot: the order edges are already covered by A1–A3, and a "stuck order" alert is demoted to inferred best practice in Section 3. ### Priority 3: Silent consumer lag **D4. Consumer lag.** Time-lag and record-lag per consumer group, with the lag exporter's own freshness shown. *Directly justified: Silent consumer lag.* **A4. Consumer time-lag.** Page when a group's time-lag exceeds its tolerance for 5 consecutive minutes. Provisional tolerances are fulfillment 5 minutes, notification 10 minutes, loyalty 30 minutes `<NEEDS-INPUT: consumer-lag-tolerance>`. Additionally, open a ticket (no page) when lag has risen continuously for 15 minutes even while under tolerance. Also alert if the lag metric itself goes missing. *Directly justified: Silent consumer lag.* - Why not too noisy: rebalances and deploys cause brief lag spikes, which the 5-minute sustain filters out. Time-lag is used instead of record count because promotion throughput varies and record counts mislead. - Why not too quiet: the rising-lag ticket catches a growing backlog before tolerance is breached, the fixed tolerance catches a steady high backlog that a slope rule would miss, and the missing-metric alert catches the case where the exporter dies and lag silently stops being reported. ### Priority 4: Alert distrust (runs in parallel, enabler for the rest) **D5. Paging-alert health.** Weekly counts of each paging alert's firings and whether each was actionable. *Directly justified: Alert distrust.* **Alert audit rule.** Over a two-week inventory, any host CPU or memory alert that did not lead to action is demoted from page to dashboard or ticket. Every remaining paging alert must have an owning team and a runbook link. *Directly justified: Alert distrust.* **A5. Minimal node-level alerts.** Keep node not-ready and disk-full-within-4-hours projection as pages, since they have clear actions and low noise. *Inferred general best practice, lower priority.* ## 5. Correlation & Debugging Workflow The workflow below assumes Phase 1 coverage (Section 7). Target durations are goals to be measured in the game day described in Section 7, not promises. **Scenario 1: a promotion-style cascade (Cascade-origin blindness).** 1. Alert A1 or A2 fires and names the edge (caller to callee) or the saturated pool, with a runbook link and the owning team. 2. On-call opens D1 and follows the degraded edges downstream to the deepest degraded edge. In the past incident this would be pricing to its database, not cart or order, which were only victims. 3. From the latency panel, click through to a slow trace for that edge (via metric exemplars if the chosen trace backend supports them, otherwise by trace search on the edge) `<NEEDS-INPUT: tracing-hosting-constraints>`. The span waterfall shows which hop consumed the time. 4. Copy the `trace_id` into Kibana. Every service's logs for that request are now one filtered view, including the slow-query log for the pricing statement. 5. Hand off to the pricing team with the trace link, replacing the timestamp-matching exercise that took 3 hours. **Scenario 2: "this order keeps spinning" (Order-trace gap).** 1. Support gives engineering an order number. 2. Search Kibana by `order_id` to find any log line, read its `trace_id`, then filter all logs by that `trace_id`. The trace backend can also be searched by the `order_id` span attribute. 3. The span tree shows where the request stopped. If a synchronous hop (payment or risk) timed out, the failing span is visible. If the order service published to Kafka and the downstream consumer span is missing, go to D4 and check that consumer group's lag. 4. If the client retried and created several traces, use `order_id` and `user_id` to group them. **Scenario 3: consumer backlog (Silent consumer lag).** 1. A4 pages (or the rising-lag ticket opens) for a specific group before any business complaint. 2. D4 shows which group and how fast lag is growing. Trace search on recent messages for that group shows whether consumer spans are slow, failing, or absent. **What stays manual.** Requests passing through services not yet instrumented lose the trace at that point, and Kafka consumers not yet instrumented have no consumer spans. Phase 1 covers only the critical path (Section 7), so engineers will still need the old manual method for the remaining services. This is a stated limitation of the phase boundary, not a gap in the design. ## 6. Tooling Gaps & Build-vs-Buy Considerations | Gap | Realistic options and honest trade-offs | |---|---| | Tracing backend (does not exist) | **Grafana Tempo (self-hosted):** fits the existing Grafana and Prometheus exemplar workflow and uses cheap object storage; adds a system for a 2-person capacity to run. **Jaeger:** mature and simple, but its storage choice matters; reusing Elasticsearch would load the cluster, and whether the logging ELK cluster is separate from the product search Elasticsearch cluster is unknown `<NEEDS-INPUT: elk-cluster-separation>`; search load must never touch product search. **Elastic APM:** consistent with ELK, but depends on license tier and still adds ELK load. **SaaS (Datadog, Honeycomb and similar):** fastest to value and least operations work, but cost scales with spans and hosts across ~35 services and promotion peaks, and data-hosting constraints may rule it out `<NEEDS-INPUT: tracing-hosting-constraints>`. No default pick is made; see recommendation below. | | Context propagation and instrumentation (does not exist) | **OpenTelemetry Java agent:** low code change for Java services (HTTP, gRPC, JDBC, Kafka clients) and injects trace context into logs. **Go services:** need explicit middleware and client wrappers (otelhttp, otelgrpc, a Kafka client hook), which is more code per service. **Service mesh:** if one exists it gives edge metrics cheaply but does not propagate trace context through application code, so it does not replace instrumentation `<NEEDS-INPUT: service-mesh-presence>`. | | Log format standard (partial) | Keep ELK. Add the JSON schema from Section 3 through a shared logging library or helper and an ELK index template; do not reindex old logs. Cost is mostly adoption work (Section 7), not tooling. | | Dependency and pool metrics (missing) | Extend Prometheus via the same instrumentation libraries. Control cardinality: no user or order identifiers as metric labels. Low cost. | | Kafka consumer lag (unconfirmed) | If not already exported, a lag exporter (kafka_exporter, Burrow or similar) is a small, low-maintenance addition `<NEEDS-INPUT: kafka-lag-metric-source>`. | | Alert routing and ownership (unconfirmed) | Depends on current alerting tooling and paging system `<NEEDS-INPUT: alerting-stack>`. Prefer reconfiguring existing alerting over buying a new one. | **Recommendation.** Standardize on OpenTelemetry instrumentation behind an OpenTelemetry Collector first, and decide the backend after a one-week spike. Instrumentation is the expensive, sticky part; the backend is comparatively swappable behind the collector, so the choice is reversible. Run the spike against Tempo, plus Jaeger or a SaaS trial if constraints allow, measuring operational effort and cost at promotion-level volume `<NEEDS-INPUT: peak-traffic-volume>`. Do not buy a comprehensive platform by default; no named incident requires profiling, RUM or APM-suite features. ## 7. Rollout & Adoption Plan **Capacity.** About 2 people for 6–8 weeks is roughly 12–16 person-weeks, shared with daily operations. That is not enough to instrument all ~35 services, so Phase 1 covers the critical path only: gateway, BFF, the six core services, payment, risk control, and the order-event Kafka path including the three consumers (roughly 13 components). Remaining services are Phase 2, ordered using the dependency graph that the first traces reveal. **Sequencing by impact versus feasibility.** | Blind spot | Impact | Feasibility | Placement | |---|---|---|---| | Silent consumer lag | Medium–high | Very high (exporter, one dashboard, a few alerts; days) | Weeks 1–2, first quick win | | Alert distrust | High (enabler) | High (inventory and policy; needs on-call buy-in) | Weeks 1–3, in parallel | | Cascade-origin blindness | Highest | Medium (needs instrumentation on the critical path) | Core of Phase 1, weeks 2–8 | | Order-trace gap | High (frequent complaints) | Medium, depends on tracing and the log schema | Partial in Phase 1: order path and Kafka hop; remaining consumers and services in Phase 2 | **Timeline.** - **Week 0–1 (decisions and spike):** tracing backend spike, trace-id coverage inventory `<NEEDS-INPUT: trace-id-coverage-list>`, draft log schema, begin alert inventory. - **Weeks 1–2:** Silent consumer lag: exporter, D4, A4. - **Weeks 1–3:** Alert distrust: audit, demotion, owner and runbook assignment, D5. - **Weeks 2–6:** Cascade-origin blindness: collector and backend, instrumentation of the critical path starting with gateway, BFF, pricing, cart and order, then payment, risk, inventory, product and coupon; edge and pool metrics; D1, A1–A3. - **Weeks 4–7:** Order-trace gap: log schema in the order path, `order_id` span tagging, Kafka context propagation, D2 and D3. - **Week 7–8 (game day):** replay the promotion scenario in a load-test environment and measure time to locate the origin; test order lookup and lag alerts with a deliberately stalled consumer. **Proposed split of the 2 people (to be confirmed against actual availability):** one on tracing, collector and instrumentation; one on log schema and library, edge and pool metrics, lag and alerts. **Provisional success measures, one per blind spot.** - Cascade-origin blindness: origin service identified within 15 minutes in the game day (provisional target). - Order-trace gap: an engineer reaches the full trace for an order in under 5 minutes. - Silent consumer lag: a stalled consumer pages before any business complaint. - Alert distrust: a majority of paging alerts rated actionable in D5 within a quarter (target to be set after the first measurement). **Adoption work beyond tooling.** Observability strategies commonly fail at adoption, so these are scheduled work, not assumptions: - **Instrumentation habits.** Provide instrumented client factories and a shared logging library so engineers do not need to remember to propagate context. Add a pull-request checklist item that any new outbound call or Kafka producer or consumer uses them. - **Log schema compliance.** Agree the schema with business teams, publish it as a one-page standard, and add a CI lint or sample check on the mandatory fields. - **Business-team capacity.** Services owned by business teams need their time to adopt the libraries `<NEEDS-INPUT: business-team-instrumentation-capacity>`. Without agreement, Phase 1 stalls at the services the platform team owns. - **Minimal unified on-call convention.** Not a full process redesign: every paging alert has an owner team, a severity, and a runbook link, so that A1–A4 route to someone. - **Postmortems.** Introduce a lightweight standard template with an "observability gap" section so future incidents feed the backlog, and run the first one on the promotion incident's findings. - **Training.** One short session (about 45 minutes) on the Section 5 workflow, using the game-day trace as the example. **Explicitly not in Phase 1:** the inferred best-practice items in Section 3, instrumenting the remaining ~22 services, service mesh adoption, and a full SLO program. ## 8. Open Questions & Data Requests 1. `<NEEDS-INPUT: trace-id-coverage-list>` — Which services (roughly half of ~35) already emit `trace_id`, and in what format: W3C `traceparent`, a custom header, or something else? 2. `<NEEDS-INPUT: kafka-lag-metric-source>` — Is Kafka consumer lag currently exported to Prometheus or visible anywhere, and for which consumer groups? 3. `<NEEDS-INPUT: datastore-metrics-status>` — Do exporters, slow-query logging or dashboards exist for the sharded MySQL, Redis and Elasticsearch clusters? 4. `<NEEDS-INPUT: elk-cluster-separation>` — Is the ELK logging cluster separate from the Elasticsearch product search cluster, and what is its headroom and retention? 5. `<NEEDS-INPUT: alerting-stack>` — What currently evaluates and routes alerts (Alertmanager, Grafana alerting, other), and what is the paging tool? 6. `<NEEDS-INPUT: service-mesh-presence>` — Is a service mesh in use on the Kubernetes clusters? 7. `<NEEDS-INPUT: service-language-by-component>` — Which of the critical-path components are Java and which are Go, and which edges use HTTP versus gRPC? 8. `<NEEDS-INPUT: caller-timeout-config>` — What are the configured timeouts and thread or connection pool sizes on each critical-path edge (gateway, BFF, cart, order, pricing, payment, risk)? 9. `<NEEDS-INPUT: baseline-latency-error-data>` — Is any historical latency or error data available (for example from the gateway or load balancer) to set baselines before instrumentation produces its own? 10. `<NEEDS-INPUT: peak-traffic-volume>` — What are peak requests per second at the gateway and on the busiest internal edges during promotions, for sizing sampling and storage? 11. `<NEEDS-INPUT: consumer-lag-tolerance>` — How many minutes of lag can fulfillment, notification and loyalty tolerate before the business impact is real? 12. `<NEEDS-INPUT: business-team-instrumentation-capacity>` — How much engineering time can business teams commit to adopting the logging library and instrumentation in the next 6–8 weeks? 13. `<NEEDS-INPUT: tracing-hosting-constraints>` — Are there budget limits, data-residency rules, or a policy against SaaS observability tools that constrain the tracing backend? 14. `<NEEDS-INPUT: spinning-order-root-causes>` — For past "order keeps spinning" complaints, were the causes ever found (synchronous timeouts, Kafka stalls, client-side polling), and how frequent are they? 15. `<NEEDS-INPUT: pii-logging-policy>` — May `user_id` and `order_id` be written to logs and trace attributes, and are there restrictions on retention or access? 16. `<NEEDS-INPUT: incident-contributing-factors>` — For the promotion incident, is the full chain understood, including whether timeout settings, retry behavior, or missing circuit breakers contributed beyond the observability gaps named here?