## 1. Service Scope & Critical User Journeys `payment-authorize` is on the revenue path: checkout page and mobile app → risk control service → card network gateway → accounting write. A failure at any step loses transactions (Finance estimate: roughly USD 8,000 of transaction volume per minute of unavailability), and three internal services (subscription renewal, refunds, reconciliation) call it too. Two journeys get SLOs: | ID | Journey | Why it earns an SLO | |---|---|---| | CUJ-1 | A consumer taps "Pay" and receives an authorization result | Consumers tolerate at most 3 s; beyond that abandonment and repeated clicking rise noticeably. Authorization success is what merchants rank first. A failure here is a lost transaction. | | CUJ-2 | A merchant is charged exactly once per customer payment intent | The second thing merchants named. Repeated clicks after 3 s (stated) and gateway timeouts (2 of the 3 incidents over 10 minutes) are plausible routes to duplicate requests; that mechanism is not yet verified. | Deliberately **not** given SLOs: - **Risk control and card network gateway latency/errors.** These are causes, not user experience. They are diagnostic dashboards linked from alerts and never page on their own. - **Internal callers (renewal, refunds, reconciliation).** They traverse the same path and share its failure modes, so their requests are counted in Authorization Success Rate and separated by a `caller` dimension. No separate SLO until there is data showing their needs differ. - **Accounting write as a standalone SLO.** It is not a journey a user sees; its failures matter through CUJ-2 and are flagged in Section 5. - **Per-endpoint or per-region SLOs.** No data exists to set them; revisit after the baseline. ## 2. Candidate SLI Selection | SLI | Journey | Definition (good ÷ valid) | Why this is the right proxy | |---|---|---|---| | **Authorization Success Rate** | CUJ-1 | Good = valid authorization requests that end in *approved* or *business decline* (issuer or risk-policy decline). Bad = any system failure: 5xx, timeout, failed risk-control or gateway call, accounting-write failure that fails the request, and any "decline" actually caused by a dependency failure (e.g. risk service unreachable → declined). Valid = well-formed requests; caller validation errors are excluded. | The consumer's and merchant's question is "did my payment get a real answer?" A legitimate decline is the system working; a 5xx is the system failing. The existing metric (share of non-5xx responses) cannot separate these, so it is a weaker proxy, usable only as an interim. | | **Authorization Latency** | CUJ-1 | Good = requests counted good in Authorization Success Rate that return within 3 s, measured from arrival at the gateway to response sent. Valid = same denominator. Failed requests are excluded so a failure is not double-counted. | The stated abandonment cliff is a threshold (3 s), not a mean. A proportion-under-threshold measure maps to it directly; an average or a lone p99 hides how many users cross it. **Weakness:** the gateway clock excludes network and client time, so the user's real wait is longer than measured. It is proposed because click-to-result measurement does not exist yet. | | **Duplicate Charge Rate** | CUJ-2 | Good = distinct customer payment intents that produce exactly one captured/authorized charge. Bad = intents that produce two or more. | It is what the merchant actually experiences. The easy alternative, counting repeated idempotency-key hits, is a weaker proxy: it measures how often the protection fired, not whether a double charge got through (a high count may even mean the protection is working). | ## 3. SLO Target Setting & Rationale **Window: rolling 28 days** = 28 × 24 × 60 = 40,320 minutes. Four weeks contain exactly four Tuesday promotional pushes, so every window sees the same weekly traffic cycle. **Authorization Success Rate: 99.9% over rolling 28 days.** *Data-derived* (provisional: 3 months of data, outcomes not yet classified). - Budget at 99.9% = 0.10% of requests. The 90-day average failure ratio is 100% − 99.93% = 0.07%, so an average month uses 0.07 ÷ 0.10 = 70% of it. The worst month (99.81%, failure ratio 0.19%) would have used 0.19 ÷ 0.10 = 190%. - If 99.93% is the equal-weight mean of three months, the other two average 99.99%: (3 × 99.93 − 99.81) ÷ 2 = (299.79 − 99.81) ÷ 2 = 99.99%, i.e. roughly 10% of budget each. Failures are concentrated in incident months, so the mean flatters the typical month. A 99.9% target passes quiet months with room and is breached by an incident-driven month, which is what the SLO is for. This inference depends on equal monthly weights; see `<NEEDS-INPUT: baseline-data>`. - Why not looser: 99.8% would have passed even the worst month (99.81%) and allows 0.002 × 40,320 = 80.64 minutes of full-outage equivalent per window. - Why not tighter: 99.95% means a budget of 0.05%, which the average month already exceeds (0.07% > 0.05%). That demands a bar current performance does not meet. - This number coincides with the commonly cited 99.9%. It was chosen from the distribution above, not from convention. - Exposure at this target: 40,320 × 0.001 = 40.32 minutes; 40.32 × USD 8,000 = **USD 322,560** of transaction volume at risk per window if burned as full-outage minutes. Whether that is acceptable is a business decision: `<NEEDS-INPUT: acceptable-volume-at-risk>`. - The 99.93% figure is a non-5xx share from gateway logs and may move in either direction once business declines and dependency-caused declines are separated: `<NEEDS-INPUT: outcome-classification>`. **Authorization Latency: 99% of requests within 3 s over rolling 28 days.** The 3 s threshold is *Requirement-derived* (consumers' stated maximum wait). The 99% proportion is provisional: `<NEEDS-INPUT: latency-histogram>`. - p99 = 2.9 s means at least 99% of requests finished under 2.9 s, hence at least 99% finished under 3 s. The target is met by construction today, but with only 0.1 s of margin at the tail. Percentiles do not reveal how far above 99% the real figure sits (it could be 99.01% or 99.99%). - Why not 95%: p95 is 1.4 s. A 95% target would tolerate up to 5% of users beyond 3 s, up to five times the slow share the data implies today (≤1%), and would bless a large regression. - Expect thin headroom: risk control slowdowns occur about 4 times per month, and Tuesday-push latency is not segmented in the data (`<NEEDS-INPUT: peak-traffic-baseline>`). **Duplicate Charge Rate: `<NEEDS-INPUT: baseline-data>`**, specifically `<NEEDS-INPUT: duplicate-charge-tolerance>`. No measurement exists, and the stated requirement ("not charging twice") is directional with no tolerance. A target of zero is not an SLO (the first event exhausts a zero budget permanently), and any other figure would be invented. Interim rule: every confirmed duplicate charge is recorded and reviewed as an incident. ## 4. Error Budget Policy **Budget arithmetic** (28 days = 40,320 minutes): | SLO | Budget | Time-equivalent at average traffic | 50% consumed | 75% consumed | |---|---|---|---|---| | Authorization Success Rate (99.9%) | 0.1% of valid requests | 40,320 × 0.001 = 40.32 min of full outage | 20.16 min | 30.24 min | | Authorization Latency (99% ≤ 3 s) | 1% of requests slower than 3 s | not meaningful as minutes | 0.5% of requests | 0.75% of requests | | Duplicate Charge Rate | not defined | not defined | not defined | not defined | Scale check: a 20-minute full outage consumes 20 ÷ 40.32 = 49.6% of the Authorization Success Rate budget. Budgets are fractions of requests, and the minutes above assume average traffic. During the Tuesday push (≈3× normal weekday traffic), a minute of outage touches about 3× as many requests and burns budget up to 3× faster than the minute figures suggest. **Phase 0: Baseline (at least two full 28-day windows; ends at the exit review in Section 7).** Management wants trustworthy data before accepting constraints, and the Search team's pilot shows what happens without a hard rule. So Phase 1 rules below are agreed now, and the exit review only asks whether the data is trustworthy yet. During Phase 0 the tiers are mandatory review triggers, not automatic freezes: - 75% consumed → service owner holds a documented reliability review within 5 business days. - 100% consumed → same review, plus a recorded decision by the service owner on whether releases continue, with the reason. **Phase 1: Enforced (after the exit review).** Applies to Authorization Success Rate and Authorization Latency, whichever exhausts first: - **50% consumed** → notify service owner and channel; top contributors reviewed at the weekly review. No restrictions. - **75% consumed** → non-reliability deployments to `payment-authorize` are paused pending a reliability review. Reliability fixes, security patches and rollbacks are exempt. - **100% consumed** → freeze all non-emergency changes to `payment-authorize` until **any one** of: (a) remaining rolling-window budget is above zero; (b) all high-priority postmortem actions for the causing incident are complete and the service owner signs off; (c) the service owner approves a written exception, recorded with its reason. - Exit path (b) exists because of the rolling window: a single large incident ages out all at once, so a 76.6-minute outage (the worst month's equivalent: 0.0019 × 40,320) would hold the budget above 100% for up to 28 days. **Duplicate Charge Rate:** every confirmed duplicate triggers an incident ticket and review. A freeze trigger is deferred until `<NEEDS-INPUT: duplicate-charge-tolerance>` is answered. **Dependency-caused burn** (card network gateway, risk control) counts fully, because users experience it. A freeze does not fix a gateway timeout, so when burn is attributed to a dependency the required consequence is prioritized work on our side (timeouts, retries, degradation handling) rather than a release freeze. Whether to formally exempt dependency-caused burn is open: `<NEEDS-INPUT: dependency-exclusion-policy>`. Exceptions and notifications need a named owner: `<NEEDS-INPUT: service-owner>`. ## 5. Measurement & Data Source Validation | SLI | Measured from | Reflects user experience? | Gap / prerequisite | |---|---|---|---| | **Authorization Success Rate** | Today: gateway-layer access logs (non-5xx share). Whether Prometheus counters carry a status/outcome label is unconfirmed (`<NEEDS-INPUT: outcome-classification>`). | **Partially.** Misses requests that never reach the gateway (client or network failures). Cannot separate business declines from system errors, so a decline caused by a dependency failure looks like a success. A failure after card network approval (accounting write) may not surface as a 5xx. | Add an `outcome` label (`approved`, `business_decline`, `system_error`, `invalid_request`), with the rule that dependency-caused declines are `system_error`. Add a `caller` label. Re-cut the 90-day logs with the same classification if fields allow. An external synthetic checkout check would capture never-reached-server failures; feasibility unknown. | | **Authorization Latency** | Today: gateway timing; Prometheus latency dashboards are percentile-based. A "share ≤ 3 s" ratio needs a histogram bucket boundary exactly at 3 s (`<NEEDS-INPUT: latency-histogram>`). | **Optimistic.** User wait = gateway time + network + client time + page overhead, so the true tail beyond 3 s is larger than measured and a 3 s gateway threshold is lenient against the user requirement. | Add a 3 s bucket. Add a real-user timer in the checkout page and mobile app from tap to result render (`<NEEDS-INPUT: client-side-latency>`). Then set the gateway threshold to 3 s minus the observed client/network overhead. Until then the 3 s gateway threshold is an interim, known to be lenient. | | **Duplicate Charge Rate** | No source exists in the inputs. Candidate: reconciliation/accounting ledger joined with card network settlement, grouped by payment-intent identifier. | **Strongest available** (it reflects what the cardholder is charged) but delayed by settlement lag, so it is retrospective and cannot page in real time. | Needs a stable intent/idempotency identifier carried from request through ledger and settlement, and a duplicate report from reconciliation (`<NEEDS-INPUT: duplicate-detection-source>`). Prerequisite, not assumed. | Before any alert in Section 6 goes live, backtest its rules against the 90 days of gateway logs. ## 6. Alerting Strategy Tied to SLOs Burn rate = observed bad-event ratio ÷ budget ratio. A burn of 1× exhausts the budget exactly at the end of the 28-day window (28 × 24 = 672 hours). Budget consumed over *h* hours at burn *B* = B × h ÷ 672. **Authorization Success Rate** (budget ratio 0.001). An alert fires only when both the long and short windows exceed the burn rate, so it stops soon after the problem does. | Tier | Windows (long AND short) | Burn | Failure ratio that triggers | Budget consumed in long window | Time to exhaust at that burn | Action | |---|---|---|---|---|---|---| | Page (high urgency) | 1 h AND 5 min | 13.44× (2% × 672 ÷ 1) | 13.44 × 0.001 = 1.344% | 13.44 × 1 ÷ 672 = 2% | 672 ÷ 13.44 = 50 h | Page on-call via PagerDuty | | Notify (low urgency) | 6 h AND 30 min | 5.6× (5% × 672 ÷ 6) | 5.6 × 0.001 = 0.56% | 5.6 × 6 ÷ 672 = 5% | 672 ÷ 5.6 = 120 h | Low-urgency PagerDuty, no wake-up | | Ticket | 72 h AND 6 h | 1× | 0.1% | 1 × 72 ÷ 672 = 10.7% | 672 h (28 d) | Ticket for the weekly review | Detection time for the page tier (1 h window reaching 1.344%): full outage ≈ 0.8 min; 10% failures ≈ 1.344 ÷ 10 × 60 = 8.1 min; 5% ≈ 16.1 min; 2% ≈ 40.3 min. A slow, steady burn near 1× never pages. It becomes a ticket, and the policy tiers in Section 4 (50%, 75%, 100% consumed) notify the service owner independently of burn rate. **Authorization Latency** (budget ratio 0.01), same multipliers: 13.44× = 13.44% of requests slower than 3 s over 1 h; 5.6× = 5.6% over 6 h; 1× = 1% over 72 h. In Phase 0 all three tiers are notify/ticket only, because the proportion is provisional and the gateway clock is optimistic. The 13.44× tier may page after the exit review. **Duplicate Charge Rate:** each confirmed duplicate opens a high-priority ticket. A burst rule that pages is deferred to `<NEEDS-INPUT: duplicate-charge-tolerance>`. **Guards and noise:** - Alerts use ratios, so the Tuesday 3× traffic does not change thresholds. It does change the cost at the boundary: 1.344% failures at 3× volume is 3× as many failed payments. Revisit after Tuesday-segmented data (`<NEEDS-INPUT: peak-traffic-baseline>`). - Add a minimum request count to the 5-minute and 30-minute windows so overnight trough traffic cannot trip the page (`<NEEDS-INPUT: traffic-trough-volume>`). - Risk control and card network gateway panels are linked from alerts as diagnostics. They page only through SLO burn. - Expected paging load, to be checked by the backtest: about 3 incidents over 10 minutes per 90 days suggests roughly one to a few pages per month. If the backtest shows materially more, the thresholds are wrong, not the on-call rotation. ## 7. Review Cadence & Iteration Plan - **Weekly during Phase 0 (30 minutes):** burn chart, top contributors, data-quality check on the new labels and histogram. - **Checkpoint 1, end of the first 28-day window (about week 4):** compare observed SLIs to targets; confirm the outcome classification works and measure how much it moves the success rate; confirm the 3 s histogram bucket; review backtest results; decide whether 99.9% stands. - **Checkpoint 2 / exit review, end of the second window (about week 8):** finalize or revise all three targets and set the Phase 1 start date. Exit criteria: (1) `outcome` label live for at least 28 days; (2) latency histogram with a 3 s bucket live and segmented Tuesday vs other days; (3) client-side latency measured, or the gap formally accepted by the service owner; (4) duplicate-detection source exists, or Duplicate Charge Rate stays explicitly provisional; (5) alert backtest completed. - **After exit:** monthly SLO review (budget consumed, incident attribution, policy actions taken); quarterly re-evaluation of targets against accumulated data. - **Event-driven, out-of-cycle:** any single incident consuming 50% or more of a budget (20.16 min-equivalent for Authorization Success Rate); any change to the Tuesday promotion cadence; a new dependent service; a change of risk control or gateway provider. - The 90-day baseline contains only whatever peaks fell inside it; check whether a larger seasonal peak is approaching before trusting headroom (`<NEEDS-INPUT: seasonal-peaks>`). ## 8. Open Questions & Data Requests Ranked by how much the answer would change the proposed targets. 1. `<NEEDS-INPUT: outcome-classification>` Of the last 90 days of responses, how many were business declines, and how many system failures were returned as non-5xx (for example, declines caused by risk control being unreachable)? Can the logs or Prometheus counters be re-cut with an outcome label retroactively? 2. `<NEEDS-INPUT: baseline-data>` What were the success rate and failure-minute totals for each of the last three months (weekly if available), with error ratio and duration per incident? This tests the 99.9% choice; the "other two months ≈ 99.99%" inference assumes equal monthly weights. 3. `<NEEDS-INPUT: acceptable-volume-at-risk>` Does the business accept about USD 322,560 of transaction volume at risk per 28 days (40.32 min × USD 8,000) as the worst case inside the SLO, or is a tighter figure wanted (99.95% = 20.16 min = USD 161,280)? 4. `<NEEDS-INPUT: latency-histogram>` What share of requests exceeded 3 s in each of the last 12 weeks, split Tuesday-push vs other days, and does the latency histogram have a bucket boundary at 3 s? 5. `<NEEDS-INPUT: client-side-latency>` How much time passes between the tap and the request reaching the gateway, and between response and result render, on web and mobile? Is any real-user timing data collected today? 6. `<NEEDS-INPUT: duplicate-charge-tolerance>` What rate or count of duplicate charges per 28 days do merchants tolerate, is any single duplicate a top-priority incident, and how many confirmed duplicates occurred in the last 90 days? 7. `<NEEDS-INPUT: duplicate-detection-source>` Is there a stable payment-intent or idempotency identifier from request through ledger and card network settlement, and can reconciliation produce a duplicate-charge report? Does an accounting-write failure after card network approval surface as an error to the caller? 8. `<NEEDS-INPUT: peak-traffic-baseline>` What were the success rate, latency distribution and request rate during Tuesday pushes in the last 90 days? 9. `<NEEDS-INPUT: dependency-exclusion-policy>` Should card network gateway and risk control failures count fully against the budget (the default in this document), and does the organization want a different consequence for them than a release freeze? 10. `<NEEDS-INPUT: traffic-trough-volume>` What is the minimum request count per 5-minute window at the overnight trough, for sizing the minimum-traffic guard? 11. `<NEEDS-INPUT: seasonal-peaks>` Are any seasonal or campaign peaks expected in the next 8 weeks that the 90-day baseline did not include? 12. `<NEEDS-INPUT: service-owner>` Who is the named service owner who receives budget notifications and approves exceptions?