## 1. Release Scope Summary & Evidence Confidence **Scope: mobile app 5.2.0.** Six change areas, labeled here so they can be traced through Sections 4–5: - **R1 – Order Tracking page:** new page replacing the old WebView page. - **R2 – Apple Pay HK/SG:** expansion of the Apple Pay payment entry point to Hong Kong and Singapore. - **R3 – Recommendation ranking:** new ranking logic for the personalized recommendation slots on the home page. - **R4 – Recommendation timeout config:** server-side change, recommendation service timeout 800 ms → 500 ms. - **R5 – Defect fixes:** 14 fixes, 2 of which are crashes carried over from the previous version. - **R6 – Firebase SDK:** bundled minor-version upgrade. **Context that sets the bar:** consumer app with global users; the annual major promotion starts next week; management wants the features live before it; a payment-path problem during the promotion is extremely costly. The team accepts a phased rollout (1% → 10% → 50% → 100%) and will not accept a payment success rate drop of more than 0.3 percentage points. Management's preference for timing is a pressure input, not evidence of readiness. **Evidence confidence: sufficient to block a release, not sufficient to clear one.** *Confirmed by supplied evidence:* - Unit test results (2,310 / 2,310 pass) and new-code line coverage (78%). - UI suite counts (181 / 186 pass) with a stated cause for each of the 5 failures. - iOS and Android main-flow regression sign-off. - Gray release crash rate (0.21% vs 0.14% for the previous version). - Open defect counts by severity (1 P2, 6 P3, no P0/P1). - Crashlytics monitoring with a 0.5% alert; order tracking instrumentation live on the dashboard. - The existence of a server-side switch for R1, the absence of any switch for R2, and the absence of an alert for R4. *Missing (each becomes an open item in Section 7):* - Real-card Apple Pay results for Singapore (sandbox only) and any Apple Pay results for Hong Kong. - Payment success rate baselines, monitoring, or halt criteria tied to the 0.3 pp tolerance. - Any item-level test evidence for R3, R4, R6, and per-fix verification for R5. - Severity of the 2 failing order tracking tests, and any mapping of the open P2/P3 defects to this release's scope. - Cause or event counts behind the crash-rate increase. - A rollback drill, an on-call runbook, and rollout stage gates. **Ceiling:** the supplied evidence is enough to establish failures and gaps, so a No-Go can be supported. It is not enough to certify any payment-path readiness, so no "Go" can be supported until the Section 7 items are closed. --- ## 2. Test & Quality Signal Review **Unit tests.** 2,310 of 2,310 passed — *Confirmed by supplied evidence*. New-code line coverage is 78% — *Confirmed*, but no coverage target and no per-feature breakdown were supplied, so it cannot be judged against a threshold or attributed to R1–R6. About 22% of new-code lines are not exercised by unit tests. `<NEEDS-INPUT: item-level-test-evidence>` **UI automation.** 186 cases, 181 passed (97.3%), 5 failed: - 3 failures are described as confirmed intermittent timeouts in the test environment. Accepted as stated; how intermittency was confirmed, and which flows those tests cover (for example, whether any touch Apple Pay), was not supplied. `<NEEDS-INPUT: ui-timeout-evidence>` - 2 failures concern empty-state rendering on the new Order Tracking page (R1), **not yet fixed**. These are tied to this release's scope. Their severity was not stated. `<NEEDS-INPUT: order-tracking-empty-state>` **Manual QA.** iOS and Android main-flow regression completed and signed off — *Confirmed*. Whether "main flow" includes the new Order Tracking page, the recommendation slots, or Apple Pay in the new regions was not stated. **Apple Pay with real bank cards.** Verified only in the sandbox environment for Singapore cards — *Confirmed as a limitation*. No evidence of any kind for Hong Kong. Real-money verification of R2 is *untested*. `<NEEDS-INPUT: applepay-real-card-verification>` **Gray release (500 people, 4 days).** Crash rate 0.21% vs 0.14% for the previous version — an absolute increase of 0.07 pp, about 1.5× relative. *Confirmed by supplied evidence* as an adverse signal. - The cohort is small, and the number of crash events behind each rate and the metric definition (users vs sessions) were not supplied, so the difference can be neither called noise nor called a confirmed regression. Under this audit's rules, an unexplained adverse signal is not cleared by being small. - Which change area is responsible is not established; R1, R3, R5, and R6 are all in the build and no evidence excludes any of them. `<NEEDS-INPUT: crash-regression-attribution>` - No payment success rate data, and no indication that the cohort included Hong Kong/Singapore users or Apple Pay transactions, was supplied. `<NEEDS-INPUT: payment-success-monitoring>` **Open defects.** 1 P2, 6 P3, no P0 or P1 — *Confirmed*. None is stated to be connected to this release's scope, so these are noted as background and are **not** counted as blockers. If the P2 corresponds to the order tracking empty-state failures, that connection is not established by the inputs. `<NEEDS-INPUT: open-defect-scope-mapping>` **Evidence status per scope item:** | Area | Test evidence supplied | Status | |---|---|---| | R1 Order Tracking | Regression sign-off (scope of "main flow" unclear); 2 UI failures open | Partially tested, with known failures | | R2 Apple Pay HK/SG | Sandbox only (SG); nothing for HK | *Untested* on real cards | | R3 Recommendation ranking | Aggregate unit pass only | *Untested* at item level | | R4 Timeout 800 → 500 ms | None | *Untested* | | R5 14 defect fixes | Aggregate unit pass; gray crash rate rose | Per-fix verification *not confirmed* | | R6 Firebase SDK | None item-specific | *Untested* at item level | --- ## 3. Operational Readiness Checklist **Monitoring and alerting** - [x] Crash monitoring exists (Crashlytics, alert at 0.5% crash rate) — *Confirmed by supplied evidence*. Adequacy caveat: the threshold is more than 3× the previous version's 0.14% and more than 2× the gray release's 0.21%, so a regression of the size seen in gray would not trigger it. - [x] Order Tracking instrumentation live on the dashboard (R1) — *Confirmed*. - [ ] Alert on Order Tracking metrics (R1) — *not confirmed*. `<NEEDS-INPUT: order-tracking-switch-and-alerts>` - [ ] Payment success rate monitoring and alerting for Apple Pay HK/SG (R2), including a signal that can detect a 0.3 pp drop — *not confirmed*. `<NEEDS-INPUT: payment-success-monitoring>` - [ ] Alerting on the recommendation timeout change (R4) — *Confirmed absent*: no corresponding alert exists. `<NEEDS-INPUT: rec-timeout-alerting>` - [ ] Monitoring for the new ranking logic (R3) and for the Firebase upgrade (R6) — *not confirmed*. Also not stated: whether the upgrade touches Crashlytics components, which would affect the monitoring this audit relies on. `<NEEDS-INPUT: item-level-test-evidence>` **Runbook** - [ ] On-call runbook for 5.2.0 failure scenarios (payment degradation, crash spike, recommendation fallback) — *not confirmed*; none is mentioned. `<NEEDS-INPUT: rollback-drill-and-runbook>` **Rollback tooling** - [x] Halt phased rollout in the app stores + release a hotfix version — *Confirmed as the stated method*. Tested: *not confirmed*. - [x] Server-side switch to turn off the new Order Tracking page (R1) — existence *Confirmed*. Tested, and what users see when it is off: *not confirmed*. `<NEEDS-INPUT: order-tracking-switch-and-alerts>` - [ ] Switch for the Apple Pay regional expansion (R2) — *Confirmed absent*. `<NEEDS-INPUT: applepay-kill-switch>` - [ ] Revert path for the R4 config change — *not confirmed*; neither the revert procedure nor whether the config is scoped to 5.2.0 only was supplied. `<NEEDS-INPUT: rec-timeout-validation>` --- ## 4. Risk Assessment by Change Area | Change area | Blast radius | Reversibility | Evidence strength | Overall risk | |---|---|---|---|---| | **R1** Order Tracking page | Medium–High: every user who checks order status, with load peaking during the promotion; not a payment path | Medium: server-side switch exists but is not shown tested; fallback behavior unstated | Weak–Moderate: instrumentation live, regression signed off; 2 UI failures open | **Medium** | | **R2** Apple Pay HK/SG | High: revenue-critical payment path for HK/SG users; whether existing-region Apple Pay shares the changed code is not stated | Low: no switch; only halt rollout + hotfix, untested, and halting does not pull back devices already updated | Weak: SG sandbox only, nothing for HK, no payment success data | **High** | | **R3** Recommendation ranking | Medium–High: home page recommendation slots for all users; failure degrades relevance, not transactions | Unconfirmed (no remote control stated; treated as hotfix-only) | Weak: aggregate unit pass only | **Medium** (rises if it interacts badly with R4) | | **R4** Timeout 800 → 500 ms | Medium–High: all recommendation requests under this config; scope across app versions not stated; tighter timeouts bite hardest under promotion load | Medium: config revert likely quick but not shown; no alert exists to trigger it | None supplied; alert confirmed absent | **High** | | **R5** 14 defect fixes (2 crashes) | Medium: spread across unspecified areas | Low–Medium: binary change, hotfix only | Weak–Moderate: aggregate pass; no per-fix verification; gray crash rate rose | **Medium** | | **R6** Firebase SDK upgrade | Medium–High: SDK-level, app-wide; affected components not stated | Low: binary change, hotfix only | Weak: no item-level results; cannot be excluded as a crash-delta contributor | **Medium** | **Unattributed signal:** the gray-release crash increase (0.21% vs 0.14%) is held against the release as a whole and is not allocated to any single area until `<NEEDS-INPUT: crash-regression-attribution>` is resolved. **Interaction note:** R3 and R4 both change what the home page recommendation slots show, and the gray release did not isolate them from each other. Evidence for either would not carry over to the combination. --- ## 5. Go/No-Go Criteria Evaluation | # | Criterion (must be true for Go) | Area | Evidence | Result | |---|---|---|---|---| | G1 | No open P0/P1 defects | All | None open (severity of the 2 failing UI tests unstated) | **Pass** | | G2 | All automated unit tests pass | All | 2,310 / 2,310 | **Pass** | | G3 | Every UI failure is fixed or shown unrelated to scope | R1 | 2 order tracking empty-state failures unfixed; the 3 timeouts accepted as stated | **Fail** | | G4 | Payment path verified with real cards in every launch region | R2 | SG sandbox only; HK no evidence | **Fail** (SG) / **Unconfirmed** (HK) | | G5 | Release crash rate not worse than previous version, or regression explained | All (unattributed) | 0.21% vs 0.14%, no attribution | **Fail** | | G6 | Monitoring live that can detect a breach of the 0.3 pp payment success tolerance | R2 | Not mentioned in the inputs | **Unconfirmed** | | G7 | Alert exists on each changed server-side path | R4 | Confirmed no alert | **Fail** | | G8 | Monitoring live for the new Order Tracking page | R1 | Dashboard live; alert not confirmed | **Pass** (visibility) / **Unconfirmed** (alert) | | G9 | Payment change can be turned off without a store release, and that path is tested | R2 | No switch; hotfix only; untested | **Fail** | | G10 | Order Tracking switch tested | R1 | Exists, test not shown | **Unconfirmed** | | G11 | Timeout change validated (latency, fallback behavior under load) | R4 | None | **Unconfirmed** | | G12 | Item-level validation for ranking, SDK upgrade, and per-fix verification | R3, R5, R6 | Aggregate only | **Unconfirmed** | | G13 | On-call runbook exists | All | Not mentioned | **Unconfirmed** | | G14 | Rollout stage gates and halt criteria defined, and workable before the promotion | All | Stages stated; gates, dwell times, and timeline not supplied | **Unconfirmed** | **Verdict: No-Go for 5.2.0 as currently scoped.** This follows from the table: G3, G4, G5, G7, and G9 are failures established by supplied evidence, and the failures concentrate on **R2** (the payment path, rated High, with no switch, real-card verification not done, and no monitoring confirmed for the 0.3 pp tolerance) and **R4** (rated High, untested and unalerted). Under the stated risk tolerance, an unswitched, real-card-unverified payment change shipping into the promotion window does not meet the bar. The remaining Unconfirmed items would each prevent a clean Go on their own. **What would change the answer.** This release becomes **Go with conditions** when all of the following are met: 1. **R2:** Apple Pay HK/SG either (a) does not reach users in 5.2.0 (removed from the build, or shipped dark behind a server-side switch that is off at launch), or (b) has a tested server-side off-switch **and** real-card verification passed in both Singapore and Hong Kong. 2. **R1:** the 2 order tracking empty-state UI tests are fixed and passing, or the QA owner records an explicit severity and acceptance, and the R1 switch is drilled with its fallback confirmed. 3. **G5:** the crash-rate increase is attributed with event counts and either fixed, gated, or shown to be within baseline noise. 4. **R4:** either the timeout stays at 800 ms for this release (the config can ship separately after the promotion), or latency/fallback validation under promotion-level load is supplied together with an alert on timeout/fallback rate. 5. **Payment monitoring:** payment success rate is monitored per region against a baseline, with a written halt rule at a 0.3 pp drop, defined before the 1% stage begins. 6. **Rollout plan:** stage gates, dwell time per stage, and an estimate of hotfix lead time including store review, shown to fit before the promotion. A runbook exists. If these cannot be met before the promotion begins next week, the evidence supports holding R2 (and R4's config change) out of the promotion window rather than shipping them into it. Splitting the scope creates a new release candidate that requires its own audit; this audit does not certify the non-payment changes separately. --- ## 6. Rollback & Contingency Readiness **Stated rollback path:** halt the phased rollout in the app stores and release a hotfix version. - **Tested: not confirmed.** No rollback drill or hotfix rehearsal is mentioned in the inputs. A designed-but-untested path is not credited as ready. `<NEEDS-INPUT: rollback-drill-and-runbook>` - Halting a phased rollout stops further exposure; it does not revert devices that have already updated. Hotfix lead time, including store review, was not supplied. **Per change area:** - **R1:** server-side off-switch exists (confirmed); never shown tested, and the user-facing fallback when it is off is unstated. `<NEEDS-INPUT: order-tracking-switch-and-alerts>` - **R2:** **no switch.** The only path is halt plus hotfix, the slowest and least certain option, on the highest-cost path. `<NEEDS-INPUT: applepay-kill-switch>` - **R3, R5, R6:** binary changes; hotfix only unless a remote control exists (not stated). - **R4:** config revert path and its speed are not shown, and with no alert there is no trigger for using it. `<NEEDS-INPUT: rec-timeout-validation>` **Phased rollout as contingency.** The team's willingness to roll out at 1% → 10% → 50% → 100% is *confirmed as a stated intent*, but it is not yet a contingency plan: stage gates, dwell times, and halt triggers were not supplied, and it was not shown that these stages are supported by each store's rollout mechanism. It is also not established that HK/SG Apple Pay volume at 1% exposure is large enough to detect a 0.3 pp drop. `<NEEDS-INPUT: rollout-gates-and-timeline>` **Stakes weighting (stated explicitly).** RELEASE_RISK_TOLERANCE describes a payment-critical path shipping immediately before a major promotion, with a hard 0.3 pp tolerance. This audit therefore treats the untested rollback path as a **serious gap**, and treats R2's missing switch as a **blocking gap**. In a low-stakes internal tool the same facts would be an improvement item. --- ## 7. Open Questions & Data Requests Ranked by effect on the Section 5 verdict. | Rank | ID | Question | Owner | Affects | |---|---|---|---|---| | 1 | `applepay-real-card-verification` | Have real bank cards been tested end-to-end for Singapore and Hong Kong (not sandbox)? Results by region, issuer, and card type. | Payments lead / QA lead | G4; R2 | | 2 | `applepay-kill-switch` | Can Apple Pay HK/SG be disabled server-side, or kept dark at launch, without a store release? If not, will it be removed from 5.2.0? | Payments lead / mobile lead | G9; R2 | | 3 | `payment-success-monitoring` | What are the per-region payment success baselines, what alert detects a 0.3 pp drop, and who acts on it? Did the gray cohort include any HK/SG or Apple Pay transactions? | Payments lead / SRE | G6; R2 | | 4 | `crash-regression-attribution` | What produced the 0.21% vs 0.14% crash rate: event counts, metric definition (users or sessions), crash signatures, and which change area (R1/R3/R5/R6) is responsible? Are the 2 carried-over crash fixes verified? | Mobile lead / QA lead | G5; all areas | | 5 | `rec-timeout-validation` | What is the latency distribution against the new 500 ms limit at promotion-level load, what does the app show on timeout, is the config scoped to 5.2.0, and how is it reverted? | Recommendation service owner | G11; R4 | | 6 | `rec-timeout-alerting` | Can an alert be added on recommendation timeout and fallback rate before launch? | Recommendation service owner / SRE | G7; R4 | | 7 | `order-tracking-empty-state` | What is the severity of the 2 failing empty-state tests, and when will they be fixed or formally accepted? | QA lead / mobile lead | G3; R1 | | 8 | `order-tracking-switch-and-alerts` | Has the R1 switch been exercised, what do users see when it is off, and is an alert configured on the live dashboard metrics? | Mobile lead / SRE | G8, G10; R1 | | 9 | `rollback-drill-and-runbook` | Has a halt-and-hotfix drill been run, what is hotfix lead time including store review, and does an on-call runbook exist for 5.2.0 failure modes? | Release manager / on-call lead | G13; Section 6 | | 10 | `rollout-gates-and-timeline` | What are the gates, dwell times, and halt rules for each stage, do the 1/10/50/100 stages work on both stores, and does the schedule finish before the promotion starts? | Release manager | G14; Section 6 | | 11 | `item-level-test-evidence` | What test or metric evidence exists specifically for ranking (R3), the SDK upgrade (R6, including whether Crashlytics components changed), and each of the 14 fixes (R5)? What is the coverage target, and what is covered per feature? | QA lead / mobile lead | G12; R3, R5, R6 | | 12 | `open-defect-scope-mapping` | Which of the open P2 and 6 P3 defects, if any, are in this release's changed areas? | QA lead | Section 2 classification | | 13 | `ui-timeout-evidence` | How were the 3 UI timeouts confirmed as intermittent, and which flows do they cover? | QA lead | G3 (caveat) |