## 1. Mandate, Audience, and Definition of Done **Outcome.** This experiment informs one decision: whether reducing the number of screens in the 5-step onboarding wizard raises the share of new signups who complete it, without lowering the share who reach their first meaningful action. If the pre-declared primary metric clears the pre-declared minimum detectable effect and no guardrail is breached, the merged-screen flow becomes a ship candidate. Otherwise the 5-screen flow stays, and the backlog in Section 6 is revisited. The design does not identify which step is at fault. That is a funnel-analysis question resolved under `<VALIDATE: step-dropoff-funnel>`. **Stakeholders (by role; named individuals to be recorded before launch).** - Variant build and Optimizely configuration: engineer. - Analysis, sizing, and SRM checks: analyst (analysis owner). - Hypothesis, prioritization, and ship/no-ship call: growth PM (proposed; confirm before launch). - Sign-off on any change to the paymentinfo step: Finance. **Requirements** | ID | Requirement | Source | |---|---|---| | R1 | Test whether a shorter wizard improves onboarding outcomes on the activation lever. | Supplied (hypothesis, stated as a hunch) | | R2 | Establish which step contributes most to drop-off, since this is currently unknown. | Supplied (stated uncertainty) | | R3 | No change to the paymentinfo step may be tested without Finance sign-off. | Supplied (hard constraint) | | R4 | Run the test in the existing Optimizely setup rather than building new tooling. | Supplied (context) | | R5 | Size and time the test from the supplied baseline (3,000 signups/month; 22% full-wizard completion), reconfirmed against current analytics. | Supplied (context); currency is a gap | | R6 | Define "first meaningful action" and its relationship to wizard completion. | Gap revealed by inputs | | R7 | Pre-declare the primary metric, MDE, guardrails, and analysis method before launch. | Gap revealed by inputs | | R8 | Fix a stopping rule, given that no maximum test duration was supplied. | Gap revealed by inputs | **Explicit exclusions.** This design does not: - test any change to the paymentinfo step, pricing, or any regulated claim; - test a "skip"/deferral option, a progress indicator, or save-and-resume; - prove that wizard length is the cause of drop-off; - prove that wizard completion equals activation; - measure long-term retention. **Solution shape.** - *Statistically determined (analysis owner):* sample size, α, power, the analysis test, SRM thresholds, and the stopping rule. - *Product judgment (growth PM):* the minimum effect worth shipping, guardrail thresholds, and the ship/no-ship call. - *Finance judgment:* whether the variant touches the paymentinfo step. - *Rule-based (pre-declared in Section 5):* which screens get merged. **Completion criteria** - **Scope and hypothesis (R1, R2):** the hypothesis and the pair-selection rule are dated before the build starts, the per-step funnel pull is attached, and the selected pair is recorded. - **Constraint compliance (R3):** Finance's written answer is on file, and a deterministic diff shows the paymentinfo screen is identical across arms. - **Tooling and baseline (R4, R5):** the experiment is configured in Optimizely, the baseline and volume are reconfirmed in current analytics, and sizing is recomputed from them. - **Metrics (R6, R7):** the activation event is defined and verified to fire, and the MDE and guardrail thresholds are ratified in writing. - **Run integrity (R8):** the pre-declared sample size and duration are both reached with no early stop, every Section 9 check is recorded, and every Section 12 action used is approved. - **Decision:** the ship/no-ship call is recorded using only the pre-declared analysis. A designed experiment alone is NOT completion (→ Section 13). ## 2. Experimenter Instructions ``` Build and run this experiment against real users and real data, not a simulation of them. Check current analytics and any prior experiment on this lever first. Reuse an existing test framework instead of building a new one for this single test. Resolve every <VALIDATE> item using current analytics or a real data pull — not internal guesswork. Implement the actual variant and actual tracking; do not stop at a written design. Run the pre-launch checks defined in Section 9 before allocating any real traffic. Once live, do not stop the test early because of an interesting-looking early trend; report a concrete blocker when progress needs a decision, resourcing, or access outside your authority. Do not report a result before the pre-declared sample size or duration is reached, invent a p-value or confidence interval, or silently change the primary metric after seeing partial data. Continue safe, low-risk setup work while a launch-critical validation is pending. Allocate real traffic to the experiment or ship the winning variant only within approved authority. Report progress using actual traffic and analysis data, and clearly separate designed, launched, and concluded status. Operational defaults (which experimentation platform, how variants are named internally) may be chosen and documented autonomously. Do not autonomously invent baseline metrics or claim an analysis was run when it was not. ``` In particular: do not assume which step drives the drop-off from the hunch that the wizard is too long — the per-step drop-off distribution is `<VALIDATE: step-dropoff-funnel>`, a validation item, not a design default. ## 3. Baseline and Assumption Ledger **3.1 Claims about current user behavior at this point in the funnel** | ID | Claim | Source | Confidence | Consequence if wrong | |---|---|---|---|---| | A1 | 22% of signups complete the full wizard. | Supplied | Medium (measurement window and completion definition not stated) | The primary-metric baseline and sample size shift. | | A2 | The wizard has 5 steps, one of which is the paymentinfo step. | Supplied | High | Pair selection and the compliance check change. | | A3 | No change to the paymentinfo step may be tested without Finance sign-off. | Supplied (hard constraint) | High | Immovable. Violating it invalidates the launch. | | A4 | Drop-off is concentrated in an identifiable step or adjacent steps rather than spread evenly. | Unverified — `<VALIDATE: step-dropoff-funnel>` | Low | The pair-selection rule has no principled target. | | A5 | Wizard completion is associated with reaching the first meaningful action. | Unverified — `<VALIDATE: activation-definition>` | Low | The primary metric may not track the lever. | | A6 | The 22% figure is current and stable week to week. | Unverified — `<VALIDATE: baseline-completion-current>` | Medium | Sizing and the power calculation are off. | **3.2 Claims about available traffic and time to a decision** | ID | Claim | Source | Confidence | Consequence if wrong | |---|---|---|---|---| | A7 | About 3,000 new signups arrive per month. | Supplied | Medium (currency not stated) | Duration to reach sample size changes. | | A8 | Volume is current, roughly steady across weeks, and all signups enter the wizard. | Unverified — `<VALIDATE: signup-volume-current>` | Medium | The 6-week plan may under- or over-run. | | A9 | Optimizely is already set up for testing. | Supplied | High | The build sequence would need a tooling step. | | A10 | A +4 percentage point absolute lift in completion (22% to 26%) is the smallest change worth shipping. | Unverified — `<VALIDATE: mde-and-guardrail-thresholds>` | Low (proposed) | The test may be over- or under-powered for the real decision. | | A11 | No other in-flight experiment touches the signup or onboarding flow. | Unverified — `<VALIDATE: inflight-experiments>` | Low | Interaction effects contaminate both tests. | | A12 | The signup population contains no vulnerable or specially regulated group. | Unverified — `<VALIDATE: population-profile>` | Low | Extra review is triggered (Section 10). | **3.3 Claims about why the problem is believed to exist** | ID | Claim | Source | Confidence | Consequence if wrong | |---|---|---|---|---| | A13 | The requester believes the 5-step wizard is too long. | Supplied (hunch, not established fact) | Low | The test reads null, and the cause is elsewhere (a specific step's friction). | | A14 | The step causing drop-off is not currently known. | Supplied | High | The funnel pull may be unnecessary. | | A15 | Merging two non-payment screens without altering the paymentinfo screen or its context does not count as a "change to the paymentinfo step." | Unverified — `<VALIDATE: finance-signoff-scope>` | Low | The variant would breach the hard constraint. | | A16 | Per-transition abandonment (not per-field friction) explains a meaningful share of drop-off, so fewer screens should help. | Unverified — `<VALIDATE: step-dropoff-funnel>` | Low | The mechanism is wrong, and a null result is the likely outcome. | **Most invalidating assumption.** A5 is the assumption whose failure would most invalidate the design. The only supplied baseline is wizard completion, and the stated lever is activation. If completing the wizard is not associated with reaching a first meaningful action, a lift in completion would be a hollow win. Section 9's first checks must probe A5 through `<VALIDATE: activation-definition>` before any traffic is allocated. ## 4. Experiment Design and Hypothesis ``` HYPOTHESIS (PROVISIONAL — core assumptions A4, A5, A16 rest on unresolved <VALIDATE: step-dropoff-funnel> and <VALIDATE: activation-definition>) If the adjacent pair of non-paymentinfo wizard screens containing the highest step-level drop-off is merged into a single screen (5 screens -> 4, no field removed, no requiredness changed), then full-wizard completion rate will increase, because fewer screen transitions reduce abandonment at each transition (traces to A1, A4, A16). CONTROL Current 5-screen wizard, unchanged. VARIANT A Identical to control except two adjacent non-paymentinfo screens render as one screen. Same fields, same validation, same required/optional status, same order of fields. Pair-selection rule (pre-declared): candidate pairs exclude the paymentinfo step; choose the adjacent non-payment pair containing the single highest step-level drop-off from the funnel pull; ties go to the pair with the higher combined drop-off. If the highest drop-off step is paymentinfo, it is logged as a finding, routed to Finance separately, and NOT tested here. The paymentinfo screen's content, fields, position relative to other required steps, and progress-indicator text must be identical across arms unless Finance's written answer (<VALIDATE: finance-signoff-scope>) says otherwise. PRIMARY METRIC Wizard completion rate = share of eligible new signups submitting all required inputs through the final wizard state, using the same completion definition and window as the 22% baseline (A1; definition to be confirmed under <VALIDATE: baseline-completion-current>). Current baseline: 22% (Supplied). GUARDRAIL METRICS G1 First-meaningful-action rate per eligible signup, same window in both arms (definition and window: <VALIDATE: activation-definition>). Breach: variant lower than control by more than 10% relative (proposed; ratified under <VALIDATE: mde-and-guardrail-thresholds>). G2 Paymentinfo-step completion rate among users who reach that step (observed only; the step is not modified). Breach: variant lower than control by more than 10% relative (same ratification). G3 Validation-error rate on the merged screen versus the control pair (diagnostic only; no stopping threshold). MINIMUM DETECTABLE EFFECT +4 percentage points absolute (22% -> 26%), proposed. Reasoning: a smaller MDE is impractical at the supplied traffic. A +2 pp MDE needs roughly 6,950 signups per arm (about 4.6 months at 3,000/month), while +4 pp needs roughly 1,786 per arm (about 5-6 weeks). Whether +4 pp is worth shipping is a business judgment: <VALIDATE: mde-and-guardrail-thresholds>. ANALYSIS (pre-declared) Two-sided two-proportion test, alpha 0.05, power 0.80, on the primary metric only, run once at the stopping point. Guardrails are used to block or flag a ship decision, never to declare a win. ``` **Evidence rule.** This design is a draft for review and sizing, not an approved test. Launch is gated by Section 12, not by finishing this section. ## 5. Segment and Variant Contracts **Contract C0: Control** - **Name:** `onboarding-5screen-control` - **Segment definition:** all eligible new signups entering the wizard during the test window. Eligible means new accounts only, excluding the segments listed below. - **Allocation percentage:** 50% of eligible traffic (ramped per Section 12: initial soak at 10% of eligible traffic, then 50% at full allocation). - **What differs from control:** nothing (this is the reference arm). - **Owner:** engineer. **Contract V1: Variant A** - **Name:** `onboarding-4screen-merged` - **Segment definition:** same as C0. - **Allocation percentage:** 50% of eligible traffic (initial soak at 10%, then 50%). - **What differs from control:** the selected adjacent non-payment screen pair renders as one screen. The pair is set by the Section 4 rule after `<VALIDATE: step-dropoff-funnel>` resolves. - **Owner:** engineer (build), analyst (assignment and metric wiring). **Assignment.** Sticky, randomized at the user or visitor level in Optimizely at signup start, with 50/50 split between arms. Users assigned during the soak phase are kept in the final analysis only if the variant, tracking, and assignment configuration were not changed after they were bucketed. Otherwise their data is discarded. **Deliberately excluded segments (Principle E)** - Existing or returning users re-entering onboarding. Their behavior differs and they are not the target of the activation lever. - Internal, test, and duplicate accounts, which would distort the baseline. - Users enrolled in any other in-flight experiment touching signup or onboarding (mutual exclusion, pending `<VALIDATE: inflight-experiments>`). - Any population flagged by `<VALIDATE: population-profile>` as vulnerable or specially regulated, until Section 10 review is complete. - No device-, channel-, or geography-specific variants are tested. Subgroup cuts, if reported, are labeled exploratory and never used for the ship decision. **Deliberately excluded variant ideas.** These are excluded because they are not supported by the supplied inputs or would breach R3, and each is logged in Section 6: - a skippable-step or deferral variant (it redefines "completion" and would contaminate the primary metric); - a progress indicator or time-estimate variant; - any change to the paymentinfo step. **Internal vs. external claim separation.** Users in the experiment may see only the merged screen's normal UI. Variant names, experiment keys, hypothesis, MDE, guardrail thresholds, and arm allocation stay internal. No user-facing copy may claim a duration or benefit (for example, "takes one minute") unless it was separately verified, and any copy touching payments or a regulated claim follows Section 10. ## 6. Prioritization Architecture and Backlog Map Scoring framework: Impact / Confidence / Ease, each 1–5, summed to a total out of 15. Scores are proposals for growth PM ratification. | # | Idea | Impact | Confidence | Ease | Total | Status | |---|---|---|---|---|---|---| | 1 | Merge the highest-drop-off adjacent non-payment screen pair (5 → 4 screens) | 4 (directly tests the length hunch) | 2 (hunch unverified; A4, A16 open) | 4 (Optimizely available, front-end change) | 10 | **Selected** | | 2 | Add "Skip for now" to a non-essential step and defer it | 4 (removes work from the critical path) | 2 (same unverified basis) | 3 (needs a deferral flow) | 9 | Deprioritized: redefines "completion", contaminating the primary metric; reconsider once activation is the primary metric. | | 3 | Add a progress indicator with step count or time estimate | 2 (may not change abandonment) | 2 (no supporting data) | 4 (small UI change) | 8 | Deprioritized: touches every screen including paymentinfo, so it likely triggers Finance review; time claims need verification. | | 4 | Change or defer the paymentinfo step | 3 (unknown until the funnel is pulled) | 1 (no evidence) | 1 (blocked by hard constraint R3) | 5 | Deprioritized: not testable without Finance sign-off. | **Selected for this cycle:** merge the highest-drop-off adjacent non-payment screen pair. It best satisfies R1 (test whether a shorter wizard helps) while staying inside R3, since the paymentinfo step is left untouched. **Backlog map** | Idea | Score | Revisit owner | |---|---|---| | Skippable/deferred step | 9 | Growth PM | | Progress indicator or time estimate | 8 | Growth PM | | Paymentinfo step change | 5 | Growth PM with Finance | **Not used for this lever right now.** These are deliberately excluded: - pricing or plan-packaging tests; - copy-only or visual-polish tests on individual screens; - multivariate combinations of the above. Any variant not scoped in Section 5 is ruled out for this test. ## 7. Validation and Data Requirements **`<VALIDATE: step-dropoff-funnel>`** (launch-critical) - *What must be established:* the names, order, and per-step entry and completion counts of the 5 wizard steps over a recent representative window, including which step is paymentinfo and whether step-level events already exist. - *How to establish it:* pull real current data / confirm with the data team. - *Dependent tasks/actions:* pair selection (T7), variant build (T8), and Section 12 action A-01. - *Status until resolved:* no stand-in. The pair to merge is undefined, and launch must not proceed until resolved. **`<VALIDATE: activation-definition>`** (launch-critical) - *What must be established:* the precise definition of "first meaningful action," its current rate and time window among new signups, and whether wizard completers reach it at a different rate than non-completers. - *How to establish it:* pull real current data / confirm with the data team. - *Dependent tasks/actions:* guardrail G1, the primary-metric relevance check (A5), and action A-01. - *Status until resolved:* wizard completion stands in as the only outcome metric, and G1 is uncomputable. Launch must not proceed until resolved. **`<VALIDATE: finance-signoff-scope>`** (launch-critical) - *What must be established:* whether Finance considers the variant a "change to the paymentinfo step." Specifically, whether a changed step count, progress-indicator text, or screen position around an unmodified paymentinfo screen counts as a change, and whether sign-off is granted where it does. - *How to establish it:* pull real current data / confirm with the data team, by routing the written scope question to Finance through the data team. - *Dependent tasks/actions:* the T9 compliance check, variant contract V1, and action A-01. - *Status until resolved:* any visible difference to the paymentinfo screen or its context is treated as requiring sign-off. Launch must not proceed until resolved. **`<VALIDATE: mde-and-guardrail-thresholds>`** (launch-critical) - *What must be established:* the smallest completion lift worth shipping, the guardrail breach thresholds (G1, G2), α, power, and the stopping rule, all ratified before launch. - *How to establish it:* pull real current data / confirm with the data team (analysis owner, with growth PM ratification). - *Dependent tasks/actions:* the sizing calculation, Section 9 readiness rows, and action A-01. - *Status until resolved:* stand-ins are +4 pp MDE, 10% relative guardrail thresholds, α 0.05, and power 0.80. Launch must not proceed until resolved. **`<VALIDATE: baseline-completion-current>`** (routine) - *What must be established:* that the 22% completion figure reflects a recent window, states its completion definition and attribution window, and is stable week to week. - *How to establish it:* check current analytics before finalizing. - *Dependent tasks/actions:* the Section 9 sizing note and the primary-metric definition. **`<VALIDATE: signup-volume-current>`** (routine) - *What must be established:* that about 3,000 signups/month is current, roughly steady weekly, and that all signups enter the wizard. - *How to establish it:* check current analytics before finalizing. - *Dependent tasks/actions:* duration planning and the stopping rule. **`<VALIDATE: inflight-experiments>`** (routine) - *What must be established:* that no in-flight Optimizely experiment touches the signup or onboarding flow, and the mutual-exclusion setup if one does. - *How to establish it:* check current analytics before finalizing. - *Dependent tasks/actions:* the Section 10 conflict check and action A-01. **`<VALIDATE: population-profile>`** (routine) - *What must be established:* whether the signup population includes minors or other groups needing extra review, and in which jurisdictions. - *How to establish it:* check current analytics before finalizing. - *Dependent tasks/actions:* the Section 10 population review. **Existing-data-first.** Existing dashboards and past onboarding experiments on this lever are checked before commissioning any new data pull. Any baseline or effect-size figure quoted is either sourced (Supplied or a cited current pull) or flagged with a `<VALIDATE>` tag, and none is invented for plausibility. ## 8. Build Sequence and Requirement Traceability 1. **T1: Pull the step-level funnel and confirm baseline and volume.** Reuse existing dashboards first. Resolves `<VALIDATE: step-dropoff-funnel>`, `<VALIDATE: baseline-completion-current>`, and `<VALIDATE: signup-volume-current>`. IDs: R2, R5, A1, A4, A6, A7, A8. Depends on: none. 2. **T2: Define the first meaningful action and pull its baseline.** Resolves `<VALIDATE: activation-definition>`. IDs: R6, A5. Depends on: none. 3. **T3: Send Finance the written scope question and record the answer or sign-off.** Resolves `<VALIDATE: finance-signoff-scope>`. IDs: R3, A3, A15. Depends on: T1 (to name the screens involved). 4. **T4: Ratify MDE, guardrail thresholds, α, power, and stopping rule in writing.** Resolves `<VALIDATE: mde-and-guardrail-thresholds>`. IDs: R7, R8, A10. Depends on: T1, T2. 5. **T5: Audit Optimizely for in-flight experiments and profile the signup population.** Resolves `<VALIDATE: inflight-experiments>` and `<VALIDATE: population-profile>`. IDs: A11, A12. Depends on: none. 6. **T6: Set up instrumentation.** Reuse existing event names where they exist; the fragment below is the minimum. IDs: R4, R6, A9. Depends on: T1, T2. 7. **T7: Apply the pair-selection rule and record the pair in contract V1.** IDs: R1, R2, A4. Depends on: T1, T3. 8. **T8: Build Variant A in Optimizely per Section 5.** IDs: R1, R4, A9. Depends on: T7. 9. **T9: Run a deterministic compliance check on the hard constraint.** Diff the rendered paymentinfo screen (content, fields, position, progress-indicator text, event schema) between arms, and confirm the diff is empty or covered by Finance's written answer. IDs: R3, A3, A15. Depends on: T3, T8. 10. **T10: Run the Section 9 pre-launch checks** (design-review, instrumentation-check, SRM plumbing). IDs: R5, R7, R8. Depends on: T4, T6, T8, T9. 11. **T11: Final pre-allocation task: report blockers on every unresolved item.** Lists each unresolved `<VALIDATE>` item by name and status, and blocks T12 while any launch-critical item remains unresolved. IDs: all. Depends on: T1–T10. 12. **T12: Allocate initial traffic (soak)** once every launch-critical item is resolved and Section 12 approves A-01. IDs: R1. Depends on: T11. 13. **T13: Run the soak-phase instrumentation and SRM checks, then expand allocation** under A-02. IDs: R5, R8. Depends on: T12. 14. **T14: Run to the stopping rule and produce the Section 13 report.** IDs: all. Depends on: T13. ``` event: wizard_screen_viewed {user_id, experiment_key, variant_key, screen_id, screen_index, ts} event: wizard_screen_submitted {user_id, experiment_key, variant_key, screen_id, screen_index, validation_error_count, ts} event: wizard_completed {user_id, experiment_key, variant_key, ts} event: first_meaningful_action {user_id, experiment_key, variant_key, action_type, ts} // defined by <VALIDATE: activation-definition> ``` **Traceability** | Requirement / Assumption | Variant / Metric | Owner | Acceptance Check | |---|---|---|---| | R1 | V1 / primary metric | Growth PM | Hypothesis and pair rule dated before the T8 build | | R2 | Pair selection | Analyst | Step-level funnel report (T1) attached to contract V1 | | R3 | V1 paymentinfo screen | Engineer | T9 diff output plus Finance's written answer | | R4 | Optimizely experiment | Engineer | Optimizely experiment config shows both arms and 50/50 split | | R5 | Sizing | Analyst | Sizing recomputed from confirmed baseline and volume | | R6 | G1 | Analyst | `first_meaningful_action` event verified firing in both arms | | R7 | Primary metric, G1, G2 | Analyst, growth PM | Written ratification on file | | R8 | Stopping rule | Analyst | Stopping rule recorded in Section 9 before launch | | A1 | Primary baseline | Analyst | Baseline pull with definition and window | | A2 | Contract V1 | Engineer | Screen inventory from T1 | | A3 | Paymentinfo screen | Engineer | T9 compliance check | | A4 | Pair selection | Analyst | Per-step drop-off report | | A5 | G1, primary relevance | Analyst | Completer vs. non-completer activation comparison | | A6 | Primary baseline | Analyst | Weekly completion series from analytics | | A7 | Duration plan | Analyst | Monthly signup series from analytics | | A8 | Duration plan | Analyst | Weekly signup and wizard-entry counts | | A9 | Optimizely | Engineer | Optimizely project access confirmed | | A10 | MDE | Growth PM | Ratified MDE on file | | A11 | Exclusions | Engineer | Optimizely experiment audit | | A12 | Section 10 review | Analyst | Population profile from analytics | | A13 | Hypothesis | Growth PM | Recorded as provisional in Section 4 | | A14 | Pair selection | Analyst | Funnel pull (T1) | | A15 | Paymentinfo screen | Engineer | Finance written answer plus T9 diff | | A16 | Primary metric | Analyst | Per-transition drop-off report | ## 9. Statistical Readiness and Launch Checks The sample size and duration below are estimates built on the baseline in Section 3. They are confirmed only once real data collection is underway. The estimate uses the standard two-proportion normal approximation at α 0.05 and power 0.80 with a 22% baseline and +4 pp MDE. That gives roughly 1,786 eligible signups per arm (about 3,572 total). At about 3,000 signups/month (roughly 100/day), 6 full weeks yields about 4,150 eligible signups. **Sizing note (UNCONFIRMED).** The sample-size estimate is UNCONFIRMED until the real baseline is checked. It uses the supplied 22% completion rate and 3,000/month volume as stand-ins, pending `<VALIDATE: baseline-completion-current>` and `<VALIDATE: signup-volume-current>`. **Stopping rule (fixed, pre-declared).** The test ends at the later of two conditions: 6 full weeks (42 days) elapsed and at least 1,786 eligible signups in each arm. No stop occurs on the basis of an interim trend. | Assumption / Req | Check level | Method | Expected result | Pass condition | Prerequisites | |---|---|---|---|---|---| | R1, A13 | design-review | Confirm the hypothesis and pair rule were written and dated before the variant build. | Dated document precedes the T8 build. | Timestamp order verified. | None | | R2, A4, A14, A16 | design-review | Review the step-level funnel pull and the recorded pair selection. | Pair selected by the pre-declared rule. | Pair matches rule output, and paymentinfo is excluded. | T1 | | R3, A3, A15 | design-review | Confirm Finance's written scope answer. | Answer on file. | Answer on file and consistent with the built variant. | T3 | | R3, A3 | instrumentation-check | T9 diff of the rendered paymentinfo screen and event schema between arms. | Empty diff (or diff covered by written sign-off). | Diff empty or covered. | T8 | | R4, A9 | instrumentation-check | Verify the experiment config in Optimizely. | Two arms, 50/50, sticky bucketing. | Config matches contracts C0, V1. | T8 | | R5, A1, A6, A7, A8 | design-review | Recompute sizing from confirmed baseline and volume. | Confirmed or revised n per arm and duration. | Analysis owner signs the recomputed plan. | T1 | | R6, A5 | design-review | Confirm the activation definition and its completer vs. non-completer relationship. | Definition on file. | Definition documented and relationship shown. | T2 | | R6 | instrumentation-check | Verify `first_meaningful_action` fires in both arms in a staging or soak sample. | Event fires for a known test action. | Event present with correct `variant_key`. | T6 | | R7, A10 | design-review | Confirm ratified MDE, thresholds, α, power. | Written ratification. | Ratification dated before launch. | T4 | | R8 | design-review | Confirm the fixed stopping rule is written. | Rule recorded. | Rule matches the text above. | T4 | | A11 | design-review | Optimizely audit for conflicting experiments. | No overlap, or a mutual-exclusion group in place. | Audit output on file. | T5 | | A12 | design-review | Review the population profile. | Profile on file. | Any vulnerable group triggers Section 10 review. | T5 | | Instrumentation QA (all metrics) | instrumentation-check | Walk test users through each arm and check every event in the schema. | Events fire once per action with correct keys. | Zero missing or duplicated events in the QA sample. | T6, T8 | | Assignment balance | sample-ratio-check | Chi-square test on arm assignment counts against 50/50, at the end of soak, each full week, and final. | Counts consistent with 50/50. | p ≥ 0.001 at each check (soak checks are low-powered and read as plumbing checks). | T12 | | Primary analysis | full-launch | Two-sided two-proportion test on the primary metric at the stopping point. | NOT RUN until Section 12 approvals granted. | Runs once, at the stopping point, on the pre-declared metric. | A-01, A-02 | | Guardrails G1, G2 | full-launch | Compare arms at scheduled checkpoints and at the stopping point. | NOT RUN until Section 12 approvals granted. | No breach of the ratified thresholds. | A-01, A-02 | | Final SRM | full-launch | Assignment-count check on the final dataset. | NOT RUN until Section 12 approvals granted. | p ≥ 0.001. | A-01, A-02 | **What each check proves.** An instrumentation-check PASS proves only that tracking fires correctly, meaning events are present, correctly keyed, and not duplicated. It says nothing about whether the merged-screen variant works. Only the completed run, at the pre-declared sample size and duration, can show whether the effect is real. ## 10. Ethics, Privacy, and Compliance Boundaries - **[HIGH — OPEN] Launching with unresolved launch-critical items.** Four items are launch-critical: `<VALIDATE: step-dropoff-funnel>`, `<VALIDATE: activation-definition>`, `<VALIDATE: finance-signoff-scope>`, and `<VALIDATE: mde-and-guardrail-thresholds>`. **No-launch rule:** no traffic is allocated while any of them is unresolved. - **[HIGH — OPEN] Materially worse experience for the tested segment.** Merging screens could raise per-screen friction and lower activation. The guardrail is G1 (first-meaningful-action rate): a breach means the variant is lower than control by more than 10% relative at any scheduled checkpoint (proposed, pending ratification). A breach halts new allocation and triggers human review; it does not declare a result. G2 (paymentinfo-step completion among those who reach it) uses the same 10% relative threshold. - **[MEDIUM — OPEN] Data boundary.** The experiment may collect and use only data the product already collects under the existing privacy terms, for randomization and outcome measurement. No new personal fields, no external or sensitive attributes, and no targeting beyond random assignment. Confirmation that Optimizely's event capture falls within the current privacy notice is required before launch. - **[HIGH — RULE] Must never be tested without explicit sign-off.** This covers any change to the paymentinfo step (Finance), any pricing or plan change, and any change affecting a regulated claim or payment-related statement. No user-facing time or benefit claim is added without separate review. - **[MEDIUM — OPEN] Conflicts with in-flight experiments.** Caught before launch by the Optimizely audit (T5) and, if overlap exists, a mutual-exclusion group. Tracked as `<VALIDATE: inflight-experiments>`. - **[MEDIUM — OPEN] Vulnerable or protected population.** Not determinable from the supplied inputs (`<VALIDATE: population-profile>`). If the signup population includes minors or another specially regulated group, launch pauses for compliance review of the variant and the data collected. **Risk unassessable until launch-critical items resolve.** The risk that merging screens lowers the first-meaningful-action rate stays unassessable until `<VALIDATE: activation-definition>` and `<VALIDATE: step-dropoff-funnel>` resolve. Neither the activation baseline nor the merged pair is known before then. ## 11. Run, Monitoring, and Stopping Plan **Build and QA (runnable now).** Instrumentation setup (T6), the Optimizely experiment shell and reuse of existing test scaffolding (T8, minus pair selection), the routine audits (T5), and the written Finance question (T3) can proceed. **Blocked on validation:** pair selection and the final variant build (T7, T8), guardrail G1 wiring, the T9 compliance check, and all traffic allocation. **Approved traffic allocation.** See Section 12. Initial soak (A-01): BLOCKED BY VALIDATION. Expansion (A-02): blocked on A-01. Ship (A-03): blocked on A-02. Early stop (A-04): blocked on A-01. **Monitoring checkpoints during the run** - *Instrumentation health:* checked daily against the schema in Section 8. - *Sample-ratio checks:* at the end of the soak (day 3), then at the end of each full week, and at the final analysis. - *Guardrails (G1, G2, G3):* reviewed at each weekly checkpoint as a safety review. These are not result reads, and no result is reported before the stopping rule is met. **Approved conclusion.** The test may conclude only when all of the following hold (AND): - the Section 9 checks passed at their stated levels; - all Section 12 actions used are complete and approved; - the pre-declared sample size and duration are both reached, with no early stop. **Bounded interim monitoring** - *Concrete low-risk check:* each day, confirm that `wizard_screen_submitted` and `wizard_completed` events fire in both arms and that every completion has the preceding screen events. A gap is an instrumentation defect to fix, not a result. - *Exclusion verification:* weekly, query assignment logs to verify that no excluded segment (internal or test accounts, returning users, users in mutually exclusive experiments) was accidentally enrolled. **Stopping.** Before assuming a stop halts allocation, confirm the actual current allocation state in Optimizely. Stopping the test does not undo a decision already made on partial data. Containment means halting new allocation and flagging the early stop as an early stop, and partial data is never treated as final. ## 12. Launch and Rollout Approval Register | Action ID | Purpose | Segment / Allocation | Expected Effect | Trigger Condition | Status | Approved by / Date | |---|---|---|---|---|---|---| | A-01 | Allocate initial traffic (soak) | Eligible new signups; 20% of eligible traffic split 10% control / 10% variant, for 3 days | Live traffic through both arms; tracking and SRM plumbing verified | Section 9 pre-launch checks pass; all launch-critical items resolved; all routine items confirmed | BLOCKED BY VALIDATION: `<VALIDATE: step-dropoff-funnel>`, `<VALIDATE: activation-definition>`, `<VALIDATE: finance-signoff-scope>`, `<VALIDATE: mde-and-guardrail-thresholds>` | ____ / ____ | | A-02 | Expand traffic allocation | 100% of eligible traffic, split 50/50 | Sample accrues toward 1,786 per arm and 6 full weeks | Soak instrumentation-check and SRM checks pass; no guardrail breach | Blocked on prior action A-01 | ____ / ____ | | A-03 | Ship the winning variant to 100% | All new signups | Merged-screen flow becomes the default | Stopping rule met; primary metric clears the MDE; no guardrail breach; Finance sign-off if any paymentinfo context changes | Blocked on prior action A-02 | ____ / ____ | | A-04 | Stop the test early | Current allocation | New allocation halted; early stop flagged | Guardrail breach or a data-integrity defect (never an interesting-looking trend) | Blocked on prior action A-01 (nothing running to stop) | ____ / ____ | Approving one action never implies approval of another. Listing a stop action (A-04) does not mean it is to be executed now. ## 13. Completion Evidence and Handoff A final report must contain: - **Design as run.** The actual variant built (including the merged pair and its screen names) and the actual traffic allocation used at each phase. - **Check results.** Every Section 9 check itemized as PASS, FAIL, NOT RUN, or BLOCKED, with one line of evidence each. - **Interim vs. final results.** Reported separately, with an explicit rule that an interim trend is never presented as a concluded result. - **Sample size and duration.** The actual sample size and duration achieved, only if the test actually ran. Otherwise the report states "NOT YET LAUNCHED." - **Section 7 status.** Every unresolved item listed by name with its status: - `<VALIDATE: step-dropoff-funnel>` - `<VALIDATE: activation-definition>` - `<VALIDATE: finance-signoff-scope>` - `<VALIDATE: mde-and-guardrail-thresholds>` - `<VALIDATE: baseline-completion-current>` - `<VALIDATE: signup-volume-current>` - `<VALIDATE: inflight-experiments>` - `<VALIDATE: population-profile>` - **Residual limitations.** Any limitation still true at handoff (for example, G1 not fully monitored because the activation window was shorter than the run, or a sizing estimate still on unconfirmed baselines). - **Next steps.** The exact next steps to continue any outstanding validation or approval item, naming the owner role for each. Fabricating statistical results, confidence intervals, or approvals is prohibited. A well-designed experiment alone is not evidence of the outcome until it actually runs to completion.