### 🔍 SECTION 1 — Operationalized Research Question **Operationalized research question (measurable restatement):** Among new sign-ups to a B2B SaaS product who are assigned to one of two onboarding variants, **does assignment to a shorter onboarding flow (3 steps) versus a longer onboarding flow (7 steps)** produce a **different 30-day activation rate**, where activation is defined as completion of a pre-specified product “activation event” within **30 days** of sign-up? **Independent Variable (IV): Onboarding flow length (manipulated exposure / assignment)** * **Construct:** Onboarding “length” operationalized as **number of required steps/screens** to complete initial onboarding. * **Levels:** * **Short** = 3-step onboarding * **Long** = 7-step onboarding * **Measurement/encoding:** Binary indicator (0 = 7-step; 1 = 3-step). * **Implementation note:** Analyze primarily by **intention-to-treat (ITT)** using random (or pseudo-random) assignment; optionally complement with a **treatment-on-the-treated** analysis using completion as adherence. **Dependent Variable (DV): 30-day activation** * **Construct:** User activation within 30 days of sign-up. * **Operational definition (recommended):** A binary event: **Activated = 1** if the user completes the activation event within 30 days of sign-up; **0** otherwise. * Activation event should be **pre-registered and behaviorally anchored** (e.g., “created first project AND invited ≥1 teammate”). * **Measurement:** Event-tracking logs; compute outcome within a fixed window beginning at sign-up timestamp. **Target population** * **Primary population:** **Newly registered accounts/users** for the B2B SaaS product during the study period. * **Unit of analysis (must be specified):** * Often **account/workspace** is more meaningful than individual user in B2B contexts (to avoid within-account interference). If individual-level is used, explicitly handle clustering. **Moderators and controls to acknowledge** * **Key moderators (theory-aligned):** * **Baseline user intent / motivation** (proxy: acquisition channel, trial vs demo-request, pricing page visits) — Fogg “Motivation.” * **Ability constraints** (proxy: user role, company size, prior product familiarity, tech sophistication) — Fogg “Ability,” Cognitive Load. * **Onboarding context** (mobile vs desktop; region/language; time-to-first-session) — impacts load and friction. * **Core controls (quasi-experimental hygiene):** * **Acquisition channel**, **plan type**, **industry**, **company size**, **user role**, **product version**, **time period** (week fixed effects), **device type**. * If assignment is not perfectly randomized, include **propensity score / covariate adjustment**. **Ambiguities to resolve before analysis (to protect hypothesis clarity)** * What exactly constitutes **“activation”** (single event vs composite)? * Is the study unit **user** or **account/workspace** (and how will interference be prevented)? * Is onboarding **mandatory** or can users skip/partial-complete (adherence definition)? --- ### 📌 SECTION 2 — Hypothesis Set Let ( p_{3} ) be the probability of 30-day activation for users assigned to the **3-step** onboarding and ( p_{7} ) for users assigned to the **7-step** onboarding. **H₀ — Null Hypothesis:** ( H_{0}: p_{3} = p_{7} ) There is **no statistically significant difference** in the 30-day activation rate between new sign-ups assigned to the 3-step onboarding flow and those assigned to the 7-step onboarding flow. **H₁ (Two-Tailed) — Non-Directional Alternative Hypothesis:** ( H_{1}: p_{3} \neq p_{7} ) There is a **statistically significant difference** in the 30-day activation rate between new sign-ups assigned to the 3-step onboarding flow and those assigned to the 7-step onboarding flow. **H₁ (One-Tailed) — Directional Alternative Hypothesis:** ( H_{1}: p_{3} > p_{7} ) New sign-ups assigned to the 3-step onboarding flow have a **statistically significantly higher** 30-day activation rate than those assigned to the 7-step onboarding flow. **Justification for one-tailed directionality (one sentence):** Cognitive Load Theory predicts that reducing extraneous processing demands during onboarding increases task completion, and the Fogg Behavior Model predicts that reducing friction increases the likelihood of the target behavior when motivation and prompts are present—together implying **shorter onboarding should increase activation**. --- ### 🧠 SECTION 3 — Theoretical Framework Linkage **Cognitive Load Theory (CLT) → mechanism (mediator):** A longer onboarding flow plausibly increases **extraneous cognitive load** (more screens, more decisions, more working-memory demands), which can elevate error rates, increase perceived effort, and raise abandonment probability before users reach the activation state. The hypothesized causal path is: **Onboarding length (3 vs 7 steps) → extraneous cognitive load / perceived effort (mediator) → onboarding completion & early product use → 30-day activation.** Even if activation is measured at 30 days, the mechanism can operate early through reduced drop-off and faster progression into meaningful product use. **Fogg Behavior Model (FBM) → mechanism (direct + moderated):** FBM posits behavior occurs when **Motivation × Ability × Prompt** converge. A shorter onboarding increases **Ability** (i.e., reduces friction) at the moment of prompt (the onboarding sequence itself), making activation-related actions more likely. This also implies moderation: the benefit of shortening onboarding should be **largest for low-motivation or low-ability segments** (e.g., low-intent channels, less experienced users), where friction is more behavior-limiting. **What the hypotheses are testing conceptually:** Primarily a **causal effect of onboarding length on activation** under an experimental assignment logic (even if labeled quasi-experimental). Secondary theory-driven extensions (optional) include: * **Mediation test:** onboarding length → perceived cognitive load → activation * **Moderation test:** onboarding length effect stronger among low-intent/low-ability users **Suggested citations — verify for current editions:** * Sweller, J. (1988). *Cognitive load during problem solving: Effects on learning.* Cognitive Science, 12(2), 257–285. * Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). *Cognitive architecture and instructional design.* Educational Psychology Review, 10(3), 251–296. * Fogg, B. J. (2009). *A behavior model for persuasive design.* Proceedings of the 4th International Conference on Persuasive Technology (Persuasive ’09). --- ### 📊 SECTION 4 — Recommended Statistical Tests | # | Test Name | When to Use | Key Assumptions | Software (e.g., SPSS, R, Python) | | - | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | 1 | **Two-proportion z-test** (difference in activation rates) | Primary A/B comparison when DV is binary and assignment is independent | Independent observations; sufficiently large (n) for normal approximation (expected successes/failures adequate); correct windowing | R, Python, SPSS | | 2 | **Logistic regression (GLM, binomial)** with covariates | When you want adjusted estimates, handle imbalance, add moderators, or include fixed effects | Correct model specification; independence (or clustered SEs if not); no severe multicollinearity; adequate events per parameter | R, Python, SPSS, Stata | | 3 | **Chi-square test of independence** (2×2) | Simple unadjusted association test between onboarding variant and activation | Expected cell counts not too small; independence | R, Python, SPSS | | 4 | **Mixed-effects logistic regression** or **GEE logistic** | If users are clustered (e.g., within accounts/companies) or repeated measures exist | Correct correlation structure; sufficient clusters; model convergence for mixed effects | R, Python, Stata | | 5 | **Survival analysis** (Kaplan–Meier + Cox model) for time-to-activation | If timing matters (not just 30-day binary), and you want to use all time-to-event info | Independent censoring; proportional hazards for Cox; correct event definition | R, Python, Stata | **Primary recommendation:** Use **logistic regression (binomial GLM)** as the primary analysis because it naturally matches a **binary DV (activated by 30 days)**, supports the **directional contrast** (via coefficient sign and one-tailed test if pre-specified), and allows **covariate adjustment and moderator testing**—especially important if the design is quasi-experimental in practice or if there is clustering by account. --- ### ⚙️ SECTION 5 — Operationalization & Assumptions Notes 1. **Measurement scales** * IV (onboarding length condition): **Nominal (binary)** — 3-step vs 7-step assignment. * DV (30-day activation): **Nominal (binary)** — activated vs not activated within 30 days. * Optional mediator (perceived cognitive load): **Ordinal/interval** depending on instrument (e.g., NASA-TLX often treated as interval with caution). * Key covariates (channel, plan type): **Nominal**; company size: **ordinal/ratio** depending on capture. 2. **Minimum sample size guidance (practical, not a substitute for power analysis)** * Plan for a power analysis using expected baseline activation and minimum detectable effect (MDE). * Rule-of-thumb: ensure enough events for stable estimation. For logistic regression, target **≥10–20 activation events per parameter** (including moderators and fixed effects). If activation is rare, you’ll need substantially larger (n). * For a plain two-proportion comparison, you typically need **hundreds to thousands per arm** in product contexts unless the effect is large; compute explicitly from baseline rate and MDE. 3. **Data collection alignment with quasi-experimental A/B** * Confirm **assignment integrity** (randomization, bucketing, exposure logging) and analyze **ITT** (assigned variant) as primary. * Predefine the **30-day window** (start time = sign-up timestamp; end time = +30×24 hours) and ensure consistent time zones and attribution. 4. **Key threats to internal validity (specific to this context)** * **Selection bias** if assignment is not truly random (e.g., rollout by segment, device, region). * **Instrumentation / tracking drift** (activation event logged differently across variants or over time). * **Interference/contamination** (users switching devices/flows, shared accounts seeing mixed experiences). * **History effects** (product changes, campaigns) coinciding with the test; mitigate via time fixed effects. 5. **Operational pitfalls** * Define whether “activation” is achievable **without completing onboarding**; if yes, clarify whether activation is logically downstream of onboarding steps. * Handle **multiple sign-ups** from the same account/domain (dedupe rules; choose user vs account as the analytic unit). 6. **Ethical considerations** * Ensure transparency in user data handling (privacy policy alignment), minimize undue friction that could harm user outcomes, and avoid manipulative prompts. In B2B, also consider contractual obligations around experimentation and data processing. --- ### ✅ SECTION 6 — Quick-Reference Summary Card ┌─────────────────────────────────────────────────────┐ │ HYPOTHESIS FRAMEWORK SUMMARY │ ├─────────────────────────────────────────────────────┤ │ Research Question: │ │ Among new sign-ups, does assignment to 3-step vs │ │ 7-step onboarding change 30-day activation rates? │ │ │ │ IV: Onboarding length (3-step vs 7-step assignment) │ │ DV: 30-day activation (activation event within 30d) │ │ │ │ H₀: p₃ = p₇ (no statistically significant difference)│ │ │ │ H₁ (two-tailed): p₃ ≠ p₇ (significant difference) │ │ │ │ H₁ (one-tailed): p₃ > p₇ (3-step yields higher rate)│ │ │ │ Primary Statistical Test: Logistic regression (GLM) │ │ Theoretical Anchor: Cognitive Load Theory; FBM │ └─────────────────────────────────────────────────────┘