Practical framework / 5 MIN READ
Keep treatment-created behavior out of the eligibility filter
Check whether a subgroup comparison preserves the original randomized question.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
A campaign test may change who clicks, starts a form or completes an update. Comparing only those people afterward can replace the randomized groups with selected groups whose differences were partly created by the treatment itself.
What you will leave with: Build an analysis-population audit that distinguishes assignment, exposure and post-treatment behavior.
Use this when
- A winning result appears only after filtering to engaged users.
- The treatment changes whether someone enters the reported funnel.
This does not prohibit all subgroup analysis; it requires a defensible design and qualified review for causal subgroup claims.
Draw the journey in time order
Write the sequence from eligibility and random assignment through exposure, clicks, intermediate actions and final outcomes. Mark which variables existed before assignment and which the treatment could influence. A field recorded later may still describe a pre-existing trait, but its availability can also be affected by treatment.
State the main estimand in ordinary language: the effect of assigning this experience to eligible visitors. Keep that population visible when stakeholders ask to remove people who did not engage. Their lack of engagement may be part of the treatment’s effect rather than irrelevant noise.
Identify the selection mechanism
Microsoft’s build-testing research shows how filtering users by update or restart behavior can produce selection bias when treatment changes that behavior. The lesson transfers as a warning to marketing analysis: a behavior-based filter needs scrutiny before it is treated as a fair causal comparison.
Ask whether the same people would enter the subgroup under either version. An appealing button may attract casual visitors who would never click the old button. Conversion among clickers could then fall even while purchases per assigned visitor rise. The selected populations answer a different question from the original randomization.
Separate descriptive diagnostics from effect claims
Report funnel behavior to explain what happened, but label conditional rates clearly. Purchases per clicker can diagnose journey composition; it does not automatically estimate the treatment’s effect on a fixed set of would-be clickers. Do not multiply selected subgroup differences into a population effect without a valid method.
For a narrower causal question, ask the analyst whether pre-treatment eligibility, a properly designed trigger or another defensible estimand is available. A trigger must not simply be a convenient action observed more often under treatment. Do not retrofit a complex adjustment to rescue a preferred result.
Preserve the planned population and disclose changes
Compare the planned analysis set with the actual extract. Audit exclusions for missing outcomes, fraud rules and logging failures, especially when the treatment could affect exclusion. Record any necessary correction and show how it changes interpretation.
Keep operational safety exclusions separate from statistical convenience. The team can stop harmful exposure while still reporting the experiment honestly. In the readout, distinguish assignment-based results, descriptive funnel measures and any specialist subgroup estimates. Give each its own denominator and limitations so a persuasive conditional chart cannot silently replace the original decision question.
Make the reasoning concrete.
A hypothetical control assigns 1,000 visitors, with 100 clickers and 20 purchases. Treatment assigns 1,000, with 200 clickers and 30 purchases. Purchases per assigned visitor rise from 2% to 3%, while purchases per clicker fall from 20% to 15%. These are illustrative point estimates without uncertainty.
Judge only clickers
Opening: Reject treatment because clicker conversion fell.
Hypothesis: The conditional funnel rate is lower.
Risk: Treatment changed who clicked, so the groups need not be comparable.
Preserve assignment and diagnose
Opening: Review purchases per assigned visitor with uncertainty, then inspect the clicker mix.
Hypothesis: The main comparison retains the randomized population.
Risk: A rollout still needs valid inference and guardrail review.
Use the planned assignment-based analysis for the main decision. Treat clicker conversion as a descriptive clue unless a specialist establishes a valid narrower causal question.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Timeline
Which fields can treatment influence?
Population
What assigned group defines the main effect?
Filter
Would subgroup membership be the same under either version?
Diagnostics
Which conditional rates are descriptive only?
Disclosure
What exclusions or analysis changes occurred?
Watch for the wrong conclusion.
- Dropping non-clickers because they seem unexposed.
- Assuming equal subgroup sizes restore randomization.
- Treating post-treatment missingness as harmless.
- Replacing the planned outcome with a favorable conditional rate.
Your next action
Audit one experiment’s filters and label every treatment-influenced condition before accepting its subgroup claims.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- Microsoft Research: A/A’/B Testing, Evaluating Microsoft Teams across Build Releases →Primary experimentation research · Supports: Shows how filtering on update or restart behavior can create selection bias when behavior differs by treatment. The marketing click example is original and does not borrow reported Microsoft performance results.