Practical framework / 5 MIN READ
Check who went missing before reading the experiment winner
Compare observed assignment counts with the intended allocation, then trace discrepancies through the data pipeline.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
A treatment can appear better because its unsuccessful users disappeared from the report. Sample-ratio checks help identify a class of problems that should be investigated before interpreting an outcome difference.
What you will leave with: Build a practical investigation path from assignment to exposure, collection, filtering and analysis.
Use this when
- A planned equal split produces unexpectedly unequal analyzed groups.
- An experiment result changes sharply after a reporting filter is added.
This is a diagnostic workflow, not a claim that equal group sizes prove randomization or measurement quality.
Use the allocation that was actually planned
Write down the configured allocation, its effective dates and the unit being counted. An experiment deliberately assigned 70% of eligible accounts to one version should not be tested against an equal split. If allocation changed during a controlled ramp, preserve the relevant stages instead of guessing one average ratio.
Count unique assigned units before looking at outcomes. Chance produces some imbalance, so a small numerical difference is not automatically a fault. Use an appropriate statistical check and investigation threshold agreed with the experiment owner. Record the tested population and whether the check applies to all assignments or a filtered subset.
Treat a warning as a data problem to explain
Microsoft describes sample-ratio mismatch as an unexpected difference between configured and observed group proportions. Its research illustrates how filtering can remove users unevenly and reverse interpretation. A warning therefore calls for investigation rather than a quick reweighting exercise.
Trace counts stage by stage: eligible, assigned, served, recorded, joined and retained in analysis. Compare losses across arms. Check whether an error, delayed upload, bot rule or eligibility filter could be affected by the treatment. The location where the ratio first changes is often more useful than the final mismatch alone.
Inspect the mechanism without fishing for a winner
Break the diagnostic counts down by a few operational dimensions such as browser, release version and day. These slices help locate a technical fault; they are not an invitation to search for a favorable business subgroup. Keep the business result secondary until the counting problem is understood.
For example, a new page might complete purchases while failing to send assignment metadata on one browser. Removing that browser from both arms after seeing results changes the population and needs an explicit rationale. A correction should restore a defensible measurement process, not simply make the warning disappear.
Decide whether recovery is possible
If complete assignment and outcome records exist, an analyst may be able to reconstruct the intended population. Document the repair, rerun integrity checks and distinguish corrected from original results. If missing outcomes cannot be recovered reliably, a fresh test may be the honest next step.
Do not force equal sample sizes by randomly discarding the larger arm unless a valid, preplanned method calls for it. Equal counts cannot repair selective missingness. Preserve the incident record, the affected period and the prevention check. That record makes the next test safer even when the current test cannot answer its business question.
Make the reasoning concrete.
A hypothetical equal-allocation landing-page test records 5,000 assignments in each arm, but the final report retains 4,900 control accounts and 4,200 treatment accounts. The scenario is invented to illustrate a diagnostic path, not an observed platform defect.
Compare only the final rates
Opening: Ignore losses and declare the higher conversion rate a winner.
Hypothesis: The difference might look commercially attractive.
Risk: Selective missingness could create or reverse that difference.
Trace the missing rows
Opening: Compare assignment, delivery and join logs by arm.
Hypothesis: The first divergence may reveal a measurement failure.
Risk: Finding a cause does not guarantee lost outcomes can be recovered.
Withhold the winner claim. Establish whether outcomes can be reconstructed under the original assignment rule; otherwise repair the pipeline and rerun.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Expected allocation
What ratio and unit were configured at each stage?
Observed counts
Where does the population first depart from expectation?
Differential loss
Which filters or failures can affect one arm more?
Recovery
Are original assignments and outcomes complete enough to repair?
Prevention
What automated check will catch the fault next time?
Watch for the wrong conclusion.
- Expecting every valid equal-allocation test to have exactly equal counts.
- Checking only people who converted.
- Discarding rows merely to make groups equal.
- Treating a passed ratio check as proof that every metric is trustworthy.
Your next action
Add an assignment-to-analysis count reconciliation to the next experiment review before presenting outcome results.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing →Primary experimentation research guidance · Supports: Defines sample-ratio mismatch and shows how missing or filtered users can undermine a randomized comparison. Passing an SRM check alone does not establish validity.