Practical framework / 5 MIN READ
Randomize the unit that can actually receive a different experience
Account for shared accounts, households and repeated visits before choosing an assignment method.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
The exposure unit, counting unit and analysis unit need not be identical, but their relationship must be explicit. A test can have thousands of actions while containing far fewer independent assignments.
What you will leave with: Map assignment, exposure and outcome aggregation so the experiment matches how customers interact.
Use this when
- Several contacts from the same company can receive campaign messages.
- A person returns on different sessions or devices.
This does not teach geographic experiment design or provide a cluster-adjusted statistical estimator.
Draw who can influence whom
Start with a simple interaction map. In a business purchase, an email recipient may share an offer with colleagues. In a household, a discount may influence a joint order. In a subscription product, multiple users may share account settings. These connections can carry the treatment beyond the person who first receives it.
Mark the plausible paths rather than assuming every exposure is isolated. Decide whether a mixed experience is acceptable, confusing or likely to contaminate the comparison. If messages can be forwarded, record that limitation even when no practical design can eliminate it completely.
Choose a stable assignment boundary
Microsoft describes organization-level assignment when users need a consistent shared experience. It also explains that dependence within organizations changes variance and power. The lesson for a marketing plan is to choose a boundary that matches the treatment, then ask for analysis that respects it.
A persistent account assignment may suit an account-wide offer; a session assignment may suit a truly transient interface question. Neither is universally better. Document what happens when someone logs in, changes devices or joins an existing account. Avoid changing the assignment merely because the person returned or because a preferred treatment has spare capacity.
Separate assignment from actual exposure
Keep a record of which eligible units were assigned and which could receive the treatment. A user may be assigned to an email variant but never open it. Comparing only openers can select different kinds of people in each arm, especially if the subject line itself changes opening behavior.
Agree with the analyst which effect is being estimated. A policy decision often concerns the effect of assigning the experience, including non-use. Any analysis restricted to exposure needs careful justification and a comparable definition in both arms. Do not quietly turn an assignment-based experiment into an engagement-based comparison.
Count information at the right level
Create a sample report showing assignments, exposed units and outcomes separately. If accounts are assigned, aggregate or model outcomes in a way that respects account-level dependence. Hundreds of employees within one account are not hundreds of independent account assignments.
Check concentration: one very large customer may dominate revenue or activity. Plan how unequal cluster sizes will be handled before results arrive. Ask whether the available number of independent units can support the decision. If it cannot, simplify the treatment or revise the question instead of presenting a precise-looking user-level calculation.
Make the reasoning concrete.
A hypothetical software vendor wants to test an account-wide onboarding offer across 80 customer organizations with many users. The offer can be shared internally, so assigning individual contacts would produce mixed experiences inside companies.
Assign contacts
Opening: Different colleagues receive different offers.
Hypothesis: This would provide many contact records.
Risk: Sharing and common account decisions undermine independence.
Assign organizations
Opening: All eligible contacts within a company share the assignment.
Hypothesis: This better matches the account-wide treatment.
Risk: Only 80 organizations are available, and their sizes differ.
Ask for an organization-level design and power assessment. Do not claim that the total number of employee records solves the small number of independent assignments.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Boundary
Who must receive a consistent experience?
Spillover
How could treatment reach another assigned unit?
Persistence
How does assignment survive return visits and identity changes?
Estimand
Is the decision about assignment, exposure or use?
Analysis
How will shared outcomes and unequal group sizes be handled?
Watch for the wrong conclusion.
- Assigning sessions while analyzing them as independent people.
- Filtering to openers after testing subject lines.
- Ignoring forwarded offers within an account.
- Treating many users in a few organizations as a large randomized sample.
Your next action
Draw the assignment-to-outcome map for one planned test and ask an analyst to identify every place independence could fail.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- Microsoft Research: Why Tenant-Randomized A/B Testing Is Challenging →Primary experimentation research guidance · Supports: Explains organization-level assignment and the consequences of within-organization dependence for variance and power. Specific Microsoft observations are not universal marketing effect estimates.