Practical framework / 5 MIN READ
Test whether two changes work differently together
Plan the joint comparison before combining separate winners.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
A new headline and a new offer can each help on their own yet work poorly together, or only help as a pair. A factorial design makes the combined conditions part of the experiment so the team can examine the interaction rather than assuming effects simply add up.
What you will leave with: Create a two-factor design brief with four valid combinations, an interaction question and a deployment decision.
Use this when
- Two related marketing changes will be shipped together.
- Separate tests suggest improvements but do not show how the changes interact.
This is a design brief for analyst review, not permission to test unsafe combinations or a universal traffic requirement.
Name the factors and the joint question
Define two controllable changes with clear versions. For example, factor A changes the explanation format and factor B changes a permitted incentive presentation. Confirm that all combinations remain truthful, accessible and operationally valid. If a combination cannot be delivered, the intended full design is unavailable.
Write the interaction question in customer terms: does the explanation’s effect depend on the incentive presentation? This differs from merely keeping unrelated simultaneous experiments from colliding. The joint effect is itself part of what the team intends to learn and use in its rollout decision.
Build the four-condition assignment plan
NIST defines a full factorial design as including every combination of the factors’ settings. With two factors at two settings each, the plan includes current/current, new/current, current/new and new/new. The number of conditions says nothing by itself about sufficient sample size.
Ask the analyst to plan assignment, exposure, sample requirements and inference for the main and interaction questions. Preserve a consistent eligible population and measurement window across cells. Test implementation so a visitor assigned to the joint condition actually receives both intended changes, without accidental extra differences.
Calculate the interaction in interpretable units
For an additive outcome scale, compare the effect of A when B is present with the effect of A when B is absent. Their difference is the interaction contrast on that scale. A percentage-point interaction can differ from one defined on a relative or other modeled scale; specify the estimand before analysis.
Show cell-level counts, outcome estimates and appropriate uncertainty. Do not conclude that an interaction exists simply because one cell comparison is significant and another is not. Directly estimate the planned contrast with the analyst’s method, while accounting for the set of questions the experiment is intended to answer.
Choose a deployable combination
Review the combined condition directly against the current experience as well as the interaction question. An interaction can be informative without making the joint treatment the best commercial choice. Consider customer protections, implementation cost and whether a simpler condition delivers acceptable value.
Avoid searching every segment and combination for a lucky winner. If the interaction is too uncertain, say so and choose a justified next step rather than claiming independence. Record which factors, levels and population were tested; findings about these exact versions do not establish a universal rule for all headlines and incentives.
Make the reasoning concrete.
A hypothetical four-cell test has outcome rates of 10% for current/current, 12% for A only, 13% for B only and 14% for both. A adds two points without B but one point with B, giving an additive interaction contrast of minus one point. No uncertainty is supplied in this illustration.
Add separate gains
Opening: Forecast 15% for the combined version from two plus three points.
Hypothesis: Separate changes appear beneficial.
Risk: The assumed additive forecast ignores the observed joint condition.
Estimate the joint design
Opening: Review the 14% combined cell and the minus-one-point interaction with valid uncertainty.
Hypothesis: The decision considers both joint performance and dependence.
Risk: Point estimates alone do not establish a reliable interaction.
Do not call the interaction proven from illustrative rates. Have the analyst evaluate the planned contrasts, then choose among valid deployable combinations using value and guardrails.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Factors
What exactly changes at each level?
Conditions
Can every combination be delivered safely and consistently?
Estimand
On which scale is interaction defined?
Inference
Which contrasts and uncertainty methods are planned?
Deployment
Which combination is worthwhile and supportable?
Watch for the wrong conclusion.
- Assuming separate winners must add together.
- Treating four conditions as four sufficient observations.
- Inferring interaction from significance in only one subgroup.
- Changing the interaction scale after seeing the result.
Your next action
Draft the four-condition matrix and the interaction contrast, then obtain design review before launching the test.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- NIST/SEMATECH: Full factorial designs →Primary statistical handbook · Supports: Defines a full factorial design as including every combination of factor settings. The marketing interaction arithmetic and operating worksheet are original illustrations, not a sample-size prescription.