Practical framework / 5 MIN READ
Ask what improvement is worth detecting before asking how long to test
Connect sample planning to commercial value and a realistic volume of eligible participants.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
The smallest effect a test can detect and the smallest effect worth acting on are different quantities. Planning is useful when it exposes that gap before the team invests in a comparison it cannot interpret.
What you will leave with: Create an analyst-ready planning brief with a baseline, meaningful effect, uncertainty requirements and feasible exposure window.
Use this when
- Someone proposes testing for a convenient number of days.
- Traffic is limited and many small variants compete for exposure.
This lesson is not a sample-size calculator and does not prescribe a universal traffic threshold.
Translate a change into an operating decision
Estimate the cost of implementing and maintaining the proposed change. Consider production effort, staff capacity, margin and customer consequences. Then identify an outcome improvement that would justify those costs. Document assumptions as assumptions, especially when each extra lead has uncertain value.
Express a rate change in both percentage points and relative terms. Moving from 4% to 5% is one percentage point and a 25% relative increase. A brief that simply says a five-percent improvement is ambiguous. Use the counting unit and outcome window that will actually appear in the analysis.
Give the analyst the right inputs
NIST’s treatment of proportion testing shows why baseline, detectable change, significance and power matter for sample planning. Its one-process illustration is not a two-arm campaign calculator. Ask the analyst to use a method appropriate to the actual design, including repeated visitors, clusters and unequal allocation where relevant.
Provide a plausible baseline range rather than a falsely exact historical rate. A traffic shift or revised eligibility rule may make last quarter’s average unsuitable. Explain the tolerated risks of false alarms and missed useful changes. If those choices are defaults from a platform, record them rather than assuming everyone understands them.
Compare statistical demand with available exposure
Estimate eligible, assignable participants per week after exclusions. Do not use impressions if the experiment assigns people, or sessions if returning visitors remain in one arm. Account for ramp-up and the time needed for outcomes to mature. A test can finish recruitment before its evidence is ready.
If the required sample exceeds what is practical, change the question honestly. Consider testing a larger meaningful difference, fewer variants or a higher-frequency outcome whose limitations are clear. A qualitative review may resolve a comprehension problem more efficiently, but it cannot replace a quantitative lift estimate.
Preserve the plan when reality changes
Record the planned sample, duration, analysis method and stopping policy. Set an early operational checkpoint for delivery and measurement problems without declaring a winner. If traffic arrives more slowly than expected, revise the schedule prospectively and explain why.
Do not respond to an inconclusive result by lowering the worthwhile-effect threshold until the treatment looks useful. Separate a design limitation from a business conclusion. The final readout can say that the study rules out large improvements while leaving smaller worthwhile ones unresolved, if the interval actually supports that statement.
Make the reasoning concrete.
A hypothetical service sees roughly 4 qualified requests per 100 eligible visits. Implementation is worthwhile only if the improvement is material enough to cover extra follow-up work. The team proposes examining a change to 5 per 100, explicitly a one-percentage-point difference.
Feasible test
Opening: An analyst checks the required sample against eligible weekly traffic.
Hypothesis: A realistic schedule may support this specific effect question.
Risk: A calculator using independent sessions may overstate available information.
Infeasible test
Opening: Recruitment would outlast the stable campaign period.
Hypothesis: A different decision or larger treatment may be more useful.
Risk: Extending indefinitely can introduce changing business conditions.
No sample count is asserted here. Obtain a design-specific calculation, then decide whether the question is worth the time and exposure it requires.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Value
What improvement changes the implementation decision?
Scale
What are its absolute and relative forms?
Baseline
Which comparable historical period supports the planning range?
Design
What is assigned and what counts as an independent unit?
Feasibility
Can recruitment and outcome maturity finish under stable conditions?
Watch for the wrong conclusion.
- Treating a default two-week duration as a power calculation.
- Confusing percentage points with relative improvement.
- Planning on total traffic instead of eligible assigned units.
- Calling an underpowered null result proof of no effect.
Your next action
Send one planning brief to an analyst before producing more variants: baseline range, worthwhile change, assignment unit and available weekly volume.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- NIST/SEMATECH: Sample sizes required for testing proportions →Primary statistical handbook · Supports: Demonstrates dependence of sample planning on baseline proportion, detectable change, significance and power. Its displayed one-process formula is not a ready-made two-arm marketing calculator.