Practical framework / 5 MIN READ
Count the chances you gave yourself to find a winner
Variants, outcomes and audience slices all create opportunities for chance findings.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
An isolated impressive result becomes less surprising when it was selected from many comparisons. A useful readout makes the search process visible instead of presenting the selected result as the only question asked.
What you will leave with: Define the decision family, separate exploration from confirmation and plan how promising findings will be checked.
Use this when
- A test includes several variants and many success measures.
- The overall result is weak but one discovered audience slice looks exceptional.
This lesson does not prescribe one correction for every study or discourage exploratory analysis.
Inventory the comparisons before launch
List the variants being compared, the decision outcomes and the subgroup claims that might lead to action. Include comparisons against control and any planned comparisons among treatments. A team that intends to pick the best of several versions is making a different decision from a team evaluating one specified version.
Write down which questions belong to the main decision and which are explanatory. Not every diagnostic chart requires the same inferential treatment, but a chart promoted into a winner claim becomes part of the evidence story. Share the inventory with the analyst before results arrive.
Choose an error-control plan that fits the claim
NIST describes Bonferroni as one way to maintain simultaneous coverage across a specified finite set of comparisons. It is an available method, not a universal best choice. The analyst should select a method that matches the decision family, dependence among measures and the cost of false discoveries.
A smaller set of well-chosen primary questions can be more useful than a sprawling test. If the team needs many variants, budget the additional sample and analysis complexity. Do not correct only the comparisons retained in the final slide while ignoring the broader search that selected them.
Keep exploratory findings useful but provisional
A surprising subgroup can reveal a valuable next hypothesis. Describe how it was found, the subgroup size, the estimated difference and its uncertainty. Avoid statements that one group benefited while another did not merely because one individual test crossed a threshold and the other did not.
Ask whether the difference between groups itself is supported. Check whether the subgroup was defined by a characteristic that the treatment could change; such post-treatment grouping can distort interpretation. Plan a fresh comparison or other suitable confirmation before making a consequential personalization decision.
Report the selection process with the result
A credible summary names the planned primary result and then labels additional discoveries. If a selected variant is advanced, explain whether the displayed estimate comes from the same data used to choose it or from new validation data. The most favorable observed estimate among many candidates may exaggerate the durable benefit.
Preserve the losing comparisons in the analysis record. This helps the next team avoid repeating weak ideas and prevents selective success stories. Decide in advance what evidence is enough to act, what merits another test and what should remain an interesting observation.
Make the reasoning concrete.
A hypothetical team tests four messages against a control and inspects ten audience slices. One message looks strong among returning tablet visitors, a small group discovered after the overall comparison disappointed.
Promote the discovered slice
Opening: Announce that tablet visitors prefer this message.
Hypothesis: A real interaction might exist.
Risk: The claim was selected from a large unplanned search.
Register a follow-up question
Opening: Test the audience interaction with a suitable design.
Hypothesis: Independent confirmation can assess whether the pattern persists.
Risk: The smaller subgroup may require substantial recruitment.
Keep the subgroup pattern as a hypothesis. Report the original decision result and the breadth of exploration instead of advertising an unqualified winner.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Family
Which comparisons can trigger the same rollout decision?
Primary questions
Which outcomes and subgroups were planned?
Method
How will the analyst control the relevant error risk?
Discovery log
What else was inspected before selecting this result?
Confirmation
What new evidence would justify a subgroup claim?
Watch for the wrong conclusion.
- Applying a correction only to favorable reported comparisons.
- Calling a subgroup difference real because only one subgroup is significant.
- Defining subgroups using behavior changed by treatment.
- Hiding an exploratory search behind a single clean chart.
Your next action
Count every route by which the current experiment could produce a winner, then mark which claims need confirmation.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- NIST/SEMATECH: Bonferroni’s method →Primary statistical handbook · Supports: Explains simultaneous inference for a specified finite set of comparisons. The lesson does not claim Bonferroni is the best method for every experiment.