Practical framework / 5 MIN READ
Test whether a cheaper option preserves enough value
Define acceptable degradation before asking whether savings justify a change.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
A lower-cost production or delivery approach may be worthwhile even if it is not better on the main outcome. The relevant question can be whether performance remains above a justified minimum, rather than whether a conventional superiority test fails to find a difference.
What you will leave with: Draft a non-inferiority brief with a defensible margin, cost evidence and explicit decision boundaries.
Use this when
- A simpler creative workflow could reduce cost but might slightly reduce effectiveness.
- A team calls two options equivalent because a superiority test was inconclusive.
This lesson is for ordinary marketing choices, not medical treatment decisions or a universal statistical protocol.
Define what must be preserved
Choose the customer or business outcome that the alternative must protect. Specify its direction, population and measurement window. For a higher-is-better outcome, express the largest acceptable decline in meaningful units, such as percentage points of valid completion rather than an ambiguous percent.
Justify the margin from customer consequences, economic value and operational requirements before seeing test results. A convenient margin chosen to make the test pass defeats the purpose. Some harms, including accessibility failures or misleading claims, may be unacceptable regardless of a favorable average outcome or production savings.
Ask the correct statistical question
Daniel Lakens describes non-inferiority as assessing whether an effect exceeds a lower bound, and distinguishes this from merely failing to detect a difference. A nonsignificant superiority result can still be compatible with commercially unacceptable harm.
Ask a qualified analyst to specify the design, margin, uncertainty procedure, sample planning and analysis population. The appropriate method depends on outcome type and operational context. Do not borrow a confidence level, sample-size rule or stopping convention from a different test and assume it establishes the required claim.
Compare the bound with the agreed margin
For a higher-is-better metric expressed as alternative minus current, examine whether the method’s relevant lower confidence bound stays above the negative margin. If it crosses that boundary, the evidence does not establish the proposed non-inferiority claim, even when the point estimate looks harmless.
Review measurement quality, implementation fidelity and follow-up completeness. Similar failures in both groups can make weak alternatives look reassuringly similar. Keep the planned analysis intact and discuss departures with the analyst. A statistically acceptable result still applies only to the population, conditions and outcome actually tested.
Evaluate savings and remaining risks separately
Verify the lower cost using a consistent scope: production effort, licensing, ongoing maintenance, customer support and any new failure handling. Avoid celebrating a saving that merely shifts work to another team. Record uncertainty in the cost estimate as well as in performance.
Use a decision table with three outcomes: acceptable preservation plus credible savings, unacceptable harm, or unresolved evidence. A bounded pilot may be appropriate when operational risks remain, but it should not be mislabeled a successful non-inferiority test. Set a review plan and preserve a practical rollback path.
Make the reasoning concrete.
A hypothetical team pre-approves a maximum one-percentage-point decline in valid completion, subject to analyst review. One analysis estimates minus 0.2 points with a relevant lower bound of minus 1.4; another would have a lower bound of minus 0.7. The first does not exclude excessive decline; the second clears the stated performance boundary only if its design is valid.
Use no significant difference
Opening: Adopt the cheaper option because superiority was not demonstrated.
Hypothesis: The observed estimates look close.
Risk: Uncertainty may still include unacceptable harm.
Use the agreed boundary
Opening: Compare the valid lower bound with minus one point, then review costs and guardrails.
Hypothesis: The decision addresses acceptable preservation explicitly.
Risk: Passing the boundary is not proof of exact equality or universal safety.
Treat the first illustrative result as unresolved for non-inferiority. Consider the second only alongside credible savings, valid methods and all required guardrails.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Outcome
What performance and population must be protected?
Margin
Why is this maximum decline acceptable?
Design
Who approves inference, sample planning and stopping?
Savings
Which full-scope costs actually change?
Decision
What leads to adoption, rejection or an inconclusive result?
Watch for the wrong conclusion.
- Equating nonsignificance with equivalence.
- Choosing the margin after seeing the interval.
- Confusing percent with percentage points.
- Trading away critical customer protections for savings.
Your next action
Write a justified degradation margin and ask the responsible analyst to design the test before collecting decision data.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- Daniel Lakens: Equivalence Testing and Interval Hypotheses →Primary author-published methodological text · Supports: Explains non-inferiority against a lower effect bound and why a nonsignificant difference does not establish negligible harm. The business margin and testing protocol require separate justification.