Causal evidence review workshop / 6 MIN READ
Ask what changes at the eligibility cutoff before calling it an experiment
Assess whether a real eligibility threshold supports a credible local comparison and what the resulting estimate can describe.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-07
A threshold can create a useful evaluation opportunity, but customers just above and below it are not automatically comparable. The decision rule, manipulation risk and other changes at the same boundary determine whether the design is believable.
What you will leave with: A regression-discontinuity feasibility brief with the running variable, implementation evidence, competing threshold changes and the population to which the result applies.
Use this when
- A program grants a benefit or service when an existing score crosses a fixed threshold.
- An analyst proposes using the eligibility boundary to estimate the effect without randomized assignment.
This is design review with a qualified causal analyst, not an estimation recipe, permission to manipulate eligibility or a claim that the local result applies to every customer.
Reconstruct the actual assignment rule
Identify the score or measure used to determine eligibility, its exact cutoff and when it was recorded. Preserve the original value rather than only an eligible flag. Check whether staff overrides, appeals or later recalculations changed who received the offer. A policy document and the implemented allocation may differ in ways that matter to the analysis.
The World Bank’s DIME handbook describes regression discontinuity as comparing observations close to an eligibility boundary, with particular attention to manipulation, bandwidth choices and local interpretation. Those principles support investigating the rule, not treating any visible jump on a chart as a causal effect.
Ask whether the boundary separates otherwise credible comparisons
Look for reasons customers or staff could precisely move a record across the cutoff. A discretionary score updated after someone sees the benefit is different from a fixed historical measure. Ask the analyst to examine the score distribution and predetermined characteristics, alongside implementation records. No single diagnostic test proves that all relevant assumptions hold.
List everything else that changes at the same threshold. If a loyalty tier simultaneously changes discounts, support access and shipping terms, the boundary may identify the combined package rather than one preferred component. Avoid naming the estimate after the easiest benefit to market. Check for missing or differently measured outcomes near the cutoff before interpreting a discontinuity in the reported metric.
Plan the local analysis before selecting a persuasive result
Ask the analyst to justify the range around the cutoff, the form of the score-outcome relationship and the uncertainty method. A wider range supplies more observations but may make comparability harder; an extremely narrow range can leave little useful information. These choices need a defensible analysis plan and sensitivity checks, not a search for the window with the smallest reported probability value.
Distinguish a sharp rule from one that changes the chance of receiving the benefit. Imperfect take-up or exceptions may require a different design and assumptions, rather than dropping inconvenient records. Preserve eligibility, actual receipt and outcome as separate fields. If the design is not credible, document that conclusion and consider another evaluation approach instead of presenting a polished regression as a substitute for identification.
Keep the conclusion attached to the boundary
Report the actual comparison, time period and customer context represented near the cutoff. Do not extrapolate automatically to very low- or high-scoring customers, a new threshold or a different benefit. Explain whether the result concerns eligibility, receipt under additional assumptions or a package of changes.
Show the relevant plots, sensitivity results and unresolved implementation questions with the estimate. A useful decision might be whether to investigate a nearby expansion or run a randomized test, not immediate rollout to the entire customer base. Preserve the original scoring and allocation records so another analyst can reconstruct the study. The goal is a bounded causal question supported by an actual rule, not a before-and-after story wearing a technical label.
Make the reasoning concrete.
A fictional membership program grants a premium tier above a historical activity score. The tier includes both free shipping and priority support, but the marketing team wants to call any outcome difference the effect of free shipping alone.
Name the result after the preferred benefit
Opening: Compare the two sides and attribute the entire difference to shipping.
Hypothesis: The finding has an attractive commercial story.
Risk: Other simultaneous tier changes and implementation exceptions remain unexplained.
Review the actual discontinuity
Opening: Document the full tier package, score integrity and local comparison with an analyst.
Hypothesis: The claim matches the variation the design can identify.
Risk: A separate design may be needed to isolate shipping.
The team evaluates the tier boundary as implemented and labels any supported result locally. It does not change customer scores or infer the effect of one component from the combined package.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Rule
What exact pre-existing measure and cutoff determine eligibility?
Manipulation
Who could alter which side a record occupies?
Co-change
What else changes at the same threshold?
Analysis
How are the local window and sensitivity checks justified?
Scope
Which customers and intervention does the estimate actually describe?
Watch for the wrong conclusion.
- Discarding implementation exceptions without understanding them.
- Selecting the local window after looking for a favorable result.
- Generalizing a cutoff estimate to the whole customer population.
Your next action
Document one proposed threshold study’s actual assignment rule and all benefits that change at the same boundary.
Sources & scope
Workflows, worksheets and fictional examples are original editorial analysis. Named research guidance supports identified method principles. This is design-review preparation for a qualified analyst, not an estimation recipe or a claim of observed business results.
- World Bank DIME Analytics Data Handbook: Research design for impact evaluation →Primary research-method guidance · Supports: Regression-discontinuity eligibility boundaries, manipulation concerns, local windows and sensitivity checks. The commercial feasibility workflow is original analysis.