Practical framework / 5 MIN READ
Review the counterfactual behind a before-and-after comparison
Use a comparison group to ask a sharper question, then examine why it might still fail.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
When a campaign starts in one market, sales may rise for reasons unrelated to the campaign. Difference-in-differences compares that change with a comparison group’s change, but the subtraction is useful only when the underlying counterfactual is credible.
What you will leave with: Prepare an analyst review brief covering timing, comparison selection, assumptions and uncertainty.
Use this when
- A regional rollout was not randomized.
- A report attributes the whole post-launch change to marketing.
This is an interpretation checklist, not a ready-made estimator or a substitute for a qualified causal analyst.
Specify the intervention and target effect
Record who received the intervention, when exposure began and what outcome is measured on a consistent basis. Define whether the question concerns the treated locations, all eligible customers or a future rollout. A broad claim about marketing effectiveness may exceed what this particular comparison can estimate.
Build a timeline of pricing, distribution, staffing, competitor activity and measurement changes. Identify anticipation before launch and spillovers into comparison locations. A control market exposed to the same campaign through commuting or shared media may no longer represent the untreated experience the analysis needs.
Make the parallel-trends assumption explicit
World Bank researchers explain that difference-in-differences relies on treated and comparison units having the same expected outcome change without treatment, potentially conditional on relevant characteristics. They also discuss complications from staggered adoption and correlated observations.
Ask why that assumption is plausible here. Examine earlier outcome trajectories and substantive operating differences. Similar historical trends are useful evidence, but they cannot prove what would have happened after launch. A comparison selected because it produces a preferred answer is especially weak. Document selection before reviewing post-treatment effects where possible.
Inspect the simple arithmetic and the actual design
For an introductory two-group, two-period summary, calculate the treated change minus the comparison change. Keep units explicit: revenue, orders or percentage points. This arithmetic helps stakeholders understand the contrast, but it does not supply a valid standard error or solve confounding.
Have the analyst choose methods appropriate to assignment level, repeated observations and adoption timing. Multiple rollout dates, changing treatment effects and few independent locations require particular care. Do not feed every customer row into a generic calculator and pretend that a market-level decision created independent customer-level assignments.
Report what could overturn the decision
Request sensitivity checks tied to credible threats: alternative defensible comparison sets, measurement breaks, spillovers and concurrent local shocks. Distinguish a robust direction from a fragile exact magnitude. Preserve inconclusive findings when plausible alternatives materially change the interpretation.
State the estimated contrast, uncertainty, population and identifying assumptions together. Recommend a bounded decision proportionate to the evidence, such as further testing or a limited rollout. Do not describe observational adjustment as equivalent to a randomized test or hide an important confounder in a technical appendix.
Make the reasoning concrete.
A hypothetical treated market rises from 100 to 130 weekly orders, while a comparison market rises from 80 to 100. The simple difference-in-differences is 30 minus 20, or ten orders. A simultaneous store-hours extension in the treated market threatens attributing those ten orders to advertising.
Attribute the full increase
Opening: Call all 30 additional orders campaign impact.
Hypothesis: The launch and growth occurred together.
Risk: Shared growth and the local operating change are ignored.
Review the contrast and threats
Opening: Use the ten-order contrast as a starting point for analyst review.
Hypothesis: The comparison removes one shared movement in this illustration.
Risk: The store-hours change can still confound the result.
Do not call ten orders a verified advertising effect. Resolve or disclose the competing intervention and obtain design-appropriate uncertainty before making a causal recommendation.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Effect
Whose effect and which period are being estimated?
Comparison
Why is the untreated trajectory credible?
Threats
What shocks, spillovers or anticipation could differ?
Method
How are timing and dependence handled?
Decision
What remains defensible under sensitivity checks?
Watch for the wrong conclusion.
- Treating subtraction as proof of causality.
- Claiming a pre-trend check proves parallel trends.
- Ignoring staggered adoption or shared-location dependence.
- Selecting comparison units to obtain the desired result.
Your next action
Write the intervention timeline and counterfactual argument before asking an analyst to estimate a difference-in-differences effect.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- World Bank Development Impact: When your difference-in-differences has too many differences →Primary researcher methodological commentary · Supports: Explains parallel trends as a counterfactual assumption and complications from differing trends, staggered adoption and correlated errors. The simplified arithmetic is illustrative, not a sufficient causal analysis.