Make the simulated assignments match the experiment that actually ran

Recover the allocation procedure, not just the treatment labels

Obtain the original assignment record and the rules that generated it. Identify the assignment unit, treatment counts, blocks or matched pairs, restrictions and assignment probabilities. Preserve exceptions and implementation deviations separately. Knowing that some stores received treatment is not enough to reconstruct how those stores could have been selected.

EGAP explains randomization inference through the set of assignments allowed by the experimental procedure and the statistic’s reference distribution under a specified null. It also discusses clustered and constrained designs. These principles support checking the simulation against the design, rather than shuffling every row in an export and calling the output an exact experiment-based test.

Name the null hypothesis and the statistic

Ask the analyst to state the tested claim in ordinary language. A common sharp null says that treatment changes no unit’s outcome; that is not identical to saying the average effect is zero while positive and negative individual effects cancel. Other hypotheses require their own justified procedure. Do not silently relabel one test as another in the executive summary.

Specify the test statistic and the direction of the comparison before selecting the most favorable output. A difference in means, a standardized contrast and a regression coefficient can behave differently. Keep the statistic tied to the outcome and assignment design. The null and statistic together determine what an unusual result means; a generic label such as nonparametric does not remove those choices.

Check the allowable assignments and computation

For a matched-pair market experiment, the simulated allocation must respect the pairing and the treatment rule within each pair. For a cluster-assigned campaign, moving individual customer rows between conditions would invent assignments that never could have occurred. Ask the analyst to demonstrate the restrictions on a small synthetic fixture before accepting the production result.

Distinguish enumeration of every permitted assignment from a finite simulation sample. Request the reproducibility details, the number of draws where relevant and an assessment of simulation error. Many simulation draws cannot create more independent experimental units or repair missing assignment records. Likewise, an exact calculation can still have coarse attainable probabilities in a small design. Avoid publishing unwarranted decimal precision.

Keep the probability separate from the business effect

Report the observed effect estimate and an appropriate uncertainty assessment alongside the test, while making clear which hypotheses the methods address. A p-value is not the probability that the campaign is effective, and failing to reject the null does not prove no worthwhile effect. The result also does not automatically resolve spillovers, missing outcomes or noncompliance; those require design-specific treatment.

Preserve the assignment specification, data version, statistic and analysis choices so the result can be reproduced. If a mismatch is found, correct the implementation and explain which conclusions changed instead of selecting whichever test looks stronger. The decision should reflect the practical effect, its uncertainty and the experiment’s limits, not the prestige of the word exact in a software output.

Make the reasoning concrete.

A fictional retailer randomizes one store within each matched pair to a local campaign. An analysis shuffles individual receipt rows across all stores and reports a very small permutation p-value.

The retailer requests a design-consistent analysis and retains the original allocation record. It reports the effect and limitations separately rather than treating the first probability as a business-impact estimate.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Allocation

   Which assignments were possible under the real procedure?

2. Null

   What exact causal hypothesis does the test evaluate?

3. Statistic

   Which planned outcome contrast is being compared?

4. Computation

   Were assignments enumerated or simulated, with what uncertainty?

5. Meaning

   What does the result establish beyond the effect estimate?

Watch for the wrong conclusion.

• Shuffling customer rows when markets were randomized.
• Calling a finite simulation exact without qualification.
• Reading a p-value as the probability the campaign works.

Sources & scope

Workflows, worksheets and fictional examples are original editorial analysis. Named research guidance supports identified method principles. This prepares a review with a qualified analyst and does not claim an observed business result or a named human reviewer.

• EGAP: 10 Things to Know About Randomization Inference →Primary research-method guidance · Supports: Assignment-based reference distributions, specified sharp nulls and statistics, exact enumeration, and clustered or constrained assignment. The review record is original analysis.