Resolve conflicting evidence by comparing the questions

Write the decision and each method’s question

State the practical choice, such as whether to expand a particular campaign in a specified market. For each evidence source, record population, period, outcome, unit, exposure scope and the quantity estimated. Attributed purchases and purchases caused by advertising can both be validly reported while answering different questions.

Include uncertainty and exclusions. A model covering national weekly sales should not be compared casually with a short regional experiment or a survey of recent customers. First identify the overlap relevant to the decision, then mark the remaining differences rather than forcing all results into one superficially comparable chart.

Check evidence quality and shared dependencies

Inspect the strongest threats to each result: assignment integrity, missing outcomes, attribution rules, model assumptions, sample coverage or respondent interpretation. Judge evidence relative to the question it can answer. A randomized study with major implementation failure is not automatically decisive merely because it uses a prestigious method label.

Map shared inputs. Two reports using the same conversion feed are not independent confirmations of its completeness. A model calibrated to an experiment does not provide wholly separate support for that experiment’s effect. Record dependence so the team does not count one underlying signal several times.

Investigate genuine conflict rather than vote

The Magenta Book’s synthesis guidance recommends integrating findings around evaluation questions and examining conflicting evidence, considering explanations and seeking additional evidence where appropriate. Apply that principle by writing competing explanations for the disagreement and checks that could distinguish them.

Possible explanations include different windows, spillovers, changing creative, lagged effects or a faulty outcome feed. Assign an owner and a finite investigation to each material issue. Do not commission endless analysis to avoid an uncomfortable result. Preserve disagreements that cannot be resolved within the available evidence and decision deadline.

Make a proportional decision and preserve uncertainty

Summarize what the evidence supports jointly, what relies on one fragile assumption and what remains unknown. Choose a bounded action when appropriate: maintain current spend, run a targeted test or expand only within a measured population. State what future finding would change that choice.

Do not average incompatible point estimates into a compromise number. If formal evidence synthesis is appropriate, have a qualified analyst define the common estimand, dependencies and method. Keep the practical recommendation separate from its confidence so stakeholders can accept a cautious action without pretending the measurement disagreement has disappeared.

Make the reasoning concrete.

A hypothetical platform attributes 100 purchases to a campaign. A regional randomized study estimates 20 incremental purchases with substantial uncertainty, while a national model estimates a different annual effect. These numbers are not three interchangeable estimates of the same quantity.

Use the evidence relevant to the decision, investigate material quality conflicts and disclose unresolved gaps. Do not label attributed purchases as incremental or manufacture agreement by averaging.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Decision

   What specific choice and population require evidence?

2. Estimands

   What quantity does each method actually estimate?

3. Dependencies

   Which reports share data or calibration inputs?

4. Conflict

   Which differences are definitional and which need investigation?

5. Action

   What bounded choice and future evidence trigger are justified?

Watch for the wrong conclusion.

• Voting among methods by stakeholder preference.
• Averaging estimates with different units or causal meanings.
• Counting shared data as independent corroboration.
• Hiding unresolved disagreement behind a single confidence score.

Sources & scope

Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.

• HM Treasury: Magenta Book, Central Government guidance on evaluation →Primary government evaluation guidance · Supports: Section 3.7 explains integrating findings around evaluation questions and investigating conflicting evidence. The marketing evidence matrix is original analysis and does not equate attribution with causal impact.