Investigate synthesis inconsistency without a universal cutoff

Identify the inconsistency that matters

Cochrane Chapter 10, checked 7 November 2026, warns that heterogeneity measures are uncertain with few studies and that a nonsignificant heterogeneity test does not establish absence of heterogeneity. Interpretation depends on effect patterns and context. Do not copy a threshold table into a universal marketing release rule.

Read the study estimates on the outcome scale the decision uses. Identify whether the concern is different magnitudes, different directions or an apparently isolated result. Keep each estimate’s uncertainty visible. A chart that uses only colored significance labels can conceal the pattern the review needs to understand. Ask the analyst for the relevant representation rather than inferring consistency from a short executive summary.

State why the observed pattern could change the recommendation. A summary that averages useful and harmful effects creates a different concern from effects that vary in size but remain operationally similar. That is a substantive judgment about the decision, not a claim that a particular statistic supplies the answer automatically.

Check errors before explaining variation

Trace a small set of discrepant estimates back to their extracted inputs. Confirm units, outcome direction, comparison labels and the relevant observation window. A mistaken reversal or an incorrect scale can create a persuasive story about differences between settings that never existed. Keep the untouched extraction and record each verified correction rather than adjusting numbers until the display looks consistent.

If the inputs are correct, examine the study descriptions for plausible explanations tied to the review question. Record which explanations were anticipated and which arose after seeing the pattern. Avoid a long search through arbitrary characteristics until one divides the studies conveniently. A plausible explanation is a hypothesis to assess, not proof that the observed variation has been causally explained.

Choose a reporting response to the remaining evidence

Consider a fictional synthesis in which the immediate-use studies suggest improvement while later-use studies are less favorable. First establish that the windows were extracted correctly and that the contrast does not simply reflect different interventions. The review might need separate reporting, a qualified overall description or a decision that an average would conceal too much. The appropriate choice depends on what the evidence and the business question support.

Ask the analyst to show what the proposed response changes and what it leaves unresolved. Do not remove a study merely because its omission makes the summary more attractive. If a sensitivity analysis is used, state the reason for the exclusion and retain the main record. Readers should see the consequence of a defensible concern, not only the cleanest remaining result.

Preserve uncertainty in the diagnosis

Write a concise account of what was checked, what was corrected and which variation remains unexplained. Separate a data error from a genuine difference in study context, and separate both from limited information about the amount of variation. A conclusion can be useful while acknowledging that the evidence is too sparse to explain the pattern confidently.

Tie the reporting choice to the original decision. If the evidence cannot support a portable single-number claim, say what narrower conclusion remains possible and which next study could address the uncertainty. Do not automatically demand more observations without identifying the unresolved contrast. Retain the diagnosis with the synthesis so an updated review can distinguish a persistent difference from a previously corrected extraction mistake.

Make the reasoning concrete.

A fictional review of workplace documentation finds conflicting study estimates. A manager wants to suppress the overall result whenever a dashboard indicator exceeds a familiar threshold, while another wants to remove the least favorable study.

The team records its diagnosis and keeps unresolved variation visible. It does not invent a universal cutoff or discard evidence for its direction.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Pattern

   What difference between study results matters to the decision?

2. Inputs

   Which extraction or scale errors were checked?

3. Explanation

   Which contextual explanation is supported or still hypothetical?

4. Response

   Would separate reporting or qualification be more useful than an average?

5. Uncertainty

   What remains unknown and what evidence could resolve it?

Watch for the wrong conclusion.

• Treating a statistic as an automatic release gate.
• Explaining a pattern before checking its inputs.
• Deleting an inconvenient study to improve the headline.

Sources & scope

Original adult business decision scenarios are explicitly fictional. Methodological claims and their applications remain source-derived. Qualified analysts own formal calculations. Cochrane-specific guidance is not a universal mandate. No actual research operations or publication occur.

• Cochrane Handbook, Chapter 10: Analysing data and undertaking meta-analyses →Primary technical documentation · Supports: Supports contextual heterogeneity interpretation and uncertainty, not universal cutoff decisions.