Choose what each person sees before comparing concept scores

Name the judgment the study must capture

Ask what the team needs to learn: whether a concept makes sense on first encounter, how alternatives differ, or what to improve before a later evaluation. Write the intended decision before choosing a survey template. A study built for detailed comparison may be useful without measuring the untouched first impression that a buyer would have in the market.

Qualtrics describes monadic tests as showing a respondent one concept and sequential monadic tests as showing multiple concepts. Its guidance discusses the different survey-length and development-stage tradeoffs. Those labels describe exposure structure; they do not establish that a particular sample or comparison is unbiased or sufficient.

Make the concept packages comparable

Use a versioned stimulus pack. Match the level of detail, visual finish and explanation where the goal is to compare the underlying propositions. If one concept needs a long operational explanation and another is instantly familiar, document that difference rather than silently editing away a genuine implementation challenge. An unrealistic polished version can win approval for an offer the business cannot deliver.

Have someone outside the concept team review each package for unsupported claims, missing assumptions and accidental clues about the preferred answer. Record exactly what participants will see, including introductions and follow-up questions. The stimulus includes the surrounding survey language, not only the hero image. Keep the material stable after fieldwork begins unless an explicit correction requires a documented version change.

Review assignment, order and burden with the researcher

For a single-concept design, ask how comparable participants are assigned to concepts and how precision will be assessed for each group. For a multiple-concept design, ask how order will be handled, how many tasks are reasonable and how the analysis accounts for repeated responses from the same person. A larger count of ratings is not necessarily a larger count of independent people.

Rehearse the experience from beginning to end. Look for concepts that become easier to understand only after an earlier explanation, repeated questions that encourage mechanical responding, and mobile layouts that make one alternative harder to inspect. Record these as design issues before looking at scores. Randomizing order can help distribute order effects, but does not make prior exposure vanish for an individual participant.

Keep the design visible in the recommendation

Report who saw what, in which sequence, using which versions. Distinguish an initial reaction from a judgment made after several alternatives. If historical benchmarks used a different design, do not put the scores on one chart as if the context were identical. A researcher can assess whether a bridge or separate interpretation is appropriate.

Use the findings to choose a proportionate next step: refine confusing claims, investigate a segment difference or commission a more focused evaluation. Preserve objections and uncertainty beside the ranking. The concept that receives the highest score may still fail a product-truth, feasibility or business-economics gate. Research informs that decision; it does not erase the other gates.

Make the reasoning concrete.

A fictional agency tests three service concepts. Every participant sees the most detailed version first, which explains the category vocabulary used by the other two. The team wants to call the final rating a clean first-impression comparison.

The research lead documents the dependency and uses the existing results for refinement. Any later first-encounter comparison receives a design appropriate to that question.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Decision

   Is this about first encounter, comparison or refinement?

2. Stimuli

   Which exact versions and surrounding explanations will be shown?

3. Exposure

   Who sees one concept, who sees several and in what order?

4. Analysis

   How are assignment, repeated responses and uncertainty handled?

5. Transfer

   Which historical scores are genuinely comparable?

Watch for the wrong conclusion.

• Treating every concept rating as an independent participant.
• Confusing a randomized order with absence of contextual influence.
• Comparing polished and incomplete stimuli without acknowledging the difference.
• Turning purchase-intent ratings directly into sales commitments.

Sources & scope

Workflows, worksheets and hypothetical examples are original editorial analysis. Platform documentation supports only identified technical principles. No measured business result or named human reviewer is claimed.

• Qualtrics: Concept Testing Program →Primary research-software provider methodology guidance · Supports: Definitions of single-concept monadic and multiple-concept sequential monadic exposure in the vendor’s program. Assignment review and interpretation safeguards are original editorial analysis.