Find out how people reach an answer before counting it

Specify the intended concept without teaching it to participants

Write an internal description of what the question is meant to measure, its reference period and the experiences that should and should not count. Identify uncertain terms and response choices before the session. Keep this specification out of the participant introduction; explaining the desired meaning first would remove the misunderstanding you need to observe.

CDC’s questionnaire-evaluation guidance describes cognitive interviewing as examining how people understand and answer questions. It distinguishes purposive problem-finding from population estimation and identifies comprehension, recall, judgment and response as useful probe areas. Those principles support a diagnostic study, not a claim that a small set of interviews establishes the percentage of all customers who misunderstand.

Observe an answer before exploring the reasoning

Present the question in its intended context and record the participant’s initial answer, uncertainty and any request for explanation. Use a planned protocol for when to probe. A researcher who paraphrases the question before the person responds may unintentionally solve the very defect the study should reveal.

Ask neutral follow-ups about the meaning used, the information recalled and the way the answer fit the available choices. Avoid praising an intended interpretation or arguing that another reading is wrong. If the participant cannot retrieve an exact amount, explore what they used instead without pressuring them to reveal private records. Separate their spontaneous explanation from an interpretation they adopt only after the moderator offers it.

Classify the problem at the point where the answer changes

Keep a record for each question and participant, using only appropriate de-identified evidence. Distinguish a misunderstood word from an unavailable memory, a difficult judgment and a missing response category. These lead to different fixes. Shortening a sentence will not repair a reference period people cannot reliably reconstruct.

Compare patterns across relevant experiences, including cases near the intended boundary. A person who cancelled yesterday may interpret “current customer” differently from someone whose membership expires next month. Do not convert a count of issues in a purposive sample into a population rate. Look for a plausible failure mechanism and contrary evidence, then identify which design change could address it without altering the intended construct.

Retest the revised question rather than approving the explanation

For each revision, record the original wording, evidence, change and expected improvement. Check whether changing one definition creates a new ambiguity or excludes an experience the measure needs. Review the whole response set and routing as well as the sentence. A clearer question with no suitable answer can still produce misleading data.

Retest the material change with suitable participants who have not been taught the previous explanation. Keep unresolved limitations visible and decide whether the question is ready for a broader pilot, needs another iteration or should be dropped. If it belongs to an existing time series, arrange a separate comparability review. Cognitive pretesting improves understanding of question performance; it does not supply the sampling design or statistical evidence for a later survey.

Make the reasoning concrete.

A fictional membership service asks how often customers used its benefits last month. Some respondents include browsing offers, others include only redeemed offers, and everyone selects an answer without asking for help.

The researcher records the competing interpretations and tests a clarified measure without explaining the desired answer in advance. The pilot’s completion rate is not treated as evidence of valid measurement.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Concept

   What experience should this question count?

2. Initial answer

   What did the participant answer before probing?

3. Process

   Where did interpretation, recall, judgment or response mapping diverge?

4. Revision

   What specific change addresses that mechanism?

5. Retest

   What evidence would show that the new wording works?

Watch for the wrong conclusion.

• Explaining the intended meaning before observing an answer.
• Counting a purposive sample as a population estimate.
• Approving a rewritten question without testing it.

Sources & scope

Workflows, worksheets and fictional examples are original editorial analysis. Named sources support only identified method definitions and principles. No population estimate, measured business result or named human reviewer is claimed.

• CDC CCQDER: Cognitive Interviewing →Primary research-method guidance · Supports: Cognitive interviewing examines comprehension, recall, judgment and response processes, with purposive problem-finding rather than population estimation. The proposed operating register is original analysis.