Question Evaluation

Question evaluation is the process of testing and analyzing how survey questions perform. The knowledge that comes from question evaluation is invaluable to survey managers and data users. Our question evaluation process poses the question: "How does the question function, and what does this imply about the contexts in which it can profitably be employed?" With an understanding of how the survey questions performs, survey managers and data users have full confidence in the validity of the data as well as an opportunity to revise the questions to better fit their study. Some of the strengths of question evaluation are listed below.

Question evaluation can determine if a question...

• Captures the intended concept: Ideally, questions should capture constructs that they were created to capture. However, it is important to know if questions are not capturing the intended constructs. Understanding the constructs that the question is capturing allows survey managers to modify the question, and allows data users to understand the potential range of constructs that may have been captured in the data. Both of these measures increase the validity of the data.
• Is interpreted consistently across respondent groups: Respondents' interpretations of a question may differ depending on their social experiences and demographic factors. For example, respondents with different education levels might interpret or comprehend question terminology differently.
• Poses problems for respondents when attempting to answer: There are many possible reasons why respondents might have difficulty answering a question. Question evaluation can help to shed light on this process.

Question evaluation should be...

• Empirical - Surveys that are developed solely from expert opinion depend on theory and assumptions rather than evidence. Contrary to this, empirical question evaluation uses direct observation as evidence.
• Transparent - The methods and analysis conducted are clearly documented, so all conclusions can be traced back to original data. This transparency increases the credibility of the findings from question evaluation studies.
• Systematic - Consistency and rigor in data collection and analysis is key to good question evaluation. The QDRL's analysis tools help to ensure a systematic and transparent analysis across all studies.

Why conduct question evaluation?

Question evaluation is a vital part of survey development. Evaluation reduces non-response error, which ensures comparable data and supports data users in their analyses of survey data. In addition, the Office of Management and Budget stipulates in Section 1.4 of The Standards and Guidelines for Statistical Surveys:

"Agencies must ensure that all components of a survey function as intended when implemented in the full-scale survey and that measurement error is controlled by conducting a pretest of the survey components or by having successfully fielded the survey components on a previous occasion."

OMB Standards and Guidelines for Statistical Surveys [PDF - 318 KB]

Evaluation Methods

Cognitive Interviewing

Other Research Methods