Expert review (method) explained

Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts.

An expert review has two primary goals:^[1]

1. Identify potential problems related to data quality and data collection so they can be mitigated.
2. Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects.

An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended.

The experts document their assessment and typically report the results in open-ended comments.^[2] They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error.^[3]

When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation.^[4]

Compared to empirical pretesting methods, an expert review is more time-efficient and less costly to complete.^[5] An expert review can also be combined with other methods to enable multimode pretesting.^{[6] [7] [8]}

See also

• Heuristic evaluation

References

1. Olson . Kristen . 2010 . An Examination of Questionnaire Evaluation by Expert Reviewers . . 22 . 4 . 295–318 . 10.1177/1525822X10379795.
2. Yan . T. . Kreuter . F. . Tourangeau . R . December 2012 . Evaluating Survey Questions: A Comparison of Methods . . 28 . 4 . 503–529.
3. Web site: Willis . G . Lessler . J . 1999 . Question Appraisal System (QAS-99) . Centers for Disease Control and Prevention Website.
4. Book: Pan . Yuling . The Sociolinguistics of Survey Translation . Sha . Mandy . 2019-07-09 . . 978-0-429-29491-4 . London . 118–124 . 10.4324/9780429294914 . 198632812.
5. Presser . S . Stanley Presser . Blair . J . Survey pretesting: Do different methods produce different results? . . 24 . 73–104.
6. Ridolfo . Heather . Thompson . Ashley . 2023-10-05 . Conducting Small-Scale Multimethod Questionnaire Evaluation . Survey Practice . en . 16 . 1 . 10.29115/SP-2023-0017. free .
7. Web site: Sha . Mandy . March 2012 . Translation Review and Cognitive Testing of ACS Language Assistance Guides in Multiple Languages . 2024-01-08 . United States Census Bureau.
8. Book: Aizpurua, Eva . The essential role of language in survey research . RTI Press . 2020 . 978-1-934831-23-6 . Sha . Mandy . 129–150 . Pretesting methods in cross-cultural research (Chapter 7) . 10.3768/rtipress.bk.0023.2004 . Gabel . Tim . free.