Box 6-1

Quality Rating Criteria for External Validity

Reach and Representativeness

  • Participation: Are there analyses of the participation rate among potential (1) settings, (2) delivery staff, and (3) patients (consumers)?

  • Target Audience: Is the intended target audience stated for adoption (at the intended settings, such as worksites or medical offices) and application (at the individual level)?

  • Representativeness (settings): Are comparisons made of the similarity of settings in the study to the intended target audience of program settings—or to those settings that decline to participate?

  • Representativeness (individuals): Are analyses conducted of the similarity and differences between patients, consumers, or other subjects who participate vs. either those who decline or the intended target audience?

Program or Policy Implementation and Adaptation

  • Consistent Implementation: Are data presented on level and quality of implementation of different program components?

  • Staff Expertise: Are data presented on the level of training or experience required to deliver the program or quality of implementation by different types of staff?

  • Program Adaptation: Is information reported on the extent to which different settings modified or adapted the program to fit their setting?

  • Mechanisms: Are data reported on the process(es) or mediating variables through which the program or policy achieved its effects?

Outcomes for Decision Making

  • Significance: Are outcomes reported in a way that can be compared to either clinical guidelines or public health goals?

  • Adverse Consequences: Do the outcomes reported include quality of life or potential negative outcomes?

  • Moderators: Are there any analyses of moderator effects—including of different subgroups of participants and types intervention staff—to assess robustness vs. specificity of effects?

  • Sensitivity: Are there any sensitivity analyses to assess dose−response effects, threshold level, or point of diminishing returns on the resources expended?

  • Costs: Are data on the costs presented? If so, are standard economic or accounting methods used to fully account for costs?

Maintenance and Institutionalization

  • Long-term Effects: Are data reported on longer-term effects, at least 12 months following treatment?

  • Institutionalization: Are data reported on the sustainability (or reinvention or evolution) of program implementation at least 12 months after the formal evaluation?

  • Attrition: Are data on attrition by condition reported, and are analyses conducted of the representativeness of those who drop-out?

SOURCE: Green, L. W., and R. E. Glasgow, Evaluation and the Health Professions 29(1), pp. 126-153, Copyright © 2006 by SAGE Publications. Reprinted by permission of SAGE Publications.

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement