Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 77
4
The Criteria and Process
for Setting Priorities
abstract: This chapter describes the prioritization criteria and process the
committee used in its review of the nominated research topics. The com-
mittee considered the balance of the portfolio of nominated research topics
across research areas, populations to be studied, types of interventions,
and methodologies. In establishing prioritization and portfolio criteria,
the committee had the benefit of examining methods and criteria used in
several past priority-setting exercises. The committee developed two types
of criteria: () condition-level criteria that relate to the significance of
specific conditions or diseases for the population as a whole or certain age
groups, and () priority topic-level criteria that include the appropriateness
of the nominated research topics for CER, information gaps, variability
in care, and gaps in translation. The committee used three rounds of vot-
ing to narrow the list of nominated topics to a manageable, high priority
portfolio. Based on this experience, the committee made recommendations
for future priority-setting projects. Prioritization of CER topics should be
a sustained and continuous process that requires the prioritizing body to
make regular reports to the Secretary of Health and Human Services, in-
volves the public in a transparent process, and is informed by robust topic
briefs and background information.
INTRODuCTION
The previous chapter described the committee’s method of obtaining
the nominated topics from stakeholders and the public, while this chapter
describes the prioritization criteria and process the committee used in its
OCR for page 78
INITIAL NATIONAL PRIORITIES FOR CER
review of the nominated topics. The first half of the chapter outlines the
various types of prioritizing criteria. The committee developed the concept
of portfolio criteria that were intended to ensure that the final priority
comparative effectiveness research (CER) topics reflect a balance of CER
questions across research area (e.g., geriatrics, neurology, psychiatry), study
populations (e.g., women, children, elderly), type of intervention (e.g.,
surgical, pharmaceutical), and study methodology (e.g., randomized trial,
observational study). The committee also developed two sets of criteria for
committee members to use in evaluating the specific nominated priority
topics: (1) condition-level criteria, including data on burden of disease and
variability in care, and (2) priority topic-level criteria, including the appro-
priateness of the nominated research topics for CER, existing information
gaps, variability in care, and gaps in translation.
The second section of the chapter includes an overview of the Institute
of Medicine’s (IOM’s) previous recommendations, criteria, and procedures
for setting research priorities, as well as the criteria and methodologies used
by other prioritization projects. The committee relied on these past IOM
and external reports to develop the criteria it used in the voting process that
established the final priority CER topics. The committee’s voting process
that was used to narrow the list of nominated topics to a manageable, high
priority portfolio is also described in detail.
The chapter concludes with a description of the lessons learned from
the committee’s priority-setting exercise and provides recommendations for
future prioritization processes in CER. The final results of the committee’s
deliberations and voting are presented in Chapter 5.
PORTFOLIO CONSIDERATIONS
As described in Chapter 3, the committee developed the majority of its
priority CER topics from the public input gathered through the web-based
questionnaire. The questionnaire required the respondents to provide de-
tailed information about each of their nominated priority topics. Respon-
dents were required to identify the primary area of study of their nominated
topic from among the 32 disease classifications, patient conditions, and
systems of care categories provided. Research areas included categories
such as oncology and hematology, geriatrics disorders, neurologic disorders,
and so on.
Respondents were also asked to identify the proposed population to be
studied, ranging from categories based on gender, age, race, special popu-
lations, and rare diseases, as well as the proposed comparators, such as
alternative and complementary treatments, behavior intervention, devices,
and pharmacological therapy. In addition, the respondents were requested
to identify a proposed methodology for their nominated research topic.
OCR for page 79
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
TAbLE 4-1 Portfolio and Priorities Criteria
Priority Topic-Level
Portfolio Criteria Condition-Level Criteria Criteria
• Research area • Prevalence • Appropriateness of
• Population to be studied • Mortality topic for CER
• Interventions • Morbidity • Information gaps and
• Proposed methodology • Cost duplication
• Variability • Gaps in translation
Potential methodologies included database review, prospective observa-
tional study, randomized clinical trial, or systematic review.
As part of the process of prioritizing the nominated research topics, the
committee, with the financial support provided by the Agency for Health-
care Research and Quality (AHRQ), created a portfolio of CER topics that
was balanced and diverse across the characteristics previously described
and listed in Table 4-1 (see next section for discussion of criteria for prior-
ity setting). The distribution of the nominated research topics according to
the portfolio’s characteristics was provided to the committee throughout
each step in the voting process. The committee weighed the balance of the
portfolio in its deliberations and selection of the final priority CER topics.
The committee selected the following four characteristics for balancing
the portfolio after careful consideration of the goals of CER. The commit-
tee determined that it is important to have a portfolio balanced in terms
of research areas because it did not want to study only those diseases and
conditions with the greatest effects on the health of the U.S. population. It
determined that it is also important to study rare diseases and conditions
that disproportionately and seriously affect subgroups of the population,
partly because the scientific opportunity may be greatest for some of these
conditions. Similarly, the committee determined that it is essential for the
portfolio to be balanced across populations to be studied. The priority CER
topics should include populations and subpopulations representing minor-
ity, racial and ethnic groups, all genders, and different age groups ranging
from infancy to the elderly.
The committee determined that the portfolio should also include a
diversity of interventions. Traditionally, much of CER has focused on head-
to-head comparisons of pharmaceutical treatments. However, the commit-
tee saw great value in extending the concept of comparison to include a
variety of interventions, including tests to screen for or monitor diseases
(e.g., imaging for cancer or during normal pregnancy), surgical techniques
(e.g., closed vs. open procedures), and therapeutic alternatives (e.g., medi-
cal therapy vs. surgical vs. radiotherapy for prostate cancer). This diversity
OCR for page 80
0 INITIAL NATIONAL PRIORITIES FOR CER
helps ensure that the portfolio covers the entire continuum of health care,
including screening and prevention, treatment of acute health problems,
chronic health problems, and palliative and end-of-life care. Additionally,
CER that examines different means of delivering health care was considered
to be an important determinant of quality and was incorporated into the
options for intervention.
The committee also decided that the portfolio should include a diversity
of methodologies. This is especially important in light of the committee’s
charge to develop a portfolio of topics that will lead to an appropriate
expenditure of the $400 million for CER in the American Recovery and Re-
investment Act (ARRA) of 2009 (P.L. 111-5). The different methodologies
vary widely in terms of resource requirements, time lines, and types of in-
frastructure necessary to conduct the research. Varying these methodologies
in the portfolio ensures that some relatively inexpensive and easy results
will be generated early on, and within the scope of ARRA. Performance of
randomized controlled trials or prospective observational trials will have
to extend well beyond the 2-year focus of ARRA.
CRITERIA CHOSEN FOR PRIORITy SETTING
In addition to the portfolio criteria intended to assess the balance
of all the priority CER topics, the committee also concluded that criteria
were needed to evaluate the individual nominated research topics. The
committee reviewed the criteria used for priority setting by other projects
and the two IOM reports discussed below, but it developed its own set of
criteria that applied specifically to setting priorities for CER. After careful
consideration, the committee recognized two levels of criteria to assess the
nominated CER research topics: (1) condition-level criteria, and (2) priority
topic-level criteria.
Condition-Level Criteria
The condition-level criteria focus on burden of disease indicators that
could be readily obtained by the committee. The committee recognized
the importance of selecting priority topics that affected a large portion of
the population (prevalence), were the leading causes of death and disease
(mortality and morbidity), imposed serious costs on patients, families,
payers, and society (costs), and had the greatest differences in treatment
used by practitioners (variability). The committee would have also liked to
include work loss due to disability as a criterion, but it could not find data
to support that criterion within the time frame of the study. Data on the
following criteria were provided to the committee for consideration in the
prioritizing and voting process:
OCR for page 81
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
• Prevalence: The number or percentage of people with a specified
condition in the United States at a given time (Appendix C, Table
C-1).
• Mortality: The number or percentage of deaths due to a specified
condition in the United States in a given time period (Appendix C,
Table C-2).
• Morbidity: The extent of illness, injury, or disability in a defined
population (Appendix C, Table C-3).
• Cost: The total treatment expenses for selected conditions (Appen-
dix C, Table C-4).
• Variability: A measure of the dispersion of data. In the context
of Appendix C, Table C-5, it refers to the pattern of variation in
admissions for specific procedures among hospital referral regions.
In the context of Appendix C, Table C-6, it refers to the pattern of
variation in admission for treatment of conditions among hospital
referral regions.
Priority Topic-Level Criteria
The committee made a distinction between condition-level criteria and
the criteria that could be applied to individual topics involving those con-
ditions (Table 4-1). For example, as shown in Appendix C, the prevalence
data provided to the committee identified the top 20 conditions for adults
and the top five for children. From this information, committee members
might determine that a particular condition targeted in a proposed research
topic is widespread and so it may be of importance to study; however,
they had little or no information on other aspects of the topic such as
whether particular procedures, clinical decisions, or delivery models were
also prevalent or appropriate for CER. The specific priority topic-level crite-
ria considered by the committee were intended to help assess the particular
questions identified in the nominated research topics, not just the conditions
and diseases. The priority topic-level criteria used by the committee include
the following:
• Appropriateness of topic for CER
o tility for decision making—Does the proposed topic include
U
populations previously excluded from trials, clinically meaning-
ful comparisons, or patient-important outcomes rather than
markers or intermediate outcomes? Does it involve direct, head-
to-head comparisons to inform the decisions of daily practice?
Is it patient-centered so that it tailors the test or treatment to
the specific characteristics of the patient? Will it enhance clinical
OCR for page 82
INITIAL NATIONAL PRIORITIES FOR CER
practice decision making by patients and physicians in everyday
circumstances and help policy makers?
o isks associated with care—There may be risks as well as benefits
R
associated with particular treatments/methods of care either in
current practice or the comparator that should be considered.
• Information gaps and duplication. CER should address gaps left by
existing research. Research gaps for selected topics can be identified
by Cochrane or Evidence-based Practice Center (EPC) systematic
reviews on the subject, which compare “what we need to know”
to make good decisions with “what we know now” from existing
studies. The description of gaps should follow the logic of the pre-
viously outlined criteria for utility in decision making:
o or suggested comparative effectiveness systematic reviews, are
F
there recent EPC or Cochrane reviews on the same subject?
o or suggested CER studies or trials, do recent EPC or Cochrane
F
reviews on the same subject identify the suggested research as
needed research (evidence gaps)?
o ave previous studies ignored patients with comorbidities? Does
H
the proposed study explicitly include them?
o ave previous studies ignored patients from special popula-
H
tions? Does the proposed study include them?
o ave previous studies made meaningful comparisons? Does the
H
proposed study include head-to-head comparisons?
o ave previous studies fully explored benefits and harms?
H
o the proposed topic redundant with current research? Data
Is
derived from a check of the short list against www.clinicaltrials.
gov and answers to the above questions.
o Variability
• The data requested from Dartmouth indicate rates of hos-
pitalizations and common procedures with high variability
across the country.
• Health care delivery—Does the proposed study address the
effectiveness of different strategies for delivering the inter-
vention? Strategies may include organization characteristics,
work patterns, or work processes.
• Gaps in translation
o oving from research to practice—Has CER been conducted on
M
the topic and recommendations made, but with limited impact
on practice? Will the proposed study be likely to improve the
implementation of the recommendations? Or identify improved
strategies for research translation?
OCR for page 83
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
DATA COLLECTION TO AID TOPIC SELECTION
The committee was unable to collect priority topic-level data relating
to the significance of the 1,268 nominated research topics to be assessed
initially, and instead, focused mainly on data to support criteria at the
condition level.
The committee selected individual data tables to aid in the voting pro-
cess. The Medical Expenditure Panel Survey (MEPS) (AHRQ, 2009) pro-
vided proxy indicators for information on prevalence, morbidity and cost,
and the National Vital Statistics Report on mortality (Kung et al., 2008).
The Dartmouth Institute for Health Policy and Clinical Practice analyzed
clinical practice data according to variation in treatment for medical condi-
tions and surgical procedures at the IOM’s request (Wennberg, 2009). Data
sources were chosen based on their year of production (with preference
given to the most recent reports), representativeness of the whole popula-
tion, and ability to provide age stratification.
Data pertaining to knowledge gaps (areas of scientific uncertainty in
terms of treatment strategies for the population as a whole or for subpopu-
lations) and funding gaps (areas with a dearth of recent or existing research
studies) were feasible to use after the first round of voting on the 1,268
nominated topics had narrowed to the list of 145. For the second round of
voting, the committee was provided several proxy indicators for knowledge
and information gaps, including the most recent systematic reviews as well
as the funding source and number of recent and ongoing clinical trials.
The AHRQ Effective Health Care Program’s issue briefs and the National
Institutes of Health’s registry of privately and publicly supported clinical
trials in the United States and abroad supplied data on perceived knowledge
gaps remaining to be addressed. Many of the second-round research topic
nominations for CER were not covered in the AHRQ issue briefs, and
committee members were informed of the unevenness of the supporting
data across topics.
The committee used the criteria and data tables as guided in voting
instructions and as summarized in cover sheets for each of the 32 broad
research areas. The cover sheets for each research area indicated whether
specified conditions within that research area were among the top-ranked
conditions by each condition-level criterion (as ranked in the tables in
Appendix C), and if the topic was listed by other national priority-setting
projects. For instance, in the cardiovascular and peripheral vascular disease
category, hypertension and hyperlipidemia were 2 among the top 20 most
prevalent diseases across all conditions for all age groups, and hypertension
along with heart conditions were 2 of the top 12 diseases according to the
morbidity proxy table, number of events, for all ages. Cardiovascular dis-
ease including stroke and hypertension were listed among the top priorities
OCR for page 84
INITIAL NATIONAL PRIORITIES FOR CER
of AHRQ’s Effective Health Care Program (Whitlock et al., 2009), Healthy
People 2010 (HHS, 2000), and the Cochrane Collaboration (Doyle et al.,
2005). The cover sheets were designed to be a quick reference to highlight
more specific subcategories or conditions within each broad research area.
An example of a cover sheet for cardiovascular and peripheral vascular
disease is presented in Appendix D. All 32 cover sheets are available at
www.iom.edu/cerpriorities.
The voting instructions for the committee listed and, if necessary, de-
fined the condition-level and priority topic-level criteria for the committee
to consider. However, there was no explicit direction on how to weigh
criteria, except that members should decide based on their own expertise
and preferences.
The selected criteria provided the committee with a framework for their
voting decisions. However, individual committee member expertise played
an important role in the decision-making process. Consequently, although
committee members were instructed to take into account quantitative data
such as prevalence, morbidity, and cost where such data were available,
the voting process had subjective elements in terms of how each member
selected their top priorities.
LESSONS FROM PREvIOuS PRIORITy-SETTING PROCESSES
IOM Reports
For this project, the most relevant IOM reports concerning priority set-
ting, Priority Areas for National Action (IOM, 2003) and Knowing What
Works in Health Care (IOM, 2008), provided the initial basis for the meth-
odology and criteria used by this committee in setting priorities for CER.
In Priority Areas for National Action, AHRQ tasked an earlier IOM
committee with recommending a list of 20 priority conditions whose im-
provement would help the nation achieve significant advances in health care
quality over the next 5 years, and then with establishing a process and set
of criteria for determining those priorities. The study committee chose a
framework, initially designed to assess care across the lifespan, developed
by the Foundation for Accountability (FACCT, which closed its opera-
tions in 2004) to organize all potential high priority conditions (Markle
Foundation, 2008). This framework consisted of four “domains of care”:
preventive, acute, chronic, and palliative; the committee also considered
system-wide interventions that would cut across these domains (FAACT,
1997). The goal was to create a final portfolio of priorities for quality
improvement that touched on the full continuum of care. Members of that
committee nominated most of the topics with the aid of stakeholder input
OCR for page 85
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
through presentations at a public workshop and a review of other relevant
priority lists (IOM, 2003).
That committee then applied three sets of criteria to the suggested top-
ics. The criteria reflected potential impact (disease burden variables), im-
provability (the likelihood the priority would address one of the six quality
aims in the 2001 Quality Chasm report), and inclusiveness (relevance to
a broad range of patients, conditions, and health care settings). The com-
mittee ranked the topic suggestions within their respective domains of care
categories and then determined the complete list, which is shown in Table
4-2 (IOM, 2003).
The committee that produced Knowing What Works in Health Care
provided further research and recommendations on the topic of setting pri-
orities. In a detailed assessment of methods used by other organizations to
identify topics for systematic reviews,1 the committee found little guidance
for designing an optimal priority-setting process. However, it did establish
several principles for future priority-setting committees (IOM, 2008):
• The process should be open, transparent, efficient, and timely.
• It should consider how evidence-based practice could help reduce
burden of disease.
• It should include cost considerations in the decision-making process.
• Its membership should include people with a broad base of inter-
ests and expertise to minimize bias and conflicts of interest.
Knowing What Works in Health Care also identified the most common
criteria used in other priority-setting efforts (IOM, 2008):
• Burden of disease (prevalence, disability, mortality, morbidity, etc.)
• Public controversy (uncertainty around the topic and supporting
data)
• Cost
• Potential impact
• New evidence that might change previous conclusions
• Existence of an evidence gap
• Unexplained variation in the use of healthcare services
1 The list of organizations includes the Agency for Healthcare Research and Quality, Blue
Cross and Blue Shield Association Technology Evaluation Center, Centers for Medicare &
Medicaid, the Cochrane Collaboration, Drug Effectiveness Review Project, Medicare Evidence
Development and Coverage Advisory Committee, the National Institute for Health and Clini-
cal Excellence, National Institutes of Health Office of Medical Applications of Research, and
the U.S. Preventive Services Task Force.
OCR for page 86
INITIAL NATIONAL PRIORITIES FOR CER
TAbLE 4-2 Criteria and Priorities for Quality Improvement
Criteria Priority List
Impact—disease burden (disability, • Asthma
mortality, and economic costs affecting • Cancer screening that is evidence-based
patients, families, communities and • Care coordination
society) • Children with special health care needs
• Diabetes
Improvability—the likelihood that • End of life with advanced organ system
systemic changes in health system could failure
improve priorities in the six quality aims • Frailty associated with old age
listed in the Quality Chasm report • Hypertension
• Immunization
Inclusiveness—equity (across ages, races, • Ischemic heart disease
gender, and socioeconomic status), • Major depression
representativeness (across spectrum of • Medication management
healthcare conditions); reach (across • Nosocomial infections
spectrum of healthcare settings and • Pain control in advanced cancer
providers); also, later included four care • Pregnancy and childbirth
domains (preventive, acute, chronic, and • Self-management/health literacy
palliative), or crosscutting • Severe and persistent mental illness
• Stroke
• Tobacco dependence treatment in adults
• Obesity
SOURCE: IOM (2003).
External Priority-Setting Initiatives
In addition to the two IOM reports, the CER committee reviewed the
following external priority-setting initiatives to select the condition-level
criteria and priority topic-level criteria: AHRQ’s Effective Health Care
Program, which identified topics for comparative effectiveness systematic
reviews (Whitlock et al., 2009); Healthy People 2010, an alliance of na-
tional and state public health agencies that developed a list of leading
health indicators and priority focus areas to set a prevention agenda for the
nation (HHS, 2000); the Cochrane Collaboration, which identified global
priorities for Cochrane systematic reviews of public health topics (Doyle et
al., 2005); the World Health Organization (WHO) Advisory Committee,
which reviewed the literature on priority setting for health care guidelines,
recommendations, and technology assessments (Oxman et al., 2006); and
the National Quality Forum (NQF), which convened the National Priorities
Partnership (NPP) to set national priorities and goals for performance im-
provement efforts in potentially fruitful areas (NPP, 2008); among others.
These external priority-setting programs identified additional criteria,
including appropriateness of a topic for CER research, feasibility of study
design, potential for change, and potential risk from inaction (Whitlock et
OCR for page 87
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
al., 2009); research that is not duplicative (Doyle et al., 2005); priorities
that are likely to improve quality of life and reduce health disparities (HHS,
2000); priorities that address the major challenges of eliminating harm and
removing waste (NPP, 2008); and interventions that would likely require
systems change (Oxman et al., 2006). Criteria chosen by the priority-setting
organizations are summarized in Table 4-3. As described above, the com-
mittee incorporated the appropriateness of the topic for CER and informa-
tion gaps and duplication into the priority topic-level criteria.
The Committee on Comparative Effectiveness Research Prioritization
also considered the methodology used by the other priority-setting groups
to arrive at its final priority CER topics including the following:
• The creation of a specific taskforce or committee to oversee and
ultimately vote on the priority questions
• An invitation for stakeholders to submit comments and priority
agendas via written or oral testimony to committee members
• The establishment of explicit priority criteria (and gathering of data
sources/information relevant to criteria) on which the committee
members were to base their decisions
• A process to refine submitted questions and gain feedback on the
revisions
• A recommendation to continuously evaluate and improve upon the
inherently dynamic and subjective priority-setting process
TAbLE 4-3 A Variety of Priority-Setting Initiatives and Their Selected
Criteria
Name Study/
Priority List Source Criteria
Identifying, Selecting AHRQ Appropriateness—applies to
•
and Refining Topics Medicare and/or Medicaid
for Comparative populations, HHS priority
Effectiveness Systematic condition
Reviews: AHRQ and Importance—disease burden,
•
Effective Health Care cost, strong stakeholder support,
Program uncertainty or controversy
surrounding issue
• Desirability of New
Research/Duplication
• Feasibility
Potential Value—potential for
•
significant health and economic
impact, change, and risk of
inaction
continued
OCR for page 88
INITIAL NATIONAL PRIORITIES FOR CER
TAbLE 4-3 Continud
Name Study/
Priority List Source Criteria
Healthy People 2010 U.S. Department of • Goal : Increase Quality and
Health and Human Years of Healthy Life—Life
Services (HHS) expectancy, quality of life: looked
at global assessments, healthy
days, and years of healthy life
• Goal : Eliminate Health
Disparities—In terms of gender,
race and ethnicity, income and
education, disability, geographic
location, and sexual orientation
• Leading Health Indicators
considered when choosing
focus areas—physical activity,
overweight and obesity, tobacco
use, substance abuse, responsible
sexual behavior, mental health,
injury and violence, environmental
quality, immunization, access to
health care
National Priorities and NPP/NQF • Eliminating harm
Goals: Aligning Our • Eradicating disparities
Efforts to Transform • Reducing disease burden
America’s Healthcare • Removing waste
Improving the Use of WHO • Problems associated with a high
Research Evidence in Advisory Committee burden of illness—in low- and
Guideline Development: on Health Research middle-income countries, or new
2 Priority Setting and emerging diseases
• No existing guidelines or
recommendations of good quality
• Feasibility of developing
recommendations—that will
improve health outcomes, reduce
inequities or reduce unnecessary
costs if they are implemented
Implementation is feasible—will
•
not exhaustively use available
resources, and barriers to change
are not likely to be so high they
cannot be overcome
• Interventions that will likely
require systems changes
• Interventions where there
might be a conflict in choices
between individual and societal
perspectives
OCR for page 89
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
TAbLE 4-3 Continued
Name Study/
Priority List Source Criteria
Global Priority The • Burden of disease, magnitude of
Setting for Cochrane Cochrane Health problem, urgency
Systematic Reviews of Promotion and Public • Importance to developing
Health Promotion and Health Field countries
Public Health Research • Avoidance of duplication
• Opportunity for action
Recommendations for Healthy People 2020 • The overall burden of the risk
the Framework and (HHS) factor or disease
Format of Healthy • The extent the burden may be
People 2010 preventable or reducible
• Cost-effectiveness of alternate
opportunities
• The net health benefit
• The synergy of different
interventions that target the same
disease
• The likely timeframe to observe
the impact
• The potential to reduce health
inequities among populations
• The willingness of public health,
private organizations, and other
collaborating entities to address
a particular health problem
and to accept accountability
for convening multisectoral
stakeholders to effect changes in
these areas.
SOURCES: Doyle et al. (2005); HHS (2000, 2008); NPP (2008); Oxman et al. (2006);
Whitlock et al. (2009).
Several groups used a two-step process to arrive at their ultimate list.
For example the AHRQ program first decided on broad priority areas and
then selected specific research questions that fit in those areas (Whitlock et
al., 2009). Similarly, the Cochrane Collaboration first chose a list of eight
broad priority topics and then formulated a longer list of more specific
review priorities (Doyle et al., 2005).
The committee incorporated the lessons learned by the IOM and ex-
ternal priority-setting projects whenever possible, including maintaining a
transparent and systematic process, involving stakeholder input, and using
many of their suggested criteria. The committee also recognized that each
OCR for page 90
0 INITIAL NATIONAL PRIORITIES FOR CER
priority-setting enterprise has its own unique needs, and so it developed its
own priority-setting process and criteria to meet the needs specified in the
ARRA legislation.
vOTING PROCEDuRES
The committee used three rounds of voting, illustrated in Figure 4-1,
to establish the CER priority topics listed in Chapter 5. Each round was
conducted using a web-based data entry system. The first round included
the 1,268 unique nominated research topics submitted by the public and
screened by the committee for nonresponsive and verbatim duplicates (as
discussed in Chapter 3). In this round, the committee was divided into five
groups and given 3 days to vote. Each group voted on the nominated re-
search topics categorized into several of the 32 unique research areas, with
each group voting on approximately 20 percent of the total nominated
research topics, and each committee member voting independently. This
design was intended to ensure that the leading nominated research topics
from each of the 32 research areas were likely to be retained in the next
round of voting, thus preserving the balance of the portfolio.
Committee members were given points equal to the number of nomi-
nated research topics assigned to their group. They were required to allocate
all of their points, and they could give as many as 10 percent of those points
to any one topic. This design allowed committee members to indicate the
strength of their preference for specific topics. The score for each nominated
topic was tallied as a percentage of allocatable points received, because the
five groups had an unequal distribution of topics and, thus, of points.
After the initial voting, raw data distributions of scores were reviewed
by the committee without knowledge of the topics or portfolio distribution.
Of the 1,268 that were voted on, 200 topics received at least 1 percent of
the available points and included 60 percent of the topics that received at
least one point by any committee member. The committee concluded that
the top 200 nominated topics represented a natural statistical break for the
second round of voting.
Three clinicians on the committee reviewed these 200 topics again for
duplications. Any topic that all three clinicians agreed was a duplicate was
removed from consideration for the second round of voting. Fifty-five top-
ics were consolidated in this process. The committee assessed the remaining
145 topics against the portfolio criteria and determined that the portfolio
was sufficiently balanced across research areas—only 3 of 32 designated
research areas were eliminated (see Table 5-1 in the next chapter).
In the second round of voting, committee members received the scores
from the first round of voting for each of the nominated topics, as well as
expanded information on research and funding gaps related to those topics.
OCR for page 91
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
Stakeholder questionnaire open to public March 6-27, 2009
2,606 recommended CER topics received from 1,758 respondents
Initial screen April 2-6, 2009 (removal of nonresponsive topics and verbatim duplicative topics)
Round 1 Voting = 1,268 Nominated Topics
Nominated topics voted on by five voting groups organized
by clinical areas April 10-13, 2009
Round 1 Results = 200 Nominated Topics
Substantive duplicate topics removed
Round 2 Voting = 145 Nominated Topics
Committee Deliberation (April 14-16, 2009)
In-depth discussion and review, consolidation of list to 129 topics,
discussion of overall portfolio considerations, and nomination of
26 new topics to fill gaps in research areas resulting in
Round 3 Voting = 155 Nominated Topics
Committee voted April 19-20, 2009
Round 3 Results = Final 100
Priority Topics
FIGuRE 4-1 Voting process and selection of priority topics.
Figure 4-1
R01511
vector, editable
OCR for page 92
INITIAL NATIONAL PRIORITIES FOR CER
Unlike the first round of voting, in the second round all committee members
voted on all 145 remaining nominated topics. Each committee member was
allocated 145 total points and could give a maximum of 14 points to any
individual topic.
At a 3-day retreat, the entire committee reviewed and discussed the
nominated topics in priority order following the second round voting. The
leading topics were discussed in detail to clarify the topics. The discussion
also allowed committee members to share their opinions and expertise
about the individual nominated topics. Through the discussion process, the
committee combined 16 of the 145 nominated topics and expanded several
of them beyond the scope of their original condition or population. For
example, topics addressing mental health issues or obesity in adults were
extended to include children and adolescents. Health care delivery topics
examining care of a single chronic disease were expanded to cover multiple
chronic diseases. In general, the committee decided that it was useful to
broaden the nominated research topics at this stage because agencies spend-
ing the CER funds will later be issuing their own more detailed requests
for proposals as part of the grant applications process, and researchers will
define the questions further when applying for those funds.
At the retreat, the committee also reached consensus on topics to fill
out or eliminate gaps in the portfolio representation. A total of 26 topics
were nominated by the committee. These topics were incorporated into the
129 remaining submitted topics without distinguishing them, providing
a total of 155 unique nominated research topics for consideration in the
third round of voting. Committee members were allocated 300 total points
for voting, with a maximum of 30 points allowed for any particular topic.
Web-based voting took place over a day and a half. The raw scores were
reviewed by the committee, and the distribution of the scores provided a
natural cutoff at 100 topics, 18 of which had been proposed by the com-
mittee. The top 100 topics all received a mean of at least 1.0 points. Topics
that fell below this threshold received zero scores from at least 60 percent
of the committee members. The final results of the voting process are pre-
sented in Chapter 5.
LESSONS LEARNED FROM THE CuRRENT PRIORITIzATION
PROCESS AND COMMITTEE RECOMMENDATIONS
The IOM committee developed several recommendations to set future
priorities based on the experience of this project.
Recommendation 1: Prioritization of CER topics should be a sustained
and continuous process, recognizing the dynamic state of disease, inter-
ventions, and public concern.
OCR for page 93
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
Health care is dynamic; new diseases and health needs can arise sud-
denly and other health problems might become insignificant when a treat-
ment is found. As new evidence is produced and gaps in evidence are
diminished, CER will need to go in new directions. New scientific tools
and techniques may open opportunities for CER where none previously
existed. A continuous process is necessary to update funding priorities as
conditions change and the impact of previous CER becomes evident. The
criteria used in the prioritization process may also need to be evaluated on
a regular basis. The prioritizing body may consider additional criteria, such
as evaluating levels of uncertainty and potential for future funding from
various stakeholders.
Recommendation 2: Public (including consumers, patients, and care-
givers) participation in the priority-setting process is imperative to
provide transparency in the process and input to delineating research
questions.
An ongoing process of CER prioritization will need to engage the
public more completely. Efficiency requires that representative stakeholder
perspectives be engaged at the most critical time points rather than every
possible step in the research process. With respect to prioritizing topics,
there are two key stages. The first is in setting criteria for choosing topics
and balancing the overall portfolio. While the committee has set forth cri-
teria here, these criteria should be revisited to ensure that they reflect the
public’s goals and values.
Recommendation 3: Consideration of CER topics requires the devel-
opment of robust, consistent topic briefs providing background infor-
mation, current practice, and research status of the condition and its
interventions.
“Topic nomination development” is the second critical stage in the
prioritization process for public input (Whitlock et al., 2009). Many po-
tentially important nominations delivered through the web-based question-
naire would have benefited from further development before voting so that
the voters had a better idea of what motivated the nomination and the
nominator and so that other contributors had a chance to expand on the
patients, interventions, comparators, and outcomes to be considered by
the research. In the future, the topic brief preparation process should be an
interactive one in which the prioritizing body gains the perspective of the
nominator and other stakeholders to better convey the context and main
points of the nomination to the voters.
The process should allow sufficient time to develop robust, consistent
OCR for page 94
INITIAL NATIONAL PRIORITIES FOR CER
topic briefs for use in voting in order to “level the playing field.” The
process should provide background information on the condition, address
current practice and policy, and document existing research in order to al-
low explicit consideration of each topic against pre-specified prioritization
criteria.
Recommendation 4: Regular reporting of the activities and recommen-
dations of the prioritizing body is necessary to evaluate the portfolio’s
distribution, its impact for discovery, and its translation into clinical care
in order to provide a process for continuous quality improvement.
The committee believes that increased transparency of the overall pro-
cess and documentation of decision making for each topic would allow
improved public participation and allow the public to revise and resubmit
rejected research suggestions for future consideration. This type of transpar-
ency is needed for any ongoing process to be responsive to public concerns
and interests and to enhance its legitimacy. Thus, the prioritization process
should produce regular reports evaluating its portfolio of potential and
selected topics for CER against a variety of criteria, including type of ser-
vice domain, clinical domain, population characteristics, and other policy
priorities such as addressing vulnerable populations and health disparities.
A rolling evaluation of the selection and prioritization processes, as well as
the return on investment of prior CER research by application throughout
the health system should be incorporated in the prioritization process to
ensure quality improvement. Ultimately, any prioritization process for a
CER Program will be evaluated by the impact of the funded research on
improving health decision making, health outcomes, and reducing unneces-
sary variation in health care.
REFERENCES
AHRQ (Agency for Healthcare Research and Quality). 2009. Medical Expenditure Panel
Survey. http://www.meps.ahrq.gov/mepsweb/ (accessed March 10, 2009).
Doyle, J., E. Waters, D. Yach, D. McQueen, A. De Francisco, T. Stewart, P. Reddy, A. M.
Gulmezoglu, G. Galea, and A. Portela. 2005. Global priority setting for Cochrane sys-
tematic reviews of health promotion and public health research. Journal of Epidemiology
and Community Health 59:193-197.
FAACT. 1997. The FACCT consumer information framework: Comparative information
for better health care decisions http://www.facct.org/information.html (no longer
accessible).
HHS (Department of Health and Human Services). 2000. Healthy People 00: Understand-
ing and improving health. U.S. Government Printing Office. http://purl.access.gpo.gov/
GPO/LPS4217 (accessed April 3, 2009).
OCR for page 95
THE CRITERIA AND PROCESS FOR SETTING PRIORITIES
———. 2008. Recommendations for the framework and format of Healthy People 2020.
http://www.healthypeople.gov/HP2020/advisory/PhaseI/sec5.htm#_Toc211942927 (ac-
cessed June 12, 2009).
IOM (Institute of Medicine). 2003. Priority areas for national action: Transforming health
care quality. Edited by K. Adams and J. Corrigan. Washington, DC: The National
Academies Press.
———. 2008. Knowing what works in health care: A roadmap for the nation. Edited by J.
Eden, B. Wheatley, B. J. McNeil and H. Sox. Washington, DC: The National Academies
Press.
Kung, H.-C., D. L. Hoyert, J. Xu, S. L. Murphy, and Division of Vital Statistics. 2008.
Deaths: Final data for 00. National vital statistics reports National Center for Health
Statistics.
Markle Foundation. 2008. FACCT legacy documents http://www.markle.org/archives/facct/
(accessed May 12, 2009).
NPP (National Priorities Partnership). 2008. National priorities and goals. Washington, DC:
National Quality Forum.
Oxman, A., H. Schunemann, and A. Fretheim. 2006. Improving the use of research evidence in
guideline development: 2. Priority setting. Health Research Policy and Systems 4(1):14.
Wennberg, J. E. 2009 (unpublished). Recommendations to the Institute of Medicine on com-
parative effectiveness research priorities. Submitted in response to a request from the
Institute of Medicine Committee on Comparative Effectiveness Research Prioritization.
The Dartmouth Institute for Health Policy and Clinical Practice.
Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2009. Identi-
fying, selecting, and refining topics for comparative effectiveness systematic reviews:
AHRQ and the effective health care program. http://effectivehealthcare.ahrq.gov/repFiles/
20090427IdenttifyingTopics.pdf (accessed June 5, 2009).
OCR for page 96
