1

Introduction

The medical research landscape in the United States is supported by a variety of organizations that spend billions of dollars in government and private funds each year to seek answers to complex medical and public health problems. The largest government funder is the National Institutes of Health (NIH), followed by the Department of Defense (DoD). Almost half of DoD’s medical research funding is administered by the Congressionally Directed Medical Research Programs (CDMRP) (Viergever and Hendriks, 2016).

In the 1993 Defense Appropriations Act, the U.S. Congress gave $210 million to the U.S. Army Medical Research and Development Command (USAMRDC; now the U.S. Army Medical Research and Materiel Command [USAMRMC]) to develop and implement a breast cancer research program. This appropriation was in response to a grassroots advocacy effort, coordinated by the National Breast Cancer Coalition, to address the growing need for research on breast cancer that would differ from that currently funded by NIH. In the 1992 Defense Appropriations Act, Congress had already given DoD $25 million for breast cancer research, although this health concern had not previously been considered to be particularly relevant to USAMRDC’s mission (IOM, 1993).

The 1993 legislation required that the new program have a mechanism for peer review of scientific merit (IOM, 1993, 1997, 2004). That single disease research program eventually evolved to become part of the DoD CDMRP, which now encompasses a broad spectrum of health conditions. The mission of CDMRP is to foster innovative approaches to

medical research in response to the needs of its stakeholders—the U.S. military, their families, the American public, and Congress. CDMRP funds medical research to be performed by other government and nongovernmental organizations, but it does not conduct research itself.

Since the initial appropriation of funds for breast cancer, Congress has called for the establishment of and has appropriated funds for many more medical research programs under CDMRP management. As of 2016, CDMRP has been responsible for the allocation of more than $9.7 billion in congressional appropriations across all of its research programs (CDMRP, 2015a) and more than $1 billion annually. For comparison, in 2016 alone, NIH received more than $31 billion (HHS, 2016). Congress provides specific funding for each CDMRP research program in the annual Defense Appropriations Act.

The major focus of CDMRP funded research is the improved prevention, diagnosis, and treatment of diseases, injuries, or conditions¹ that affect service members and their families, and the general public. The research programs are advocacy driven and they address a broad spectrum of health conditions from relatively uncommon ones such as Duchenne muscular dystrophy and neurofibromatosis to more common conditions such as breast cancer and prostate cancer. The CDMRP research programs also address a variety of aspects of these health conditions ranging from biological research to diagnosis, treatments, and caregiver needs. The hallmarks of CDMRP include reviewing applications for research funding using a two-tiered review process, and involving consumers throughout the process. CDMRP states that it “fills research gaps by funding high-impact, high-risk and high-gain projects that other agencies may not venture to fund” (CDMRP, 2016a).

In 2016, CDMRP listed 29 health conditions or areas of medical research for which Congress has appropriated funding. Some of the congressional appropriations for CDMRP have been targeted to medical conditions that predominantly impact the military population, such as the conditions addressed by the Joint Warfighter Medical Research Program, the Gulf War Illness Research Program, the Orthotics and Prosthetics Research Program, and the Military Burn Research Program. Research outcomes from programs that focus on health conditions resulting from military deployment, such as treatment of burns, improved prosthetics, and long-term care for multiple-system trauma, also can have applications to other affected populations, such as first responders and vehicle crash victims. Other health conditions in the CDMRP program portfolio

___________________

¹ The committee decided to use the term “health condition” in this report in order to succinctly represent the phrase “disease, injury, or condition” as used by CDMRP, but it acknowledges that there are important distinctions among the three categories.

include autism and various cancers (particularly breast cancer and prostate cancer) as well as relatively rare diseases such as amyotrophic lateral sclerosis, Duchenne muscular dystrophy, neurofibromatosis, and tuberous sclerosis complex (see Box 2-1 for a complete list of CDMRP research programs for 2016). CDMRP also manages core funds for several DoD military health research programs.

CDMRP receives thousands of funding applications each year. Since 1992, it has received more than 90,000 research applications and has awarded more than 12,000 grants or contracts (CDMRP, 2015a). “To ensure both scientific excellence and programmatic relevance, the CDMRP administers a two-tier review process, consisting of scientific peer review and programmatic review” (CDMRP, 2016g). CDMRP considers the first tier in the process to be peer review in which applications are evaluated on their scientific and technical merit. The second tier is programmatic review where applications are assessed for their relevance to programmatic goals. Both tiers of review are conducted by panels that are composed of scientists and clinicians from governmental and nongovernmental organizations (e.g., academia, pharmaceutical industry), consumers, and, for certain programs, active-duty service members. CDMRP defines consumers as “patients, survivors, family members, or caregivers of people living with a disease, injury, or condition, who serve as representatives of consumer advocacy, support, or military organizations” (CDMRP, 2015a). These consumers contribute as active participants in the review process. One of their roles is to keep the needs of patients and their caregivers at the forefront of the scientific discussions and to help remind scientists of the human dimensions of the health condition. CDMRP states that it strives for transparency in its review process by providing updated information on its public website, such as lists of funded applications with funding amounts, panel rosters, and research accomplishments, including publications and patents.

PRIOR REPORTS ON CDMRP

In response to the 1993 congressional appropriation for breast cancer research, the USAMRDC sought advice from the Institute of Medicine (IOM) on how to best manage a research program aimed at reducing the incidence of breast cancer, increasing survival rates, and improving the quality of life for those diagnosed with the disease. The IOM developed a funding strategy for the breast cancer appropriation and recommended that CDMRP adopt a scientific peer review process modeled on the one used by NIH (IOM, 1993).

The 1993 IOM committee further recommended that the USAMRDC adopt several programmatic goals to guide its research program. The

goals emphasized channeling research funds in directions that stimulated innovative ideas and scientific excellence, involved interdisciplinary research, and enhanced the use of existing research resources. The 1993 committee also recommended that the USAMRDC adopt a programmatic investment strategy (i.e., a plan on how the money would be awarded) that would focus on training and recruitment, infrastructure enhancement, and investigator-initiated research. The committee also suggested that those investment strategies be implemented through the use of a two-tiered peer review process to ensure scientific excellence and programmatic relevance. The two-tiered process would consist of study sections convened by USAMRDC to review applications for scientific and technical merit and a subsequent review of all scored applications by an “advisory council” (now called the programmatic panel) to assess programmatic relevance. USAMRDC put into place a review process modeled on the recommendations from that report.

In late 1995 the USAMRMC asked the IOM to review the implementation and progress of the Breast Cancer Research Program (BCRP) since 1993. Specifically, the IOM was asked to (1) review the portfolio of breast cancer research that had been funded by the Army’s BCRP as well as breast cancer research supported by other public and private funding agencies; (2) provide an analysis of the BCRP as it had been implemented in response to the 1993 IOM recommendations, in particular, program management and program achievement; and (3) provide recommendations for the identification of important but underfunded areas in breast cancer research that might be targeted by the program in the future. That committee found that for the two funding cycles (1993/1994 and 1995) for which data were available, the BCRP supported research that was not fundamentally different from that supported by other funding agencies such as NIH (IOM, 1997). At the time of the resulting report in 1997, there were no data on which to evaluate the outcomes of the BCRP-funded projects.

The 1997 committee noted that unique aspects of the USAMRMC program included the involvement of consumers at both levels of review—scientific merit review and programmatic review—and its ability to quickly change direction and goals on a year-by-year basis. Over the years, BCRP had made changes to the investment strategy and programmatic goals recommended in the 1993 IOM report. The new focus was on funding innovative ideas in the absence of preliminary supporting data and on supporting multidisciplinary research with translational potential. The 1997 committee found that this new approach had the potential for experimentation and innovation that exceeded what traditional organizations such as NIH were able to do. Nevertheless, the committee had concerns about the lack of an oversight structure and the fact that, given that the USAMRMC did not have in-house expertise in breast cancer

research, all the funding decisions were based on recommendations by a group of outside experts.

The 1993 and 1997 committees made several recommendations for the nascent BCRP (see Box 1-1), of which many, but not all, have been implemented by CDMRP over the years. Some of the early recommendations were directed at CDMRP administrative practices, including those of its support contractors, others focused on the structure and functions of the two review panels, and still others addressed the program’s investment strategies themselves. Importantly, both committees encouraged transparency in the program processes and recommended that scientific excellence should be the hallmark for the review and selection of research applications.

As the conflicts in Afghanistan and Iraq escalated during the early 2000s, given the competing demands on the military budget, Congress became concerned about funding increases for CDMRP. In 2004, at the request of Congress, DoD asked the IOM to assess the possibility of augmenting CDMRP funding from nonfederal sources. The subsequent report, Strategies to Leverage Research Funding: Guiding DOD’s Peer Reviewed Medical Research Programs (IOM, 2004), determined that it would be pos-

sible to leverage nonfederal funds from foundations and other private organizations, but only to a limited extent for certain types of research that are not the main focus of CDMRP. That 2004 IOM report has less relevance to the present committee’s task.

CHARGE TO THE 2016 COMMITTEE

Congress has encouraged coordination between DoD and NIH with regard to selecting, funding, and conducting scientific research. To determine if such coordination is taking place, the Senate Committee on Appropriations, in Senate Report 113-211 on July 17, 2014—which accompanied the Department of Defense Appropriations Act, 2015—directed DoD to contract with the National Academies of Sciences, Engineering, and Medicine for a study on CDMRP’s management of research. The National Academies’ committee’s formal statement of task is in Box 1-2.

In response to this task, the committee focused on the CDMRP processes used to select applications for funding across and within the research programs. In response to discussions with staff on the Senate Appropriations committee, the National Academies’ committee further refined its task to emphasize the evaluation of the review processes for those research programs that receive funding only from the DoD appropriations language. The review process for research programs that are administered by CDMRP on behalf of the DoD joint program committees (JPCs; see Chapter 2 for more information on those programs) was not considered in detail as those programs’ initial research prioritization and planning activities are different from those of the programs specified in the appropriation language. The committee was not asked to determine the value of the outcomes of the individual research programs, the effectiveness and impact of the applications that are funded, or the management of awards after the review process is complete. The committee also

notes that it did not review or evaluate the website or instructions for submitting applications via the electronic biomedical research application portal (eBRAP) system, as this system is used by other DoD health and research programs in addition to CDMRP.

COMMITTEE’S APPROACH TO ITS CHARGE

The National Academies’ committee was composed of experts in medical and scientific research, program and grant review, consumer advocacy, evidence-based medicine, and clinical trials; two members had previously served on CDMRP review panels. The committee’s information-gathering efforts consisted of open sessions to hear from DoD, CDMRP, and other interested parties; written questions to CDMRP on its processes and procedures; literature searches; and a solicitation of input. Each of these efforts and the limitations the committee encountered in its information gathering efforts are discussed briefly in the following sections.

Open Sessions

The committee began its task by holding two public sessions. At the first open session, the committee heard from representatives of CDMRP, including the director and the program managers for the Breast Cancer Research Program and the Psychological Health/Traumatic Brain Injury Research Program. These individuals provided an overview of the CDMRP review process and the specifics of their respective research programs. The committee also heard from representatives of the Department of Veterans Affairs (VA) Office of Research and Development and from a program director from the National Institute of Neurological Disorders and Stroke who serves on the CDMRP programmatic review panel for the Neurofibromatosis Research Program. These latter presentations focused on the interactions between the CDMRP research programs and NIH and VA. During the second open session, the committee arranged for presentations and panels to learn about

• liaison activities between CDMRP program managers and CDMRP programmatic panel members from NIH, VA, and advocacy organizations for the Spinal Cord Injury Research Program and the Amyotrophic Lateral Sclerosis Research Program;
• CDMRP’s approach for holding a stakeholders meeting for a new CDMRP research program, the Tick-Borne Disease Research Program;

• activities of current members of peer and programmatic panels for various research programs, such as the Gulf War Illness Research Program and Tuberous Sclerosis Complex Research Program; and
• the responsibilities of a former program manager for the Prostate Cancer Research Program who also spoke about the uniqueness of the program.

Information Requests for CDMRP

A major information-gathering activity for the committee was the generation of specific questions for CDMRP leadership and staff. At the request of the DoD contract officer representative, all committee questions were submitted to both the DoD contract officer representative and the director of CDMRP following the committee meetings and written responses were provided by the director of CDMRP or her designate. CDMRP also provided many of the documents requested by the committee that were not publicly available on the CDMRP website such as several vision setting booklets, one landscape document, one programmatic review guideline document, performance work statements for each of its contractors, and internal review process checklists. Other CDMRP documentation was available on its website (cdmrp.army.mil), such as annual reports, program announcements (current and archival), program summaries, lists of peer and programmatic reviewers, and a searchable database of funded awards and publications. All materials provided by CDMRP are included in the public access file for this study. However, in response to the committee’s request for CDMRP documentation such as standard operating procedures, guidance, handbooks, detailed job descriptions, templates, and checklists, that specified the functions or activities to be performed by CDMRP research program staff, particularly program managers and science officers, CDMRP replied that “the internal SOP/handbooks/checklists are not available for public dissemination” (Salzer, 2016b).

As discussed in greater detail later in the report, CDMRP relies on two support contractors to help manage each tier of its review process. However, in response to the committee’s request for contractor instructions that they use internally to identify peer reviewers and programmatic panel members, as well as peer reviewer or programmatic reviewer guidance, handbook(s), instructions, timelines, and access to the orientation webinar, CDMRP provided the following information:

Thus, the two support contractors declined to discuss their activities with the committee and did not supply any documentation regarding their role in the CDMRP review process. In response to the fourth set of committee questions, CDMRP asked its contractors for input on four questions regarding the selection of and qualifications for peer and programmatic scientist and consumer reviewers. The contractors’ responses to those specific questions were included with CDMRP staff responses to the committee’s questions and are cited in Salzer (2016d). Because the contractors’ responses were part of the full set of CDMRP responses, they were not individually cited.

Literature Searches

The committee conducted a literature search to identify publications that described or evaluated the CDMRP. The committee also assessed the peer review processes at NIH and VA. Both government agencies have extensive websites that explain their processes for applying for funding for scientific research and how those applications are reviewed. In addition, the committee also considered other governmental and nongovernmental organizations that fund medical research and how they go about establishing research priorities and reviewing and funding applications. The committee focused on organizations that might have overlap with research funded by CDMRP, such as the Centers for Disease Control and Prevention, the American Cancer Society, Susan G. Komen, the National Multiple Sclerosis Society, the Alzheimer’s Association, and the Brain and Behavior Research Foundation.

Solicitation of Input

The open sessions were very helpful in providing information to the committee on the responsibilities of CDMRP program managers and to a lesser extent, of programmatic panel members. However, the committee lacked an understanding of how both peer and programmatic reviewers saw their experiences with the CDMRP review process. To address this need, the committee decided to solicit input. The qualitative results of this solicitation of input are discussed throughout the report where appropriate.

The committee kept the input request brief and open ended in order

to give reviewers the opportunity to express both positive and negative impressions and to make any recommendations they thought would assist the committee in its task. Box 1-3 shows the input that was requested by the committee.

The committee chose to use the 2014 review cycle as the basis for the solicitation. Academies staff members compiled names of all the peer and programmatic reviewers who were listed on the CDMRP website as having served on either type of panel during that review cycle (accessed March 16, 2016). In total, 2,556 peer reviewers from 20 programs (7 programs did not list any peer reviewers) and 309 programmatic reviewers from 26 programs (1 program did not list any programmatic reviewers) were identified on the CDMRP website. Email addresses for the reviewers were obtained from Internet searches or in a few cases, by contacting the organization that the reviewer represented. In total, 2,356 reviewers (82%)—2,095 peer and 261 programmatic—were contacted via email and asked to provide open-ended information at a SurveyGizmo^® link.

The solicitation of input was posted for 3 weeks and a reminder email was sent to all reviewers 1 week before the solicitation was closed. Over that time, the committee received 652 unique, anonymous responses for a response rate of approximately 27% (all of these responses are available in the committee’s public access file). The committee notes that responses that were duplicative (based on Internet protocol address and time stamp) or that indicated a reviewer type but did not provide a written response to the committee’s questions, were not included in the public access file or counted toward the response rate. The breakdown of the responders by self-reported reviewer type and organizational affiliation is given in Table 1-1. (Note: Tables 3-1 and 3-2 in Chapter 3 show the organizational

TABLE 1-1 Results of Solicitation of Input by Reviewer Type and Organizational Affiliation

^a Served in both roles, but in different years.

^b Two or more of the categories.

^c Four incomplete responses were not assigned to any category.

affiliations of all peer reviewers and programmatic reviewers in 2014 based on the study staff’s assessment of their affiliations as reported on the CDMRP website; for this reason Table 1-1 cannot be compared with the tables in Chapter 3.) Reviewers may have served as both peer and programmatic reviewers, but not in the same year. Consumer reviewers were counted as representing advocacy organizations. By far, most respondents were peer reviewers from academic institutions.

Limitations of the Committee’s Information Gathering

The committee was constrained in its evaluation of CDMRP by the lack of cooperation on the part of the two CDMRP contractors that provide administrative support for the programmatic panels and peer review panels, respectively. As the contractors are responsible for selecting and obtaining all peer and many programmatic reviewers, for ensuring that the reviewers have no conflicts of interest, and for training and compensating the reviewers, this lack of cooperation made it difficult for the committee to assess some critical aspects of the review process. While the CDMRP website contains considerable information on many of CDMRP’s processes and requirements, specific information on the qualifications, selection, recruitment, and training of reviewers (particularly scientist reviewers) and topics such as conflict of interest and compensation were not available. CDMRP did provide its own general descriptions for many of these topics, although contractor handbooks, training materials, and other documentation were not available for review. Furthermore, neither the committee nor the National Academies staff observed any of the peer or programmatic review meetings, vision setting meetings, or stakeholders meetings.

The committee’s solicitation of input had a response rate of 27%. Because of the nature of the solicitation, no statistical analyses were performed on the responses, and thus all references to responses to the solicitation are qualitative and are included to illustrate or corroborate points the committee heard from other sources. Furthermore, the committee did not ask any of the reviewers whether they were funded by CDMRP at the time of the solicitation or at any other time, or had ever been an applicant to a CDMRP program. Finally, the committee did not seek out and talk with any applicants, whether funded or not funded.

ORGANIZATION OF THE REPORT

Chapter 2 of this report provides a general overview of the CDMRP review process and sets the stage for the later chapters which discuss each step in the review process in more detail. The roles and responsibilities

of CDMRP staff and contractors are described in Chapter 3. To the extent available, the qualifications, selection, training, and other administrative aspects of the peer and programmatic reviewers are also considered in this chapter. Chapters 4, 5, and 6 then discuss the procedures for and inputs and outputs of the CDMRP review process, beginning with vision setting in Chapter 4, moving to peer review in Chapter 5, and finishing with programmatic review in Chapter 6. In response to its task, in Chapter 7, the committee considers how other organizations, particularly NIH and VA, establish research priorities and how CDMRP might coordinate its funding initiatives and priorities with those and other governmental and nongovernmental medical research organizations. Chapter 8 summarizes the committee’s findings on the CDMRP review process and its coordination of research priorities with NIH and VA and makes recommendations for improving those activities. Brief biographical sketches of the committee members are provided in Appendix A and Appendix B contains the agendas from the committee’s two open sessions.