Summary

Each year the United States spends billions of dollars on medical research. The largest government funder is the National Institutes of Health (NIH), which received more than $31 billion in 2016. The second largest funder is the Department of Defense (DoD) through its Congressionally Directed Medical Research Programs (CDMRP). In 2016, CDMRP received congressional appropriations of almost $1.5 billion to support 29 individual research programs ranging from breast cancer to military burns. Annual appropriations for most of the research programs have been relatively consistent since their inception, although the number of programs and the CDMRP overall budget have grown over the years.

The origins of CDMRP are in the 1993 Defense Appropriations Act, in which the U.S. Congress gave DoD $210 million to establish a breast cancer research program. Subsequently, DoD asked the Institute of Medicine (IOM) to advise it on the best approach for establishing and managing the new program. The IOM responded with guidance on a funding strategy for the program. It also recommended that CDMRP adopt a scientific peer review process modeled on the process used by NIH. In 1997, the IOM reviewed the DoD’s progress in implementing the 1993 recommendations and provided additional guidance on enhancing the program’s areas of research and its review processes. The current committee reviewed the conclusions and recommendations from both those earlier committees. It found that many of the earlier recommendations have been implemented by CDMRP, including the establishment of a two-tiered review process for scientific merit and for program relevance, the provision of peer review

summary statements to applicants, communicating to the scientific community about the role of consumer reviewers, and the inclusion of programmatic evaluation criteria in program announcements.

COMMITTEE’S STATEMENT OF TASK AND APPROACH

The CDMRP review process has not been assessed since the 1997 IOM report. Thus, in 2014, the Senate Committee on Appropriations, in Senate Report 113-211—which accompanied the Department of Defense Appropriations Act, 2015—directed the DoD to contract with the National Academies of Sciences, Engineering, and Medicine to conduct a study of CDMRP’s research management. The statement of task for the National Academies’ committee is in Box S-1.

The committee focused on the CDMRP processes used to select applications for funding by the research programs, specifically, those programs that receive funding only from appropriations to CDMRP. The review process for those research programs that are administered by CDMRP on behalf of the DoD Defense Health Agency’s Joint Program Committees was not considered in detail as they have different research prioritization, planning activities, and approval processes from those of other CDMRP programs. The committee was not asked to assess the outcomes, effectiveness, or impact of any research programs, or the management of awards after the application review process is complete.

The National Academies’ committee was composed of experts in medical and scientific research, program and grant review, consumer advocacy, evidence-based medicine, and clinical trials. The committee approached its task by gathering information via open sessions, from written questions to CDMRP on its processes and procedures, and from literature searches. It also conducted a solicitation of input from past

reviewers. And it heard from Senate Appropriations Committee staff about the congressional intent for the study.

Although CDMRP leadership and staff responded to the committee’s series of questions, the committee was constrained by a lack of some documentation from CDMRP staff and the almost complete lack of information from its two contractors, who provide administrative and managerial support for the programmatic and peer review panels. This was a substantive omission because the contractors are responsible for selecting, recruiting, training, and compensating peer and programmatic reviewers, and ensuring they have no conflicts of interest. In spite of this lack of information, the committee believed that it was able to judge the effectiveness of the CDMRP review processes.

THE CDMRP REVIEW PROCESS

CDMRP views itself as a leader in advancing medical and scientific research and filling research gaps “by funding high impact, high risk and high gain projects that other agencies may not venture to fund” (cdmrp.army.mil). It emphasizes that its 29 research programs (see Box S-2) and the projects they fund must be relevant to the health of members of the

military and their families. Each research program has its own specific vision and mission statement.

The CDMRP review cycle takes about 2 years from the annual appropriation of funds through award negotiation and implementation; however, the committee’s evaluation focused on the application review cycle, the left side in Figure S-1. All CDMRP research programs follow five major steps for soliciting, reviewing, and funding applications (in yellow in Figure S-1) as summarized below:

1. Vision Setting: Vision setting, conducted annually by the programmatic panel, occurs in the first 1–4 months of the program cycle to identify research gaps and define an investment strategy to address those gaps. The investment strategy identifies the award mechanisms to be used for each research program; the award mechanisms result in program announcements that drive the application process. If the appropriation is for a new research program, CDMRP holds a one-time stakeholders meeting before the first vision setting meeting.
2. Program Announcement Release: A program announcement is developed and released by CDMRP within 2–5 months of the vision setting meeting. Program announcements notify the research community that new funding opportunities are available from that program. They list the submission requirements as well as the review criteria that will be used for each application.
3. Pre-Application Screening and Invitation to Submit Full Application: Applications may be submitted to CDMRP only in response to a program announcement; no other applications are accepted. Most programs use a pre-application step to reduce the number of full applications to be reviewed. Pre-application requirements are specified in the program announcement and are typically submitted to and screened by the programmatic panel during months 5–8. Applicants who meet the pre-application criteria are invited to submit a full application.
4. Peer Review: Between months 7 and 11, full applications are received and undergo review for scientific and technical merit by the peer review panel; this is the first tier of the two-tier review process. The results of peer review (both numeric scores and narrative summaries of each application) are submitted to the programmatic panel.
5. Programmatic Review: Following peer review, applications are reviewed by the same programmatic panel as in steps 1 and 3 to determine programmatic relevance and portfolio balance, in addition to considering the scientific merit scores and summaries

from peer review. Programmatic review (months 9–14) serves as the second tier of the two-tier review process and results in a list of applications that are recommended for funding. Once the list of funding recommendations is completed, it is submitted to the U.S. Army Medical Research and Materiel Command or the Defense Health Agency for approval. Applications that are approved for funding move into the award negotiation and management phases.

PARTICIPANTS IN THE PROCESS

CDMRP uses an integrated program team to manage each research program and facilitate the communication and coordination of program activities. The integrated program team is led by the CDMRP program

manager and includes other CDMRP program and support staff as well as staff from both the programmatic and peer review contractors. Both contractors are tasked with identifying, recruiting, training, and compensating the scientist and consumer reviewers for their respective review panels, managing the meetings, and preparing pre- and post-meeting materials, such as minutes. The peer review contractor is also responsible for maintaining the electronic biomedical research application portal (eBRAP) for application submissions and notifications.

Each research program has one programmatic panel and at least one peer review panel. Both types of panels are composed of scientist and consumer reviewers, and all panels have a chair. Programmatic and peer review panels may also include ad hoc and specialty reviewers, depending on the award mechanisms (e.g., clinical trials) and the number of applications to be reviewed. In 2014, a total of 309 programmatic reviewers and 3,195 peer reviewers participated in CDMRP’s review process.

The names and affiliations of programmatic and peer reviewers for all research programs are publicly available on the CDMRP website. The website also provides information on how to become a consumer reviewer. The minimum qualifications to serve as a scientist reviewer are not publicly available.

Programmatic panel members typically serve 2-year terms and panel chairs typically serve for 1–3 years, but there is no limit on the number of terms that members or chairs can serve or on the consecutive terms they can serve. The committee appreciates that for some health conditions there may be relatively few organizations that are able to provide consumer representatives for programmatic panels, but it encourages CDMRP to strive to recruit more consumer reviewers from a variety of organizations in order to improve turnover and expand participation on the panels.

Unlike the programmatic panels, peer reviewers are recruited to serve a 1-year term, but they too may serve for more than one term. The committee commends CDMRP’s effort to recruit a high percentage of new reviewers each year (40–60% for both scientist and consumer reviewers) while maintaining some continuity in panel membership.

Although the contractors own the training materials for programmatic and peer reviewers, CDMRP does approve them, and CDMRP program managers participate in the training process. Conflict of interest information for applicants is included in program announcements and the contractors provide conflict of interest training to recruited reviewers. The contractors also check for conflicts of interest between reviewers and the applications they are to review, but no information on conflict of interest is available on the CDMRP website.

CONSUMER ENGAGEMENT

Consumers are engaged at all levels of the CDMRP process, from obtaining appropriations from Congress for new and existing programs and establishing a vision for a program to reviewing applications for scientific merit and programmatic relevance. This level of consumer engagement is distinctive among government research funding agencies.

CDMRP uses mentors for consumer reviewers on peer review panels to supplement the reviewers’ training, explain the review process to them, and assist them with their initial scoring and critiquing of applications. The committee finds this to be a good approach for new consumer peer reviewers, but it notes that mentors are not provided to either consumer reviewers on programmatic panels or to scientist reviewers on either panel. The committee suggests that CDMRP consider offering mentors to these other reviewers.

On the basis of information heard at its public sessions and from its solicitation of input, the committee concludes that the inclusion of consumers in both tiers of the review is a positive aspect of the CDMRP review process that can benefit scientists and consumers alike.

FINDINGS AND RECOMMENDATIONS

For some health conditions, such as breast cancer and neurofibromatosis, CDMRP has been a substantial funder of research for many years. CDMRP also has dedicated funding for health conditions that primarily affect service members, such as military burns and orthotics and prosthetics.

On the basis of information from a variety of sources, the committee concludes that, in general, the CDMRP review process is effective in dispensing research funding across its programs and is not in need of extensive revisions. However, the committee did identify four interconnected areas where it hopes its recommendations will improve the CDMRP review process. Specifically, these areas are

• the development of a strategic plan for each research program,
• more formal coordination between CDMRP and NIH and VA,
• greater transparency of the CDMRP review process, and
• improved standardization of CDMRP’s business practices.

Strategic Plans

All CDMRP research programs receive 1-year funding, with no guarantee that Congress will appropriate money for any program in any given year. The committee heard from CDMRP leadership and several stake-

holders that this funding uncertainty precludes developing a long-term strategic plan. However, the committee finds that most of the research programs, once established, continue to be funded each year. This consistent funding supports the need for a strategic plan for each program that establishes research goals and promotes coordinated research efforts with other organizations. A strategic plan should identify and evaluate programmatic research priorities for the next 3–5 years; specify the research initiatives, including award mechanisms, that are expected to achieve those long-term goals; and provide flexible approaches to build on past successes and address failures and gaps. The plan should also describe the resources (e.g., human, institutional, technological, and financial) needed to implement the initiatives. The committee understands that having 1-year funding makes it difficult to guarantee that long-term goals can be achieved; however, the lack of a strategic plan means that each year a research program must establish its priorities anew, making it difficult to track program progress, and develop strengths and expertise placing increased reliance on the institutional memory of the long-serving programmatic panel members. Furthermore, the committee does not consider a research program’s vision setting booklet or landscape document to be equivalent to a strategic plan.

The committee recognizes that developing a strategic plan is not a simple task. To leverage resources, a stakeholders meeting (discussed later) held every 3–5 years that includes scientific experts and consumers from a variety of governmental and nongovernmental organizations could develop the initial framework for a strategic plan, with subsequent stakeholders meetings to update it as necessary.

Recommendation: Each CDMRP research program should develop a strategic plan that identifies and evaluates research foci, benchmarks for success, and investment opportunities for 3–5 years into the future. The plan should be re-evaluated and updated as necessary at the end of that interval. Each strategic plan should specify the mission of the program, coordination activities with other organizations, research priorities, how those priorities will be addressed by future award mechanisms, how research outcomes will be tracked, and how the outcomes will inform future research initiatives.

Coordination of Research Priorities

The committee was tasked by Congress with evaluating CDMRP’s coordination of research priorities with NIH and VA. The coordination of medical research priorities and funding among federal agencies and

with nongovernmental agencies can help ensure that research dollars are going to fund critical research within an organization’s area of expertise or focus, reduce administrative costs, and avoid unnecessary duplicative research. CDMRP has no legislative mandate to coordinate its research priorities or funding with other governmental or nongovernmental organizations to meet these goals, but it does so informally and to a varying extent across individual research programs. The committee finds that coordination is best accomplished when all the involved organizations work together. In other words, although CDMRP may attempt to coordinate its research priorities with NIH and VA, those organizations must also be willing to coordinate with CDMRP, and currently there is no requirement or incentive for them to do so. CDMRP’s coordination efforts generally consist of (1) informal discussions between program managers and scientific colleagues at NIH and VA, (2) having representatives from NIH and VA and other interested organizations on CDMRP programmatic panels, (3) presentations from NIH and VA researchers at the annual vision setting meetings, and (4) participation of the CDMRP program manager on a variety of interagency groups with other governmental and nongovernmental members.

During the vision setting and programmatic review processes, the programmatic panel assesses what other organizations are currently funding and the potential for CDMRP to fund duplicative research. This assessment is possible, in part, by having representatives of those organizations (e.g., NIH, VA, American Cancer Society) participate as programmatic panel members; however, the committee finds that some programmatic panels have no representation from organizations that the committee believes would have valuable input for that program, such as a VA representative on the panel for the Prostate Cancer Research Program. Representatives from other major funders for a specific health condition, whether represented on the panel or not, may be asked to give briefings at the vision setting meeting. While these activities are helpful, the committee finds that the participation of CDMRP program managers in research reviews at nongovernmental organizations, if appropriate, might strengthen the CDMRP knowledge base of research funding and current or anticipated activities by both organizations.

The committee finds that one approach to systematically informing programmatic panels about research funded by other organizations is to require that vision setting booklets include a section on research initiatives from other major organizations as well as information on the sources (e.g., people, databases) that were consulted in developing the booklet. Publishing the vision setting booklets on the CDMRP website prior to the meeting would give the knowledgeable public an opportunity to provide input to the program as well.

The CDMRP Procedures to Avoid Research Duplication document informs applicants, awardees, other funding agencies, and the public about how CDMRP identifies and avoids funding duplicative research. There are several points throughout the CDMRP program cycle where duplication of research is checked using the NIH RePORTER database, including at application submission, peer review, programmatic review, and the negotiation and monitoring of funded awards. Additionally, principal investigators and all key personnel are required to submit a comprehensive list of previous, current, and pending funding support. All CDMRP awards are also entered into the Federal RePORTER database and are available for public scrutiny. The committee notes that unless these databases are kept up to date by the respective users, it will be difficult to identify duplicative research.

The committee recognizes that coordination efforts are not cost neutral. Additional CDMRP staff and contractor time and expertise may be required to implement these efforts, although the committee notes that some CDMRP research programs already undertake some of these efforts, such as consistently having representatives from NIH or VA on their programmatic panels. However, the development of a long-term strategic plan for each CDMRP program that includes input from other major funding organizations could reduce the risk of funding duplicative research and enhance opportunities for identifying complementary or collaborative research.

Recommendation: Where there is a commonality in substantial research efforts by other organizations, whether federal or nongovernmental, CDMRP should have a formal mechanism to coordinate with these entities in a predictable, consistent, and standardized manner each year to learn of substantial or new areas of research on the health condition being funded or considered for funding by those other organizations.

Transparency

CDMRP has made many aspects of its review process publicly available on its website (cdmrp.army.mil). For example, the website describes how to become a consumer reviewer, lists available funding opportunities for all the research programs, identifies peer and programmatic reviewers and applicants who have been recommended for funding, and publishes many program documents, including annual reports and research highlights.

There are four notable areas of the CDMRP review process, however, where transparency is lacking: stakeholders meetings, contractor support

activities and policies, the use of ad hoc and specialty reviewers, and feedback from programmatic reviewers. The committee concludes that improving transparency in these areas would help engage the best scientists, researchers, and consumers and improve CDMRP’s selection process for reviewers, and thus the applications that those reviewers recommend for funding.

Stakeholders Meetings

The committee finds that the CDMRP process for identifying stakeholders for a particular health condition is not subject to public input. Stakeholders meetings are not publicly announced either before or after they take place; the public is not invited to attend the meeting or to submit information; and the outcomes of the stakeholders meeting are not made public at any time. Although the CDMRP program managers stated that they try to obtain input from all relevant stakeholders, the committee does not see the need to limit public access to the stakeholders meeting to only invited attendees, as that may preclude input from interested parties, particularly those with singular or dissenting viewpoints.

The committee finds that there are a variety of ways to obtain public input for stakeholders meetings such as announcing meetings on the CDMRP website, allowing Web-based participation, and encouraging electronic input from stakeholders. Having a periodic stakeholders meeting would also allow new information and stakeholders to be included in the process and provide an opportunity to refocus the program as necessary.

Recommendation: Stakeholders meetings for each research program should include an opportunity for public engagement prior to, during, and after the meeting, using a variety of mechanisms (e.g., Web-based). Furthermore, such meetings should be held about every 3–5 years, and notices about such meetings should be broadly announced in advance.

Contractor Support Activities and Policies

The committee understands the need for external support contractors to assist CDMRP program managers with many peer review and programmatic review activities. However, the committee has concerns about the contractors’ lack of transparency and the unavailability of their materials for review. First, scientists are recruited by the contractors, but the minimum qualifications to serve are not publicly available.

Second, by not owning its training materials or the peer and program-

matic reviewer survey forms and evaluation results, CDMRP has ceded these key responsibilities to its contractors. The criteria for reviewer conflicts of interest are also owned by the contractors and are also not available publicly. Although CDMRP reviews and approves the training, conflict of interest criteria, and the reviewer evaluation materials, these are all claimed as business products of the contractors and thus do not appear to be subject to any external review.

The committee believes that CDMRP would benefit in several ways if it increased the transparency of its review process. By making information such as conflict of interest requirements, the minimum qualifications to serve as a scientific reviewer, and compensation policies publicly available, CDMRP would be better aligned with other leading research organizations such as NIH, and would demonstrate its efforts to comply with the President’s call for more transparency and public participation in federal government operations. Finally, by owning its training and other materials, CDMRP could maintain continuity in its processes should contractors change over time.

Recommendation: To improve its transparency and business practices, CDMRP should determine how best to obtain the training and evaluation materials used by its contractors so that the materials may be periodically reviewed and revised as needed. Furthermore, at a minimum, CDMRP should establish and make publicly available its qualification criteria for serving as a scientist reviewer—both peer and programmatic—and its conflict of interest policies.

Use of Ad Hoc and Specialty Reviewers

The committee appreciates the need for ad hoc and specialty reviewers to complete the review process in a timely and effective manner. However, the committee found only a few programs on the CDMRP website that identified these additional reviewers.

As ad hoc and specialty peer reviewers may provide an overall score for an application and, thus, have a substantial impact on its rating, the committee finds the lack of acknowledgement of these reviewers to be puzzling. Providing the names of the ad hoc reviewers and more information on the CDMRP website on how and when ad hoc and specialty reviewers are used and selected would increase public confidence in the uniformity of reporting reviewers in the peer and programmatic review process.

Recommendation: CDMRP should consistently and publicly identify ad hoc and specialty reviewers as is currently done for peer and programmatic reviewers.

Feedback from Programmatic Reviewers

Applicants do not receive any feedback on their pre-applications if they are not invited to submit a full application. The committee suggests that providing applicants with a standardized statement as to why the programmatic panel rejected their pre-application would improve the transparency of the program. Applicants who submit a full application receive the scores and summary statements from peer review, and the committee finds this to be an appropriate level of feedback for that review step.

However, the committee is concerned about the lack of feedback to applicants from the programmatic review step. CDMRP has stated that some applicants may receive brief “snippets” after programmatic review, along with the results of the peer review, that indicate why they were not recommended for funding, but not all applicants receive snippets, and there do not appear to be any criteria as to who receives them and what information they contain.

To address this lack of transparency in programmatic review, the committee suggests that after the programmatic review is complete and applicants have been notified, the list of applications recommended for funding be available electronically, such as through eBRAP, to the peer reviewers who reviewed those applications. The committee also finds that improved feedback from programmatic reviews to applicants would assist them in revising their applications for possible resubmission, if appropriate.

Recommendation: Applicants whose pre-applications are rejected should receive a standardized statement explaining why. A programmatic review summary that includes more than simply the panel’s funding decision should be provided to applicants along with the peer review scores and summary statements. Furthermore, peer reviewers should be informed of which applications were recommended for funding by the programmatic panel.

Standardization

CDMRP has made an attempt to standardize many of its processes across its research programs, such as introducing the use of consistent

terminology and formats for program announcements, standardized application submission instructions, and a searchable database of funded applicants. The committee recognizes that standardizing processes across programs that vary so widely in terms of funding and goals is not an easy task. Nevertheless, it finds two areas—scoring criteria and term limits—where CDMRP might improve transparency and standardization.

Scoring Criteria

Each CDMRP program announcement contains pre-application screening criteria, the numeric evaluation criteria for peer review, and the numeric or narrative criteria to be used for programmatic review. The variability in the scales used to rate applications makes it difficult to generalize across programmatic ratings and contributes to concerns about the transparency of the programmatic review process.

Other organizations that review research applications have developed scoring systems that might be considered by CDMRP, such as the 9-point rating scale used by NIH for both criterion scores and the overall score. The committee finds that these alternate scoring approaches might be preferred by the peer and programmatic reviewers as well as by the applicants. Furthermore, the use of a wider and whole-number scale might prevent some of the bunching of scores that sometimes occurs with the current CDMRP scales.

Recommendation: CDMRP should consider updating and standardizing its scoring system to reflect current review practices and to reduce confusion among reviewers and applicants.

Term Limits

CDMRP does not impose term limits on members of either peer review or programmatic panels, however, the committee suggests that CDMRP consider standardizing the terms of service for panels. For example, reviewers might serve for a specific term length (e.g., 1–3 years) with multiple, but finite, renewals. This call for term limits does not mean that other representatives of the same organization may not participate on the panel. The committee recognizes that long-serving panel members provide institutional memory, but it also finds that new perspectives, knowledge, and diversity would improve the review process.

Recommendation: CDMRP should have standardized limits on both the terms of service and the number of consecutive terms

that peer and programmatic panel members, including chairs, may serve.

CONCLUSIONS

CDMRP is a well-established medical research funding organization, covering many health conditions of concern to members of the military and veterans, their families, and the general public. Although it has grown substantially in terms of the breadth and complexity of its research programs, its management practices do not appear to have evolved to keep pace with this growth. In general, the committee found the CDMRP processes for reviewing and selecting applications for funding to be effective in allocating funding for each research program, and not dissimilar to those used by NIH and other research organizations. However, the committee also found that several improvements to the CDMRP review process—the development of strategic plans, enhanced coordination with other organizations, better transparency, and more standardization—would help align CDMRP with NIH and similar organizations. The committee’s recommendations are interconnected—that is, improving standardization can also result in better coordination and transparency, and a strategic plan will encourage coordination and transparency.

Although it is worthwhile to periodically review any research program to ensure that current best practices are being used and to identify any needed updates or gaps, the committee emphasizes that there has been no known concerted effort by an external entity to determine whether the research funded by the various CDMRP research programs have in fact helped them accomplish their individual missions. Thus, perhaps the more fundamental question remains unanswered: Has CDMRP, even with a well-conducted review process, produced the innovative and impactful research it strives to fund?

This page intentionally left blank.