This chapter focuses on the processes that the Congressionally Directed Medical Research Programs (CDMRP) programmatic panels use in reviewing applications after the peer review (discussed in Chapter 5) is complete. The same members of the programmatic panel who started the review cycle and established the program’s vision (see Chapter 4), come together once again to carry out another tier of review and to recommend applications for funding (see Figure 6-1). Once the peer review for scientific and technical merit has been completed, the programmatic panel reviews all applications for programmatic relevance and portfolio balance.
After the full applications have been received in response to the invitation letters and the applications have been peer reviewed, a programmatic review meeting is held. Prior to this meeting, the program manager reviews and approves the peer review summaries and scores for the applications and sends them to the programmatic review support contractor. The program manager also completes a review of the administrative notes from the peer review meeting, provides input on ad hoc reviewer recruitment, manages potential conflicts of interest (COIs), and finishes any verification of application eligibility or other issues identified at peer review (Salzer, 2016b). The programmatic review contractor assigns each programmatic panel member to be a primary or secondary reviewer for
each application; each assignment is approved by the program manager. Both reviewers for a particular application make a preliminary determination prior to the plenary meeting on whether that application should be recommended for funding.
As with the vision setting process, before the programmatic review meeting, the program manager and the contractor may schedule a teleconference with the panel chair to discuss the programmatic processes, award mechanisms, and any other program specific issues, such as conflicts of interest. The program manager also prepares briefing slides for the meeting, which include the funds available for research, key elements of the award mechanism(s), the numbers of applications received per award mechanism, the investment strategy, and the order of review and processes (Salzer, 2016b).
Programmatic Review Criteria
All applications to a CDMRP research program undergo programmatic review on the basis of the criteria specified in the program announce-
ment. Each program announcement includes review criteria for both peer review and programmatic review. Programmatic reviews focus primarily on the following criteria:
- Peer review scores and summary evaluations of the applications, and
- Relevance to the mission of the Defense Health Program and the annual requirements for the specific research program, including the following subcriteria:
- Program portfolio composition,
- Relative impact and innovation,
- Adherence to the intent of the award mechanism, and
- Programmatic relevance in relation to the research program’s overarching challenges and focus areas (if applicable).
There may be additional topics to be considered for the subcriteria. For example, the Prostate Cancer Research Program’s population science impact award subcriterion “adherence to the intent of the award mechanism” has additional areas for consideration such as “Do the proposed outcomes of this project have high potential for substantial impact for men with or at risk for prostate cancer?” and “Is the proposed sample of sufficient size to demonstrate statistical significance, and is the statistical expertise of the study team clearly described and evident in the study plan?” For the programmatic review criteria, reviewers can make comments that are available to their fellow panelists, but the criteria do not receive scores as is done in the peer review process. Primary and secondary programmatic reviewers do, however, give each application a preliminary overall score on the basis of the criteria. Programs may use a programmatic review form for each award mechanism which is completed by reviewers prior to the meeting. However, the preliminary recommendations may be revised during the meeting in response to the discussions.
Some award mechanisms have a two-stage programmatic review process, such as the biomarker development award for the Prostate Cancer Research Program and the Breast Cancer Research Program’s era of hope scholar award. This process and the criteria are clearly described in the program announcement. In the first stage of this process, applications are rated against all or some of the criteria listed above. Principal investigators who were highly rated in Stage 1 are then asked in Stage 2 to make oral presentations to the programmatic panel. Stage 2 criteria, which include a list of standardized questions or topics to be addressed in the oral presentations, are more detailed and program-specific than are Stage 1 criteria. After the oral presentations, the principal investiga-
tors participate in a question-and-answer session with the programmatic panel members.
The committee notes that in response to its solicitation of input, some peer reviewers reported that they did not understand how peer review scores were used in programmatic review and that there appeared to be a “disconnect” and lack of transparency between peer review and programmatic review. The programmatic review guidelines for the 2014 Prostate Cancer Research Program do provide the basic scoring for peer review, but there is little information on what is meant by such category descriptors as “outstanding” or “fair,” nor is it clear what weight, if any, programmatic reviewers give to the ratings and evaluations of peer reviewers compared with the other programmatic criteria.
Programmatic review is almost always conducted via in-person meetings, although teleconferences may occasionally be used. Meetings begin with a discussion of possible or real COIs among the programmatic panel members. At the meeting, the panel is reminded that this second tier of review is not a re-evaluation of the scientific merit of the applications, as that was previously determined by the peer review panel. The CDMRP program manager answers any questions about the program review cycle, the investment strategy, the award mechanism, and applications and summary statements. In addition, the CDMRP science officer may attend the programmatic review meeting to answer questions about currently funded awards and to take notes to assist with the upcoming negotiations with principal investigators whose applications are recommended for funding (Salzer, 2016b).
Programmatic panel members begin their task with a review of the program announcements, the award mechanism summaries, and programmatic criteria—which may be included in the Programmatic Review Guidelines that were sent to panel members by the contractor prior to the meeting—and the peer review summary statements and scores. The discussion of the applications begins with those that have been assigned to primary and secondary reviewers. The program manager sets the “assigned for discussion” range for programmatic review based on the results of peer review scores, topic areas, focus areas, and other key elements specific to the program. The program manager may also designate applications that are assigned for review but are not assigned for panel discussion unless a programmatic panel member champions the application after reviewing it. These unassigned summary statements are available for review by the panel members prior to and during the meeting.
Any summary statement may be discussed at the meeting if championed by a panel member.
At each panel meeting, the chair asks the primary and secondary reviewers about their preliminary numeric recommendation (if used), funding decision, and comments (MOMRP, 2014). For each application, the primary reviewer leads the discussion of the strengths and weaknesses of that application; the secondary reviewer then comments, followed by a discussion with the entire panel. The chair then calls for a vote of “fund,” “not fund,” or “maybe” for each application. A majority vote is required to pass a motion; applications without a majority vote are revisited at the end of the session and discussed until a majority vote is reached.
After all assigned submissions have been discussed, the panel chair invites members to champion any applications that had lower peer review scores, but which may be programmatically relevant for another reason. The panel then prioritizes the applications for funding and the chair, with contractor support, makes a list of applications recommended for funding, and a list of alternate applications that might be considered for funding if money becomes available or applications on the recommended list are not funded for any reason. The chair then signs off on the recommended-for-funding list and the alternate funding list.
Although each application is initially judged on its own programmatic merit, the CDMRP Programmatic Review Guidelines state that this is a comparison-based review, that is, once each application has been discussed for its own merits, applications are then compared with each other across the pool of submissions to identify those that best meet the program needs (programmatic relevance and portfolio balance). In general, the recommendations of the programmatic panel are based on the ratings and evaluation of the peer review panels, the adherence of the application to the intent of the award mechanism, the programmatic relevance of the application, and the overall composition and balance of the program’s portfolio (Salzer, 2016a). Thus, the individual merits of each application are considered, but not all applications that receive the highest scores in peer review are necessarily recommended for funding, as is clearly stated in the program announcement. Reviewers can revise their scores for any application during the panel discussions. The committee has learned from some programmatic reviewers who responded to the solicitation of input that this is often a struggle for them, but it notes that this situation may be unavoidable, given the high level of expertise on the panel.
After the programmatic review meeting, the contractor provides the CDMRP program manager with a draft list of applications that have been
recommended for funding along with those to be placed on the alternate list and on the do-not-fund list, plus the notification letters for all three categories. The program manager evaluates the accuracy of the lists and checks final application budgets against the research funds that are available. The program manager then creates a decision briefing to accompany the lists of applications recommended for funding and the alternate list that is sent for review and concurrence by the Defense Health Agency and the U.S. Army Medical Research and Materiel Command commanding general. If applicable, the joint program committee may review and concur on the funding lists as well.
The notification letters for the applications are then approved by the U.S. Army Medical Research Acquisition Activity grants officer, after being reviewed by the program manager. Upon approval, the letters are posted on eBRAP, and the applicants are informed via email of their status. The program manager also sends the list of investigators recommended for funding and their organizations to the CDMRP website contractor for posting and verifies the draft post before it is finalized and made public.
In addition to the notification letter, all applicants receive a principal investigator information paper, which explains the different funding opportunities and provides information on the number of applications per opportunity, the funding rate, and how the two-tiered review process works. All applicants also receive a copy of their peer review summary statement, which contains a description of the application’s strengths and weakness. Some applicants may receive a “snippet” with their notification letter that explains the reasons for their funding status. For funded applications, the snippet may state an issue or a contingency for award, such as the inclusion of a statistical plan or a data monitor. For not-funded applications, particularly for those that received a high peer review score, the snippet may explain why the programmatic panel recommended that the application not be funded, such as it did not meet the intent of the award mechanism, there were concerns for portfolio balance, or the application did not meet programmatic relevance. The inclusion of a snippet is at the discretion of the program manager (Salzer, 2016c). CDMRP did not provide any examples of snippets to the committee. In response to the committee’s solicitation of input, peer reviewers responded that the lack of a summary statement from the programmatic review, in combination with the presence of the summary statement from peer review, could lead to misunderstandings and incorrect assumptions. The committee finds that this inconsistency and lack of transparency in who receives a snippet and under what circumstances to be confusing and unnecessary. It suggests that a summary statement from the programmatic review should be provided to all applicants and that it would be particularly helpful for
applicants who received high scores for scientific merit and impact but whose applications were not recommended for funding.
Lastly, after the meeting, the contractor and program manager assess aspects of the programmatic review that need improvement or that went well (Salzer, 2016b). The CDMRP program manager may also coordinate with the contractor on feedback regarding the programmatic panel members’ academic level or equivalent position, expertise, ability to communicate ideas and rationale, interaction in a group setting, and ability to debate and present an opposing view in a professional manner. CDMRP reports that there is no standardized format or criteria for evaluating panel members because of the varying nature of each program (Salzer, 2016c).
This page intentionally left blank.