A Review of the Department of Defense's Program for Breast Cancer Research (1997)

The 1993 programmatic vision of the DOD BCRP was to ''provide investigators the opportunity to explore new approaches to understanding breast cancer and relieving or eliminating its toll on individuals and their families" (IOM, 1993). As discussed in Chapter 4, over time the IP (Integration Panel) has refined this vision to support investigations that promise dramatic breakthroughs that could lead to breast cancer eradication. While the current vision emphasizes higher-risk research, funded proposals must nonetheless demonstrate solid scientific judgment. The committee believes that this evolution remains consistent with the IOM's original vision, and commends the Army and the IP for the initiative in refining the vision.

The BCRP is unique among breast cancer funding sources. It includes participation of consumer representatives on peer review panels at both levels of grant application review while a flexible management framework allows relatively quick changes in direction. These unique features have positive aspects because they connect the BCRP with highly interested constituents and provide great opportunity to respond to new research breakthroughs. On the other hand, this funding flexibility necessitates strong strategic planning and program oversight and evaluation capabilities. The committee is concerned that such capabilities have not been demonstrated.

Also problematic are the requirements imposed on investigators by the DOD's regulations for applications and grants management. Voluminous documentation is required for human use, animal use, hazardous materials use,

and environmental and safety analysis before an award can be made. The committee applauds the Army for being responsive and streamlining these application appendix items to some extent.

The BCRP is organized as described in Chapter 4. The U.S. Army Medical Research and Materiel Command (USAMRMC) Program Management Team (PMT) currently contracts with two organizations, Science Applications International Corporation (SAIC), which organizes and provides staff support for the activities of the IP, and United Information Systems, Inc. (UIS), which coordinates and supports the scientific peer review process. SAIC and UIS leadership work on the Fort Detrick, Maryland, campus in close proximity to the PMT. The PMT performs grants management in-house with some assistance from SAIC. This organizational structure acknowledges the Army's limited expertise in managing scientific peer review of competitive grants programs in areas not directly relevant to the military mission.

The PMT is to be commended for devising a structure that appears to be working well overall while keeping overhead costs under 10%. Nevertheless, this structure has room for improvement. For instance, the decision-making process is not clear to the public, and the lines of communication between grant applicants and BCRP organizational components are generally cumbersome.

As Chapter 4 describes, USAMRMC contracted in FY 1993/1994 with the American Institute of Biological Sciences (AIBS) to conduct the first-tier peer review (the Army had used this contractor for a similar but smaller task in its FY 1992 breast cancer program). After completion of the FY 1993/1994 award process, the peer review contract was rebid. UIS was the successful bidder. The committee commends the IP and PMT for identifying obstacles and difficulties in the initial round of the peer review process and acting quickly to correct the problems.

The IP took its original direction from the 1993 IOM report and subsequently created a charter describing its official designation, objective and scope, purpose, duration of terms, tasks and duties, panel composition, conditions of panel appointment, method of selection of the chair, executive committee and subcommittees, recommendations, and the types and due dates of reports.

The committee believes that the IP represents a new and imaginative concept in planning and monitoring a research grants program. The committee regards the IP as unique because it functions as both a second-tier review panel and as a council in the NIH model, serves as a subcontractor to an administrative services contractor to the USAMRMC, and reports to largely nonscientific administrators within the Army. The committee judged the IP to be highly effective in performing all functions envisioned for the advisory council by the 1993 IOM committee. This success appears to be a result of the remarkable dedication and high quality of the members of the IP and their widely diverse expertise. The committee has concerns, however, about whether the Army will be able to

continue to recruit individuals with the expertise, time, and level of commitment necessary to sustain the current level of responsibility assumed by the IP.

As the Army has recognized, the operations and outcome of the BCRP should be monitored for quality and productivity. Periodic reviews are one mechanism by which this can be accomplished. The committee has concerns, however, that outside review on an ad hoc basis may prove to be insufficient in the future. If the BCRP evolves into an ongoing program with stable funding, then consideration should be given to establishing a permanent advisory committee to the Army, independent of the IP and contractors, to assure that the program continues to function at its current high level.

Funds for breast cancer research were originally requested by the National Breast Cancer Coalition (NBCC) from nonmilitary research funding sources. Thus, the large appropriation in the DOD budget was not expected. Congress, scientists, and the public were unsure as to whether this was a "one-time" program or whether it would be continued in following years. Each year since that time, the NBCC has lobbied Congress successfully for funds to maintain a significant breast cancer research program. Because of the necessity for annual congressional approval, however, ensuring the long-term stability of the program requires considerable effort each year on the part of the advocates. Furthermore, such uncertainty seriously compromises any long-term planning efforts. Congressional earmarks for specific breast cancer topic areas (e.g., imaging methods and cancer centers) and appropriations for other nonmilitary medical research (such as osteoporosis, neurofibromatosis, and prostate cancer) all threaten the stability of the BCRP.

The considerable delays that have occurred between congressional action and release of the funds to USAMRMC have resulted in short time periods between release of the BAAs and the deadlines for application submission. This short interval severely handicaps the PMT, IP, SAIC, UIS, peer reviewers, and, perhaps most importantly, the scientists preparing applications. For example, investigators responding to the "Dear Colleague" letter sent out by this committee (see Appendix C) indicated that there might be more and better applications if there were more time between the publication of the BAA and the submission date. The extraordinary delay in transferring funds from the USAMRMC to the institutions of the investigators whose projects have been selected for funding are also of great concern. for the program to be effective in recruiting and selecting the most meritorious and innovative research proposals, the appropriated funds should be made available in a more timely manner, and sufficient staff and other resources dedicated to this purpose.

The BAAs have described the program goals and application process and requirements, as well as evaluation criteria, in sufficient detail. The committee believes that more applications in the areas requiring multidisciplinary teams—such as clinical research, epidemiology, and psychological, social, and health services—could be elicited if more time were allowed for preparation and there were a mechanism for resubmission of unfunded grant proposals. In addition, some respondents to the "Dear Colleague" letter found compliance with the budgetary and regulatory procedures unnecessarily cumbersome.

As contractor for the first-tier peer review process, UIS has produced several documents useful in the peer review process, including the USAMRMC Procedural Manual for Executive Secretaries (UIS, 1996). Setting up the process involved communication among the PMT, IP, and UIS. The IP charter lists as one of the panel's tasks "quality control recommendations and advice and guidelines on selection and implementation of the Peer Review Panels" (USAMRMC, 1996c). Thus, the IP was kept informed by the PMT of the members and chairs selected for the panels and provided feedback. The committee agrees it is important that oversight is built into the process of selection of the peer review panels, and that the IP should not have day-to-day oversight of this process. Their general oversight is beneficial, however, given the lack of scientific expertise in the infrastructure of the BCRP. As the charter is currently written, it is not within the purview of the IP to be involved in the selection of the executive secretaries or members of the core directorate. Since the core directorate selects the executive secretaries, who in turn recruit the panel chairs and reviewers, this group is key to the success of the peer review program and requires oversight at the highest level of the PMT.

As noted earlier, the time constraints under which the program has operated have impeded the optimal design of review panels and reviewer recruitment. All review panels were ad hoc as a result of the year-by-year nature of the program, and some panels had to be formed before applications were received by the PMT. This situation could be vastly improved if standing review panels were created. The advantages of standing review panels include greater reviewer familiarity with the procedures and aims of the program and members of such panels can provide historical perspective and continuity in judging applications in relation to the quality and content of prior proposals.

In the early years of this new program, the IP was troubled that technical merit scores were uneven across review groups. To alleviate this problem, in FY 1996 the IP provided an orientation to its program goals for review panel

members. This was necessary because the IP instructed reviewers on how to score proposals for nontraditional criteria, including innovation, novel ideas, and gain-versus-risk potential. These steps may improve the quality of peer review and increase consistent scoring within a panel, but scoring differences among review panels are well known and certain mechanisms (including assignment of percentile ranks) have evolved to deal with them.

The first tier of peer review, nonetheless, received high marks and was compared favorably to the scientific peer review conducted by other large agencies by most of the grantees who responded to the "Dear Colleague" letter and by members of the IP who provided testimony to the committee.

A few executive secretaries testified that, in their opinion, communications between branches of the review infrastructure were inadequate, and that program goals were not always made explicit. In addition, names of previous grantees were not made available in a timely fashion for possible recruitment to panels. The value of technical writers was also questioned by some. To improve lines of communication, the PMT established an Executive Secretary Liaison Subcommittee of the IP to develop an orientation on the new program vision for executive secretaries and other peer review participants in order to foster a shared program vision between peer and programmatic review agencies (USAMRMC, 1996c). This group has met at least three times in 1996. The committee noted that no mechanism appears to be in place to formally evaluate the executive secretaries themselves.

SAIC has the responsibility for assembling nominations for the IP and participating in members' selection as well as organizing the IP meetings, including recording the minutes. The committee reviewed the IP composition over its three-year existence and commends SAIC for the quality of scientists and consumers comprising the IP. The committee found the IP meetings are documented in detail, with SAIC producing the following:

• USAMRMC Orientation Handbook for New Integration Panel Members (SAIC, 1996), and

• DOD Breast Cancer Program "At-a-Glance Synopsis" slide show and talking points provided to IP members as a guide for outside lectures (USAMRMC, 1996c).

The IP, as a subcontractor to SAIC, performs the role envisioned in the 1993 IOM report for the advisory committee. It provides multiple critical functions in the areas of advisory input and programmatic review. The major

tasks and duties of the IP, as enumerated in its revised 1996 charter are listed in Chapter 4.

While the PMT has final decision authority, the IP wields considerable power in determining the direction of the program. The IP refines program focus and investment strategy, makes funding decisions on individual applications, and carries out oversight of the entire program. The 24-member IP includes internationally recognized leaders in their fields—investigators in basic and clinical sciences, physicians, epidemiologists, health care delivery specialists, and three to four consumer activists or other concerned laypersons knowledgeable in breast cancer issues. The qualifications of the individuals on the current IP are very impressive and the IP members appear to be intensely committed to their tasks. The IP is directed by an executive committee consisting of the chair, the chair-elect, the chair emeritus, other members, and a consumer. This composition appears to ensure continuity in leadership and is considered an asset to the program. The committee does not recognize any potential benefit of increasing the number of consumers on the IP to more than four. The 1996 amended IP charter specifies "at least three or four" (USAMRMC, 1996c).

Funding recommendations are made to the full IP by discipline-oriented subgroups that consist of six to eight members and at least one consumer.

The IP's reliance on the priority scores assigned by the scientific review panels should ensure that the outstanding applications will receive funding. In the less-than-outstanding range, because of the lack of standardization across study sections (all of which operate on an ad hoc basis), percentile scores are of limited usefulness for ranking proposals across different study sections. This process still leaves room for potential inconsistencies. Some applicants whose proposals were not funded despite a high ranking were disgruntled because they had not been informed about the criteria used by the IP for establishing priorities for funding, and they viewed the process as an arbitrary rather than objective one (Wadman, 1996).

Funding limitations set for each subgroup are based on the investment strategy developed by the IP in advance of the meeting. For FY 1993/1994, the 1993 IOM report-recommended allocations were followed. The number of applications received in each category has been the starting point for developing these allocations. The strategy is finalized at the IP meetings when funding decisions are made. Although the IP (and its subgroups) agree that their function is not to carry out scientific and technical review, the exact criteria by which these IP subgroups make their funding recommendations that are not in strict priority score order were not clearly spelled out in the FY 1993 and FY 1995 BAAs. The criteria appear to have been more clearly defined in subsequent funding cycles. Some of the respondents to the "Dear Colleague" letter reported being mystified by "secret criteria." This lack of clarity undermines confidence in the program and may be a deterrent to potential qualified applicants. In

response to such concerns, the FY 1996 BAA included revised and more detailed evaluation criteria, specific to the type of proposals requested.

Presentations made to the committee by Army representatives indicated that the criteria in FY 1995 were:

• programmatic relevance, that is, direct application to breast cancer, opportunity to produce a breakthrough, and diversity of the research portfolio;

• limitation on funding duplicative proposals; and

• adherence to program investment strategy.

In addition to programmatic balance, the FY 1996 criteria articulated also included "originality and innovative nature of proposal," "timely translatability to prevention, screening, diagnosis, prognosis, treatment, and health care delivery," and potential for "ultimate eradication of breast cancer" (USAMRMC, 1997).

These revised evaluation criteria illustrate the recent shift in the program's mission, vision, and programmatic goals, which is described in Chapter 4. It will be recalled from Chapter 4 that the IOM (1993) listed among programmatic goals:

• bring new investigators into the field;

• encourage communication across disciplines and collaborative studies;

• encourage research that extends scientific advances into new strategies for prevention, detection, diagnosis, treatment, and ongoing patient care;

• support excellent ongoing research and promising yet underfunded research areas;

• stimulate research on the obstacles to widespread dissemination of proven detection methods and diagnostic and therapeutic interventions; and

• enhance the use of existing resources and encourage the development of new resources.

After the FY 1993/1994 and FY 1995 experience, the IP issued a more focused statement in May 1996 stating that the mission of the BCRP should be to eradicate breast cancer, and the vision of the BCRP was to expedite and facilitate breakthroughs in breast cancer research, support innovative, risk-taking research demonstrating solid scientific judgment, and support research that will translate into advances in breast cancer prevention, diagnosis, and treatment. This focused vision was translated into the decision to limit research grants to either 5-page IDEA applications with no preliminary data required, or large multidisciplinary research programs of "translational" potential (RTP) while continuing to fund individual research training awards for people at all levels (predoctoral to sabbatical). Some committee members were concerned

that the 5-page IDEA application would not be adequate to describe epidemiological or health services research. Aside from the 1997 IOM committee review, there is no mechanism for an independent evaluation of these decisions and their consequences for the BCRP.

Some members of the committee had reservations about the open-ended nature of this focused vision. While breakthroughs are of course welcome, they are also rare. Most scientific progress is built on carefully crafted research resting on the foundations of prior evidence. The success of this research program relies on its both fitting into a context of adequately funded traditional research and supporting "high-risk" research that is also meritorious, well thought out, and well defined. Moreover, the focus on eradication unintentionally excludes projects that do not relate directly to curing the disease, such as those addressing quality-of-life issues.

Throughout most of its 3-year history, the IP has demonstrated adherence to the congressional mandates and has translated the mandates in the BAAs for the scientific community. The IP, along with the USAMRMC, is responsible for a breast cancer program viewed as successful by this committee; yet despite the IP's flexibility and inclusion of consumers at every level (see "Consumer Participation" section below), the funded portfolio of research grants for 1993/1994 and 1995 has not yet taken on the unique character envisioned by both the IOM (1993) and the IP. With the change of focus on multidisciplinary studies and innovative ideas, the portfolio is likely to acquire uniqueness in the 1996 and 1997 funding cycles.

The U.S. Army Medical Research Acquisition Activity (USAMRAA) may take up to 8 months to complete award negotiation and processing, which must be finalized by the end of the fiscal year following the fiscal year of the appropriation. DOD regulations governing grants appear to require as much effort as NIH contracts. Responses to the "Dear Colleague" letter indicate some grantee frustration with the time and effort required to complete the process. From the standpoint of scientific investigators, it would be highly desirable to streamline the grants awards process and management to be more like that of NIH. At this time, for example, the USAMRAA does accept institutional assurances regarding human subjects, animal welfare, and environmental and safety compliance. However, applicants are required to submit to an independent, and duplicative, set of military specific procedures.

Although part of quality assurance in grants management involves annual reporting and review, the committee understands that within the BCRP this process is cumbersome and requires detailed follow-up forms. These forms have been revised frequently on an as-needed basis, as demonstrated to the committee by USAMRMC, but they do not appear to have been systematically used to monitor the progress of the program, and the results of any progress reports were not made available to the IOM committee. No mechanism appears to have been developed for systematic evaluation of the success of the BCRP, for example, by tracking new investigators attracted to the field. The committee does note, however, that the IP and USAMRMC have planned a conference for late 1997 for all participants in the program, and that they plan to disseminate research results in a computerized database.

Under PMT direction, UIS and SAIC are coordinating a project recommended by the IP to evaluate the impact of consumer participation on the peer review panels. The committee commends this joint effort and the willingness of all parties to work together on this important project. While the results of the questionnaire study were not available, the committee heard testimony from consumers, peer review panel members, and IP members who felt very strongly that consumer participation was valuable in both levels of peer review. DOD observers were also enthusiastic regarding the consumer advocate's role. Not only do consumer participants return to their communities and report about the peer review process thus fostering understanding and communication between scientists and the general public, their presence during the review serves to remind basic scientists of the human component of this disease and the need for more research on psychological and social aspects, and health care delivery.

The committee questions why the Army uses different definitions for "consumer" for the IP and scientific review panels (see Chapter 4). If the Army feels it is important to have a survivor perspective and that the survivor participants should represent a constituency, they should apply the same parameters to both panels' members.

The investment strategy recommended in the 1993 IOM report was followed closely in the FY 1993/1994 funding cycle. It entailed a balance

between funding for research projects, recruitment and training, and infrastructure enhancement. The changes made in subsequent funding cycles, as outlined in Chapter 5, were based on sound reasoning. With the changing emphasis on eradication and on innovative high-risk research projects with breakthrough potential, the number of IDEAs awarded increased from 54 in 1993/1994 (with 30 recommended by the IOM report) to 75 in 1995 and 145 in 1996, while new investigator and traditional investigator-initiated research awards that are based on prior research accomplishments were decreased in 1995 and discontinued in 1996. The proportion of the total allocation for IDEA proposals not requiring preliminary/pilot data increased from 2% in 1993/1994 to 53% in 1996. This policy was intended to support research in underexplored areas and to encourage the entry of new investigators and researchers from other areas into the field of breast cancer research.

Of the many 1993 IOM committee recommendations, the BCRP had narrowed its focus to "the importance of channeling the research funds in directions that stimulate innovative ideas [and] involve interdisciplinary research" (IOM, 1993). The committee believes that the BCRP has succeeded in identifying a niche that is unique and makes it different from programs supported by other funding agencies. The high-risk investment in the IDEA category has to be considered experimental. It is important to develop evaluation criteria for this program, such as tracking publication records and follow-up traditional grants awarded, along with tracking the careers of investigators who were attracted to the field by the BCRP's innovative mechanisms.

The committee was concerned that the term "translational research" became an FY 1996 funds allocation category even though it has no universally accepted definition. While the committee agrees that multidisciplinary research that encourages basic, clinical science and public health investigators to work more closely together should be encouraged, it believes this should be a proposal evaluation criterion rather than a funding category. Good basic research will have an impact on clinical practice, and good clinical research will shape basic science. Neither type of research should be restricted by artificial time lines and definitions, such as "timely translatability."

The committee agrees with the decision to eliminate support for infrastructure enhancement after the first funding cycle in view of low-scoring applications and support available from other sources. The committee views continuation of funding training and recruitment awards at all levels as an important investment strategy toward the mission.

The 1993 IOM recommendation to sponsor interdisciplinary meetings "to bring together people of diverse perspectives and expertise to think about areas related to breast cancer" "as a source of serendipitous collaborations and as the genesis of creative high-risk proposals" was not followed in any funding cycle. These meetings were envisioned as involving small groups of people with

diverse backgrounds. The large conference, mandatory for all BCRP grantees, that the Army is planning for late 1997 (mentioned above) has quite a different purpose. It appears to be primarily a means of monitoring research progress. The committee views the original IOM recommendation to fund small interdisciplinary meetings that foster cross-fertilization as important and consistent with the current BCRP mission.

As discussed in Chapter 5, the 1993 IOM committee recommended that funds not be restricted to proposals that deal solely or directly with breast cancer, but, instead, that funds be allocated to support the best proposals as long as the work is relevant to at least one of the six fundamental questions that the committee identified. It explained that many of the discoveries that have benefited breast cancer patients arose from research that did not address breast cancer directly. The current committee's review of abstracts of funded projects in the 1993/1994 and 1995 funding cycles, determined that the portfolio covers research and training topics that are responsive to all six questions posed in the IOM 1993 report. As Chapter 5 also points out, the questions are sufficiently broad so that almost all abstracts could be classified into one or more categories, with the exception of epidemiology. Because the fundamental question that was directed to epidemiology had focused on the need for research relating epidemiological findings to changes occurring at the cellular and molecular level (''molecular epidemiology"), the committee created an additional category for standard epidemiological research projects. The fundamental areas of breast cancer research thus defined (see Table 5-11) appear adequately covered in the FY 1993/1994 and FY 1995 cycles (Tables 5-12a,b and 5-13a,b). The data in Tables 5-1 and 5-2 reveal that funding in these areas was proportionate to the number of proposals received. Thus, neither scientific reviewers nor IP members appeared to be biased against proposals in these research areas and an effort should be made to increase the number of submissions.