National Academies Press: OpenBook

A Review of the Department of Defense's Program for Breast Cancer Research (1997)

Chapter: 4 U.S. Army Breast Cancer Research Program

« Previous: 3 Non-BCRP Support for Breast Cancer Research
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

4
U.S. Army Breast Cancer Research Program

The Army's Breast Cancer Research Program (BCRP) has evolved over the past 5 years from a small research program pursuing interservice research on breast cancer screening and diagnosis into an organization pursuing a broad-based, competitively awarded research portfolio covering all areas of breast cancer research, with approximately $500 million appropriated to it by Congress over the 4-year period. In its brief history as a peer-reviewed, competitive grants program, the BCRP has reviewed over 7,000 research proposals and developed a diversified $465 million research portfolio of approximately 800 projects distributed to public and private research institutions across the United States and internationally. In 1995, the USAMRMC asked the IOM to assess this program. To provide the context for interpreting the conclusions and recommendations of the IOM Committee on Breast Cancer Research, this chapter provides an overview of the history, structure, processes, and vision of the BCRP. In Chapter 6, the committee comments on selected aspects of the BCRP.

HISTORICAL OVERVIEW

The BCRP was designed in response to a mandate from Congress to ''promote research directed toward reducing the incidence of breast cancer, increasing survival rates, and improving the quality of life for those diagnosed with the disease" (USAMRDC, 1993). When Congress increased the fledgling BCRP's budget from $25 million in FY 1992 to $210 million in FY 1993, the Army asked the IOM to provide recommendations for development of a breast

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

cancer research program and investment strategy. As discussed in Chapter 1, in response, the IOM suggested that funds be allocated among three broad programmatic areas—infrastructure enhancement, training/recruitment, and investigator-initiated research—and that applications within these areas be evaluated using a two-tiered review system (IOM, 1993). The first tier of the review system would review applications for scientific merit and the second tier would make funding decisions regarding those applications on the basis of programmatic relevance. Following the IOM recommended investment strategy, the USAMRMC developed a Broad Agency Announcement (BAA), which was released in September 1993 to invite submission of proposals.

Congress extended the BCRP in FY 1994 with an additional $30 million appropriation, stating that "this funding should be used to continue the fiscal year 1992 and 1993 breast cancer research program in accordance with the standards outlined by the Institute of Medicine recommendations." The congressional report stated "the conferees agree that the Department (of Defense) should continue this important program in future budget requests" (Committee on Appropriations, 1993).

In 1995, Congress and the Secretary of the Army directed the USAMRMC to conduct another breast cancer research initiative, similar to the 1993/1994 program. The appropriation of $150 million for FY 1995, however, was associated with some changes in programmatic priorities—$20 million earmarked for research in mammography/breast imaging and $15 million for breast cancer centers, leaving $115 million to support other breast cancer research. The $20 million earmark for mammography was intended to take advantage of new applications of military technology that could facilitate automated mammography screening. The goal was to "improve and verify the accuracy of breast imaging in institutional and community environments" (USAMRMC, 1995b). The other earmark, $15 million for breast cancer centers, was designed to "support the development and enhancement of patient-centered care that incorporates strategies for increasing patient accession in clinical trials" (USAMRMC, 1995c) at three geographically dispersed centers. Based on these programmatic goals, a supplemental BAA (USAMRMC, 1995c) was released on June 15, 1995, inviting applications.

IOM PROGRAMMATIC VISION (1993)

The 1993 IOM committee sought to "provide investigators the opportunity to explore new approaches to understanding breast cancer and relieving or eliminating its toll on individuals and their families" (IOM, 1993). The recommended programmatic strategy was intended to provide guidance to the USAMRMC on how to bring new talent into the field and foster innovation in

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

breast cancer research. The IOM committee outlined the following programmatic aims:

  • bring new investigators into the field, both junior and established;

  • encourage communication across disciplines and collaborative studies;

  • encourage research that extends scientific advances into new strategies for prevention, detection, diagnosis, treatment, and ongoing patient care;

  • support excellent ongoing research and promising yet underfunded research areas;

  • stimulate research on the obstacles to widespread dissemination of proven detection methods and diagnostic and therapeutic interventions;

  • enhance the use of existing research resources and encourage the development of new resources;

  • encourage women and minorities to apply for grants;

  • encourage investigators to address in their research protocols the needs of minorities, elderly women, and low-income, rural, and other underserved populations;

  • include women and minorities in the advisory council and study section memberships (IOM, 1993).

The IOM committee envisioned a broad portfolio of investigator-initiated research, articulating the following questions to provide examples of the range of research initiatives considered relevant:

  • What genetic alterations are involved in the origin and progression of breast cancer?

  • What are the changes in cellular and molecular functions that account for the development and progression of breast cancer?

  • How can endogenous and exogenous risk factors for breast cancer be explained at the molecular level?

  • How can investigators use what is known about the genetic and cellular changes in breast cancer patients to improve prevention, detection, diagnosis, treatment, and follow-up care?

  • What is the impact of risk, disease, treatment, and ongoing care on the psychosocial and clinical outcomes of breast cancer patients and their families?

  • How can investigators define and identify techniques for delivering effective and cost-effective health care to all women to prevent, detect, diagnose, treat, and facilitate recovery from breast cancer (IOM, 1993)?

The strategy further outlined the approximate amounts to be allocated to each area, funding mechanisms within each programmatic area, and the approximate number of individual awards funded by each mechanism (see

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

Table 4-1). A variety of investigator training and recruitment awards were proposed to attract junior-level, mid-career, and senior-level investigators into the field—predoctoral training programs, predoctoral and postdoctoral fellowships, "instant sabbaticals" for established scientists willing to change their research course, career development awards, and interdisciplinary meetings. Infrastructure enhancement awards were intended to improve access to research resources. Investment targets in this area included enhancement of existing cancer registries; registries of high-risk women; DNA resources; transgenic mouse husbandry; banks of tumor samples, breast tissue, and cell lines; information systems; and shared resources.

Within the category of research projects (see Table 4-1), the IOM committee recommended creation of three types of award mechanisms to attract both junior and senior researchers as well as individuals who were already in the field and others who were new to it. New Investigator Awards (NIAs) would sponsor junior-level investigators and were conceived as equivalent to the NIH First (R29) award. Innovative Developmental and Exploratory Awards (IDEA) were intended to stimulate innovative ideas while acknowledging that some may not lead to successful results. The IDEA awards would be directed toward a variety of disciplines for scientists possibly lacking the pilot data necessary to submit a traditional research proposal (IOM, 1993). In 1996 the application process for IDEAs was streamlined so that only five pages are required to describe the proposed research in terms of background, hypothesis, and methods. If pilot data are available, they should be included but are not required. In addition, an abstract, statement of relevance, biographical data, statement of work, other sources of funding, institutional assurances, and appendixes are still required to enable a complete review of the application. The IDEA awards would provide investigators an opportunity to explore a new area of research as well as to test the research worthiness of an idea. Investigator-Initiated Awards (renamed Other Investigator-Initiated Awards [OIAs] by the BCRP) would be similar to the NIH R01-type grant; the IOM committee anticipated that the majority of the BCRP awards would fall into this category.

The 1993 IOM committee recognized that an important key to the program's success would be its ability to attract applications from talented scientists in many different fields and settings. Thus, it recommended that the Army issue a BAA notifying the scientific community of the availability of awards, including amount of funding, programmatic priorities, application requirements, submission procedures, and review criteria. In following these recommendations, the USAMRMC set aside up to $8 million for historically black colleges/universities and minority institutions (HBCU/MIs) (USAMRDC, 1993).

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

TABLE 4-1. 1993 Institute of Medicine Recommendations for Breast Cancer Research Program Programmatic Investment Strategies

Training and Recruitment—up to $27 million

Predoctoral training program—$4 million

Predoctoral fellowships—$3 million

Postdoctoral fellowships—$6 million

Instant sabbaticals—$2.5 million to $5 million

Career development awards—$8 million

Interdisciplinary meetings—up to $1 million (cap)

Infrastructure Enhancement—up to $21 million

Enhancement of existing cancer registries—up to $10 million

Registries of high-risk women—up to $2 million

DNA resources (clones, DNA markers, etc.)—up to $2 million

Transgenic mouse husbandry—up to $1 million

Banks of tumor samples, breast tissue, and cell lines—up to $2 million

Information systems—up to $3 million

Other innovative shared resources—up to $1 million

Research Projects—at least $151.5 million

New Investigator Awards—up to $15 million per year

Innovative Developmental and Exploratory awards (IDEA)—up to $4.5 million

Investigator-initiated grants (R01-type)—at least $132 million

 

SOURCE: IOM, 1993.

Two-Tiered Review Process

To decide which applications would be funded, the 1993 IOM committee concluded that "the best course is to set up a peer review system that reflects many of the traditional strengths of existing review systems but that is tailor-made to accommodate the goals and the novel and complex program the committee has proposed" (IOM, 1993). It recommend a two-tiered review system, whereby newly constituted study sections would first review the proposals for scientific and technical merit. Applications receiving high scores would subsequently be forwarded to an advisory council which would assess them individually and as a group for their relevance to programmatic goals. Scientific excellence would be the primary criterion for awarding grants and programmatic relevance would be secondary—"that is, when the Advisory Council receives two excellent proposals but can only fund one, the award should go to the proposal that best meets the programmatic goals" (IOM, 1993).

The committee stated that "if the program is to distribute funds more widely to a new mix of scientists and ideas, then it must have study sections specifically

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

attuned to the programmatic goals" (IOM, 1993). Most study section members should have experience in biomedical review and scientific expertise in one or more of the disciplines under review. Because the committee wished to encourage funding for a wide mix of scientists and ideas, it recommended that the program utilize study sections comprising scientists from a range of disciplines, career levels, and perspectives, with special consideration given to appointing women and minorities. Study section chairs should be senior scientists who are widely recognized as experts in their fields, with experience as technical reviewers. They should also be receptive to, and "indeed enthusiastic" about, the program's broad goals of attracting new scientists and stimulating innovative ideas. Study section members would be identified by the BCRP administrator primarily from lists of the American Cancer Society (ACS), Agency for Health Care Policy and Research (ACHPR), the National Cancer Institute (NCI), and the National Science Foundation (NSF). The administration would also seek nominations through an open process using a broad array of channels.

The IOM recommended three study sections related to training and recruitment (predoctoral training programs, predoctoral and postdoctoral fellowships, and instant sabbaticals and career development awards); three study sections for infrastructure (registries, physical reagents, and information systems); and six or more study sections meeting in two review phases organized around the themes of basic, clinical, and public health sciences. A representative of each study section, preferably the chair, would attend advisory council meetings when decisions on awards were made.

The 1993 IOM committee envisioned that meritorious applications would be reviewed by a single advisory council comprised of 16 to 18 members appointed to advise the managers of the BCRP. Members would represent multiple disciplines, geographic regions, and institutional settings, and be at different career levels and primarily non-military. Breast cancer survivors should be included among consumer representatives; others might include a family member of a survivor, a member of a high-risk family, or others with specific interests and perspectives related to breast cancer.

The major proposed tasks of the advisory council were far reaching, extending to monitoring of the program (IOM, 1993).

Program Administration and Management

On the subject of the program's administration and management, the 1993 IOM committee recommended that "the first and most crucial task for successful management of this program is for the Army Medical R&D Command to choose the individual who will serve as the program administrator

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

or manager. The committee recommends that the Command choose a strong manager with extensive experience in biomedical peer review" (IOM, 1993).

BCRP IMPLEMENTATION, 1993-1996

In 1993 the U.S. Army chief of staff assigned the Army Medical Research and Development Command (later called the Army Medical Research and Materiel Command [USAMRMC]) to direct the BCRP and appointed a program director from within the ranks of the command. The program director developed a multifaceted program infrastructure made up of an Army program management team (PMT), a contractor overseeing peer review, peer review panels, a contractor overseeing programmatic review and grants management, and the Integration Panel (IP)—originally conceived of as the advisory council by the 1993 IOM committee (see Figure 4-1).

This section discusses the roles and responsibilities of each program component, specific accomplishments that address the congressional mandates and 1993 IOM report recommendations, and implementation of the program.

Scientific Oversight

The U.S. Army BCRP is under the scientific oversight of the Armed Services Biomedical Research Evaluation and Management Committee (ASBREM). The ASBREM is co-chaired within the office of the Secretary of Defense and has representation from all three military departments. The PMT is located at Fort Detrick, Maryland, and has 11 full-time employees, including a program director, deputy director, two program managers, and several support staff. Recognizing it lacked the infrastructure necessary to review breast cancer research applications and funded projects, the Army established contractual arrangements with the private sector to assist in administering the grant review and management for the BCRP. For the FY 1993/1994 funding cycle, the Army contracted with the American Institute of Biological Sciences (AIBS) to administer the first-tier peer review performed by the study sections., Starting in 1995, after the IP recommended that the contract be rebid, the Army contracted with United Information Systems, Inc. (UIS) to administer the first tier of peer review, with AIBS retained to assist with the annual review of funded projects. Study section panel members and chairs are subcontractors to UIS (and were to AIBS).

Beginning in 1993, the Army contracted with Science Applications International Corporation (SAIC) to administer the second-tier programmatic review performed by the Integration Panel (IP). SAIC's principal responsibilities in this regard include identifying prospective IP members and

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

FIGURE 4-1. USAMRMC Breast Cancer Research Program organizational chart.

assisting USAMRMC in evaluating them and providing logistical and administrative support to the IP (e.g., meeting arrangements, preparation of draft documents, and facilitation of programmatic review). IP members are subcontractors to SAIC.

The Integration Panel

The IP serves as the advisory council recommended by the IOM (IOM, 1993). It met for the first time in September 1993. Its charter was developed and adopted in December 1994, and subsequently amended in March 1996. The revised 1996 charter outlines the main purpose of the IP as to "provide timely advice and counsel to SAIC in its role to support the U.S. Army Breast Cancer Research Program Director for its development of biomedical research and related management of appropriated funds to realize programmatic goals for breast cancer research" (USAMRMC, 1996c).

The major tasks and duties of the IP, as outlined in its charter, include:

  1. recommendations for a research investment strategy within the guiding framework of the IOM;

  2. advice on the content and type of solicitation announcements and on the timing and number of solicitations;

  3. recommendations on proposal format;

  4. quality control recommendations and advice and guidelines on selection and implementation of the peer review panels;

  5. review of the results of the peer review panel deliberations and comparison of scorings/rankings across panels;

  6. decisions, if applicable, on the percentage of applications to be recommended for funding from each peer review panel;

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
  1. assistance in overall program evaluation;

  2. budget modifications of proposals recommended for funding;

  3. recommendations on whether funds should be transferred from one category to another;

  4. recommendations for selection of proposals for funding by matching scientific excellence with programmatic goals set across the major program areas;

  5. recommendations on plans for (PMT) dissemination of information on program progress (USAMRMC, 1996c).

These are all in keeping with the recommendation of the 1993 IOM committee. The IP is a multidisciplinary group of scientists, clinicians, health care professionals, and consumer advocates, all with extensive knowledge, experience, or both in breast cancer issues (USAMRMC, 1996c). Also in accordance with the 1993 IOM committee's recommendations, the IP has had a significant role in other aspects of the program's operation—advising the Army on the content and type of BAAs, number of solicitations, timing, and application formats; providing advice and guidelines on the operation of the study sections; and monitoring the program. It has not been involved in selection of specific study section members or selection of chair and vice-chair of study sections, although it has had the names of these individuals made available to it.

Congressional language each year has mandated that the largest component of the BCRP be peer-reviewed research. The BCRP's investment strategy has been established by the Army, after recommendations from the IP. The IP has also recommended to the Army procedures for peer and programmatic review.

Executive Committee

The charter of the IP specified formation of an executive committee to provide direction and guidance to the BCRP when full IP membership participation was not practical. For 1993–1995, IP executive committee membership comprised a large, self-identified subset of the IP who were interested in day-to-day IP decision making. Starting in July 1996, the IP decided that the executive committee's role should "perhaps be more formal" (USAMRMC, 1996c) and implemented the IP executive committee (EC) as originally provided for in the charter. Specifically, the EC was defined as including "the IP chair (who will also serve as the executive committee chair), IP chair elect, and IP chair emeritus. At the direction of the chair, others, to include at least one consumer advocate, will be appointed to attain approximately 25% of the full panel membership" (USAMRMC, 1996c). The

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

timing of the IP and IP executive committee meetings has varied depending on the workload and year. The IP meets approximately monthly, usually by teleconference, with in-person meetings held mainly during programmatic review periods.

Programmatic Goals

Congressional language and the original 1993 IOM report's emphasis on using program funds to stimulate new ideas and attract new talent have guided the IP's actions. The FY 1993 and FY 1994 program priorities, set by the Army, essentially adhered to the IOM committee recommendations, in both dollar allocations and funding mechanisms. In FY 1995, the IP recommended several changes in the overall scope of the program, and these were subsequently adopted by the Program Management Team. (See Chapter 5, "Research Projects" section, for a more detailed discussion.)

As recommended by the IP, the USAMRMC decided that the FY 1995 BCRP would emphasize subject areas and disciplines that were not heavily invested in during FY 1993/1994, provided that high-quality, programmatically relevant proposals were received in these areas. These areas included epidemiology, psychosocial/behavioral sciences, and alternative medicine. Although funding of projects in the heavily invested areas of molecular biology, cell biology, and mammography would continue, the extent of investment in these areas would be limited to those projects judged to be highly innovative or very likely to have a positive impact on breast cancer prevention, treatment, and cure.

The IP recommended that the 1995 BAA be reworded to make clear that the BCRP seeks proposals of all types, including those concerning "all areas of basic, clinical and epidemiologic research including all disciplines within the basic sciences, the basic health sciences, the clinical sciences, as well as public health, economics, social sciences, psychosocial sciences, quality [of] care, nonconventional therapies, occupational health, nursing research, environmental concerns, and conventional therapies" (USAMRMC, 1995b). The IP also recommended that the BAA be distributed more widely, and that program notices be published in major journals. Finally, the IP recommended that the BCRP retain the $800,000 funding limit for research grants. It was recommended, however, that the BAA state that for larger studies, such as those in epidemiology, larger grants should be considered.

The FY 1995 BAA emphasized the desire for innovative proposals to stimulate and reward speculative but especially promising and creative ideas that may or may not yield a big payoff (USAMRMC, 1995b). It also encouraged the submission of proposals in breast cancer prevention. In addition, based on its

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

belief that studies of breast cancer etiology are an important area of research, the IP recommended that the new BAA state: ''(S)tudies identifying etiologic agents and leading to better strategies for prevention, early detection and treatment are encouraged" (USAMRMC, 1995b). In both the FY 1995 and FY 1996 BAAs, the BCRP made special note that "proposals addressing the needs of minorities, elderly, low-income, rural and other under-represented populations are encouraged" (USAMRMC, 1995b, 1996a).

A significant shift in priorities and funding strategies, including a redefining of the program's mission, occurred in 1996. While formerly oriented toward research on breast cancer prevention, detection, treatment, and quality of life, the mission of the BCRP explicitly shifted towards breast cancer eradication. The original BCRP mission, as detailed in the BAAs from 1993 and 1995, was to "promote research directed towards reducing the incidence of breast cancer, increasing survival rates, and improving the quality of life for those diagnosed with the disease" (USAMRDC, 1993; USAMRMC, 1995b). The 1996 decision to redefine the BCRP mission to "promote research directed toward eradicating breast cancer" (USAMRMC, 1996a) was based on the desire to convey to the scientific community and the public that the program had a single, important goal which it intended to reach by sponsoring a research agenda that was not being addressed by conventional funding sources. The objective of the mandate to eradicate the disease was to be achieved by emphasis on innovation and new ideas, bringing new investigators into the field, focusing on under-represented areas, and fostering multidisciplinary approaches. To foster innovation, increased funding was directed to IDEA grants, but no particular research area was identified in which innovation was specifically to be encouraged. Members of the IP, in meetings with the 1997 IOM committee, emphasized that this change in mission is not intended to discourage research in areas that do not have the potential for leading to eradication (e.g., treatment and survivorship). (See further comment on this point in Chapter 6.)

Maintaining the central theme of innovation, the IP also stressed in 1996 the need for accelerating progress in translating scientific discoveries into practical approaches for the prevention, detection, and treatment of breast cancer. The 1996 BAA called for "proposals which will foster new directions, address neglected issues, and bring new investigators into the field of breast cancer research. The central theme is innovation. Scientific ventures that represent unattempted avenues of investigation or novel applications of existing technologies are highly sought" (USAMRMC, 1996a). The 1996 BAA also stated that "[w]hile the program wishes to encourage risk-taking research, such projects must nonetheless demonstrate solid scientific judgment" (USAMRMC, 1996a).

To implement the 1996 program, with its emphasis on eradication, innovation, and translatability, the IP made important changes in its investment

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

strategy and its distribution of funds to specific award categories. The new investment strategy and award mechanisms were intended to solicit and support investigations which promised to forge dramatic breakthroughs in the field. Although it wished to support projects that potentially would deliver large gains toward breast cancer eradication, the IP recognized that some of the more innovative "high gain" projects might also be high risk (i.e., they might not achieve their stated goals).

In light of its 1996 themes, the IP elected to support three types of awards in 1996: training, IDEA, and Research with Translational Potential (RTP) awards. It allocated approximately $20 million to continue the program's efforts in the area of scientist training and recruitment. It increased the number of IDEA grants and the size of individual awards from former years: A total of $40 million was allocated and award sizes were increased to $300,000 for periods of up to 3 years. IP members were not, however, in unanimous agreement on the recommendation for an increased focus on IDEAs during the FY 1996 funding cycle. Some argued that focusing on IDEAs would not be responsible because there were insufficient numbers of scientifically meritorious IDEA proposals received during the FY 1993/1994 and FY 1995 funding cycles. These opponents also argued that an emphasis on the IDEA approach meant ignoring research in genetics and other basic science research areas which have the potential to lead to the eradication of breast cancer.

The third category of awards supported in 1996 were for projects identified as having translational potential—that is, those with promise for moving rapidly from research to application in the areas of breast cancer prevention, detection, treatment, or health care delivery (bench to the bedside), or from clinical observations to basic research (bedside to bench). These RTP (Research with Translational Potential) awards were intended to support interdisciplinary projects. To avoid duplication of breast cancer research funded by the NCI, the IP recommended the creation of a larger category of awards targeting translational research. These translational research awards would be larger than the IDEA awards.

The IP recommended eliminating the Other Investigator-Initiated Award (OIA) and New Investigator Award (NIA) categories in the 1996 program, reasoning that OIAs and NIAs are virtually identical to NIH R01 and R29 grants. Because recent discoveries in breast cancer genetics and other basic science areas have garnered substantial attention, the IP anticipated that these research areas would continue to receive substantial funding from the NCI. The panel reasoned that the BCRP had heavily funded OIAs in previous years so that it was not necessary to continue such investment.

Defenders of the OIAs argued that new discoveries in genetics provide opportunities in underrepresented areas such as genetic testing, counseling, and breast cancer prevention. Many of the opportunities created by these new

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

discoveries are in clinical and epidemiological research, areas that had been underrepresented in previous BCRP funding cycles. Important studies in these areas, however, require funding beyond the scope of IDEAs.

Procedurally, the new vision has had several implications. First, administrative aspects of the application review were revised (e.g., the text of IDEA grant applications was shortened from 17 pages to as few as 5 pages and preliminary or pilot data requirements were eliminated). Second, the first-tier peer review procedures were modified. The FY 1993/1994 and FY 1995 criteria for reviewing proposals included scientific and technical merit; originality and innovativeness; relevance to breast cancer; appropriateness, feasibility, and adequacy of the approach; experimental design/methodology; qualifications, expertise, and experience of the investigator; resources and environment; and reasonableness of the budget in relation to the proposed research. In FY 1996, the principal criteria for the IDEA grant review became originality and the innovative nature of the proposal, followed by the previous years' criteria. RTP review criteria were similar to those used for the IDEA awards, and also included "timely translatability" (SAIC, 1996) of the proposed research. The scoring process was modified accordingly. Second-tier review was changed to include more documentation of the decision-making process and a greater programmatic emphasis on innovation. To familiarize all concerned with the newly adopted review criteria and program objectives and to assure that these changes were understood at every level of review, the IP provided an orientation program to the executive secretaries and members of the primary review panels.

Discussion by the committee of some of the above-described changes in the goals, strategies, and criteria for the 1996 program appears in Chapter 6.

THE REVIEW PROCESS

Grant applications submitted to the BCRP are first screened for fulfilling administrative requirements (e.g., completion of a "bubble sheet" containing key investigator and application information and adherence to Army application regulations—page limitations, completeness of forms, date of submission, proper number of copies, and proper format). Applications are transferred to the support contractor (AIBS in FY 1993/1994 and UIS in FY 1995 and FY 1996) for data entry, review and referral, and assignment to an appropriate scientific review panel.

Tier 1: Scientific Peer Review

As noted earlier, the USAMRMC contracted with AIBS for the first-tier peer review for the grant review cycle FY 1993/1994. Because this committee

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

had almost no information available to it from AIBS, this section focuses on only the two most recent grant cycles, FY 1995 and FY 1996, which were administered by UIS.

The main operational and scientific responsibility for the program resides with the UIS project director. The project director, who reports directly to the PMT, manages a three-level organization for peer review comprising the core directorate (level 1), the review panel executive secretaries (level 2), and the review panels and chairs (level 3) (see Figure 4-1). The project director also has overall responsibility for the administrative and logistical operations for the entire peer review effort. UIS has produced detailed orientation materials and guidebooks for executive secretaries that include definitions for IDEA grants and translational research as well as for scientist and consumer reviewers.

The core directorate is composed of five former senior federal grants management experts with substantial experience in administering peer review. The UIS project director and the BCRP director work closely with the core directorate, but are not members. The core directorate provides strategic planning and scientific oversight for the first tier of the peer review program. The PMT assures the coordination of this component with the IP programmatic review. The directorate identifies and recruits executive secretaries who in turn recruit panel chairs and reviewers. In response to the numbers of applications in each program and topic area, the core directorate determines the disciplinary foci and number of panels needed. It has responsibility for on-site meeting supervision during the review process and quality control of the summary statements (i.e., summaries of the proposed project and its critique). A referral committee comprising the core directorate and selected executive secretaries examines all the grant applications and assigns applications to specific panels. Panel assignment follows comprehensive referral guidelines that are prepared by the core directorate. This procedure optimizes internal consistency of the scientific content of the review panels and reviewer workloads and minimizes potential bias and conflict of interest issues.

The primary responsibility of the executive secretaries is to ensure that each proposal receives a competent, thorough, and fair peer review. They are responsible for recruiting panel chairs and reviewers, administering the panel meetings, and editing the review summary statements. Executive secretaries retained by UIS are reported to be mostly former NIH or NSF scientific review administrators with experience in the management of scientific review procedures.

UIS reported to the 1997 IOM committee that it has made efforts to recruit panel chairs who are distinguished scientists in their fields who have peer review panel experience. Executive secretaries and IP members usually identify nominees and the USAMRMC approves them. The chairperson should have prior peer review experience and be able to lead the panel toward appropriate

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

recommendations. In contrast to the IOM 1993 committee recommendations, no vice-chairs were appointed. Each panel includes 15 to 20 people, with consideration given to geographical mix, career level, and inclusion of women and minorities. In 1995, 30% of the scientific reviewers were women. Consumer panel members were introduced in 1995, with two consumers on each panel. All consumer panel members were women. Panel members may be identified by the executive secretary through online literature searches and by experts in the scientific fields of interest. Consumers are nominated by their respective organizations; an individual cannot self-nominate. In FY 1995, there were 42 separate panels, covering 14 different disciplines (e.g., cell biology, epidemiology, health care delivery) and comprising 42 executive secretaries, 42 panel chairs, 560 scientist panel members, 85 consumers, 18 ad hoc members, and 18 teleconference members. All scientific peer reviewers had doctorate degrees, and there were 42 government observers as well (USAMRMC, 1996c). Ad hoc reviewers may be sought as needed on a limited basis. These reviews are either done on-site, by teleconference, or, in rare circumstances, by mail.

The executive secretary assigns scientist panel members as primary or secondary reviewers on approximately 10 applications, although members are responsible for being familiar with all submitted. Some executive secretaries have allowed members to chose the proposals for which they would serve as primary or secondary reviewers. Scientist reviewers are asked to prepare written evaluations of assigned proposals prior to the panel meeting and provide these to the research technical assistant (RTA), a contract staff person who provides administrative and logistical support for the review process. Consumer panel members are each randomly assigned to review approximately 15 proposals, but are encouraged to read and comment on all of the panel's proposals. They present comments (oral and written) to the panel and the executive secretaries, but they do not serve as primary or secondary reviewers. After discussing each application, panel members (scientists and consumers) complete standardized application evaluation forms and assign scores. A technical writer, assigned to each review session by UIS, prepares a draft summary statement that is submitted to the executive secretary for final review and editing.

Before 1996, the scoring process followed the NIH system. In the FY 1996 grant cycle, it was revised to create a two-part score—a merit rating based on global priority score (1 equals best to 5 equals worst) and a rating using the evaluative criteria in the BAA for the award category (10 equals best to 1 equals worst) (see Figure 4-2). The committee's comments about the first-tier review process appear in Chapter 6.

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

Tier 2: Programmatic Review

Prior to the start of the second-level review, the IP decides on an approximate investment strategy. From 1993 to 1995, funds were allocated first by type of award (e.g., training, research); then, within each award category, funds were tentatively assigned in rough proportion to the areas covered in the applications submitted (e.g., clinical research, epidemiology). Thus, if approximately 60% of all applications were in the basic sciences, then about 60% of the total research funds available were tentatively allocated to basic science proposals before the second-level review.

In FY 1993/1994 and FY 1995, the IP reviewed all summary statements that received a technical merit score of between 1.0 and 2.9, indicating quality in the outstanding to good range. Titles of all proposals scoring 3.0 to 5.0 were made available to the IP as well, and the associated summary statements are available for IP review if necessary.

The IP considers whether the proposal was reviewed by the most appropriate scientific review panel and in the most appropriate review category and whether the scientific merit score given reflects the information contained in the summary statement. The IP reviews applications first in subject area subgroups and then in full committee. The subgroups are formed to be consistent with the disciplinary content of the submitted proposals (e.g., molecular biology or clinical and experimental therapeutics). Subgroups include IP members with the appropriate background, consumers, and ad hoc reviewers (included in the programmatic review in 1995 and 1996) in areas where additional committee experience was needed and to alleviate the workload. Applications are assigned to subgroups based on information in the descriptive scannable bubble sheet completed by each investigator at the time of submission.

A standard set of review criteria was developed for subgroup reviews. The subgroups, and the IP itself, aim to develop a research portfolio that is scientifically excellent, innovative, representative of different fields and disciplines, and inclusive of women, minorities, and diverse geographic areas (N.B.: In 1996, proposals were received from Australia, Canada, Dubai, England, Iceland, Ireland, Israel, and Sweden, as well as the United States). In 1996, relevance to the program's goals of eradication, innovation, and translatability were added to the review criteria.

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

FIGURE 4-2. Peer review scoring system. SOURCE: USAMRMC, 1996c.

For FY 1993/1994 and FY 1995, proposals with the highest technical ranking (i.e., the top 25% of proposals or the highest scoring proposals in each subgroup) were automatically recommended for funding without IP review. This approach was stopped in 1996. Based on the initial investment strategy, each IP subgroup is assigned an approximate dollar amount to spend; it ranks applications in turn until the allocation plus an additional 30% is spent. The proposals that were automatically funded were included in each subgroup's allocation.

In 1996 every proposal scoring 1.9 or better received full subgroup review. Proposals scoring 1.9 to 2.4 were reviewed by a primary reviewer and were called up for full subgroup discussion if (1) the first-tier peer review had generated a high standard deviation of ratings or a minority opinion, (2) the principal investigator represented a minority group or the research addressed an underserved population, or (3) the project was considered to be especially innovative or relevant (USAMRMC, 1997).

Each subgroup brings its recommendations to the full IP for discussion and integration with recommendations of the other subgroups. The full IP examines the overall portfolio, including subject matter covered and diversity of investigators and institutions, and makes its final decision. Proposals that deal with research topics that are underrepresented in the BCRP portfolio, and those judged not to have been appropriately reviewed, are given special consideration. Applications are discussed in rank order, special considerations assessed, and funding recommendations made.

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

The IP's final funding recommendations are provided to the PMT. In April 1996 the IP adopted the practice of preparing a full summary of the programmatic review for all proposals received, with particular emphasis on high scoring proposals that were not funded or low scoring proposals that were recommended for funding. Additionally, the IP decided to review separately and create a funding pool for applications from historically black colleges and universities and other minority institutions (HBCU/MIs). In the past, proposals submitted from HBCU/MIs were reviewed collectively with all other proposals, but ranked separately when the award selection was determined (USAMRDC, 1993).

Consumer Participation

A unique aspect of the BCRP is consumer participation. As noted by the 1993 IOM committee, efforts by the National Breast Cancer Coalition (NBCC), a grassroots advocacy group, have been instrumental in the appropriation of funds to the DOD for the breast cancer program. NBCC involvement has led to an increased appreciation of the importance of consumers in the allocation of research resources.

Consumer participation in the BCRP occurs at both levels of review. Consumers are members of the first-level peer review panels, where they read proposals, present their opinions after the primary and secondary reviewers' presentations, assign scores, and have full voting privileges. Consumers are also members of the IP, thereby serving in the second level of peer review. There is a difference in the definition of consumer at each of these levels, however. To be defined as a consumer on a peer review panel, an individual must have been diagnosed with breast cancer and must represent a constituency (i.e., she must be nominated by an organization with relevance to breast cancer). Consumer positions on the IP can also be held by breast cancer survivors' family members, members of high-risk families, and others with specific interests and perspectives related to breast cancer. IP consumers need not represent a constituency.

Two consumers serve on each panel for first-tier peer review. They are chosen by SAIC, which solicits names for service by writing letters of invitation to hundreds of advocacy and consumer groups. (Letters went out to approximately 750 groups in 1995.) Each group can nominate up to two of its members, and each nominee is asked to complete an application and write an essay. Essays are scored by three SAIC staff scientists and, based on this scoring and other factors, consumers are chosen for service on the panels. Any consumer who happens to have a scientific background is placed on a panel reviewing grant proposals outside her area of expertise to avoid confusion

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

between the "science" and "advocacy" roles of that panel member. Consumers receive all grant proposals prior to a panel meeting and are assigned a subset for reading. They have the same scoring responsibilities as scientist members, but they are not primary or secondary reviewers, nor are they required to present formally to the panel on the proposals.

In 1996 a mentor program for consumer members on the first-tier peer review panels was initiated. Its goal was to pair a new consumer with someone who had previously served on a USAMRMC panel. Questionnaires have been developed and administered to evaluate the mentor program and other consumer aspects of the USAMRMC program. The results are not yet available, but individual testimony from both consumers and scientists who served on panels was universally favorable regarding the role consumers played in the first tier of the peer review process.

The IP charter of 1994 specifies that "three or four" members of the 24-member IP must be consumers or nonscientists with a specific interest in breast cancer. This is identical to the recommendations in the original 1993 IOM report. In the 1996 amended charter, however, this was changed to "at least three or four" (USAMRMC, 1996c). Scientists who have a special interest in breast cancer from a nonscience viewpoint are asked to function as either a scientist or a consumer, but not both. Consumer IP members interviewed by this IOM committee spoke with enthusiasm about their role on the IP. Consumer input is a core element of all IP deliberations. Each IP session is opened with a moment of silence dedicated to a person who is living with or who has died from breast cancer.

Award Negotiation and Processing

Once the USAMRMC commanding general has approved the IP's programmatic recommendations, they are submitted to the comptroller for request and review of supplementary materials and for funding verification. With funding verified, the proposals go to the U.S. Army Medical Research Acquisition Activity (USAMRAA). USAMRAA is the contracts office for the USAMRMC and negotiates the final award amount with the principal investigator and processes the award payment.

The Regulatory Affairs Office of the Army also reviews the grant proposals for protocols on human use, animal use, and safety and environmental compliance. Similar to NIH policy, the DOD accepts institutionally approved protocols for human subject and animal use. However, applicants also have to comply with a DOD-specific set of requirements and regulations in those areas. During the FY 1993/1994 funding cycle, all the grant applications submitted to the BCRP went to the Regulatory Affairs Office at the same time they were

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×

being reviewed for scientific and programmatic merit. Starting with the FY 1995 award cycle, this practice was changed to include only grant proposals that had received a review score of 1.0–3.0 following the first-tier peer review process.

Over the course of the Army review process, statements of work are finalized and time lines are established. Site visits are not routinely made for these awards; they occur only on special occasions. The process is conducted by USAMRAA employees in accordance with applicable Army procurement regulations; contract support is not used. Since USAMRAA staffing was not increased to handle the additional work load of the BCRP, this step may take as long as 8 months to complete after IP approval of the funding.

Monitoring and Evaluation of Progress

USAMRAA also has the responsibility for receiving and reviewing annual and final reports from the principal investigators, ensuring that they conform to the negotiated statements of work. These reports are due within 30 days of each award date anniversary and within 30 days of the grant ending date. Outside contractors (SAIC and AIBS) review the reports for scientific content and for adherence to the statement of work in reference to progress, deficiencies, compliance, and conformity to the requirements and format specified in the grant. Final approval or disapproval of each report is done by the contracting officer representative and is based on evaluation of the report, reviews, and comments. This representative also reviews all reports for potential licenses and patents, which are required to be processed through the judge advocate general office. Results of these reviews are reported back to grantees, sometimes with a request to change the report. Approximately 85% of these reports are approved, with about 15% requiring revision (USAMRMC, 1996c). Although adherence to the proposed statement of work is mandatory, changes may be requested by the principal investigator during the course of the work. These formal requests are processed by USAMRAA, and, if approved, result in amendments to the grant or contract agreements.

In addition to this formal evaluation process, the USAMRMC has scheduled a major conference for November 1997 called "An Era of Hope," and has requested attendance and sharing of results by all investigators funded by the U.S. Army BCRP. (See Chapter 6 for comments by the committee.)

Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 45
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 46
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 47
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 48
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 49
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 50
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 51
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 52
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 53
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 54
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 55
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 56
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 57
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 58
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 59
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 60
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 61
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 62
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 63
Suggested Citation:"4 U.S. Army Breast Cancer Research Program." Institute of Medicine. 1997. A Review of the Department of Defense's Program for Breast Cancer Research. Washington, DC: The National Academies Press. doi: 10.17226/5808.
×
Page 64
Next: 5 The Funded Portfolio of the 1993/1994 and 1995 BCRP Award Cycles »
A Review of the Department of Defense's Program for Breast Cancer Research Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

According to current statistical data, one in eight women will be diagnosed with breast cancer. The five-year survival rate for breast cancer patients has improved in recent years, but the overall mortality rates have changed little. In 1993 Congress allocated $210 million for breast cancer research as part of the Department of Defense budget. An Institute of Medicine (IOM) committee was convened at that time to advise the U.S. Army Medical Research and Development Command on strategies for managing a breast cancer research program. This book evaluates the program's management and achievements to date. Although it is too early to evaluate the program in terms of breakthrough results and new insights produced by the funded projects or investigators, this book documents the process used to select research proposals for funding and analyzes the portfolio of funded projects in terms of their responsiveness to the recommendations and fundamental questions articulated in the 1993 IOM report.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!