National Academies Press: OpenBook
« Previous: A Brief History of ADAMHA and Previous Studies of Its Organization
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

3

Methodology

OVERVIEW

The questions posed by Congress do not lend themselves to a completely objective approach; by nature they include subjective evaluations. Nonetheless, the committee gathered as much data as possible within the constraints offered by time and funds and supplemented that information with the expertise of diverse committee members. The committee also developed the following criteria by which to evaluate the questions posed by the study:

Organizational goals and funding—Does co-administration enhance or detract from:

  • clarity of organizational mission;

  • level and growth in funding at the institute (bureau or office) level and at the agency level;

  • relationships with significant constituency groups.

Effectiveness of organization—Does co-administration enhance or detract from:

  • effectiveness of services programs;

  • effectiveness of research programs;

  • efficiency of research and service programs;

  • coordinating overlapping programmatic research and service priorities and eliminating unwarranted duplication of research and service projects;

  • avoiding gaps in research and service programs.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

Management—Does co-administration enhance or detract from:

  • capacity to recruit and retain talented leadership;

  • organizational management functions, for example, planning, priority setting, budgeting, and program evaluation;

  • effectiveness of dissemination of research findings into clinical practice, as well as the translation of clinical issues into research priorities.

The committee used two approaches to obtain and analyze the information pertinent to the questions being addressed:

  1. categorical analyses (i.e., case studies of disease-specific research and service development and demonstration programs conducted by different agencies, which are used to analyze the effects of differences in organizational arrangements; and

  2. functional analyses of operational activities that are thought to be most sensitive to differences in organizational structure.

Thirteen background papers (3 case studies and 10 functional analyses) were commissioned to help the committee grapple with the criteria listed above. These 13 papers provide the basis for the analysis, conclusions, and recommendations in the chapters that follow.

CATEGORICAL ANALYSES (CASE STUDIES)

The committee commissioned three case studies on the following topics: (1) Alzheimer's disease, (2) substance-abusing pregnant women, and (3) dopamine research related to schizophrenia and Parkinson's disease. Several criteria were used for choosing the three studies. As a whole they were to represent:

  • significant areas of research for both the National Institutes of Health (NIH) and the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA);

  • an aspect of service development in the Public Health Service (PHS), either in ADAMHA or the Health Resources and Services Administration (HRSA), and of prevention programs in NIH, ADAMHA, or the Centers for Disease Control (CDC);

  • research in different stages of maturity in terms of providing a basis for treatment options;

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
  • different types of research (e.g., social and behavioral, biomedical, clinical, epidemiological, health services research, and demonstrations);

  • different funding patterns and trends; and

  • differences among the external forces that shape federal programs (e.g., constituency groups, congressional interest, and social forces).

The case study approach permitted discrete, detailed, yet cross-cutting analysis of the many issues facing the committee. Analysis of the details of research grants within discrete categories also provided a manageable means for assessing duplication in research programs. Each case study was supervised by a task force composed of experts in the field, and each was written by an individual knowledgeable in the field.

Underlying Principles for the Case Study Method

The case study method, with its ability to collect, integrate, and interpret large amounts of information from a variety of sources, seemed to provide the most promising means for developing an analytic framework to understand the complexities of policy development for health-related research and service programs. The case study method has its limitations, however: time and resources preclude more than a few intensive case studies, and no case study, by itself, can be representative of the full spectrum of research and service programs, or the full spectrum of institutes, within NIH and ADAMHA. It is therefore necessary that the case studies as a group represent different types of research and services programs and projects, a variety of organizational approaches, special populations, involvement of different levels of government, varied kinds of financing, and different political climates in which research and service programs have developed over time.

Each of the three case studies chosen by the committee represents a number of issues with regard to stages of research and services and the linkage between federal research and service programs:

  1. Alzheimer's Disease. This case exemplifies early stages of research and the difficulty of linking research to service development programs when the underlying disease process is not sufficiently well understood to be treated. It focuses on how approaches to research and services differ from cases in which service provision is more

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

directly related to research findings. Because research is funded by a number of PHS institutes, coordination of research priorities was a focus in this case.

  1. Substance-Abusing Pregnant Women. This case exemplifies a combination of several stages of research: one mature area of research (smoking), one area of research approaching maturity (fetal alcohol syndrome), and one new research problem (cocaine-addicted mothers) with overlapping issues. Collaborative efforts by agencies (HRSA and the former Office of Human Development Services, each of which has responsibility for the development of policy and programs related to the impact of maternal addictive behaviors on children) were explored in this case. Issues of replication and duplication of demonstrations (administered by the Maternal and Child Health Bureau in HRSA, and by the Office of Substance Abuse Prevention [OSAP] in ADAMHA) can be assessed in this case as well.

  2. Dopamine Research (Parkinson's Disease and Schizophrenia). These are two diseases for which the etiology is unknown and for which pharmacological treatment for symptomatic relief is widely used but rarely completely effective. Research focuses on pharmacological interventions, and services focus on chronic, long-term care. This case study explored communication among basic scientists addressing the same area of research across institutes and agencies; it also examined the translation of clinical findings into the service delivery sector (i.e., drug development) and the translation of clinical observations from the service delivery sector into new research priorities.

A standard outline was used to guide the case study writers (Figure 3-1 ). The findings of the case studies are presented below, following a brief description of the nomenclature and the process used to classify research grants within the case studies. The nomenclature is described in detail in Appendix B . This classification activity served two purposes: (1) to help paint a broad picture of the research portfolios of the agencies in selected categorical areas (i.e., the case studies), and (2) to serve as the basis of the analyses of replication and duplication (see Chapter 5 ).

NOMENCLATURE AND CLASSIFICATION

A significant problem arose with regard to classification of research projects. As described in Appendix B, the committee found no acceptable, universal nomenclature for describing research and services programs. It therefore developed its own for use in this study,

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

FIGURE 3-1 Outline for Case Studies

A standard outline was used by each of the case study writers for collecting and analyzing information. It is described below:

  1. What are the current research, service, and prevention programs?

    1. Allocations

    2. Loci—agencies, institutes, bureaus, etc.

    3. Foci—basic research, clinical applications demonstrations, services development

    What are the separate objectives of current programs? If objectives were aggregated, do they amount to a national policy, or are there conflicting objectives generated from different constituencies?

  2. How did programs develop historically? What groups or factors were involved in the historical development of the programs and in establishing their objectives?

    1. Particular leaders

    2. Constituencies

    3. Scientific discoveries

  3. How do we evaluate research, service, and prevention programs in terms of the two objectives of the study? Possible outcomes include, but are not limited to, the following:

    1. Research and research-supported programs are effective, or the obverse

    2. Service development and demonstration programs are effective, or the obverse

    3. Research is having a transfer impact that helps promote service effectiveness or efficiency, or the obverse

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
  1. Service feedback is informing the development of the research agenda (e.g., services per se, service demonstrations, clinical applications, etc.), or the obverse

  2. Is there evidence of public/private exchange of knowledge from basic biomedical research through health services research and service delivery?

  3. Is there favorable or unfavorable duplication, or complementarity of research (e.g., research reaching different populations, healthy competition, etc.) versus waste and turf battles?

  1. Summary

    To what extent can outcomes be attributed to co-administration or the lack of it?

    To what extent can outcomes be attributed to ad hoc phenomena such as leadership, important events including scientific events, politics of researchers, or politics of health status constituency groups (health care providers, patients, families, etc.)?

based on an extensive search of the literature related to classification systems for health research.

Information for classifying research grants within the case study areas was obtained from the NIH Division of Research Grants (DRG), which manages the Computerized Retrieval of Information on Scientific Projects (CRISP), a database of all PHS-funded grants. CRISP includes extramural grants, research contracts, and cooperative agreements, as well as intramural projects. Each grant is described by key words, which allow the grants to be identified by topic, rather than by institute or funding mechanism.

Grant information for selected years (1975, 1978, 1982, 1985, 1987, and 1989) was classified according to the committee's nomenclature (see Chapter 5 ). Grant titles were sufficient for classification in many instances; abstracts and lists of the key terms provided additional information. CRISP lists subprojects falling under program

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

project grants, research centers, and general clinical research centers separately; the committee classified these individually.

In some years, the grants for a particular case study were so numerous as to warrant sampling. In Alzheimer's disease, for example, one of every three grants in 1982 and one of every seven in 1985–1989 were used for classification purposes. For fetal alcohol syndrome related to the maternal addictive behavior case study, one of every two grants for the years 1982–1989 were classified. With these exceptions, all other years and areas of research were assessed in their entirety.

Difficulties in Grant Classification

Classification of grants within case study areas presented several difficulties. Grant listings were not available in a tape format, which precluded electronic transfer and extensive manipulation of the data. This computerization problem emerged only after the study had been under way for close to a year, and there was insufficient time to make other arrangements. The records were therefore entered by hand, which limited the amount of information available and the depth of the committee's analyses.

A second problem was lack of information about the dollar amounts of some awards. For example, CRISP contains no information on dollar amounts for intramural research projects. In addition, the CRISP listing of awards for subprojects of a multiproject grant (e.g., a clinical research center or a program project grant) shows the average of all subproject awards on the grant—not the actual amount for a particular subproject. This problem weighs heavily in any analysis, since centers and program project grants constitute a sizable portion of research funds. Furthermore, the costs of different kinds of research vary greatly, making comparisons difficult. (For example, epidemiological or health services research based on existing data sources is considerably less costly than research requiring many animals, particularly subhuman primates, or expensive equipment.)

Given these difficulties with ascertaining the actual funding appropriated for research projects, the committee moved to an investigation of the number of grants funded in a particular category of activity. Data on this parameter were considerably easier to assemble because NIH uses the number of grants in an area as part of its overall planning and budgeting process. Nevertheless, even this seemingly straightforward information was not always available.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

For example, it was difficult to retrieve data from the CRISP system on projects in certain categories. The committee requested special searches for health services research, demonstrations, 1information dissemination, epidemiology, and social and behavioral research. Yet many of the grants identified by CRISP within these categories bore no relationship to the topics the categories appeared to cover. For example, a search of the health services research area identified grants on epidemiological assessments of Alzheimer's disease, patient education programs, effects of the endocrine system on aggression in rats, training programs, assorted surveys, and general core support components of clinical research centers. The search on information dissemination programs in schizophrenia retrieved health status research grants on information processing and sensory system integration in schizophrenia.

In these two instances, the committee sought other sources of information on which to base its deliberations. In the first case, it arranged a meeting of federal health services research administrators to discuss various aspects of health services research, including the effects of different organizational arrangements on programs. The meeting included representatives from the National Institute of Mental Health (NIMH), the National Institute on Alcohol Abuse and Alcoholism (NIAAA), the National Institute on Drug Abuse (NIDA), the Office for Treatment Improvement (OTI), the Office of the ADAMHA Administrator (OA), the Agency for International Development (AID), the Department of Veterans Affairs (VA), the Agency for Health Care Policy and Research (AHCPR), and NIH. In the second case, the committee based its discussion of information dissemination activities on an evaluation of five information dissemination programs in the PHS and on information from interviews for the case studies.

In defense of the CRISP system and the Division of Research Grants (DRG), which is charged with its operation and maintenance, the committee notes that the system was not designed specifically to provide the information being sought. On the other hand, as the only systematic source of information for both NIH and ADAMHA, the committee used it for classification, supplementing where possible with data gathered through other means. The system was most problematic for those research and research-related categories to which the PHS devotes the smallest amount of resources. It was most valuable for its information on basic and clinical biomedical research.

In sum, the information assembled by the committee through its classification effort paints a broad picture of the PHS and is not an exact accounting of agency research portfolios. The five main categories used in the following discussions are (1) health status

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

research, (2) health interventions research, (3) systems development, (4) services, and (5) information dissemination (see Appendix B for details). The areas the committee found most amenable to classification were health status research and health interventions research (with the exception of health services research and research demonstrations).

CASE STUDY OF ALZHEIMER'S DISEASE

A large proportion of Alzheimer's disease (AD) research is health status research (due to a lack of discoveries immediately applicable to interventions) and research in humans (due to the lack of satisfactory animal models for AD). 2The large influx of funds for AD research (Figure 3-2 ) has greatly increased the knowledge base regarding

FIGURE 3-2 Grants for Alzheimer's disease research. The number of research grants on aging has grown dramatically since 1975, as have the grants in dementia and Alzheimer 's disease. The data suggest that the vast majority of dementia research is on Alzheimer's disease and that Alzheimer's disease is a large component but not the only focus of aging research.

SOURCE: National Institutes of Health CRISP system.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

the molecular and chemical nature of the disease. Health interventions research has not progressed as rapidly (although there currently are many promising leads) because the health status research is not at an appropriate stage for translation into medical interventions. The major intervention for people suffering from AD is long-term custodial care. Due to the stress of chronic care, which is provided in most cases by the family, research programs on caregiver stress are also being developed.

The National Institute of Mental Health (NIMH), the National Institute of Neurological Diseases and Stroke (NINDS), and the National Institute on Aging (NIA) all sponsor significant research programs in AD. Many other NIH institutes also fund research germane to AD. Coordinating mechanisms, such as the DHHS Alzheimer's Disease Council and the congressionally appointed Advisory Panel on Alzheimer's Disease, serve to limit (but probably not totally eliminate) duplication in AD research between institutes and agencies. Current institute personnel and grantees interviewed for the case study firmly believe in the usefulness of these mechanisms for increasing communication and coordination and in decreasing duplication. As discussed at length in Chapter 5, the relatively young knowledge base about the cause and manifestations of AD means that most seemingly duplicative research projects or programs are in fact replicative and/or complementary.

Organizational arrangements were not considered a significant factor in the development or success of either AD research or service programs. Factors that have contributed to the success of research in AD include the foresight and devotion of particular institute personnel, an infusion of research funds, attention by key members of Congress, and biomedical and technological breakthroughs. The main factor in the inadequate service programs for AD patients is the lack of a national long-term care policy in the face of a widespread disease for which the only treatment is long-term care.

Co-administration of AD research and service programs does not exist at the agency level; rather, it is the Secretary of Health and Human Services who oversees agencies that support AD research and that provide social services and reimbursement for services. As a prominent AD researcher and clinician points out, there is little co-administration and integration of research and service programs at the federal level, as well as little integrated knowledge. Barriers to interdisciplinary research efforts exist within federal agencies as well as among professional groups and clinical sites. In fact, lack of coordination at the national level offers disincentives to organizations

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

such as teaching hospitals that attempt to bridge fields and integrate disciplines at the local level.

CASE STUDY OF SUBSTANCE-ABUSING PREGNANT WOMEN

Health status research on the effects of substances of abuse on the fetus has been fairly well developed and includes studies in animals and humans. 3 Health interventions research has received much less attention. Research in this field is a small subset of research in the fields of pregnancy and addiction (Figure 3-3). However, treatment and prevention programs supported by the Office of Substance Abuse Prevention (OSAP) are not related to or seemingly supported by a substantive research base.

FIGURE 3-3 Grants for research on substance-abusing pregnant women. The data show that research into pregnancy has increased slightly since 1975. Research into addiction decreases drastically between 1978 and 1982 but has increased even more dramatically since then. Research into substance-abusing pregnant women is a small portion of pregnancy research and of addiction research.

SOURCE: National Institutes of Health CRISP system.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

In contrast to the AD case study, the concern about replication, duplication, and complementarity in programs for substance-abusing pregnant women centers on demonstration programs, not health status research. Interviews for this case study revealed two main concerns in this regard. First, there is little attention to the need to replicate research demonstrations before widespread dissemination and implementation. Second, little or no coordination occurs within the three ADAMHA units working in this area (the National Institute on Drug Abuse [NIDA], OTI, and OSAP). Demonstration programs within OTI and OSAP that are applicable to substance-abusing pregnant women are frequently administered by the same state agency or in the same city with little or no communication between the ADAMHA personnel overseeing these projects. Likewise, HRSA programs for primary care are not integrated effectively with addiction treatment programs administered through ADAMHA.

Co-administration for substance-abusing pregnant women can be found at two levels: the administrator of ADAMHA, and (when considering HRSA, primary care activities and NIH, and research on child development) the Assistant Secretary for Health, as well as the Secretary of Health and Human Services, in light of the recent creation of the Administration on Children and Families. The potential for programmatic overlap and duplication in demonstration programs and service system development activities requires attention and coordination of objectives by an official who sees all the players— namely, the Assistant Secretary for Health.

Co-administration or any specific organizational arrangement of research and service programs at the agency level was not considered to be a factor in the problems identified in this case study. Rather, the decision to give ADAMHA responsibility for programs for these women was believed to be insufficiently considered. When responsibility for programs for substance-abusing pregnant women was placed in ADAMHA, it focused the concern on substance abuse exclusively and seemed to ignore consideration of the population of pregnant and substance-abusing women using primary care programs (administered by HRSA). Other relevant concerns identified in the case study include stigma attached to alcohol and drug use, particularly in pregnancy; the general lack of attention to women's issues in federal research and service programs; congressional ambivalence about the mission of ADAMHA and the relative importance of federal and state program planning; insularity in federal agencies; and the decision to deal with ADM problems largely outside the primary health care arena.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

The case study also faulted NIDA for not pursuing treatment interventions research sooner and more aggressively. Its response was slow even after the public health problem of addicted and affected babies was documented. Considerable responsibility for this failure could be attributed to neglect and to conflicting priorities between biomedical research and the development of interventions.

CASE STUDY OF DOPAMINE RESEARCH: SCHIZOPHRENIA AND PARKINSON'S DISEASE

The role of dopamine in the etiology and treatment of both schizophrenia and Parkinson's disease has been studied for over 20 years (Figure 3-4 and Figure 3-5 ). 4 Health status research and health interventions research for both these diseases are well developed. The research includes a great deal of both animal and human research. There is no discernible difference in the quality or technological sophistication of the research currently sponsored by NIMH for schizophrenia and by NINDS for Parkinson's disease. This is in contrast to the 1970s and even early 1980s, when many constituency groups felt that the NIMH research was inferior to the biomedical research conducted in NIH, despite substantial evidence to the contrary. Many observers attribute the recent success of the NIMH research programs to its declining responsibility for administering service programs, although a number of such activities still remain in NIMH.

There is also no evidence that these research programs and projects duplicate each other. Rather, the health status and interventions research programs focused on dopamine in the two institutes are competitive and complementary. Both institutes have high regard for the value of fundamental research and, when appropriate, this research contributes to new interventions research and new interventions. Many years of health status research have contributed to the development and evaluation in the past few years of very promising pharmacological interventions for both schizophrenia and Parkinson's disease (clozapine and deprenyl, respectively).

Health services research, demonstration projects, and service system development are supported by NIMH for the severely mentally ill. NINDS does not support any of these activities, limiting its activities to biomedical research. NIMH provides a nurturing environment for health services research and research demonstrations. Services demonstrations and service system development activities receive less support.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

FIGURE 3-4 Grants for Parkinson's disease and for dopamine research. This comparison of the growth in research grants for all dopamine research, all Parkinson's disease research, and research on the role of dopamine in Parkinson's disease shows that research on dopamine has increased steadily since 1975. Research into Parkinson's disease increased particularly after 1985. The biggest factors in this growth are the discovery of the MPTP model of parkinsonism and research fundamental to neurotransplantation. Research specific to the role of dopamine in Parkinson's disease is only a fraction of the research in either Parkinson's disease or in dopamine and has increased far less than either of them. The role of dopamine in parkinsonism was already a priority in 1975, and rapid growth in this particular field would have occurred prior to 1975.

SOURCE: National Institutes of Health CRISP system.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

FIGURE 3-5 Grants for schizophrenia and for dopamine research. The number of research grants in schizophrenia has increased almost threefold since 1975, reflecting the increased attention in NIMH to severe mental illness. Schizophrenia was designated the foremost research priority at NIMH in 1986, which undoubtedly accounts for the big increase in grants between 1985 and 1989. Research grants in the role of dopamine in schizophrenia have also increased greatly between 1975 and 1989, although they are only a fraction of the total research in schizophrenia.

SOURCE: National Institutes of Health CRISP system.

Co-administration is seen, to a minor degree, at the level of the director of NIMH for schizophrenia and at the level of the DHHS Secretary for Parkinson's disease. Co-administration was not thought to be a major factor in the success of the research programs or in the relative lack of attention to service programs. (The success of the health services research program in NIMH was an exception to this opinion; it was suggested that the history of service development and demonstration programs in NIMH provided fertile and nurturing ground for health services research.) Rather, the leadership of the research institutes, the missions of the institutes and agencies, and the desires of constituency groups played the major roles in federal programs for schizophrenia and Parkinson's disease.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

FUNCTIONAL ANALYSES

In addition to the case studies, 10 analyses were conducted in functional areas that were considered most sensitive to differences in organizational structure. Study staff prepared some of the analyses; other papers were commissioned. Brief summaries of their findings follow.

Planning, Priority Setting, and Budgeting

The policymaking system is complex, changing, convoluted, and (since public financing is involved) inherently political.5 In fact, there are multiple systems operating simultaneously, frequently with only minimal coordination. The major dichotomy that seems to arise is between what might be termed program planning (i.e., the content of the programs) and the budget planning and review process (which determines how much money will be available for a particular activity). This dichotomy is particularly evident in the research programs of ADAMHA and NIH, where the budget review process does not become involved with the actual content of the research programs (which is left to the individual institutes, divisions, and disciplinary study sections).

Priority setting involves the entire complex process by which decisions are made concerning the direction and level of support for federal programs. There is no single coherent system that can be labeled priority setting; rather, it is the result of myriad discrete activities (involving congressional committees, the administration, the research and service communities, and individual program managers).

At the level of Congress and the administration, the annual budget is the only “plan.” Budget decisions are largely incremental, and the most important single factor in determining the current year's budget is the past year's budget. Exceptions are relatively rare and reflect either major policy issues (such as concern about drug abuse and AIDS) or the specific concerns of key individuals in Congress or the administration. In the budgeting process, there are relatively few discussions of trade-offs between programs, especially between programs of different types. At no level of budget review, above the programs themselves, is there evidence of an analytic or fact-driven approach to determining resource allocations.

With regard to budget increases, there is a de facto PHS policy to treat the research programs of ADAMHA and NIH similarly. Breakthroughs or advances in the state of science generally command more

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

attention and give rise to budget increments greater than those adopted for the overall research budget. However, any differences in the final appropriations for the various institutes tend to reflect congressional interests and the effectiveness of various constituency groups. The content of research programs is rarely considered in the budget process except at the level of individual programs. Discussions in Congress focus instead on such issues as the number of new and continuing grants, types of awards, indirect cost rates, and pay lines.

Organizational Arrangements for Health Services Research in Federal Agencies

A number of federal agencies devote significant resources to health services research programs: (1) the Office of Veterans Affairs (VA), (2) AHCPR, (3) ADAMHA, and (4) the Agency for International Development (AID). 6 Health services research programs are located in very different parts of the organizational structures of each of these agencies. Important considerations include the interaction of health services researchers and service providers, collaboration within and across agencies, flexibility in allocating resources, broadening the constituency for health services research, equality of health services research and other research, stable leadership, and the policy orientation of research.

Integral to this analysis was the identification of three types of demonstration projects: (1) research demonstrations, (2) service demonstrations, and (3) technology introduction demonstrations. Health services research methodology has much to offer all three types of demonstration projects and therefore needs ties to agencies, institutes, and offices administering either research or services programs. Likewise, demonstration projects of all three types benefit from ties to health service research, which serve to strengthen the evaluation of costs and effectiveness. To some degree, demonstrations and services development programs administered by HRSA have suffered from a lack of capacity in that agency to conduct health services research.

No single organizational arrangement was identified as superior to all others; health services research is such a broad category that the specifics of the research topic need to be considered. However, certain general pros and cons can be identified. The advantages of placing responsibility for health services research and demonstration programs at the division level within an institute or office include the ability to work more closely with service providers in setting a

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

research agenda, equality with other divisions (which increases the ability to collaborate across divisions), and the depoliticalization of demonstrations. The disadvantages are loss of visibility for service demonstrations, difficulties in reaching across agencies from the division level for collaboration, and isolation from policymaking.

The advantages of placing the responsibility at the institute or agency level are a broader constituency for health services research, less difficulty in reaching across agencies to collaborate, and greater visibility for demonstrations. The disadvantages are that a broader constituency for health services research can lead to more difficulty in reaching down into the division to promote collaboration, and that increased policy sensitivity of demonstrations leads to difficulties in objectively administering the programs.

Research and Service Demonstrations and Block Grants

Demonstrations are used in DHHS for a wide variety of purposes. 7 Broadly, a demonstration may be defined as “a project involving the use of an innovation and operating at or near full scale in a realistic environment for the purpose of: (1) formulating national policy, (2) improving national programs, or (3) promoting the use of the innovation.” 8

Demonstrations can be either research oriented or service oriented. Research demonstrations serve to develop, test, and evaluate health service activities, as well as to foster the application of existing knowledge through experimental studies in services settings. Service demonstrations develop and provide new services for rehabilitation, treatment, and prevention of illness. In research demonstrations, services must be designed around a set of hypotheses; in services demonstrations, evaluating specific interventions is secondary to demonstrating the ability to develop and provide a set of new services to a specific population. In addition, two other types of demonstration activities are administered in ADAMHA and HRSA (see the definitions in Chapter 4 ). This paper discusses several paradigms for the research–services continuum in which demonstrations play an important role. The large recent increases in appropriations for demonstration programs in ADAMHA have been attributed to congressional displeasure with the declining federal role in service development that followed enactment of the block grant program.

The passage of the Omnibus Budget Reconciliation Act of 1981 substantially altered the administration of many of the major service

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

programs in the PHS, consolidating individual categorical services programs under broad-based block grant programs. PHS agencies that administer block grant programs are ADAMHA, HRSA, and CDC. Although mandated by the same legislation, historical and administrative differences between the block grant programs within PHS agencies have had a direct bearing on the relationship of research and services in the respective programs. The office and personnel responsible for administering the block grants changed repeatedly in all three PHS agencies, beginning in 1982. Currently, OTI is responsible for administering the drug abuse portion of the block grant, while NIMH administers the mental health portion of the same program.

Changing the funding mechanism from categorical grants to block grants significantly altered the federal role in health services delivery. The message sent to the agencies was to work under a “hands off” policy. In addition, the subsequent dismantling of national data collection efforts meant that there is no information about how the block grant funds are being used, who is being served and how, and what needs remain.

Information Dissemination

This paper reviewed the literature on information dissemination activities and described several information dissemination activities sponsored by PHS agencies and institutes including the following:

  • Depression/Awareness, Recognition, and Treatment Program;

  • National High Blood Pressure Education Program;

  • Diabetes Control Activities in High-Risk Populations;

  • Treatment of Early-Stage Breast Cancer; and

  • Alcohol Warning Labels. 9

Information dissemination efforts vary greatly, depending on the stage of the knowledge being disseminated, the audience to whom the effort is targeted, the resources dedicated to it, and the measures used to evaluate it. The single most important determinant of the success of any information dissemination activity is the charisma, dedication, and leadership qualities of the person spearheading the effort. Current organizational arrangements did not affect dissemination efforts. This lack of effect was seen when the dissemination efforts required collaboration between agencies or within agencies, or when agency

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

and institute dissemination efforts were directed specifically at practitioners in the field.

The problem most frequently identified was the difficulty in evaluating the success of dissemination programs. Much of the evaluation is inferential; efforts at increasing professional knowledge are easiest to evaluate, whereas activities aimed at the general public are extremely difficult to evaluate. For example, evaluation is reportedly a missing element in the information dissemination process related to diabetes. CDC is attempting to find funding to do a longitudinal study of the impact of its training materials; federal funding for such an effort is unlikely, however, and the agency is seeking foundation support.

There is little or no support in the federal system for “classical” evaluation, in which the plan for data collection and monitoring is developed and implemented at the beginning of the program. Rather, evaluation is usually attempted at a later stage, retrospectively, as an effort to “look back and figure out what the program did.” In the absence of a clear commitment to evaluation, and with few resources available for the collection of data and for other activities on which effective evaluation depends, it is not surprising that little is known about the relative effectiveness of the various methods of information dissemination and their impact on targeted populations.

Constituency Group Relations with Federal Agencies

This paper discusses constituency groups and their relationships with research and service agencies in the PHS, focusing on the effects on constituency groups of federal organizational arrangements for the administration of research and services programs and the related issues of mission and leadership. 10 Perceptions varied, but the interviews suggest that a consensus exists on the importance of (1) clarity of agency missions, as well as stability of program placement and leadership, and (2) the need for coordination of programs across agencies.

Both the interviews and the history of different federal programs suggest that organizational arrangements are of particular importance when a constituency group perceives a need for a focal point for its concerns, an increase in the priority given to an area, and/or increased linkage between research and service activities. A focal point is more important to constituency groups interested in services

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

development than it is to groups interested in science policy because science policy is represented in all NIH and ADAMHA research institutes, as well as in the NIH Office of the Director and the ADAMHA Office of the Administrator. Arrangements may also be of concern when they are perceived to affect program funding. Organizational arrangements per se, although important, were not found to influence the style of a constituency group's involvement with an agency—that is, how it goes about the business of building relationships with programs and the repertoire of advocacy techniques that it uses.

Linkage Mechanisms Between Research and Service Programs

This paper describes and analyzes linkage between six research and service programs in federal agencies. 11 The six examples provide evidence that the classic research and development continuum does not apply to all situations in which research and services programs are supported or administered by federal agencies. No single model can be used to describe the complex relationships among different types of research, evaluation, and the operational environment in which services are delivered. Each situation contains volatile policy controversies; in each the relationship between services and research is variable, and the specific formulation depends on the nature of the policy area, differences in federal policy roles, the nature of the target population, and the type of research required.

The paper also presents evidence that organizational location may be an artifact of history and, as such, is difficult to relate to measures of technical program effectiveness. There is evidence that research units can establish linkages and boundary-spanning activities with the world of operations and services if and when the work of the organization is perceived to be important enough to demand new modes of operation.

The Research Process: Replication and Duplication

Concern over potential duplication of research between two large research agencies spurred the second charge to the committee. 12 This paper uses examples to explain the need for replication as part of the scientific process. It describes situations in which duplication

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

knowingly occurs and is not to be discouraged. It also explains the importance of sponsoring complementary research, which sometimes may appear to be duplicative.

Duplication of research projects probably occurs, but it is not felt to be a major problem. Mechanisms exist to decrease, but certainly not eliminate, wasteful duplication. These mechanisms include pressures brought to bear by the scientific community, the peer review process for grants and journal articles, computerized databases for the published literature, administrative mechanisms in the Division of Research Grants, and inter- and intra-agency coordinating mechanisms for areas of research that are particularly cross-cutting and extensively investigated.

History of the Public Health Service

Several important and relevant historical trends can be observed in the evolution of the agencies within the PHS. 13Although it has been 17 years since the last major reorganization, the “new federalism” has had a decided impact on the organization of PHS agencies. Although the larger structure of the PHS has remained relatively unchanged, this impact is revealed through changes in the internal structure of the agencies, particularly those responsible for the administration of block grant programs.

The change from categorical grants to the block grant system resulted in a significant decrease in federal leadership in implementing domestic policies. Although service activities have been deemphasized since 1981, ADAMHA continues to have funding mandates for services and demonstration programs. Within the past 5 years, congressional authorization for demonstration and services programs that target specific diseases or populations has increased significantly.

Authorizations, Missions, and Appropriations

Congressional authorization and mission statements influence the culture and activities of government organizations. 14An agency's mission drives both its culture and its structures. The PHS mission with regard to administration and funding for health care services programs, historically and in the present, remains less clear than its biomedical research mission.

Interviews and case studies revealed significant confusion on the agency level about federal expectations for services funded through

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

block grants administered by states and localities. In addition, continuous changes in the organizational placement of services demonstrations and block grant programs administration have had some negative effects on constituency group relationships with federal agencies and have raised questions about program effectiveness.

Allocations to Health Research and Service Programs

Analyses of research and service allocations were conducted (1) to establish trends over a 17-year period within NIH, HRSA, CDC, ADAMHA, and the PHS as a whole and (2) to assess perceptions that changes in allocations had differed over time for agencies and institutes. 15In the past 8 years, research budgets have increased by similar rates for NIH and ADAMHA. Although research budgets increased in ADAMHA institutes subsequent to the initiation of the block grant program, there is no evidence of causality or direct correlation between the two events.

In the past 5 years, block grant and service demonstration allocations have increased massively across the PHS. The interpretation of those interviewed is that the increase in appropriations for demonstrations reflects congressional discomfort with a declining federal role in service development. Some suggested that research has benefited from recent social perceptions of drug abuse as a major societal problem, others that discontinuation of institute responsibility for administration of services and services development programs has allowed institute leadership to focus on the quality of research and on increasing research allocations.

HRSA has suffered the most in its allocations over the last 17 years. Its decreasing allocations have been attributed both to congressional and PHS ambivalence about the services mission of the PHS and to its limited ability to evaluate the costs, effectiveness, and efficiency of the programs it funds.

NOTES

1. Institute of Medicine staff report, “Demonstrations and Block Grants in the Public Health Service,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service,

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

Springfield, Va. * Demonstrations presented yet another set of problems in that it was difficult to categorize demonstration projects as either research or service demonstrations on the basis of the information in the system.

2. C. E. Blixen, “Case Study on Alzheimer's Disease,” prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

3. L. V. Klerman and M. A. Johnson, “Case Studies of Substance-Abusing Pregnant Women, Their Infants and Children,” prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

4. K. Stratton, “Parkinson's Disease and Schizophrenia: Dopamine and Beyond,” prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

5. R. Schmidt, “Research Planning and Priority Setting in the Alcohol, Drug Abuse, and Mental Health Administration,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.; R. A. Walkington, “Planning, Priority Setting, and Budgeting in the Public Health Service,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

6. S. Williams, “Health Services Research in Federal Agencies,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

7. Institute of Medicine staff report, “Demonstration and Block Grants in the Public Health Service,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

8. T. K. Glennan, Jr., The Management of Demonstration Programs in the Department of Health and Human Services (Santa Monica, Calif.: RAND, 1985).

9. Institute of Medicine staff report, “Information Dissemination in the Public Health Service,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

10. P. Kaye, “An Analysis of the Effects of Organizational Arrangements for the Conduct of Research and Services on Constituency Group Relations with Federal Agencies,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

11. B. A. Radin, “Linkage Mechanisms in Services and Research,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

12. J. Fox, “Report on Scientific Methodology,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

13. C. C. White and R. S. Hanft, “The Changing Relationship of the National Institutes of Health and the Alcohol, Drug Abuse, and Mental Health Administration,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.; R. A. Walkington, “The Health Resources and Services Administration: Evolution and Current Programs,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

14. Institute of Medicine staff report, “Authorizations, Appropriations, and Missions,” prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×

15. R. A. Walkington, “Allocations in the Public Health Service,” paper prepared for the IOM Committee on Co-Administration of Service and Research Programs of the NIH, ADAMHA, and Related Agencies, 1991; available from the National Technical Information Service, Springfield, Va.

*

For readers interested in obtaining copies of these papers, the full address of the National Technical Information Service is 5285 Port Royal Road, Springfield, VA 22161; telephone 703-487-4650.

Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 43
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 44
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 45
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 46
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 47
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 48
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 49
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 50
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 51
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 52
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 53
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 54
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 55
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 56
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 57
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 58
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 59
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 60
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 61
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 62
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 63
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 64
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 65
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 66
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 67
Suggested Citation:"Methodology." Institute of Medicine. 1991. Research and Service Programs in the PHS: Challenges in Organization. Washington, DC: The National Academies Press. doi: 10.17226/1871.
×
Page 68
Next: Co-Administration of Research and Services »
Research and Service Programs in the PHS: Challenges in Organization Get This Book
×
Buy Paperback | $45.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This volume looks at the administration of research and service programs in the U.S. Public Health Service to determine whether these programs fare better when administered jointly or through separate agencies. It uses case studies and analyses of programs and administrative processes, together with the results of more than a hundred interviews with top-ranking government officials and representatives of concerned organizations.

The book also focuses on the extent and effects of program and project duplication, replication, and complementarity in the research activities of the National Institutes of Health and the Alcohol, Drug Abuse, and Mental Health Administration.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!