The Context of AIDS Programs at NIAAA, NIDA, and NIMH
In order to analyze the AIDS research programs of NIAAA, NIDA, and NIMH, one must understand the larger context in which they have been operating. The most significant elements of this context from the point of view of this study are the passage of the ADAMHA Reorganization Act of 1992 (PL 102–321), which separated ADAMHA's research and services entities into two different agencies (NIH and SAMHSA, respectively), and the NIH Revitalization Act of 1993 (PL 103–43), which assigned the NIH Office of AIDS Research (OAR)—housed in the office of the NIH director—new budgetary authority over the AIDS programs of all NIH institutes. These two major legislative events occurred while the committee's assessment of the ADAMHA AIDS programs was under way, and, although they were not intended to be the focus of this report, they are an important part of the overall context in which AIDS research at NIAAA, NIDA, and NIMH was conducted in the recent past and will be conducted in the future. Also, the changes will have considerable impact on the institutes' AIDS programs and their connection to AIDS services programs elsewhere in the Public Health Service (PHS). Because these changes are quite recent, much of their potential impact can only be suggested.
There are other contextual factors that also were influenced by the ADAMHA reorganization and the new authority of the NIH OAR, and that in turn affect the nature of the AIDS programs of NIAAA, NIDA, and NIMH. These are the overall and AIDS-specific
budget process and the grant review process. It is important to understand the budget process because it governs the activities of all federal agencies, by circumscribing the financial constraints under which they operate from year to year. The review process is equally important to understand because it influences the scientific identity of the institutes by determining which specific research projects get funded.
THE REORGANIZATION OF ADAMHA
As outlined in Box 5.1, the organization of ADAMHA (and its predecessors) has always been problematic (IOM, 1991b). This is largely because, unlike the rest of the Public Health Service, ADAMHA included research, service, public health, and training activities all in one agency. The debate about the advantages and disadvantages of various options for administering all of these components continued (with periods of greater and lesser intensity) until Congress passed the ADAMHA Reorganization Act in 1992, which for the first time separated services and research into different agencies. On October 1, 1992, NIAAA, NIDA, and NIMH were moved organizationally to NIH, and a new agency—the Substance Abuse and Mental Health Services Administration (SAMHSA)—was created to manage the former ADAMHA service functions.
According to individuals both inside and outside of the Department of Health and Human Services (HHS), a number of complex reasons explain why the separation of research and services occurred successfully in 1992 and not earlier. First, the number of new programs and the budgets for these programs (especially those related to substance abuse) increased rapidly during the later part of the 1980s. As the agency grew, it had a more difficult time balancing the competing and sometimes conflicting missions of research and services (including the conflicting demands of constituency groups). Although internally the agency increasingly focused on research (by the late 1980s, the ADAMHA administrator and the directors of all three institutes were research scientists), the external community and HHS were more concerned about service and prevention activities. Under Bernadine Healy, NIH, which previously had fought against incorporating ADAMHA's research institutes, changed its position and actively sought the transfer of NIAAA, NIDA, and NIMH to NIH. The Office of National Drug Control Policy (ONDCP) in the White House also strongly favored the reorganization and believed that the research
orientation of ADAMHA made it difficult to respond to immediate policy imperatives related to services—a concern also voiced by Fred Goodwin, administrator of ADAMHA, in his reorganization proposal. In addition, the budgets of the services programs grew enough to be viable as a free-standing agency within the PHS. At the same time, ADAMHA research administrators feared that if budgets were limited, the service programs would be favored at the expense of research activities. In short, a combination of factors fostered a political climate that favored reorganization.
STRUCTURAL EFFECTS OF THE ADAMHA REORGANIZATION
Overall, the reorganization appears to have had a limited effect on the research programs of the three institutes. Their organizational structure and staffing have been left largely intact, and their review process remains the same as it was at ADAMHA for the period FY 1993 through FY 1996, as mandated by the reorganization legislation. Figure 5.1 shows the structure of ADAMHA prior to the reorganization. NIAAA, NIDA, and NIMH were primarily focused on research, whereas OTI and OSAP focused on services programs. Prior to reorganization in 1992, ADAMHA had 2,186 full-time employees (FTEs). Nearly three-quarters of the ADAMHA FTEs (1,602) were related to the research programs and the remaining one-quarter (585) was related to service activities. The majority of the AIDS funding and staffing was and remains in the research institutes. (See Chapter 6.)
Figure 5.2 shows the current organization of SAMHSA. In 1992, there were 656 staff positions, which represented the 585 service positions from ADAMHA plus approximately 70 additional positions added during the reorganization, mainly in the newly created Center for Mental Health Services. The majority of these service positions came from the former OTI, OSAP, and the office of the administrator, but some were originally non-research functions in the three institutes that were subsequently transferred to SAMHSA.
Box 5.2 highlights the programs and activities that were transferred from ADAMHA to SAMHSA as a result of the reorganization. Of these 24 programs, only three are directly related to AIDS: AIDS health care worker training (moved from NIMH to CMHS); AIDS health care worker training/AIDS hotline (moved from NIDA to CSAT); and the service delivery demonstrations (also moved from NIDA to CSAT).
Figure 5.3 shows the organization of NIH after the reorganization. The reorganization added approximately 1,600 positions to NIH, all but 31 assigned directly to the three institutes. These 31 positions were research-related positions from the Office of the Administrator at ADAMHA and were assigned to the Office of the Director at NIH.
Although the reorganization appears to have had limited direct effect on the AIDS research programs of the three institutes, the Congressional Report language accompanying the Reorganization Act raised concerns about the amount of attention the three institutes devoted to AIDS. The bill required NIDA and NIMH to each create an Office on AIDS, which:
… shall be responsible for the coordination of research and determining the direction of the Institute with respect to AIDS research related to: (1) Primary prevention of the spread of HIV, including transmission via drug abuse [sexual behavior in the case of NIMH]; (2) drug abuse services research [mental health services research for NIMH]; and (3) other matters determined appropriate by the Director.
The bill made no specific mention of NIAAA.
Box 5.2 ADAMHA Programs Transferred to SAMHSA as a Result of Reorganization
During the 1980s, NIAAA, NIDA, and NIMH faced management demands to develop, implement, coordinate, and evaluate a rapidly growing, complex program of AIDS activities. However, ADAMHA never had an official, authorized, functioning AIDS office. Because HHS was considering reorganization, ADAMHA created an acting AIDS office rather than a permanent one. In addition, this office was headed by a part-time acting coordinator, who also directed the NIMH AIDS office. At no time during its existence did ADAMHA budget more than two or three positions for central AIDS management. Overall, the ADAMHA AIDS activity was seen as collegial and cooperative and not as directive or managerial. Consequently, each institute at ADAMHA devised its own way to manage its AIDS program, which operated quite differently from every other: NIAAA had only one or two staff acting as AIDS coordinators on a part-time basis; NIDA had an official AIDS coordinator located within one of its regular divisions; and NIMH had a formal Office on AIDS Programs with a full-time director. (Further discussion of the institute's AIDS offices appears in chapter 6.)
AIDS RESEARCH AND THE NIH REAUTHORIZATION
Simultaneous with the ADAMHA reorganization, NIH underwent changes in its leadership, and under the NIH reauthorization bill passed in 1993 developed a much more centrally managed AIDS research effort. The NIH reauthorization bill, which was signed into law June 10, 1993, contains a number of provisions concerning AIDS research supported by NIH. Some of these provisions specifically affect the planning and budgeting process for NIH. Although the law had been in effect for many months by the time this report was written, a number of provisions regarding the budget were not yet operational, given the extended time involved in the budget process. Additionally, NIH had not completed the process of implementing other legislative requests, and therefore it was not yet clear how the revamped AIDS research program would operate.
The intent of the legislation was to strengthen the central control and management of AIDS research by NIH (through the Office of AIDS Research) while maintaining research direction in the hands of individual institutes, centers, and divisions (ICDs). Despite the tensions between proponents of central control and dispersed operation, it is believed by advocates for the new OAR
authority that the arrangement will create better coordination of NIH AIDS activities.
The 1993 NIH reauthorization bill builds upon previous NIH attempts to coordinate and focus its AIDS research while preserving the scientific independence of the various institutes. The Senate report accompanying the bill indicates that the legislation intends to ''mandate completion of an AIDS research plan that formally states priorities and resources needs for every aspect of the nation's biomedical and behavioral research effort." The plan should ensure that overlapping expertise and information is shared and coordinated between HIV and other research endeavors by determining an overall AIDS research budget level as part of the overall budget process.
The legislation continues on a path of greater central coordination that began in 1982 with the creation of an inter-institute coordinating group to track NIH AIDS research activities and to share information among the institutes involved in AIDS research. Previous actions had included the establishment of an NIH AIDS Executive Committee in 1984, the appointment of an NIH AIDS coordinator in 1985, the administrative creation of OAR in April 1988, and the legislative authorization of OAR in November 1988. Although since its inception OAR has had an important voice in shaping the NIH AIDS program, this strength principally has resided in its incumbent director, rather than in the inherent authority of OAR. The actual statement of function for the OAR describes a classic staff office responsible for advising, supporting, coordinating, recommending, and fostering the NIH AIDS research program (IOM, 1991a).
The 1993 NIH Revitalization Act specifically provides new and greater authority to OAR with regard to the AIDS budget. The Act requires a comprehensive plan for the expenditures of appropriations, and authorizes an emergency discretionary fund for the director of OAR. According to the legislation, the plan must: provide for the conduct and support of all AIDS activities at NIH; prioritize the various AIDS activities, which are required to have objectives, measures, and a time frame; ensure that the budget is allocated according to the plan; review the plan annually and update it as needed; ensure that approval of specific projects and ongoing operation remain with the individual institute, center, and division directors; and ensure that the plan includes a range of research (including basic, applied, intramural, extramural, investigator-initiated, NIH-directed, and behavioral and social science research).
This comprehensive plan is meant to be the basis for developing the annual budget requests for AIDS research. The director of OAR (not the director of NIH) is required to develop a full-funding budget that provides for the "… amounts necessary for the agencies of the National Institutes of Health to carry out all AIDS activities determined by the Director of the Office to be appropriate, without regard to the probability that such amounts will be appropriated" for each fiscal year. This budget (called the "professional judgment budget" or "bypass budget") is to be submitted from the director of OAR directly to the president for review and submission to Congress. The NIH director, the secretary of HHS, and the AIDS Advisory Council should all have the opportunity to comment on, but not change, the budget before it is submitted. In addition, the director of OAR is required to submit another budget package as part of the normal federal budget cycle. This budget package is to include the full-funding budget, a budget based on current activities and those initiatives for which there is the most substantial need, and other budgets as appropriate. The director of NIH and the secretary of HHS are required to consider all such budget estimates when recommending their annual budgets to the president.
When AIDS funds are appropriated to NIH by Congress, they will go directly to the director of OAR. The director will allocate to the individual institutes, centers, and divisions all of the funds received in accordance with the previously approved comprehensive plan for expenditures and appropriations. Funds to support existing AIDS activities, including "… projects or programs for which the Agencies have made a commitment of continued support," will be allocated by the OAR director within 15 days of his or her receipt of the funds. The director will then allocate the rest of the AIDS appropriations to the institutes, centers, and divisions within 30 days, if possible. This process will be in place beginning FY 1995.
The second major change in the budgetary process created by the NIH reauthorization bill is the emergency discretionary fund from which the director of OAR, in consultation with the AIDS Advisory Council, may make expenditures to support the AIDS research activities authorized by the legislation. This fund is designed to allow the NIH to respond to rapidly changing research problems and priorities, and grew out of concerns that inflexible budget rules may prohibit important and time-sensitive research (such as research on multi-drug resistant tuberculosis). The Act provides for the appropriation of up to $100 million in each of FY
1994, 1995, and 1996, and provides limitations and safeguards on the use of the funds. In particular, the projects supported by the funds must be peer reviewed; the funds must be used for a particular set of AIDS activities that constitute either a new project or additional AIDS activities for an existing project and for which the director has determined that there is a significant need; they cannot be used for more than two years for a specific set of activities; they cannot be used for purposes that were disapproved by Congress either explicitly or implicitly; and they cannot be used for construction or renovation, nor can they be invested. The director of OAR is required to submit to Congress annually a report describing how and why the funds were used during the previous fiscal year.
THE BUDGET PROCESS
Funding of ADAMHA and NIH AIDS activities over time has taken place within the larger context of the overall federal budget process and has been directly affected by it. Appreciating the length and complexity of the budget process is critical for understanding the problems faced by rapidly changing and expanding research areas such as AIDS and by organizations that are in the process of restructuring, as were ADAMHA and NIH during the course of this study.
The overall budget process has been relatively stable over time and generally has operated similarly at ADAMHA and NIH. Although there were minor changes from time to time during the 1980s and early 1990s, and even with the reorganization of ADAMHA, the larger budget structure of PHS, HHS, the Office of Management and Budget (OMB), and Congress remained the same. In addition, PHS and HHS always had an unwritten policy of treating the research budgets of NIH and ADAMHA essentially the same.
At any one point in time, an institute (whether at ADAMHA, SAMHSA, or NIH) must consider its research program in relation to three separate budgets. This is illustrated in Figure 5.4, which displays the current budget calendar for an NIH institute.
At the end of calendar year 1993, the institute was in the midst of utilizing its current (FY 1994) funds (Execution), which were allocated through September 30, 1994. At the same time, the institute was involved in presenting and defending (Justification) its next year's budget (FY 1995) to reviewers ranging from OAR at NIH, HHS, OMB, and various committees within the Congress, as
well as to various interest groups who become more involved as the budget becomes public. Starting approximately in February 1994, the individual institutes and the NIH central budget office (and OAR) began specific plans (Formulation) for the FY 1996 budget, which will be in effect from October 1995 to September 30, 1996. In addition to dealing with issues directly related to the FY 1994 and FY 1995 budgets, the institute had to consider future budgets in relation to the "NIH Five Year Plan for HIV-Related Research" (the AIDS plan described above), the overall NIH Strategic Plan, and other cross-cutting plans that the individual institute may have had. The multi-year nature of this process is also demonstrated in Figure 5.5, which focuses on the FY 1994 budget planning cycle that began in November of 1991 and governed institute spending through September 30, 1994. (An additional complexity that is not shown in Figure 5.5 is that prior to reorganization, the White House Office on National Drug Control Policy reviewed the drug-related portions of the ADAMHA budget. In addition, the National AIDS Program Office [NAPO] assumed responsibility for reviewing the AIDS budget requests of all PHS agencies beginning in FY 1990.)
With this general overview of the budget process and its complexity in mind, one can now look at the more specific experiences of ADAMHA and NIH budgeting as they relate to AIDS funding.
ADAMHA BUDGET PROCESS, FY 1983 TO FY 1992
The budget process for ADAMHA remained relatively stable since FY 1983, when the first funds were appropriated for AIDS
research. There were minor changes in timing and procedures over the years, but these were largely superficial. The budget process normally began with a planning meeting in the fall between the administrator of ADAMHA and the assistant secretary for health, where overall concerns and priorities would be reviewed. Within the guidelines that may have resulted from that meeting, individual institutes and offices began their budget process in January (approximately 21 months before the fiscal year) by outlining major themes and proposing new program concepts or initiatives. This process generally took about three to four months to complete,
and was informed not only by science, but also by the existing commitment base (the amount of money already committed to multi-year projects), administration policy, congressional earmarks, and the prioritization process within the specific institute. Budget decisions also were influenced by historical budget trends and the consideration of balance between mechanisms (extramural, intramural, grants, contracts, research project grants, and center grants), and the balance between AIDS and other research. Unlike other PHS organizations, the ADAMHA institutes also had to consider the balance between research and services.
The AIDS budgets from the various institutes were reviewed by the administrator of ADAMHA with the help of an analyst in the planning office. The administrator, with advice from the senior staff, would decide on an overall budget request (AIDS and non-AIDS) to submit to the Public Health Service.
The ADAMHA budget process recognized that the agency was unique in its mix of research and services programs, and the budget attempted to implement the concept of an integrated ADAMHA mission, where progress depended on linking research and services with national leadership and advocacy on substance abuse and mental health issues. This concern fundamentally differed from budgetary concerns at NIH, where there was little worry about the level of services budgets and programs in other agencies (such as Health Resources Services Administration or Centers for Disease Control and Prevention).
Historically, within ADAMHA, the three institutes approached planning and budgeting for AIDS in very different ways. NIAAA developed its overall research budget and then determined what portion was related to AIDS research. NIDA, with a large investment in AIDS research but without a formal organizational structure for AIDS programs, developed its AIDS and non-AIDS budgets jointly. NIMH, with a formal Office on AIDS Programs, developed an AIDS budget separately from its non-AIDS budget and then put the two together to develop an overall NIMH budget. The manner in which HHS treated AIDS funding, either as part of various agency budgets or as a separate entity, also affected how the institutes structured and developed their budgets.
In 1991 the budget process at NIAAA, NIDA, and NIMH became more complicated with the uncertainty about whether or not the institutes would be moved from ADAMHA to NIH. Even after the decision to reorganize was made, the institutes still were involved in implementing the FY 1992 budget as part of ADAMHA. Also, until the fall of 1992, they were developing and presenting
their portions of the proposed ADAMHA budgets for FY 1993 and FY 1994. It was only in September of 1992 (the eve of FY 1993) that the three institutes began to be incorporated into the overall and AIDS-specific NIH budget processes. Obviously, the length of the budget process under normal circumstances, coupled with the type of sudden discontinuities caused by the ADAMHA reorganization, made planning difficult for all institutes and agencies involved.
WORKING WITHIN THE NIH PROCESS
Given the lead time between budget formulation and execution, most of the institute- and agency-level budget planning for FY 1993 and FY 1994—the first two years that NIAAA, NIDA, and NIMH were organizationally part of NIH—was conducted while the institutes were part of ADAMHA. The institutes first participated in the NIH budget process in September 1992, when they worked with NIH to develop materials for the HHS FY 1994 budget submission to OMB. Thus, while the three institutes had been organizationally part of NIH since October 1, 1992, the FY 1995 budget was the first NIH budget to include NIAAA, NIDA, and NIMH.
The institutes joined in the FY 1994 NIH budget process after the major internal HHS decisions about the NIH AIDS budget had been made. However, the former ADAMHA institutes were still affected by those decisions. The PHS budget request sent by the new (Clinton) administration to OMB provided $10.67 billion for NIH, including $1.3 billion for AIDS research. The allocation of the AIDS budget among the various institutes, offices, and centers was made by the director of OAR (and his staff) with the consensus of the director of NIH. The NIH director then made the final overall allocation of the NIH budget within the departmental mark, using the NIH strategic plan and her sense of where limited resources could best be utilized. While the administration proposed a 3.3 percent overall increase for NIH, including a 21 percent increase for AIDS, the allocations for the individual components ranged from negative numbers to significant double digit increases.
The FY 1994 budget review process was basically the same as it had been in recent years. Over the past few years, the major difference between the AIDS budget process and the overall process occurred before the preliminary budget was submitted by NIH. In FY 1994, these initial activities occurred before NIAAA,
NIDA, and NIMH formally became part of NIH. Also in FY 1994, the process was somewhat confused because of the change in administrations.
The major difference with the review of the AIDS budget by the NIH office of the director (OD) was the level of detail and the focus on science. Most OD-level budget review focused on mechanisms and appropriate balance, while the AIDS review was conducted on a project-by-project or, at the least, area-by-area basis. The OAR director reviewed the plan and made recommended cuts. The institutes could appeal to him to reconsider specific decisions, or the institutes also could appeal to the NIH director. After this phase, the OAR director took the proposed AIDS budget request to the NIH director and associate directors for final review. At this stage, the review broadened to include issues of program balance, mechanism balance, and magnitude of resources requested (both AIDS and non-AIDS), as well as the question of scientific opportunity. Again, unlike the rest of the NIH budget, the AIDS review did include some consideration of specific projects.
According to staff at NIAAA, NIDA, and NIMH, the FY 1994 review process was more top-down and the AIDS and non-AIDS processes were more similar than they had been in the past. NIH was given a departmental mark for AIDS and for its total budget that it then allocated to the institutes based on judgments made by the OAR director for AIDS and by the NIH director for the total budget. Apparently, the decisions in both cases were made based on judgments about where the scientific payoffs would be greatest.
In addition to considering issues of overlap and duplication, appropriate use of mechanisms, program balance, and scientific opportunities, the OD also considered the specifics of projects or project areas, the quality of the science, and the likelihood of progress when determining the NIH AIDS budget allocations. In FY 1994, as in all preceding years, once the institute received its budget allocation, the determination of which specific projects to fund was mostly a function of the grant review process, which is described next.
THE GRANT REVIEW PROCESS
The ADAMHA Reorganization Act of 1992 provided that the ADAMHA peer review systems, advisory councils, and scientific advisory committees utilized remain in effect through FY 1996 (ending September 30, 1996). The report language indicates that,
while the three institutes should become full members of the NIH research community, they should also retain their independence and integrity. In addition, it recognized that the institutes' current review procedures were developed over time to meet the specific and complex needs of the alcohol, drug abuse, and mental health fields, and they provide optimal specificity for the wide array of neuroscience, behavioral, clinical, and service research responsibilities (Senate Committee Report 102-131).
In most ways, the review procedures utilized by the former ADAMHA research institutes are very similar to the procedures at NIH. Both NIH and ADAMHA have used a dual review system that separates technical and scientific assessment of projects from subsequent policy decisions concerning programmatic, scientific areas in which projects will be supported. In addition, the scientific review process is kept separate from funding to ensure that program officials are not involved in making determinations on the scientific merit of research applications. These operational procedures had evolved as part of overall development of PHS policies for extramural research grants—a joint activity of various PHS agencies.
Procedurally, the review process at NIH and ADAMHA are also very similar. All ADAMHA and NIH grant applications are mailed to the Division of Research Grants (DRG) in its role as the central receiving point for PHS research grants. In addition, DRG's major management and scientific data systems (IMPAC and CRISP, respectively) have always incorporated information about the ADAMHA research grants. The application form and basic instructions for submitting a research grant are the same for both organizations.
The major difference in processing grants is that at the former ADAMHA institutes, all grants are reviewed at institute-specific initial review groups (IRGs), while at the rest of NIH, most grants are reviewed by DRG study sections. At NIH, only the larger and more complicated grants (centers and program projects) are reviewed by institute-established special review groups (SRGs). In both cases, however, all grants receive their second-level review by the advisory council of a specific institute, and funding decisions ultimately are made by institute staff (with the director's approval).
The first level of grant review is conducted by technical experts, largely from outside the federal government, and is designed to evaluate competing applications based on scientific and technical merit. The second level of review is conducted by advisory councils to assess the quality of the first level review and to offer
recommendations based on relevance of the research to the institute's mission. The recommendations of both levels of review are advisory to the federal government and the final funding decisions reside with the institute director. Funding decisions are based not only on scientific merit and policy consideration, but may also consider administration policy, funding availability, and other factors.
At the first level of review, each application is considered at a meeting of the IRG (also called a "study section") and is either given a priority score (if it contains "significant and substantial merit") or not recommended for further consideration (NRFC). Priority scores provide a number for perceived quality by the IRG and they range from outstanding (100–150) to acceptable (350–500). To help equalize the diversity of rating styles (hard and soft graders) and grade creep (the increasing proportion of applications receiving scores of 150 or better), percentile scores are calculated using the priority score for each application.
Once the first level of review is complete, the reviewers prepare a summary statement (or pink sheet) for each application. These are sent to the National Advisory Council for the second level of review. This ensures that the scientific review was appropriately conducted and generates funding recommendations to the institute. If the council disagrees with the IRG on issues related to scientific and technical merit, it may recommend that the application be referred to the same or a different IRG for further consideration. If the IRG makes the same recommendation following reconsideration, its decision is final.
The recommendation by the advisory council completes the formal ADAMHA and/or NIH review of an application. After each council meeting (they usually occur three times each year) the institute makes a funding decision and prepares a funding plan or pay plan, which is based primarily on the scientific merit of the projects (as indicated by their percentile ranking), but also on the availability of funds and balance among research areas. The funding plan is a list of projects recommended for funding by the program office or division and approved by the institute director.
Since FY 1992, NIH has used a performance standard, called the "success rate," to measure the quantity and quality of grants funded over time. The success rate is the total number of competing research project grants (RPGs) funded divided by the total number of competing RPG applications received. The success rate varies by institute and year, and it is determined not only by the number
and quality of applications submitted but also by available funding.
The success of AIDS research applications has varied from institute to institute. At NIMH, AIDS grants (traditional investigator-initiated grants—R01s) have had higher success rates than non-AIDS grants; however, those success rates fell from 35.1 in 1989 to 13.5 in 1993 (Figure 5.6). At NIDA, AIDS grants (all RPGs) had higher success rates than non-AIDS grants in 1990, 1991 and 1993, but lower success rates in 1989 and 1992 (Figure 5.7). In general, NIDA RPGs consistently have had relatively high success rates (in the range of 28.4 to 41.1). Since the number of AIDS applications at NIAAA has been relatively small, the success rates may fluctuate dramatically from one year to the next (Figure 5.8). For example, in FY 1990, NIAAA received three competing AIDS applications and funded all of them. During the following year, NIAAA funded 6 of the 19 applications it received. In general, data for all institutes indicate that success rates for all research are declining, which is more likely a result of shrinking budgets and rising per-grant costs than of declining quality among grant proposals submitted.
NIH advisory committees include any committee, council, task
force, or group that is established to provide recommendations or advice on policies or other issues related to the missions of the NIH institutes. There are four distinct types of committees: scientific and technical peer review committees for research grant, cooperative agreement, and contract proposals; boards of scientific counselors to review the intramural research programs; program advisory committees to advise on specific research programs and future needs; and national advisory councils to offer advice on policies and programs and to conduct the second level of peer review. Membership on NIH advisory committees is based on demonstrated leadership and excellence in behavioral and biomedical research. Public members are chosen for an active interest in the particular mission of the committee.
NIAAA, NIDA, and NIMH each has its own National Advisory Council, Extramural Science Advisory Board, and Board of Scientific Counselors. Each institute also has several review committees (IRGs), including at least one separate review committee for AIDS research. These AIDS-specific review committees were primarily established in response to the mandatory expedited six-month review for AIDS applications (as opposed to the standard nine-month review).
NIAAA uses two scientific review committees for all applications: Alcohol Psychosocial Research Review Committee and Alcohol Biomedical Research Review Committee. All AIDS applications are reviewed by a subcommittee of the Biomedical Review Committee, called the Immunology and AIDS Subcommittee, which has twelve committee members with scientific expertise in medicine, microbiology/immunology, pathology, sociology, psychology, public health, behavioral science, cellular biology, and psychoimmunology.
NIDA has five research review committees, one of which is the Drug Abuse AIDS Research Review Committee. This committee has two subcommittees: the Biomedical and Clinical Subcommittee (DAAR-1) and the Sociobehavioral Subcommittee (DAAR-2). The twelve members of DAAR-1 represent pediatrics, pharmacology, epidemiology, anthropology, psychiatry, neurology, neuropsychology, and psychology. The thirteen members of DAAR-2 represent the fields of epidemiology, sociology, psychology, psychiatry, other social and behavioral sciences, and addiction medicine.
NIMH has more than twenty separate review committees. The Mental Health, AIDS, and Immunology Committee has two subcommittees for reviewing AIDS applications: the Behavioral, Clinical, and Psychosocial Subcommittee and the Psychobiological, Biological,
and Neuroscience Subcommittee. The ''psychosocial" subcommittee includes five psychologists, six psychiatrists, one epidemiologist, and representatives from the fields of public health, prevention, community research, and family studies. The "neuroscience" subcommittee includes expertise in neurology, psychology, psychiatry, physiology, pharmacology, microbiology, and immunology.
Of nearly 200 review groups of the DRG at NIH, there are seven study sections for AIDS and Related Research. All but two of them are focused on biomedical aspects of HIV/AIDS and primarily include representation from the following fields: experimental and clinical immunology, molecular biology and genetics, virology, microbiology, and clinical medicine. One study section is focused exclusively on the neuropsychologic, neuropathologic, and neurophysiologic analysis of HIV-infected individuals and includes expertise in neurology, neurobiology, psychology, pathology, and clinical medicine. Just one study section focuses on behavioral medicine and related disciplines, but it primarily includes representatives from medicine, specifically nursing, general medicine, behavioral medicine, public health, pediatrics, and psychiatry.
The second level of review at NIAAA, NIDA, and NIMH is currently conducted by the National Advisory Council of each institute. Although the ADAMHA Reorganization Act mandated that the review processes (both at the first and second levels) be maintained through 1996, it is not yet clear how AIDS research applications at the three institutes will be reviewed when this period ends. (As this report was being written, it was reported that the NIH director had requested that the former ADAMHA institutes enter into negotiations with the NIH Division of Research Grants earlier than 1996 and that the institutes agreed [Federation of Behavioral, Psychological and Cognitive Sciences, 1994].)
Given the composition of the NIH-wide AIDS study sections, some have expressed concern that applications related to the biobehavioral and social-behavioral research foci of NIAAA, NIDA, and NIMH—that is, cross-disciplinary focus—will not fare well should the three institutes be subject to the overall NIH review process. Indeed, this was a significant reason for the four-year retention of the institutes' pre-NIH review systems. While it is too soon to determine if this concern is well founded, the committee is aware that it is widespread among the institute program staff and the external research community. Furthermore, given
this study's finding about the extent of cross-disciplinary research at NIAAA, NIDA, and NIMH, the committee believes that this concern deserves particular consideration.
Under current NIH procedures, described above, each member of a review panel assigns to a proposal a score from 1 (best) to 5 (worst). These ratings are averaged, giving equal weighting to each. Proposals are then ranked according to the resulting "priority score" (after multiplying by 100) and characterized by the resulting percentile. Generally speaking, proposals are funded in order, starting with the lowest percentile. In a competitive field, that might mean funding up to the 6th or 8th percentile. As a result, the only proposals likely to be funded are those receiving extremely good ratings from the great majority of panel members.
This procedure is inherently prejudicial to innovative and collaborative proposals. Unless some adjustments are made, the agencies will continue to discourage proposals involving new disciplines and multiple disciplines, or cooperation between investigators and community-based organizations. Unless instructed otherwise, responsible panel members likely would regress their scores for innovative proposals toward the mean of the scale. Because it is inherently harder to predict how such research will turn out, it is difficult to have the confidence needed to rate it as extremely good. As a result, a proposal would have to show an extraordinary degree of innovation in order to compensate for this inherent conservatism.
Review panel members also are likely to regress the scores that they give to proposals involving disciplines other than their own. They are not in a position to know what "extremely good" is in another discipline. As a result, a single-discipline proposal will be less competitive to the extent that it is judged by specialists from other disciplines. A cross-disciplinary proposal is almost guaranteed to have portions that will be unfamiliar to each reviewer.
In cases in which priority scores are compared across programs, single-discipline panels are more likely to have proposals with extremely good priority scores than are mixed-discipline panels. That could be interpreted as evidence that the single-discipline panels receive better proposals. However, it could just be that they have more unitary (or insular or ideological) evaluative standards. The more powerful a discipline is at an agency, the more likely it is to have its own review panel. Politically weaker disciplines are more likely to be combined into mixed-discipline panels. As a result, a naive interpretation of priority scores across panels is likely to create a situation where the rich get richer.
These problems arise without any deliberate attempt to create barriers to new disciplines and approaches, which might want a share of the resources allocated to a topic such as AIDS. Rather, they provide an honest way of ensuring that such prejudices are part of the evaluation system. Those interested in preserving business as usual can make candid evaluations, then let the system do the discriminating. There may be individual members of review panels who are aware of these threats, and make various efforts to overcome them. However, the extent and impact of those efforts is unknown.
These problems may be addressed either by working within the existing procedures or by changing them. Within the system, one possible solution is to weight ratings by the disciplinary competence of the raters. Although that change might reduce the (inadvertent or deliberate) censorship of other disciplines, it would not solve the problems of innovative or cross-disciplinary proposals. Another possibility is to rescale scores so that they are standardized within different categories.
More fundamental changes might include convening special review panels, affording votes to ad hoc reviewers, and dedicating funds to unconventional proposals (so that they compete amongst themselves). In implementing these proposals NIH would have to consider the appropriateness of existing review staff for creating the appropriate panels and procedures. Also, such proposals would have to be in accord with the scientific principles of evaluation (often called psychometrics).
As this report was being written, the current NIH director established a series of inter-NIH panels to examine the ways in which the peer review process could be streamlined and more innovative projects rewarded. Although this action reportedly was prompted by the "reinventing government" paradigm of the Clinton administration, it is not intended to focus solely on speeding up the process, but rather will take into account some of the issues about study section composition and scoring raised above (Federation of Behavioral, Psychological and Cognitive Sciences, 1994).
In deciding whether it is worth the effort to undertake such reforms, the primary concern in the context of this report should be the effects such reforms will have on AIDS research and the AIDS epidemic. However, an important secondary consideration should be the relationship between the agencies and the scientific community. The current evaluation system alienates scientists whose work is treated prejudicially, in particular, those who attempt to cross disciplinary boundaries. Not only will this reduce
their willingness to work on AIDS-related problems, but it also will damage the reputation of existing research. Over time, that may imperil continuing public support for business as usual, if the federal research establishment is perceived to be unresponsive to the requirements of the AIDS epidemic.
CONCLUSION AND RECOMMENDATIONS
The development and management of the AIDS programs at NIAAA, NIDA, and NIMH takes place in a larger context. This context includes legislative and budgetary processes that are often in flux but still wield control over the resources available to manage the institutes' programs. In recent years, the most significant changes in this regard have been the structural reorganization of ADAMHA, which moved the institutes to NIH, and the new budgetary authority of the Office of AIDS Research at NIH, which changed the nature of how dollars will be allocated for AIDS research at the individual institutes. These changes were superimposed on the already complex budget process that governs all federal spending and together produced a climate of uncertainty for the management of the AIDS research programs at NIAAA, NIDA, and NIMH.
To maintain some stability, the institutes were allowed to retain their existing grant review procedures for four years (through FY 1996). This was seen as important for ensuring the continued support of behavioral and social research that historically had been better at ADAMHA than at NIH. The fact that this situation will be revisited in FY 1997 raises some concerns about how the cross-disciplinary research so important for advancing knowledge in HIV prevention and intervention will fare in the future.
RECOMMENDATIONS RELATED TO THE CONTEXT OF AIDS RESEARCH
5.1 The committee recommends that NIAAA, NIDA, and NIMH develop new programs and grant review procedures to encourage and facilitate innovative, collaborative, and cross-disciplinary proposals.
5.2 The committee recommends that the NIH task force charged with streamlining peer review consider alternative scoring schemes that would favor cross-disciplinary and innovative research proposals.