Assessing the Value of Research in the Chemical Sciences (1998)

Mary Groesch

National Institutes of Health

At the National Institutes of Health (NIH), the response to the Government Performance and Results Act (GPRA) is being developed and coordinated by the Office of Science Policy, within the Office of the Director. The GPRA was enacted to strengthen public confidence in government; to improve program management, effectiveness, and accountability by focusing on results; and to facilitate and improve congressional decision making with respect to program budgets. This paper describes some of the actions NIH is taking to implement the GPRA.

There are three required components of the GPRA. The first is an agency strategic plan—a 5-or-more-year plan that states the agency's mission and its long-term goals. The plan must be updated at least every 3 years. For GPRA reporting purposes, the agency that is required to put forward a strategic plan is the Department of Health and Human Services, not NIH. We are, however, included within the HHS strategic plan. In addition, we have been working on our own GPRA strategic plan to guide the development of our performance plan.

The performance plan is the second component of the GPRA. In this case, NIH and HHS both are required to develop a plan. A significant amount of our effort has therefore gone into developing an annual performance plan, which is submitted along with the yearly budget request. The performance plan outlines our expected program outcomes and means and sets forth performance goals and performance indicators for each. The third component of the GPRA is the performance report, which includes an assessment of how well we did in meeting our goals.

The NIH strategic plan for GPRA is straightforward in concept, although the implementation of these concepts is anything but simple. Our mission is to sponsor and conduct research that leads to better health for all Americans. This is reflected in two long-term goals, to advance medical knowledge through research, and to enhance and maintain our research capacity.

We accomplish these goals through the activities of three core programs, and this is the level at which the results are aggregated. We decided to aggregate the programs from our 24 institutes, centers,

and divisions into three core programs: Research, Research Training and Career Development, and Facilities.

The GPRA allows an agency to either aggregate, desegregate, or consolidate its program activities in the way most appropriate to performance reporting. We consider it essential that the activities of our 24 different institutes, centers, and divisions be aggregated. Our rationale for this with respect to NIH's research programs is simple. Disease is typically systemic. It is influenced by multiple factors and affects more than one organ or body system. Therefore, expertise across a wide range of disciplines is necessary to establish the mechanisms and etiology of disease and to develop strategies for diagnosis, treatment, and prevention. Multidisciplinary expertise necessarily cuts across institute boundaries; therefore, we have to combine the efforts of many of our institutes to make progress on any particular disease or disability. Following the same line of reasoning, training and facility needs are also not unique to any one particular institute or center. Therefore, we must coordinate and collaborate these efforts across NIH to ensure that we can meet our long-term goals.

To assess the performance of the three aggregated programs, we have established for each a framework of expected outcomes and program means. As the name implies, the expected program outcomes are the expected, tangible results of NIH programs. The program means reflect the process and management activities that we undertake to support the conduct of our programs and to enable us to achieve our goals. For each of the expected program outcomes and program means, we have set performance goals that identify what we expect to accomplish. Each goal is accompanied by a set of performance indicators, which will be used to measure our success in achieving it. I will discuss a few examples of the expected program outcomes, the program means, and the performance goals and indicators that we are considering for our research program. I use the term "considering" because everything we are doing for the GPRA is very much in a state of development. What I tell you today may not necessarily be what will soon be forwarded to Congress.

We have defined two broad expected program outcomes that are expected. The first is increased understanding of normal and abnormal biological functions and behavior. The second is improved prevention, diagnosis, and treatment of diseases and disabilities. For each expected program outcome in the research program, we have defined performance goals, many of which are not assessable by quantitative measures.

As previous speakers have pointed out, agencies whose missions encompass fundamental science face unique challenges in implementing the quantitative evaluations that seem to be preferred under the GPRA. This is an issue to which we have devoted much time and effort. We have had extensive internal discussions and have also sought substantial input from outside groups. In particular, the National Science Foundation has been very helpful to us. They have generally been somewhat ahead of NIH in the GPRA process, and we have benefited from their insights, experiences, and lessons learned as they, too, have grappled with GPRA.

First and foremost, we concluded that just because something can be counted does not mean it should be reported. We decided that a combination of both qualitative and quantitative performance goals and indicators is the most meaningful response for NIH. We think that strictly numerical results are neither feasible nor sufficient to capture the breadth and impact of NIH's research activities. Although conventional scientific and research metrics can be relevant, they measure only some of the dimensions of output. These metrics provide important data, but they alone cannot assess the full scope of the quality of our work, its relevance, and the impact of our research program on human health that

is necessary for evaluating NIH. For these reasons, qualitative measures will play a major role in how we report on NIH's performance.

For example, we are developing narrative descriptions of our research accomplishments. This places a specific incremental advance into its larger context—that is, describes what was previously known and unknown, the nature of the accomplishment, its contribution to understanding and improving human health, its significance to advancing a field or fields of science, the next steps in the research, and, where possible, the economic impact of the advance.

Nonetheless, we plan on using quantitative goals and indicators whenever possible. Genomic research, for example, lends itself to this approach better than much of the other research that we conduct and support. For example, one of our goals is to complete the development of important genomic resources, such as the DNA sequence of the human genome and the genomes of several other important model organisms. Quantifiable indicators for this goal include our progress toward completing the sequence of the human genome by 2005; progress toward our goal of completing the sequence of zebrafish by 2002; progress in sequence tagging the genes expressed during tumor development; the number of new gene sequences that have been added to the GenBank; and progress in completion of the rat genomic libraries, genetic map, and expressed sequence tag map, with a target date of 1999.

I have touched on the expected program outcomes portion of the research program and the goals and indicators we have developed for those. The other portion of our performance plan is program means: the way that we get to our expected outcomes—that is, the management activities. The HHS and OMB have encouraged us to provide specifics, stressing the use of metrics wherever possible for this portion of GPRA reporting.

In terms of the program means, the types of means goals and indicators that we are considering include priority setting, grants administration and peer review, communication of results, technology transfer, management and administration, and collaboration and coordination. Some of the specific performance goals for grant administration and peer review, for example, could be to ensure that submitted grant applications receive fair and appropriate review, to improve and enhance the electronic research administration, to improve and enhance communication with the extramural community, and to improve customer service by expediting and processing the award of grant applications. Some of the indicators for improving customer service include expediting and processing the award of applications, implementing pilot projects of different ways to process applications, implementing on-time procedures to eliminate unnecessary submission of administrative information, and developing pilot studies for referring grant applications to program offices.

The third component of GPRA is assessing NIH's performance. The law calls for an external review of our performance. Although this section sounds like it could be the most relevant to the discussion for this meeting, unfortunately it is the part that we cannot address because we are still at the drawing board. We are exploring who will do the assessment and how it will be done. However, I can at least describe some of the ideas that we are considering.

Our current plans call for NIH program assessment to be conducted by a subcommittee of the advisory committee to the director of NIH. Such an assessment subcommittee might include one or two individuals from the advisory committee as liaisons, as well as grantees, members of the lay public, patient advocacy groups, professional societies, other federal agencies, and various members of the NIH staff.

How the assessment group will function and what form its report will take are still under discussion.

For example, what will the external reviewers assess every 3 years—all NIH research, or several selected research programs or areas? Or, will they look at areas of research emphasis established by the NIH director? These include six broad categories: the biology of brain disorders, new avenues for development of therapeutics, genetic medicine, new approaches to pathogenesis, new preventive strategies against disease, and advanced instrumentation and computers in medicine and research. Another possibility is a combination of the above.

Although NIH is well along the path to developing a response to GPRA, a number of issues have yet to be decided.