Assessing the Value of Research in the Chemical Sciences (1998)

Patricia M. Dehmer

U.S. Department of Energy

If you can look into the seeds of time, And say which grain will grow and which will not, Speak then to me, who neither beg nor fear Your favors nor your hate

—Shakespeare, Macbeth, I:iii, 58-61.

Efforts have intensified in recent years to define value as it relates to scientific research, to determine the tools and metrics by which that value can be quantified, and to assess the results of scientific research by using these tools and metrics. The incentive derives in part from a series of laws and executive orders, including the Government Performance and Results Act (GPRA) of 1993, that focus on performance management. This paper presents an overview of how these issues have been addressed within the Office of Basic Energy Sciences (BES) at the Department of Energy (DOE).

The BES program is prototypical of a large, diverse, and robust fundamental research program that exists within a mission agency. Part of its value derives from this special role, which is neither that of pure curiosity-driven research programs supported, for example, by the National Science Foundation nor that of applied research and development (R&D) programs supported by industry. Rather, the BES program supports fundamental research with a long-term objective. That this is not immediately seen as an oxymoron is testimony to the collective successes of the BES program and of other basic research programs that still exist within mission organizations.

This discussion relies on many thoughtful studies in performance measurement and assessment of basic science. Special acknowledgment is due to those who, in the early years of preparation for GPRA, spoke obvious truths about measurement and assessment of basic science while under pressure to derive measurement systems that tally results. Among these truths, several are of particular import: that the societal outcomes of basic research—new ideas and knowledge—are often unpredictable and may not be immediately apparent; that the measurement of a system will perturb it, often in ways that are unexpected; and that it is important to measure what matters, which frequently includes attributes that cannot be quantified.

This paper is not intended to be a scholarly review of performance measurement and assessment; therefore, references to important contributions made by others are not collected here. Rather, based on the literature and on personal interactions, the present discussion is a summary of the resulting philosophy, thoughts, and actions of those who manage the day-to-day activities of the large BES program.

Even in the absence of GPRA and similar federal actions, performance measurement and assessment would exist within BES, because it is a part of good management practice. However, as with other organizations that support basic research, understanding and implementing performance management is most definitely a work in progress; thus, the obvious caveats apply.

With an FY 1997 appropriation of approximately $650 million, the BES program within the DOE's Office of Energy Research (ER) is one of the nation's primary sponsors of fundamental research in materials sciences, chemical sciences, geosciences, plant and microbial sciences, and engineering sciences. The program funds more than 2,400 researchers at 200 institutions nationwide and supports 17 major national user facilities. Included among these are the four major synchrotron radiation light sources; four neutron sources; and a number of specialized facilities for electron-beam micro-characterization, materials synthesis and processing, combustion research, pulsed radiolysis, and ion beam studies. Over 4,500 users, including hundreds of industrial scientists from about 100 U.S. companies, are accommodated at the major BES scientific user facilities each year. Detailed descriptions of the BES programs and the facilities as well as links to other related sites can be found on the BES home page at <http://www.er.doe.gov/production/bes/>.

In creating a portfolio of basic research in support of DOE's energy mission, BES has evolved an eclectic set of programs, many of which are cross-disciplinary or multidisciplinary, yet all of which have strong disciplinary grounding. In the area of the chemical sciences—the area addressed by this Chemical Sciences Roundtable workshop—BES research programs include topics such as analytical chemistry; atomic, molecular, and optical science; batteries and fuel cells; chemical kinetics; combustion dynamics; electrochemistry; heavy element chemistry; homogeneous and heterogeneous catalysis; organometallic chemistry; photochemistry; photosynthetic mechanisms; separations science; solar energy conversion; and thermophysical properties.

The first step in performance management is defining the goals of the organization. As Yogi Berra allegedly said, ''If you don't know where you are going, you won't know when you get there." Therefore, the BES approach to performance measurement and assessment relies heavily on the formulation of and subsequent adherence to the mission, goals, and tenets of the program.

The mission of the BES program, which derives both from the Energy Policy Act (EPACT) of 1992 and from the DOE and ER Strategic Plans, is as follows:

The goals of the BES program—which, in turn. derive from EPACT, the DOE and ER Strategic Plans, and the BES mission statement—are to do the following:

•  To maintain U.S. world leadership in those areas of the natural sciences and engineering that are

• relevant to energy resources, production, conversion, and efficiency and to the mitigation of the adverse impacts of energy production and use;
•  To foster and support the discovery, dissemination, and integration of the results of fundamental, innovative research in these areas;
•  To provide world-class scientific user facilities for the nation; and
•  To act as a steward of human resources, essential scientific disciplines, institutions, and premier scientific user facilities.

BES measures performance in four areas that together characterize its special role. The first three relate to the fundamental tenets or principles of BES, which correspond directly to the goals described above. These tenets are (1) excellence in basic research; (2) relevance to the energy mission of the agency and, moreover, to a comprehensive national energy agenda; and (3) stewardship of research performers, essential scientific disciplines, institutions, and scientific user facilities. Combining and sustaining these tenets are the management challenge—and it is a significant challenge—of BES. Therefore, the fourth area that BES evaluates is program management.

The principles embodied by the words excellence, relevance, and stewardship together determine the BES program portfolio, guide its funding choices, and ultimately provide the socioeconomic value of the program. Indeed, these three words—and the attendant principles—may well encompass the value of R&D activities performed in all sectors. Of course, each organization must determine its own set of activities within the three categories according to its mission and goals and must ultimately provide value to different constituencies. For example, industrially funded R&D activities provide value to shareholders, while federally funded R&D activities provide value to U.S. taxpayers—that is, to the nation as a whole. Research agendas, time scales, and risk factors are correspondingly different.

BES measures performance in the four areas described above in a number of ways, which separate naturally into four categories: (1) peer review; (2) indicators or metrics (that is, things that can be counted); (3) customer evaluation and stakeholder input; and (4) other assessments, which might include cost-benefit studies, case studies, historical retrospectives, and annual program highlights. The resulting BES performance measurement matrix is shown schematically in Figure 10.1.

During FY 1997, this matrix of "activities to be measured" versus "measurement techniques" was chosen as the framework by which BES would formalize performance measurement. A number of activities, sometimes quite unique from one another, can exist within an individual cell of the matrix. For example, the upper left cell shows peer review as a tool (in fact, it is the tool) for measuring excellence of basic research programs. Peer review can include mail peer review; panels of experts assembled to review simultaneously a number of proposals in a given subject area; site reviews by visiting committees; and special reviews overseen, for example, by the federally chartered Basic Energy Sciences Advisory Committee (BESAC) or by the National Research Council. Quite often more than one type of peer review method is used for a given program; this is particularly true of large programs and the scientific user facilities. Figure 10.1 also shows schematically that not every measurement technique is equally important to each activity. For example, peer review is very important for assessing excellence in basic research, but metrics and customer evaluation are not. Conversely, metrics and customer evaluation are very important tools for assessing the effectiveness of the scientific user facilities.

Ongoing planned and potential new activities related to performance measurement in BES are all accommodated by this four-column by four-row matrix construct. Furthermore, this matrix provides a

FIGURE 10.1

Office of Basic Energy Sciences performance measurement matrix.

comprehensive overview of performance measurement and allows a ready assessment of strengths, weaknesses, and gaps in performance measurement activities. The BES matrix is a modification of a simpler one derived by the Army Research Laboratory (ARL) for the measurement of its own performance. In July 1994, the Office of Management and Budget designated the ARL as a Pilot Project for Performance Measurement; as such, the ARL was the only R&D laboratory included in the GPRA pilot process. The mission, goals, and objectives of the ARL are different from those of BES. As a result, the row and column headings of the ARL matrix are naturally different from those adopted by BES.

A number of activities that might be considered essential or "foundation" performance measurement activities have long been in place in BES. These include peer review of research programs and customer surveys of the scientific user facilities. However, literally dozens of other activities and indicators can be envisioned for inclusion in the performance measurement activities. It quickly becomes apparent that an important management challenge is choosing a few significant items to target for special attention. Different activities and indicators may be added to the matrix or may be emphasized in successive years. In this way, a balanced system of performance measurement will evolve. For the next few years, BES has committed to select a few activities each year that address different aspects of performance measurement and that, taken together with ongoing activities, will strengthen performance measurement.

For example, during FY 1997, BES began two major activities designed to strengthen and formalize performance measurement in the future. These were (1) codification of the peer review process for research at the DOE laboratories, using a process analogous to that described in 10 CFR 605 (the Code of Federal Regulations) for the university grant program and (2) the development of a new survey tool

for the 17 BES scientific user facilities, in collaboration with the facility directors and the facility user coordinators. In addition, a number of other activities related to performance and to the management of basic research programs were conducted; for example, BES initiated a pilot study to assess the culture that promotes excellence in basic research at the DOE laboratories.

Details of some of these and other activities relating to performance measurement in the four areas to be measured are given below. However, this is not meant to be an exhaustive discussion of each cell within the matrix.

Excellent basic research produces new knowledge and ideas that endure, that change the way people think, and that are widely used by others. Intuitively, we suspect that there is value associated with the production of new ideas and knowledge, particularly as knowledge and ideas affect products and processes. Many studies have been done that link outputs of basic research with positive societal outcomes. More recently, economic theories have been proffered that suggest that ideas and knowledge are even more powerful drivers of change and of socioeconomic value than previously thought. Professor Paul M. Romer of Stanford University' s Graduate School of Business is well known for his work in this area.

Managing for excellence in basic research, that is, for new knowledge and ideas, is tantamount to managing for the unexpected. Every major organization that supports basic research has faced this conundrum. Over the years, peer review has emerged as the dominant (and perhaps the only valid) tool for measuring the technical excellence of basic research. The formal peer-review process used by the BES program is that used by the Office of Energy Research for its extramural grant program; this process is summarized in 10 CFR 605. The BES program has adapted 10 CFR 605 so that it may also be applied to the research programs and the scientific user facilities in the DOE laboratory system. First among the criteria used for selection of proposals is scientific merit. BES will soon begin reviews of its own management to assess the implementation of the peer-review system, including timeliness of decision, methods of review (mail, preproposals, panels, combinations), and demographics of reviewers.

The BES program not only assesses the degree of excellence of the basic research that it supports by using the standard techniques encompassed by peer review, but it also seeks to understand and foster the culture that promotes excellence in basic research. In a formal study headed by Gretchen Jordan of Sandia National Laboratories (SNL), the BES program is seeking to identify and assess the institutional and other factors that foster an environment for excellence in research in the DOE laboratories. The short-term goals of this study were first to identify the factors that promote excellence in basic research and then to develop a tool (a survey) to assess and improve the environment. These steps have already been completed by a panel of working scientists from SNL and managers from both SNL and BES. The resulting self-assessment survey has been beta tested on two groups at SNL. Perhaps not surprisingly, the most important factor determined by the panel was to hire and retain the best. Longer-range goals for this study include exploring the differences in the environment required to foster excellence in basic research, applied research, and technology development, and to compare and contrast research environments in national laboratories, universities, industries, and other types of U.S. institutions. A not-so-subtle hidden agenda was to remind researchers and managers that excellence in basic research is a BES priority. An unanticipated result of this effort has been a very high level of interest at other DOE laboratories, with several volunteering to participate in the study. As with all work supported by the BES

program, it is expected that this study will be published in the archival literature and will contribute to the body of knowledge directed at conducting and managing basic research.

Making basic research relevant requires that the BES program set basic research directions in keeping with the DOE missions. The program must also promote the transfer of the results of basic research to contribute to DOE missions in areas of energy efficiency, renewable energy resources, improved use of fossil fuels, reduced environmental impacts of energy production and use, science-based stockpile stewardship, and future fusion energy sources.

To bring about research relevance, BES sets strategic research directions through working relationships with other DOE programs; research workshops involving input from the scientific and technical communities; the promotion of open information transfer and exchange of ideas between the basic and applied research communities; and, finally, the sponsorship of selected high-impact research collaborations and partnerships. Individual research projects are funded based on peer review, as discussed above.

DOE's national laboratory system plays a special role in the ability of BES to effectively integrate basic and applied research by providing opportunities to co-locate activities at these sites. For example, about one-third of the scientists supported by the BES program at the DOE national laboratories also receive support from at least one of DOE's technology programs. BES also aggressively fosters the integration of basic and applied research through the formation of "real" and "virtual" laboratories that bring together researchers with different backgrounds, expertise, and problems.

An example of a "virtual" laboratory is the Center of Excellence for the Synthesis and Processing of Advanced Materials. The center involves 12 DOE laboratories as well as several universities and industries. The eight projects currently under the center's umbrella (which includes topics such as metals forming, metals joining, and high-efficiency photovoltaics) involve many disciplines and require that each participant bring unique expertise and talents to the collaborations. The Center Steering Committee is composed entirely of representatives of industry and the DOE technology offices.

An example of a "real" laboratory is the Combustion Research Facility (CRF) at Sandia National Laboratories, Livermore, California. The CRF, which is one of the BES scientific user facilities and is operated by the BES Chemical Sciences Division, houses 20 laboratories. These include the Turbulent Diffusion Flame Facility, the Burner Engineering Laboratory, the Multi-fuel Combustor, and numerous laboratories for chemical dynamics, chemical kinetics, imaging of turbulent reacting flows, spray combustion, internal combustion engine studies, and coal research. Projects under way range from fundamental studies of combustion-generated pollutants, to development of new laser diagnostic techniques, and applied studies of processes in internal combustion engines. Basic research programs at the CRF are supported by BES, and applied research programs are supported by various programs in the DOE technology offices, including the Offices of Energy Efficiency and Renewable Energy, Fossil Energy, and Defense Programs. About 25 percent of CRF users are from U.S. corporations; the other 75 percent come from universities and national laboratories.

Another way that BES fosters integration of basic and applied research is through the newly initiated Partnerships for Academic Industrial Research (PAIR) program. This program is designed to encourage and facilitate research partnerships between academic researchers, their students, and industrial researchers. As discussed above, the BES program, through support of basic research co-located with applied research at the DOE laboratories, has had considerable opportunity to observe that both basic and applied researchers contribute to problem definition, discovery, and understanding and that

the transition from discovery to development and deployment is not linear. The PAIR program encourages similar interactions between basic and applied researchers in academia and industry. Additionally, the PAIR program is intended to encourage universities to consider novel research activities and to foster faculty participation in nontraditional partnerships, which may have been discouraged in the past. The PAIR program requires evidence of a working relationship between the academic and industrial researchers and further requires that a graduate or postgraduate student spend at least 4 weeks a year in the industrial setting. Research funds are provided to the academic partner only.

As shown in Figure 10.1, assessment of these efforts incorporates the full range of measurement tools. For example, peer review of programs may involve representatives of the DOE technology programs and industry; BES advisory groups and steering committees will have similar membership. BES tracks the number of cooperative research and development agreements (CRADAs) that have resulted from BES-supported work; for example, BES funding has led to 120 CRADAs, which extend the basic research to applications and development. In addition, there are literally hundreds of collaborations between BES researchers and industrial researchers.

BES also supports a formal study led by Professors Barry Bozeman and David Roessner of the Georgia Institute of Technology to identify and measure the value to industry of research supported by BES. This study uses an exploratory approach known as R&D Value Mapping, which involves measuring a variety of project attributes (for example, resources devoted to the project, or number of industrial participants) against outcomes. In particular, the work employs a modified case study design to identify the impacts and benefits that industry experiences after interacting with basic research projects supported by BES. Moreover, the work identifies the industry impacts. Benefits will be causally linked to a series of upstream factors that can be influenced by DOE program managers, such as project funding mix and choice of mechanism for industry interaction. This method combines the strengths of case studies with those of systematic, quantitative analysis. This work, which identifies the factors that foster relevance in basic research, is philosophically analogous to the study described above, which seeks to determine the factors that foster excellence in basic research.

Finally, as an adjunct to the formal measurement and assessment techniques discussed above, it is also appropriate to use more qualitative retrospective analyses and annual highlights. When properly executed, these narratives can be compelling summaries of the relationships between the outputs of basic science and the outcomes that affect society. For example, based on a collection of more than 800 summaries of interactions involving BES researchers and industrial researchers (BES funds almost no industrial research directly), BES recently published a booklet, Basic Energy Sciences—Serving the Present, Shaping the Future, that gives an overview of the many areas in which basic research affects U.S. industry.

Stewardship requires that BES establish and maintain stable, essential research communities, institutions, and scientific user facilities. For example, BES serves as the nation's primary or sole supporter of a number of important subdisciplines such as heavy element chemistry, natural and artificial solar energy conversion, catalysis, organometallic chemistry, combustion-related science, separations science, neutron science, radiation chemistry, and radiation effects in materials. Maintaining these communities is an important responsibility of BES.

Furthermore, BES has a major responsibility for the planning, construction, and operation of major national user facilities and for encouraging the use of these facilities in areas important to BES activities, and also in areas that extend beyond the scope of BES activities, such as structural biology, environmen-

tal science, medical imaging, rational drug design, micromachining, and industrial technologies. Approximately 40 percent of the total BES budget is given to the operation of its 17 user facilities. Considerable additional funds go to support research at the facilities and to develop the next generation of tools, instruments, and facilities themselves.

Over the years, formalized processes have evolved to plan for new or upgraded facilities and to assess the progress of facility construction. These processes, for which ER and BES are well known, involve substantial participation from the scientific and technical communities. The planning process for new facilities frequently involves multiple workshops and symposia over many months or years and may involve hundreds of participants from the scientific community. Similarly, construction project reviews involve participation from the scientific and technical communities throughout the life of the construction project.

As an example, ER recently conducted a review of the BES-supported Conceptual Design Report (CDR) for the $1.333 billion Spallation Neutron Source project proposed for construction at Oak Ridge National Laboratory. This project, which will produce a next-generation spallation neutron source for neutron scattering, had been recommended in the mid-1980s as one of four major scientific user facilities needed for materials science and related disciplines. The need for and justification of this project have been validated numerous times since then by committees at both the National Research Council and DOE. The purpose of the CDR review was to assess the technical feasibility of achieving the proposed design and goals, the credibility of the associated cost and schedule estimate, and the adequacy of present and planned management arrangements to accomplish the scope of work. The DOE committee included 60 experts in the areas of project management, accelerator physics, front ends, linac systems, ring systems, neutron sources and targets, experimental systems, conventional facilities, environment and safety, and cost and schedules. This review was conducted by a group independent of BES and reported to the director of ER. These reviews (known as "Lehman Reviews" after Dr. Daniel Lehman, the director of ER's Division of Construction Project Management) are emulated by many other organizations that oversee large construction projects.

For those facilities in operation, BES uses the full range of measurement tools to assess performance. Peer review is conducted both by BES and, when warranted, by special panels charged by BESAC. In addition to peer review, both metrics and customer evaluation are important tools for evaluating operating facilities. Because of the increased interest in the large BES facilities (the synchrotron radiation light sources and the neutron sources) during the past few years by the Office of Management and Budget (OMB), the Office of Science and Technology Policy (OSTP), and the user community, it became apparent that a comprehensive, standardized survey tool was needed for the BES facilities. That tool was developed during FY 1997 in collaboration with the directors and user coordinators of the 17 user facilities. Not surprisingly, it was extremely difficult to agree on an ideal survey—one that was comprehensive, relevant to all the facilities, had standard definitions (for example, an appropriate definition of a "user" in a time when many users have collaborators and when increasing numbers of users gain access to facilities via remote electronic connections), and was sensitive to the increased effort needed by facilities and their users to complete such a survey.

The resulting BES survey has several parts: (1) a user-satisfaction minisurvey that includes questions on availability, reliability, dependability, and service provided by the facility; the outputs of research from the facility (papers, patents, students trained, new collaborations formed, and so on); and other impressions, such as safety-related issues; (2) user demographics; (3) areas of research (that is, scientific disciplines) and funding sources for the research; and (4) budget and operation data of the facility, including hours of operation, fraction of the facility in use, and beam line statistics for those facilities that have beam lines. The survey will be beta tested for FY 1997 (the facilities will provide as

much information as they can) and put into effect in FY 1998. Thereafter, it will be completed annually at the end of each fiscal year.

The characteristics of individual basic research projects supported by Basic Energy Sciences are often indistinguishable from those of projects supported by the National Science Foundation. For both organizations, achieving scientific quality is the primary goal, and merit review based on peer evaluation is the predominant vehicle for assessing excellence. To understand the differences between the two science programs, we need to compare the groupings of research projects within the broad scientific disciplines. As discussed in this paper, the rationale of how the BES program portfolio contributes to the Department of Energy's mission areas of energy efficiency, energy resources, environmental quality, and national security becomes clearer when we examine the specific mechanisms by which research areas are determined within BES and the performance measures by which the success of the BES program is ultimately determined.

A large part of BES's value is derived from the explicit goal of integrating the basic research supported by BES with applied R&D activities supported by other parts of DOE or by industry. In managing this difficult challenge, the BES program also shoulders a strong stewardship responsibility for maintaining stable, essential research communities, institutions, and scientific user facilities that are needed for new and improved energy technologies.

Thomas A. Manuel, Air Products & Chemicals Inc.: I would like to pick up on Dr. Dehmer's statement about what do you do if someone challenges you to say the world will end if you cut my budget 5 percent. This is not the end of the world. This sort of thing has been going on in the chemical industry for a long, long time. In fact, it is actually a healthy thing, because it doesn't talk about cutting 5 percent out of the objectives or goals. What it talks about is getting 5 percent more efficient in attaining those goals. So, it drives the argument from outcomes, which are long term and difficult to measure, as we have been saying, down to the work process itself. I think that it is healthy for government, as well as industry, to have this kind of challenge and to respond to it. From my experience in much smaller organizations and aggregates than BES, there is easily 5 percent inefficiency every year in many of the things we do.

Patricia Dehmer: I agree. At least, I think I agree. First of all, we separate the impacts of budget reductions into two categories. One is for the facilities, and the other is for the principal investigator programs. For the facilities, it is fairly easy to quantify what is going to happen with an n percent budget decrease, because it is usually very nonlinear, and an n percent decrease is usually a 2n or 3n decrease in the number of operating hours, and so forth. The facility operations is 40 percent of our budget. For the individual investigator program, it is more difficult to determine the impact of a budget reduction.

I tend to agree that most large programs have 5 percent inefficiency. But they do not have that year after year, and that is what has been happening for the last several years. During this period, Basic Energy Sciences has had an essentially flat budget, and, given inflation, we have had to make such reductions every year. Complicating the situation is "congressional direction" (more colloquially known as "pork"), which also forces us to make reductions in our peer-reviewed programs. The effect is usually

small but not insignificant. For example, this year we effectively had a 3 percent cut because of congressional direction.

So the system does have flexibility and you can make rational cuts, but you can't do it on an ongoing basis without severely affecting the programs. I think many basic research programs in the federal government are at the point where they can no longer make perturbative cuts to get rid of inefficiencies. Instead, you will start seeing entire programs eliminated. That is essentially where we are right now.