Research Restructuring and Assessment: Can We Apply the Corporate Experience to Government Agencies? (1995)

For industry, the research goals are clearly identified. Essentially, the goals are to make a product the customer likes and to sell it at a profit. The goals of NSF are not so clear, except in general terms. The main goal is to increase the capabilities of scientists in the service of the nation, as implied by the agency's name. That covers a broad spectrum of items in the NSF portfolio—infrastructure, education, research, and so on.

What the corporations have done over time is to get their research organizations to move in line to the strategy behind the corporation. In addition, they have separated the research and development organizations —decentralized development with the business units to integrate the planning—so that research is more involved with its partner, its customer within the corporation, the business unit. What NSF can learn from this approach is that it must begin to form partnerships with its customers. NSF also needs to engage the U.S. research community in a discussion of what it wants from its science and technology infrastructure; it should not think of itself in isolation from these other organizations. NSF also needs to engage other funding agencies in terms of what their role is in funding engineering and science research, as well as education.

A simplified view is that the companies begin by defining their purpose; they interpret that in terms of what they must do to accomplish their purpose; they decide what forms of knowledge and technology would advance that set of purposes; and then they sort through that and determine what should be worked on and supported. NSF starts with the problem of not being quite sure of those initial steps, but it is always useful to go back and read the legislation. There is a clear statement of the NSF mission, which ought to be the place to begin. What are the measures of whether the national health, prosperity, and welfare are being advanced? What are the connections—past and current—between what is being sponsored and these broad goals? Where and how is NSF-sponsored research being communicated to and used by its customers, the taxpayers? The Foundation ought to be able to choose metrics that will communicate these linkages and that will make some sense.

There is another industry perspective that is very much in line with what was discussed earlier. For those who do not know, the Industrial Research Institute (IRI) is the U.S. association of corporate research directors, and its member companies are responsible for a very large percentage of the industrial research done in this country. All of the corporate presenters at this workshop are members of the IRI, for example.

NOTE: The material in this chapter is synthesized from the comments of the discussants. It has been organized into the topics of the workshop, and the speakers' identities have been removed. These comments should be viewed as the opinions of the discussants and not as a workshop consensus.

One of the committees of the IRI is looking into the whole issue of metrics, primarily from the point of view of trying to understand how to evaluate the performance of R&D. Of course, it is looking not only at research, but also at the entire R&D process, for a number of different reasons. A lot of our corporate research budgets are under attack, and the directors of research need to be able to contribute good information about why these activities should be funded. Another reason is one of understanding more clearly what is going on in order to improve the quality of the processes that we are using and to improve our contribution.

The IRI experience is highly relevant to the issues discussed at this workshop because this organization has faced many of the same problems. NSF can learn a lot from the companies that have been successful in making a transition. Those companies have paid attention to understanding who their customers are, how they create value for those customers, and how to measure that value. They have become very good at improving the value they generate and at being able to demonstrate that they are doing so. That is highly persuasive in terms of obtaining funds. This process should be recognized as a real-world trend that is not going to go away.

Nevertheless, it is important not to make too strong an analogy between industry and NSF funding. One of the statements repeatedly made by industry is that long-range research is being cut back, and industries are looking more to the universities and to the government to support that kind of work. NSF should be funding exactly the kinds of research that industry is jettisoning now because of the very metrics that industry is using. To apply the same kind of metrics without a lot of correction would result in NSF's mimicking the decisions that have been made in industry when, in fact, the opposite decisions need to be made by government if it is to maintain an appropriate equilibrium. We need to be especially careful about how we calibrate what is happening in government against what is happening in industry.

One final point. We have been talking about industry success, but some of that success was at a cost that basic researchers want to avoid. The large corporate labs are much smaller than they were a decade ago. The cuts being discussed with regard to NSF funding are not nearly as extreme by comparison.

The forces for change are strong and they are having an impact. The industrial speakers discussed how they have responded, but those forces are affecting NSF and research universities differently than industry. There is a great tendency, a great pressure, especially with the end of the Cold War, for NSF to spread its support of research uniformly. This is not a new pressure. In 1947, when the Senate was contemplating a law to establish NSF, there was a debate between Senators Fulbright and Russell from Arkansas and Georgia, and Senator Saltonstall of Massachusetts. Senators Fulbright and Russell said, “You have got to put something in the legislation to make sure that this research support gets distributed properly around the country; otherwise Harvard and MIT will have it all within six months.” Senator Saltonstall said, “Well, you want the best national security system, you want to buy the best research, and you are not interested in that other part.” Senator Saltonstall won the argument, but Senators Russell and Fulbright were right and those forces have not gone away. Much more recently, there was a debate in the Senate about earmarking. Senator Inouye of Hawaii was a strong proponent of that. He argued that peer review was inherently unfair because it favored the haves over the have-nots and that everybody should have an equal break.

So the question is, Will these forces become more important now that the military security rationale is gone? The answer probably is yes. Here is one way in which those forces for change are going to act on research universities. The reason this country has the world' s best research universities is not because we are smarter than everybody else, but because we have a different system to execute that research. There is a direct link between the quality of research and infrastructure support, but many of the components of the research process are currently being questioned.

So what should NSF do? It may have to become more elitist than it is now. From a political standpoint, it cannot be simply elitist; it needs to take some resources and make it clear that those are to support whatever is truly excellent wherever it may be found. At the same time, there are other needs in the country that must be recognized.

One concern is that the separation between research and education is unnatural and damaging to NSF. Getting them back together and articulating this in a way that the people who are not researchers understand should be considered in this introspective process. It may be that everyone wants to separate them because no one trusts researchers to actually care about education. We all know, however, that research and graduate education are

closely coupled, that you cannot have one without the other; so we need to publicize that fact to make sure that Congress and the public are aware of their value.

A major issue raised by all the corporate presenters was the need for a change in the research culture and all of the processes that were put in place to change that culture. We have to understand that it is not so much the NSF culture that needs changing as it is the culture of academic researchers that is going to have to be changed. We have talked about the fact that the universities' culture has changed very little in the last 40 years. They did not have to change because defense was a sufficiently strong motivation for the country to support universities without the need for further accountability.

We need to carefully separate the “sale” or the explanation of the value of science and technology research from that of the NSF role in particular. NSF does not fund all science and technology, and the civil government funds only a small percentage of engineering, for example. Most of that is done by the Department of Defense. NSF is going to have to sell science and technology research, however, and then explain what its particular role is in that larger context.

The Government Performance and Results Act (GPRA) should not be viewed as an obstacle, but rather as an opportunity, because it allows NSF to review its goals and improve its activities. One of the main purposes of GPRA is to demonstrate the value of what government is doing. The example of the government's data and information activities in environmental research provides some useful lessons in this regard.

First, it turns out that in looking at information system design and information policy for research in global environmental change, there has been a void with respect to what the appropriate assessment methodology is for the issue being studied. We have had to look at how assessments are conducted in research and specifically in environmental research. It has been very difficult to identify the requirements in that area because the standards and the methodology for that type of assessment are found in all sectors of the economy. One of the reasons that research programs in global environmental change are in trouble in Congress is that there has been very little communication between the research scientists, especially physical research scientists who are trying to determine how the earth system functions, and other stakeholders around the country who are interested, from their different applications perspectives, in what is happening to the environment. Thus, one of the values of an environmental data and information network would be to provide a mechanism for the research community to communicate on an ongoing basis with the people who conduct policy assessments and other kinds of assessments that depend on this scientific research.

Second, the data that the government has generated in its research program have to be made public. This is not the kind of information about which gradations of value should be established, so that various categories of users would be charged different amounts to use it. The government has to have a budget for archiving and distributing the data to all public users and to find some way of paying for this, through either budget allocations or some other simple charge on industry, and so forth. But this information has to be public, and it should not be restricted in its use.

Third, agencies such as the National Oceanic and Atmospheric Administration (NOAA), National Aeronautics and Space Administration (NASA), and NSF need to serve as a broker between all of these stakeholders or users and the people doing the scientific research. NASA and NOAA should continue to download scientific data to their constituents and make the data available to the public. In the case of NSF, this could be done through the universities. A great demand exists for these research data—a large constituency wants access to them. For example, there are researchers in the West who are interested in water resources, forestry, agriculture, and weather and who need access to the data so that they can apply them, in their particular situations, to understand these processes and provide a better base for policy and business decisions. The greatest demand for global environmental change research data comes from educational institutions around the country that want to use them for education. Because there is a very large constituency that wants access to all of the research that is being generated, there ought to be a systematic way of networking this information, distributing it, and forming a constituency of users that will demonstrate more fully the value of this research. One suggestion arising from this is for NSF, NASA, and NOAA to design an ongoing process through which they can provide easy access to all of the research being generated to demonstrate its value directly, rather than through the third-party metrics we have been discussing. There is no better mechanism to measure usefulness than having a constituency out there that is actually using the information and tying the research community directly into that.

At the same time, it is important not to make it sound as though we are starting from scratch and that NSF has never done any of this before. In fact, it goes through this exercise every year—it is called the budget process. Bill

Harris, in his directorate, has to decide the relative merits of mathematics, physics, chemistry, and astronomy—all of which have different justifications to some degree. Even within physics, funding condensed-matter physics is different from funding solid-state physics. NSF must decide why one program and not another should be given increased funds. Part of what is necessary is building on that and communicating the way in which those decisions are made. This is something that members of Congress have requested, not because they have been trying to second-guess it, but rather to get a sense of how priorities are set and decisions are made.

If we are really serious that education is the most important goal to measure, then the way research is being done may have to change. There are different steps involved in this, but the first steps include activities that go on all of the time and simply have to be articulated more clearly.

It is very important to understand not only what you measure, but for whom. Who is the recipient of the measurement? In the case of NSF, obviously, the first customer is NSF and the research community, which need to take these measurements, do something with them, and learn from them. The second and different kind of customer for these measurements is OMB, by which we mean the administration. A third, equally important, customer is Congress. The same measurements or the same parameters that satisfy OMB may not be meaningful to Congress. The fourth category of customer is U.S. industry. The final customer that should be identified is one that we often neglect—that is, the public, the taxpayer. If these measurements are not useful, or are not intelligible to the taxpayer, then we have a major problem. There appears to be increasing disenchantment in the general public with science and technology, and the research community is not doing enough to counteract that trend.

NSF thus needs to recognize more explicitly that, ultimately, the taxpayer is the customer. This is a very difficult customer to talk to. It is necessary to find surrogates to represent that customer because the typical citizen does not understand what we do. If the scientific community fails to recognize that the taxpayer is the customer, however, we are finished. The taxpayer is also the ultimate customer for the Social Security Administration, for example. But the metrics of performance that the Social Security Administration probably uses will be things like what percentage of checks get delivered on time, how responsive it is to customers coming in the front door, and that sort of thing. The analogue of that, on the part of NSF, would be how fast it processes grants, but that alone is not sufficient. NSF is going to be held to a different standard because it has to demonstrate both that what it does is of value to society and that it does this well. Everybody generally accepts that Social Security is a good thing, whether it is or not, but that is not necessarily the case with basic research.

One of the things that NSF could possibly do better is to transfer this problem of demonstrating and articulating the value of research to university researchers themselves. The citizens of the United States have generously supported research for 40 years. Instead of treating this support as an entitlement, researchers should acknowledge explicitly that they are fortunate to be in this situation; they should show the public what it gets for its money, and they should devote some time to K-12 education and the like.

The corporate speakers all noted that they had to go through a cultural shift. One of them said that once your principal supplier goes and things change, you have to change too. Similarly, NSF may be tempted to recast the arguments for basic science in a different market setting so that it can preserve the core values that are so important. But it probably does not need to recast the arguments, just to demonstrate them more clearly. It remains the same set of arguments. NSF actually never had that much of a Cold War basis compared to much of the other research funding in both government and industry.

In the case of industry, different stakeholders in a firm clearly are interested in different types of metrics. It is the IRI's experience, in polling its members, that the owners of firms and the stockholders are not interested in metrics that look at how the R&D process is conducted. They are interested in metrics that deal with the creation of value, and those metrics must be framed in terms that the group perceives as relevant. Measures of value that are not relevant to them do not demonstrate the creation of value. Another area in which owners of businesses, and the CEOs, tend to be more interested is integration of the R&D effort with other corporate objectives.

It is clear that NSF and the scientific community have a lot to learn from industry. At the same time, it is important not to push the analogy too far for reasons already stated above and because NSF does not have such a simple bottom line. The speakers from industry told us that while the customer was the bottom line, the bosses really are the bottom line. Industrial researchers may react to their customers, but the bosses call the shots and they

are the people who decide where the research is going. NSF's boss is Congress. The entire scientific community has done an inadequate job of justifying public support of research, and we must improve.

The metrics that are critical, whether in industry or government—and especially in the context of budgetary cuts—are those that can communicate how you are contributing value to the full agenda of the firm, the government, or whatever the entity is.

It is important to emphasize that measurements do not always have to be quantitative. Some quasi-quantitative indicators, such as the one used by IBM to show customer satisfaction in bar graphs, are perfectly reasonable. In fact, NSF could depend on industry to give it a good reading, using such bar graphs, about satisfaction with the output.

In addition, the focus should be on trends rather than on absolute measurements. You can see much more when you look at the first derivative of a function, and sometimes the second derivative is even better if you can find it. So, we should look at trends and not just absolutes.

One point that was made this morning was that any of these measurements could distort the research agenda, but we should relax about this. Heisenberg made that point many years ago—that when you measure something, you distort the system. So be it. That does not mean that you should not measure. It just means that you should be careful what and how you measure, and you should be knowledgeable about the results and the consequences of your measurement.

NSF should not measure 100 parameters. Six good ones may be enough, and we must not confuse quantity with quality. All these parameters and measurements should have a competitive kind of value. You should be able to go out and compare yourself against industry and against other research organizations. This is a competitive world, and there is nothing wrong with that. It is competition that drives a lot of the energy in a system. We have to be clear that whatever we put in place, we need to have some comparisons. How well are we doing compared to someone else? This is a very important parameter.

Industry is rediscovering that, by the way. It was neglected for 20 or 30 years, especially by U.S. industry. The Japanese understood it well before us. They reintroduced benchmarking, and it has helped U.S. industry to focus on that again. But this measure applies to government and to the university community as well.

It is important to determine what your basic guiding principles are and to stick to them. Then you will know what to do as things change. The principles established by NSF in its new strategic plan are good, and NSF follows them very well right now.

It also has been said several times that NSF is very good at reviewing proposals, but not at evaluating results. However, every renewal proposal has to state what was done initially with the money and what the researcher is going to do. That then goes out for peer review, and comments come back about how good the past work was. It would be easy, therefore, to extract all of the results of the research and report that to Congress, but that is not the sort of thing the general public can understand, or what Congress wants. Members of Congress do not want to be mired in this, nor do researchers or NSF, but it would be a simple additional step to try to use such information to clarify the significance of the results according to established criteria.

If one looks at peer review of proposed research, more often than not, when you actually read the reviews, they turn out not to be forward-looking evaluations of the proposed research, no matter what was written. They are really retrospective track-record evaluations of a person. You can argue whether that is good or bad, but for people with a track record, it happens quite often.

With regard to corporate metrics, IRI has tried to develop some generic models that would apply across all kinds of industries. The institute has divided metrics into five categories. These categories are perhaps somewhat artificial, and several of them are the same. The first category of metrics is outcome measures of value creation. Outcome measures really relate to the final output of what you are trying to achieve—value creation. The second set of metrics deals with integration of the R&D effort with other corporate objectives. In the NSF case, this would involve how the agency relates to other government programs. The third area is portfolio distribution metrics. How do you distribute what you are doing across various opportunities? The fourth set, which can be referred to as

measuring the asset value of the technology base, emphasizes that investment in research creates an asset. The intellectual property that is developed provides future value for the firm. This deals with the creation of core competencies and critical capabilities for movement in desired directions. The fifth category of metrics involves those that deal with the efficiency of the R&D process itself.

One study, done by the Rand Corporation for the Air Force, used corporations as a model in developing assessment metrics. Four areas were addressed: relevance, impact, quality, and productivity. The study identified six metrics that the Air Force labs could use to evaluate their research programs. The characteristics that were used to evaluate these measures were whether they were indirect or direct measures, whether they were input oriented or outcome oriented, whether they were quantitative or qualitative, and whether they would be costly or inexpensive to implement. The measures were adopted from the literature, from IRI, and from measures that have been used in the defense organizations.

Information technology is opening whole new frontiers with respect to the science of information. This is the kind of research that Xerox talks about—it has a research lab for the managers of this research. NSF might consider beginning a process of studying the relationship between investments in human endeavor and their ultimate outcome. That is not a measure that one can conjure up and use by next fall, but it might help remove some of the mystery. How do you use information system technology over the next decade to learn, in a more quantitative way, in a more traceable way, what the impact of NSF funding has been? Such research could be a major contribution to the era of information technology, on the threshold of which we as a society now stand.

It is quite clear that goals must be established before metrics make sense. Therefore, when we start talking about what the goals of NSF-supported research are, the issue of “strategic research” will have to be addressed.

What is happening in the universities is closely coupled both to what NSF does and to what NSF needs to do. A university teaches people and prepares them for their lives. Research is performed at universities because that is an important element of how we train people for their futures. Most research universities consider the distribution of a professor's time to be about 50:50 between research and teaching. Good research and good teaching are so interconnected as to be inseparable. We need good research to teach well. Poor research leads to bad teaching and inadequate preparation of students. We need, on the other hand, good teaching to do good research. So we are in a very tight circle.

One issue to consider is whether or not the mode of teaching is changing, as well as how it ought to change. For instance, we need to train more broadly aware people whose knowledge base is wider. The multidisciplinary center mode of research that NSF supports has been effective. One of the main customers is industry, because most of the students we train in science and engineering go into industry. So, in the long run, one question to ask is how satisfied the leaders of industry are with the students we turn out. Are we turning out flexible young people who can be connected to what goes on in a company, who are bright and not fettered by a narrow education, and who can move nimbly from one problem to another? We need to ask this question of industry. As a matter of fact, many people in industry already know the answers, because they recruit heavily at certain universities and not at others. Obtaining good students who are capable of advancing a company's needs presumably can be quantified, but it really takes years because these people have to be allowed to perform for a while.

One of the principal outputs of research supported by NSF is the young individuals who graduate and go to work either at an academic institution or in industry, and who bring with them the training and education that they received at a university with the support of NSF. Usually, these individuals also carry with them the latest knowledge in terms of techniques and of where the frontiers are; they have also developed the art of asking good questions, which is ultimately what research is all about—knowing both how to pose the question and how to answer it. They eventually do have an impact on industry. Industry would not survive without a constant inflow of these bright, young people. So one thing to ask, if one is looking for something to measure, is how well educated the individuals are who go into industry or into academe. This could be done by some form of a selective audit of the industrial sector and the academic sector. Similarly, one could ask young academics whether research that was done under NSF sponsorship really boosted their careers.

An important issue to track in education is the progress of foreign-born Ph.D. recipients in the United States. For instance, non-U.S. citizens earned a little less than half of the Ph.D.s in the physical sciences awarded in the United States in 1993 and almost 60 percent of the Ph.D.s in engineering. What are their career tracks and their impact on R&D in the United States and elsewhere?

Looking at education and students as the most significant area for measuring contributions differently than by NSF, however that is finally done, is a good idea and important to emphasize. Anything that NSF can do to actually

encourage university administrations to focus more on students will be useful. Congress also thinks of universities as educational institutions. Most members of Congress have been at least to college; very few of them, however, have been in a lab. Education is something that people feel they can understand. We need measurements that reveal both the problems and the successes of our educational system. It is not enough simply to measure the number of students that graduate: just measuring the fact that bright people went in and bright people came out does not really tell much about what happens at the university level. It also may be useful to measure the ones that got away, the students that you would like to have had.

An output of NSF-sponsored research is research papers, or the archival literature. This is part of the intellectual resource pool that we all draw upon, whether we are in academia or industry. This capital in most cases has a limited lifetime. If it is to be used, it probably will be used in the first one to five years, and possibly another five years after that; beyond this time, the likelihood of utilization is rather small. One could ask, for example, How good is the quality of that output? Does it have an impact? Is it ever cited? If it is ever cited, what is the citation level over time? This is certainly something that is measurable and can be of some value.

Other questions that one can ask about research involve the significance of the work. Did it advance the frontiers of the field within which it was done, or did it simply reinforce existing paradigms? Did the work introduce or create new tools or techniques? Did it enable others to do better work or novel types of work that could not have been done before? Did the work lead to new physical or mathematical insights, or enrich and widen the scope of inquiry? One also could ask if the research performed under NSF sponsorship has had a direct and immediate impact on a company. Was something learned through the research that was immediately implemented in industry and provided a competitive advantage?

Another type of question could focus on the integration of knowledge. Through the integration of knowledge, was it possible to address real-world problems of various types? Did the work contribute to advancing established national goals and priorities? This is an issue that can be identified at the proposal stage, because the goals and priorities, at least in a broad context, have been written down. One can ask whether the proposed work is likely to contribute to these goals and priorities, as well as looking at it in retrospect.

Figure 8.1 makes the point that researchers have a good story to tell, but that we have not been telling it in quite the right way. What this figure from an NRC Computer Science and Telecommunications Board report (Evolving the HPCCI [High Performance Computing and Communications Initiative] to Support the Nation's Information Infrastructure, National Academy Press, Washington, D.C., 1995) shows is that federally funded research and development has stimulated the creation of new ideas and highly profitable businesses in computing any communications technology.

There are several messages that come out of this. First, this figure, which tracks the results of research started in 1965, shows that it has taken a long period of steady investment to bring computing and information technologies to market. Second, it debunks the linear model. If you look at the record, you can find examples in which we have gone directly from university research to product development. Product development is going along in parallel with industry-funded research. It demonstrates that sometimes there is no difference between basic and applied research, or where the research is done. We need to educate the public and the current Congress in particular about the value of scientific research.

We are recognizing that, in fact, it is a difficult job to develop performance metrics. But, in a way, the National Science Foundation has a uniquely favored position at the moment. There is a predisposition to believe that NSF does something good and useful and that there is intrinsic value to basic research. One can view the need to develop serious performance objectives or measures as a real opportunity for the NSF. This is one message that is important not to lose in this process.

Nevertheless, the main question is why the government should allocate discretionary funds to NSF, rather than to something else. And it can be anything else. This is one of the ways in which NSF is very different from industry. Funding for NSF really involves a trade-off against everything else in government and, at the appropriations level, particularly against Housing and Urban Development, the Department of Veterans' Affairs, and the Federal Emergency Management Agency, all of which can make an argument for more immediate relevance. If people are homeless right now, can't you wait a year to put money into NSF and put it into HUD right now? That is a tricky kind of question and there may be no metrics to answer it.

The supporters of NSF in Congress want something that they can use to demonstrate the eventual gain from investments in research, and they do not necessarily want a very specific number. But our congressional representatives are not ignorant, and the measurements and related justifications have consequences. For example, in the

1980s, the way to sell the need for university funding was to assert that everything supports economic development. Congress took that to heart, and earmarking mushroomed. Rhetorical strategies can have consequences, because people may be more credulous or respond differently than those who design the strategies initially expect.

Another point is that we have a large set of measurements in the research arena, but they are primarily input measurements. How much money do we spend? How many people do we hire? How many Ph.D.s do we use? What we are lacking are measures of output and outcome, and the focus needs to be on that. NSF began publishing science and engineering indicators in 1971 with the purpose of establishing a long time series of data that could be useful to research managers and educators. That has been done, but essentially with indicators of input. The indicators thus also ought to include a few metrics that gauge output and outcome.

A set of metrics needs to be developed that can track and help explain the reasons for changing the research portfolio—why something is being dropped or added, or why the relative weight of something that relates to the creation of knowledge or advances in technology is changing. The Foundation should think about how to do that more effectively, because Congress and the research community have always wanted the answers to such questions.