6
Performance Assessment Activities at NSF
Anne C. Petersen
We are all aware of the concerns that have been expressed about the research enterprise in this country. Research universities are the primary place in which NSF invests its money to conduct research, so we feel very much involved with these issues. It is clear that we must be accountable for the money that we get from the federal government. That is going to be even more clear with tight money. We expect that the best picture for at least the near term is that we would have flat budgets, but it is possible that the picture will be much worse than that. We have heard figures for next year going as low as 30 percent reductions in R&D budgets. The lowest we have heard for NSF is 20 percent. We hope that none of that comes to pass, and that we can continue to have as much money as we can to do all of the good things that we do.
The National Science Foundation, for some time, has been very efficient in how it discharges its responsibilities to the nation. The Foundation spends only 4 percent of its budget on operations, which is the lowest amount of any federal agency, unless you include subagencies. For example, the Advanced Research Projects Agency (ARPA) is able to operate a little more efficiently; of course, the Department of Defense handles all of its administrative work, and the agency awards very large grants, giving NSF a lot of its work to do. NSF appears even more efficient when you compare it to private foundations, where the average is about 10 percent for operations. So we are very proud of the fact that we operate efficiently. We intend to continue that and to try to be even more efficient, if we can, because we want to be able to distribute as much of the money as we can, rather than keep it for primary operations.
Since I have come to the Foundation, we have been continuously involved in planning efforts and looking at all of these assessment issues. NSF developed a strategic plan recently, which followed on the heels of the major administration report, Science in the National Interest (OSTP, Washington, D.C., 1994). The NSF strategic plan, NSF in a Changing World (NSF, Washington, D.C., 1995), is very consistent with that report. You will recall, if you had a chance to look at it, that we articulated the following goals: (1) maintain leadership in all aspects of science, mathematics, and engineering; (2) enhance connections between fundamental research and national goals; (3) stimulate partnerships that promote investments in fundamental science and engineering and the effective use of physical, human, and financial resources; (4) produce the finest scientists and engineers for the twenty-first century; and (5) raise the scientific and technological literacy of all Americans.
NSF has developed some core strategies for meeting these goals, which focus on developing the human infrastructure and the physical infrastructure, fostering tighter integration and interaction between research and education, and pursuing partnerships. We can never achieve alone all the goals that we have established. We are a small agency, but we really want to make sure that the investments that we make have as much impact as possible. One of the ways to do that is to cooperate with industry and with other partners.
In terms of looking at both what we are doing and how we are doing it, the recent review led by the Vice President has been a very interesting one. We call it NPR-2, or National Performance Review, the second stage. It
asks us to address two questions. The first one is, Should the federal government be engaged in the various missions or businesses in which it has been? If the answer to that is yes, should our agency be engaged in those kinds of programs, or could they be handled more effectively in some other area of the federal government? When I first read this plan in the newspaper I thought, well, the answers to those questions are easy; they are “yes” and “yes.” But as we looked more carefully at the questions, we found them to be very interesting. We really put everything on the table and examined everything we do to answer both those questions. We did come up essentially with “yes” and “yes,” after going through all of the work, but subsequently with a much richer understanding of what NSF 's role is. We do believe that there is a federal role in funding basic research, and that is probably one of the least controversial things we do. But there is some disagreement even about that role. There are some who feel that industry should support even basic research.
On the education side, there are probably more who wonder why NSF is involved in education, especially at the K-12 level. We are involved only in science and math education, but we are involved in it at all levels. We focused more on this issue when it became clear that the production of future scientists and engineers was at stake. There is a critical need for this country to have a scientifically literate public and to have future workers who will be able to have command of specific technologies and a specific knowledge base. These are very important things in which NSF should play a role.
In that effort, as in many of our other efforts, we really see ourselves as making high-risk, high-gain investments. We do not want to do things that are low gain. We also feel that it is important for us to be playing a catalytic or stimulating role. We like to be working at the frontiers. We think that historically and into the future, that is where NSF operates best, and hope that we will continue to be able to do it.
We think of what we do as investments, and we make our research investments based on evaluating proposals up-front. It is much more difficult to track specific outcomes from everything we fund. Probably the closest to a real look at the outcomes of research is a number of studies that have examined the economic benefits of funding and research. We recently benefited from a review that Laura Tyson, the President 's economic adviser, did of all these studies, many based on industry research. She talked about a 30 percent specific (or private) economic gain. If you look at the broader (or social) economic gain, however, it was closer to 50 percent. That is a very good return on any investment. We think it helps document why investment in research should continue.
Industry is unlikely to spend much more than it already spends in funding basic research because it is very difficult to appropriate the results of those investments—because basic research usually has broad impact in many different fields. We therefore believe it is the particular role of the federal government to make those investments.
How we evaluate the outcomes or impacts of those investments in other ways is largely the focus of this workshop. We know we need to provide more evidence of these impacts. I believe we can do it. We just have to find the best ways.
There are ways of evaluating educational programs. For example, one outcome that many would be interested in is whether we have increased the achievement of school children in K-12 and whether we have better degree recipients coming out of our educational institutions at other levels. There surely are many others.
Another thing for us to look at is the effectiveness of different ways of awarding money to researchers. We talk about this in terms of modes of research support. We give grants to individual investigators. We make grants to aggregates such as centers, or sometimes to institutions for particular kinds of programs in an important area. We should be able to look at those outcomes as well, and perhaps even compare the relative effectiveness of different kinds of approaches. Right now, we have a mixed portfolio. My guess is we will want to continue to have a mixed portfolio, but it really would be helpful to know if there are clear differences in effectiveness among the different kinds of approaches.
We have some evaluations going on right now of some of our centers. The centers have a much broader mission than any particular grant does, so we cannot compare them directly, but it will be interesting to see if funding to centers really meets the goals we have for them. To put it simply, we want to continue to stimulate outstanding work and to support that work at the frontiers of science and engineering. We want to do a similar thing in education. And we want to assess the outcomes without stifling the very thing we are trying to stimulate. I think that is part of the difficulty here.
We could think of many methods of measurement, but we want to be sure that whatever we do does not have some negative effect. An example that always comes to mind is the use of publications as an index of individual faculty member productivity. That has led, at least in some institutions, to ridiculous results, in terms of whether it was really measuring something important or whether people simply started writing lots of articles about things that
nobody wanted to read about. This is one of the effects we must be concerned about in that kind of effort. Perhaps there will be downsides that we cannot anticipate. We know that we will need to watch this process as it evolves, but the challenge now is to find ways in which we can assess our efforts without ruining the very process that we are supporting.
GENERAL DISCUSSION
DR. HACKERMAN: Is it certain that we no longer need the Vannevar Bush report to tell us what to do?
DR. PETERSEN: I think most people feel that it still is a very important document for what we are doing.
DR. HACKERMAN: Does it cover the complex things that we are now involved in that probably did not exist in 1950?
DR. PETERSEN: I think what has happened, which perhaps was not anticipated, is the dramatic effect that federal research funding has had on academe. I would argue that not all of the effects have been good, but we need to be sure. I think that is part of the reason there is a demand for accountability. We need to be sure that the investment that we make at the federal level is reinforcing the right thing.
A couple of things come to mind. First, the federal government has stimulated research universities to value research so much that they have come to devalue the educational mission. That clearly has been a big mistake. The federal government needs to help correct this imbalance. Second, universities also are trying to be more accountable. Research grants are one of the major indicators that universities use for research productivity. What you see quite clearly over the past 20 years in research universities is that the nature of federal investment in different fields has dramatically influenced the future of those fields. So, in most research universities, for example, the humanities have languished and do not have the status and activity that they once did. This contrasts sharply with European countries. We should take the time to reflect on whether current research universities reflect our beliefs about what an outstanding university should be. If the answer is no, we need to work together—the federal government and universities—to redirect our efforts.
DR. HACKERMAN: Should the research guidelines ask for a paragraph on the educational impact statement?
DR. PETERSEN: They do.
DR. HACKERMAN: Has this had an impact on education?
DR. PETERSEN: There is a question about how much that statement is taken into account in the peer review process. I think that the debate is actually broader than that. For example, if we want to have an effect on the way universities do business, should we do it by tinkering with the merit review or granting mechanism, or should we give money in different ways? The National Academy of Sciences' report on graduate education is out today. It makes a strong recommendation for NSF to put more money into institutional training grants as a way of affecting graduate education. The outcome of the debate will be interesting to see—whether that is the right mode, or whether asking for attention to education on research grants will do the trick.
DR. FRIEDMAN: I would like to come back to something that we discussed earlier in comparing industry to NSF. You know, for industry, in some sense, it is very simple. Industry does not have any mission; it has goals. For instance, the goal of John McTague is to sell cars with the good technology of today, with customer satisfaction, and with a profit. So we know what it is. The managers of the company can ask the technical people how to implement it. It may take five years to redirect the corporate research culture, but that is a tremendous process, as we heard from the presenters. The world is changing at a ferocious pace, as Jim McGroddy said this morning, and the culture must change at the same time and at least the same pace so that the company survives.
The situation with NSF is different. NSF has a mission. NSF does not know who the customer is. Is it Congress? Is it the research community or industry?
It is not clear how you are going to actually implement the mission. It is not clear who the competition is; is NIH the competition, or veterans, or NASA? How are we going to make it easier for those who make funding decisions to evaluate what we are doing and how are we going to communicate our mission?
DR. PETERSEN: I think it is especially complicated because the impacts of any specific research projects are long term, 15 years or so; so 3- and 5-year reviews are not helpful there. I think you are heading in the right direction in terms of who the customer is. The money comes from the taxpayer. We are competing with other ways to spend federal dollars. If we think about Congress, right now, there are many, especially among the freshmen, who really want to balance the budget. In addition, there are many who want to reduce the federal role in everything. So, in that arena, if we believe that it is important for the federal government to be investing in research, we have to make
our case to those people. Then, among the activities that are appropriate for the federal government, we are competing with the other agencies to which those funds could go. They are extremely diverse. We have to be able to say that putting money in basic research is as important or more important to society than putting that money in housing or other things. We compete directly in our congressional committees with the Veterans Administration (VA), Housing and Urban Development (HUD), and NASA, among others. We must understand this. I think most surveys show that the public has positive attitudes about fundamental research, but it is very difficult for the public to know or understand all that we do.
DR. BROWN: But isn't it an interesting question? If the primary goal of NSF is to create the intellectual infrastructure for the country, then what is the optimal amount? Maybe, you have framed the question backwards. How do you go about determining what the right amount is? Do not view it as competition initially. One can make the argument that we cannot actually use a great deal more funding effectively. I think we would all agree that if we increased NSF funding a hundredfold, it would not be particularly productive for the nation. If you could give a principled argument for a threefold increase, you could make a strong case, but we cannot. We argue that we want more, or people tell us we should have less. That is a different kind of argument than what the right amount is. How do you go about framing that analysis? Why isn't that a research project?
DR. McTAGUE: There may be no absolute answer. If you go back over the history of mankind, there were plenty of years when the answer was that zero was plenty.
DR. BROWN: Absolutely. But you start thinking about the problem differently, that is all I am saying.
DR. FRIEDMAN: I would like to ask Bill Harris another question, because we heard a lot this morning about a change in culture in industry. From your point of view, has the culture of NSF changed in the last four years?
DR. HARRIS: I think universities have probably changed over the last four years, and NSF is in many ways a mirror of what is going on in society. The universities are uniquely important to our society. The investment we choose to make is based upon our respect for education, learning, and knowledge. Framing the argument in that way, around people, might help the NSF argument better. I worry that if we base the argument only on research, it will not be understood by everyone. Most of the R&D budget is military, and from that, you get a widget you can see. So a natural public response might be to wonder what NSF makes or what it does. Well, our immediate products are people and knowledge, which energize this society and create wealth. I like very much what John Seely Brown has been saying about trying to recast the discussion, change the vocabulary, and make some of these transformations.
DR. HACKERMAN: You are saying put education first or research?
DR. HARRIS: If you think of education first, and the products being the people in the universities, you will have a win-win situation for the society and it will support you.
DR. HACKERMAN: Whatever you get in science is a bonus.
DR. HARRIS: It is a bonus.
DR. GOLDSTON: That has been a big change over the last 45 years. The NSF budget in the 1950s and early 1960s was devoted almost entirely to graduate stipends and constructing buildings and facilities to expand the campuses. It did not become what we now take to be NSF until the late 1960s, by and large.
The other point, in terms of picking a number—that is really what Erich Bloch did. He got everyone to agree to double the NSF budget. You could argue about how, but that approach worked once. I do not know if it would work a second time, even if you wanted it to.
DR. HARRIS: What we are trying to talk about is what is the best way to help people understand the value of this investment. I think that is what performance assessment is mainly about. If we could simplify the argument for external consumption and adopt some of the suggestions that Xerox and others have provided for us, we would probably be in a stronger position. Education and research must be coupled. They are integrated and that is the strength of our system. When we separate the words, we add to the confusion.
DR. FRIEDMAN: There is one obvious problem in that regard, and that is how much research is really necessary for education? It is connected to the question, What is education?
DR. HARRIS: My focus would be on people who are products of the university, both students and faculty. If you could personalize some of these things with anecdotes that a congressperson or the average citizen might understand, it would help explain our investments in research better. We do not make cars. We make the people who will make the cars eventually.
DR. McTAGUE: The trouble is that you do not make the people who make the laws.
DR. CEHELSKY: One of the most interesting comments that was made this morning by several people is that we have to be careful about the use of quantitative measures. That is rather refreshing. I think that it states a true precept. It is something that responds to a lot of pressure that we are feeling within the federal government, certainly at NSF, to be much more quantitative about assessing what we do, evaluating our performance, and justifying our existence. I found it interesting that this is a concept that industry seems to adopt for itself, in other words, that you cannot quantify everything that you do, and you cannot assess everything quantitatively. But I think it poses a dilemma for us. Industry ultimately works on the basis of profit, and there are some very clear goals. Industry works in terms of profit and clear objectives—products and processes that are productive by fairly common-sense measures, for the most part. How do we translate that to the government arena, and particularly to NSF, where the product is so open ended and where the process itself, even when we measure it, does not satisfy us in terms of yielding results? For instance, citations give us an indirect measure, an artifact perhaps, but they do not really tell us what the taxpayer gets out of this investment. This occurs because the ultimate investment is an investment in productivity, the economy, and social returns. So we get back to the very difficult question of how we measure, or alternatively, how we convince the public, that in fact we have made a wise investment of those monies that have come into the federal treasury, even if we cannot really measure the ultimate impact to any level of specificity.
DR. McTAGUE: I think there are a couple of points. One of them has to do with quantification. If there is a number, people will misuse it. The most fundamental function of government probably is national security. I do not see anyone attempting to quantify national security in some sense. It represents a qualitative judgment. You do not ask how many nonwars you get per dollar, for example, or how many non-body bags you get per dollar. No one would suggest putting a quantitative measure on national security as an outcome. You might put a quantitative measure on it as a process. Are we being efficient about procurement? The answer is no, et cetera. So here is an area in which people make qualitative judgments. Is the world more secure because of what we are spending, for example? Is what we are spending on national security getting in the way of other things that we want, in terms of quality of life? People make qualitative judgments on the big issues. So there is no reason why you cannot convince people that qualitative judgments apply equally to other fundamental roles of government, but you have to decide for yourself what your role really is. Is your main role educating very good technical people? Is your main role educating the general population on the nature of technology in a democratic society? Is your main role strengthening the university system? Is your main role developing more and better researchers in universities? Is your main role improving the impact on the economy? Is it having an impact on other social issues, such as health, environment, and public safety? Is it expanding the human vision? Is it all of the above? The first thing you must do is decide what your own role is.
