can have great technology assets, you can be lined up with your corporation' s strategy, and you can even be delivering value, but are you delivering value in the most effective way possible? What does this mean? It means: Do I have processes in place that lead me through projects quickly and that allow me to use my intellectual property efficiently and effectively? Or do I have barriers between my organizations so that effectiveness is impaired? Is my staff well motivated? All of those are important metric issues. We would really like to measure all of them.

Metrics may not date all the way back to the Stone Age, but when we embarked on this project, we did a bibliographic search—as any good scientist would—and discovered that there were references to metrics that go back at least 400 years. There were R&D metrics 400 years ago, when the princes of Europe were sponsoring R&D. They were asking the question: Are we getting our money's worth? So any of us who are sitting in this meeting today who are feeling picked on can take some comfort in the fact that we have been picked on for a long, long time.

But what's happened to metrics since those days? I would suggest that most of the metrics have ended up in the landfill of history. Why is that? There are three crucial tests that metrics must pass if they are to be supported by our wider customer base. By customer base, I mean the people who pay the bills—customers and corporate sponsors in industry, and citizens and Congress in public institutions.

One of those tests is the question of relevance. Is the metric that I am trying to use relevant to my organizational mission, objectives, strategies, and so on? This means that the metrics used vary depending on the type of organization you belong to. Though I realize that many leading companies do indeed think of publication as a critical metric, it is one that has not found much favor in the IRI. Surveys of IRI membership find this metric quite low on their list. The general opinion is that, if it is publishable, it is probably not giving much proprietary advantage. On the other hand, in an academic institution, this could be a highly relevant metric—it is important to make any knowledge gained known. In short, it is critical to align the metrics of value creation with the objectives of the organization.

The second test is credibility. My favorite examples are the wonderful metrics (and I believe some of us in this room may have used them in the past) where we said, ''Let's do a self-assessment." So we got together with our chief scientists and rated each other on how we were performing on a scale from zero to 10, zero being "dumb as a stick" and 10 being "should have won the Nobel Prize, but the committee was unaware of my work." All the scientists rate each other 9.5 and then present this self-evaluation to the business unit, which says, "Yeah right." This process lacks credibility. There are a number of other metrics that have great potential for gaming, but I think you understand the issue. Credibility is a big issue, especially if you are trying to develop metrics that are meaningful to your customers.

The last test is one that particularly appeals to me, and that is complexity. It is important for the metrics to be reasonably simple and easy to calculate. If they are not, we could end up having our whole research laboratory working on metrics rather than on science, and our preference is to have people working on science. I will add, however, that engineers tend to like complexity. We like to have a table of 60 numbers or more, to multiply this matrix, invert the matrix, and the like, but unfortunately this activity can be fairly destructive. Even if the metric is theoretically sound, it should be tolerably easy to calculate and, more important, intuitively easy for our customers to understand.

In sorting through this maze, there are a number of bright lights to guide us. This light comes from a number of sources. Some very good academic research has been conducted in the last 10 or 20 years that has begun to uncover the factors that lead R&D projects to commercial success. Which new products have been successful, which have failed, and what's the reason for each? What are the practices in R&D (particularly in business R&D) that lead to success versus failure? The increased focus that has recently been put on this area is beginning to lead to some metrics that have value.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement