Assessing the Value of Research in the Chemical Sciences (1998)

James W. Mitchell

Lucent Technologies

Research, even when academically targeted for the most esoteric knowledge-based regime, was once widely considered an unquestionably valuable enterprise. This era of basic research, relatively unrestrained by cost, was driven previously by defense-based needs. However, basic research within government agencies, industrial corporations, and to some extent within universities has been under intense scrutiny to assess the impact and payoff. Today, more research efforts are being channeled into pursuits for meeting industrial, national, and societal needs. With this current emphasis on tangible returns on investments in research, viable methods are being sought for evaluating and maximizing the impact of research programs. For the chemical science and technology areas, many approaches for evaluations exist. All are diversely dependent on the management perspectives within the institutions in which the research is conducted. Even within the same institution, flexible variations will occur within different organizations. For example, the methods used within the Materials, Reliability, and Ecology Research Laboratory of Lucent Technologies have a strong legacy in the corporate culture of Bell Laboratories, but detailed practices are different from those of other organizations within Bell Laboratories in which chemistry-related research is done.

The Materials Laboratory has responsibility for a substantial portion of the chemistry and chemical engineering research and all of the materials chemistry research associated with ceramics and metallurgy. For this organization, chemical research may be examined on three levels. At the corporate level, the issue for management is determining the value of the chemical R&D enterprise. On the organizational level, a primary interest lies in formulating methods for optimizing the value and impact of projects within the chemical R&D program. Analysis and prioritization of the portfolio of projects are needed to maximize the value and impact of the work. For individual scientists, the most valuable process is one that assists the experimental researcher in determining which one of many research ideas to pursue. Evaluation of projects, once initiated, is somewhat more straightforward and strongly parallels the assessments used in prioritization. Although the best way to evaluate chemical research at each

level is seldom unequivocal, it is certain that valued chemical research will have been scrutinized to ascertain its specific merit and impact.

From the corporate point of view, the chemical research organization is valued when it fulfills strategic corporate purposes. These may include creating breakthroughs, developing rapid innovations, broadening technological capabilities, driving new business, and supporting or expanding existing businesses.

Institutionally, the chemical organization may also be valued in direct relationship to the perceived effectiveness of its managers. Evidence of effective research management is sought in three regimes. Execution of advanced R&D that generates technology and products for the next generation of business needs is paramountly important. The formation of partnerships with business units either to create prototypes of new products and technology platforms or to jointly develop low-cost manufacturing technology objectives is an imperative as well. In administrative and policy areas, the implementation of processes that really work and the development of technology roadmaps are other expectations. Effective managers are also adept in people interaction dynamics or people skills, a talent that is difficult to describe but easy to discern when it is present (or absent).

There is considerable debate about the efficacy of metrics for determining research effectiveness. However, it is almost universally true that valued research will have been assessed by some type of method to measure its effectiveness and productivity. The most frequently applied metrics for gauging research effectiveness include output evaluations. This involves compiling a matrix of outputs (patent awards, inventions, intellectual property, percentage of revenues traceable to research, and so on) and assessing the value of the matrix of outputs. This quantitative, short-term approach to assessing research has several inherent limitations with respect to gauging the breakthrough and longer-range innovation potential of the research organization. To accomplish this assessment more effectively, the structure and fitness of the research organization are scrutinized and examinations made to determine whether the chemical R&D organization is doing the right things. Are there joint projects with business units? Are research strategies aligned with corporate and business strategies? Are innovations as well as inventions targeted? Are breakthroughs and sustaining products, processes, and technologies under development? By examining these and other questions, the corporation determines whether research programs are correctly targeted. This approach, when coupled with an outcome matrix, provides a more global assessment of research effectiveness.

The assessment of the value of research has a number of contingencies at the organizational level. In some cases, the genesis of the research imparts its own value. Thus, the method of evaluation may depend on how the research was started. For example, was it inspired by a strategic customer need? Did the market organization identify a window of opportunity? Did a management directive from a business unit or research organization fuel the R&D program? Sometimes, serendipity accompanying an ongoing investigation leads to a focus on new objectives. In addition to a dependence on its origin, the value of research is altered as the project moves through various phases. As the economic potential of the project increases in the direction shown in Figure 4.1, the risk decreases, and the value judgments used to assess the research change as it moves through each phase.

FIGURE 4.1

Phase transitions of research projects.

Clearly, the most interesting and challenging opportunities for evaluating research are associated with the analysis and prioritization of the portfolio of projects maintained by the chemical research organization. Within an R&D organization, there are always more pressures to engage in activities than there are resources to use. The business units tend to continuously concentrate on projects that will generate revenue within the next year or two. Even though there are development organizations within the business units, research manpower is continuously sought for short-term tactical projects. There must therefore be creditable methods for deciding what projects to undertake. An effective prioritization process must accomplish the following objectives:

•  Determine the value that the materials chemistry research organization delivers to the corporation;
•  Apply to both joint business unit derived projects and research initiated projects;
•  Establish specific criteria for judgment;
•  Guide deployment of resources; and
•  Optimize the value that the materials chemistry research organization delivers to the corporation.

The process should allow work to be scrutinized and determine why it is being executed. The prioritization analysis should assess this balance between long-term and near-term R&D projects to ensure that the long-term health of the company is balanced by the priority pressures of short-term tactical R&D.

The process must include suitable criteria for making the judgments mentioned above. Applying these criteria increases the value to the corporation by pinpointing low-priority projects for termination.

More resources and the best people can then be associated with the highest-priority and most complex projects while investments are minimized elsewhere.

A team within the materials chemistry organization was assigned the task of designing a project prioritization process that could be applied to develop a portfolio that would deliver the maximum value to the corporation. The group had four levels of management (directors, department heads, supervisors, and research scientists and engineers). The ultimate process designed by the team needed to apply equally as well to projects initiated by the research organization and to those done jointly with the business units. Application of the prioritization process should create a portfolio of research projects balanced with respect to the long versus short term, high risk versus low risk, high value potential, maturity, and emerging products and technologies.

The portfolio should contain projects executed independently by the research organization. In some cases, the business units may not even be interested in the project because it is at a stage well below demonstrating feasibility or commercial viability. The research organization must be responsible for innovation of long-term new technology directions for the corporation. Other projects, initiated by the business units, are joined by research team members. Still other projects within the portfolio are conceived and executed jointly with business colleagues.

The ground rules used for formulating the research prioritization process include the following:

•  Delivers maximum value to Lucent;
•  Applies to projects that exceed a set number of people;
•  Is widely supported as a tool for achieving greater effectiveness; and
•  Does not replace management judgment.

The team decided that the four most important areas to be scrutinized by the process include these:

•  Strategic fit: Does the project fit the corporation's and organization's strategy?
•  Economic potential (including resource expenses);
•  Probability of technical success; and
•  Probability of commercial success.

Each criterion is subdivided further, creating a matrix in which each element can be rated for each project. For every entry in the matrix, a team can actually evaluate a portfolio of different research activities. Obviously, there must be background information about each project, and each member of the evaluating team must be familiar with each research project being assessed.

The criteria that emerge (up to four elements for each attribute) provide a framework for attaining qualitative insights, pinpoint areas of disagreement among the evaluating team, and help to resolve disagreements. By simply owning a prioritization process, an organization considerably broadens the scope of criteria used in assessing the importance of various research activities. The application of the prioritization process, while not quantitative, does ensure that a full range of considerations are made during the comparative analysis of projects. The process does not replace managerial judgments. For example, it easily pinpoints excellent or terrible projects. However, the distinction between many good to very good projects is still small, and decisions based on other judgments are usually needed to rank-order a portfolio. Applying a prioritization process does help an organization to decide which projects to enhance, which ones to eliminate or reduce, and which new projects to add.

Even when the chemical research organization still depends on the individually motivated scientist as a source of significant innovation, methods are still valuable for helping the bench research scientist to determine what projects are likely to provide the most value. Encouraging (not requiring) the individual scientist to conduct a self-assessment of his or her ideas and projects by applying a more global corporate perspective is useful. Established scientists with excellent track records need little guidance. Their personal radar for project selection far exceeds any prescription. With new employees, however, there is an opportunity to increase the awareness of specific insights that may enhance their selection of projects to optimize value to the organization. For example, a scientist should be able to describe the intended research using language that a nontechnical person can understand. One should be able to determine what is the intended value and impact of the work. Even if the project is at the most fundamental edge, the scientist should know that the intended customer is the scientific knowledge base. If the work is intended to have value, the scientist needs to be able to identify the path to value. The scientist should also make a prediction of the probability for success. For example, if the research investigation were a complete success, who would care about the results? One should also ask how much the research would cost. If the project has applications, how much more would be incurred before the business unit customer could extract value? A final question: How long would it take to conduct the research?

By executing a self-assessment before initiating a research study, the bench researcher may eliminate some of the random-walk uncertainties that could lie ahead. Such an approach is not universally practical but may assist the individual in thinking more thoroughly about an intended project before making the decision to discuss it with management. This self-appraisal helps assess whether an intended research investigation falls within the broad context of having potential value.

Ultimately, the objective of evaluating research is to improve the effectiveness of research and (in the case of industry) to enhance value for the corporation. If the research is applied, the value can be more easily determined by using economic assessments. For fundamental research, the practical value may be unknown at the onset. However, value is still present in the form of brand enhancement, equity in reputation, and advertising, through publishing excellent scientific work. Several guidelines are useful for optimizing the value of chemical research:

•  Informed research managers should have access to information regarding where contributions could be made. Scientists and engineers need to be continuously provided with this information. Direct communication is necessary to permit individuals to volunteer to work in exciting areas rather than being assigned to projects.
•  The organization needs to value high-risk, long-term research. Bell Laboratories today puts even greater emphasis on breakthrough research than ever before. The origin of the project may or may not have involved a business unit. However, when appropriate, these projects are executed jointly so that business units may take advantage of an innovative accomplishment.
•  Research self-appraisals are useful and should be encouraged.

Organizations and individual scientists should identify paths to value as early as possible in a research investigation. Frequently, the research scientist working alone does not consider cost and complexity

benchmarks that have to be passed if a project concept is to be cost-effective and useful. When work is applications oriented, cost and complexity should be considered very early on to the extent possible. Where appropriate, market analysis and business case developments are done at the onset of experimental work. Including business units and manufacturing teams as early as possible during the research project is also highly productive.

In closing this discussion, I decided to present a nuts-and-bolts look at evaluating research. As Donald Kash showed, Lucent Technologies also has large, high-level, breakthrough research projects, and international global research as well. However, rather than cover specific research accomplishments in these areas, I sought to cover practical approaches for evaluating and valuing materials chemistry research.

The contributions of members of the Lucent project prioritization team are greatly appreciated. The author also thanks Mel Cohen for contributions and concepts for self-assessment by bench scientists.

David A. Hounshell, Carnegie Mellon University: You reviewed the criteria for prioritization, which included at least four areas: strategic fit, economic potential, and so on. They are scored on a scale, and you said that a number of people do the scoring. How do you avoid "group think" in the prioritization? How do you allow for contrariant thought, particularly with the more junior research staff as opposed to the senior staff?

James W. Mitchell: This process is executed by supervisors and department heads, so we don't involve the actual researcher in this assessment process, but rather a team of managers. To some extent, this draws out a lot of the discussion, because the individuals doing the prioritization are not usually the ones executing the projects. The process is also not necessarily applied to all of the projects in a given research area, like chemistry.

Jack G. Kay, Drexel University: There is one type of research that hasn't yet been addressed, and I wondered how you would justify it or categorize it. The type of research I am wondering about, for example, is the research that led to the realization that chlorofluorocarbons underwent a photolysis process and ended up chewing up the ozone layer. You don't have a product to sell; in fact, you've eventually gotten rid of some products that were previously sold. How do you evaluate that type of research?

James W. Mitchell: I gave you particular details on processes that apply mostly to situations where we could see some path to value. In the case of simply contributing to the scientific knowledge base, then you use other principles. There is a corporate culture at Bell Laboratories that says it is impossible for any manager to look at a given project and make an absolute decision that there is nothing of value that can come from it. So one looks at the upper 10 percent of the research population, and there is a fraction in that population that can do almost anything they want. They are the scientists that you depend on to make the decisions about what basic research ought to be going on. They merely need to present a well thought out idea to a manager, and then that manager's job is to see to it that they get the resources needed to pursue that research.

The percentage of that type of research that can be pursued is certainly smaller now that we are competitive in every aspect of our business than it was in previous years, when we had a monopoly and only had to worry about how best to spend money. So we depend on the best scientists to make those decisions and to pursue them.

Lawrence H. Dubois, Defense Advanced Research Projects Agency: As a former Bell Labs employee, I can say this is a radical change from the way things used to be done. Have you taken the criteria and the concepts that you talked about here and gone back 5, 10, 15 years and applied them to some of the developments there to see if they really work? That is, would these criteria and concepts have identified in the past those technologies that proved to be important to the corporation in the future?

James W. Mitchell: No, we've not gone back in the past and looked at case studies. When you apply these approaches, you can easily pick out the winning projects and the losing projects. In the middle, there are going to be a number of projects that will be closely rated, and then you have to use managerial judgment to prioritize those.

So let me also make it clear that we do not apply this process across the board in every phase of the research organization. I have applied this approach to certain projects, for example, in one particular area where we had many, many projects and wanted as rational an approach as possible to determine which one of those to deemphasize. But no, we have not gone back in the past and applied the process. That would be an interesting project to undertake.