Industrial Technology Assessments: An Evaluation of the Research Program of the Office of Industrial Technologies (1999)

The objective of OIT's research programs is to work with U.S. industry to improve energy efficiency, reduce waste, and increase productivity. The IOF strategy is intended to improve OIT-industry partnerships, ensure the relevance of research projects, encourage industry participation, and facilitate the commercialization of developed technologies. The stated long-term goals of OIT are to achieve a 25 percent improvement in energy efficiency and 30 percent reduction in emissions for the IOF industries by 2010 and a 35 percent improvement in energy efficiency and 50 percent reduction in emissions by 2020 (OIT, 1998a). The elements of OIT's strategic plan are listed below:

• focus on energy-intensive and waste-intensive industries
• establish partnerships with industry
• apply the IOF strategy to direct industry and government resources to areas of greatest need
• support new "vision" industries
• use integrated teams to accomplish program goals
• design flexible and responsive programs
• strengthen the education and training of OIT staff
• monitor progress and performance
• conduct comprehensive planning to guide future program strategy

The present study focuses on OIT's transition to the IOF strategy. To implement the strategy, OIT has: (1) facilitated the preparation of industry visions and technology road maps, (2) initiated cooperatively funded R&D projects identified in the visions and road maps, (3) conducted generic (or crosscutting) R&D projects, and (4) disseminated research results and program benefits. The

committee's assessments of OIT's program implementation and management are presented in this chapter.

The committee believes that the most significant accomplishment of the IOF initiative has been the development of a consensus by industry groups on research priorities through the preparation of industry visions and technology road maps. However, vision documents and road maps must be "living documents," that is, flexible enough to accommodate changes in the technological and business climate. Maintaining the consensus by updating the industry vision and technology road maps will be critical to the future of the IOF program. Examples of how current processes benefit all stakeholders are listed below:

• Industries are gaining monetary support for research that they have determined to be important.
• DOE is funding high-priority research projects needed by industry and is no longer perceived as funding projects arbitrarily.
• "Industry research agendas" assure researchers that their research is important to the industry.
• Government and industry share the cost of research.
• Researchers and their universities or not-for-profit organizations have intellectual property rights under the Bayh-Dole Act of 1980 (P.L. 96-517).

The agriculture IOF team, established in late 1997, has issued a vision document for plant/crop-based renewable products based on a workshop initiated by the National Corn Growers Association (OIT, 1997a). The DOE-industry compact was signed on February 23, 1998. The goals set forth in the vision document include the following:

• Displace at least 10 percent of petroleum with plant/crop resources as the basic building block for consumer products by 2020 and provide the concepts needed to displace as much as 50 percent by 2050.
• Establish a plant/crop-based manufacturing infrastructure.
• Establish partnerships between industry, government, and academia for R&D that can lead to market opportunities and ensure that processes and systems are commercially viable.

Two road-mapping workshops were held in mid-1998 to focus on uses of existing crops and modified plants, which was a key technology identified in the vision. The road map was published in February, 1999 (OIT, 1999).

The aluminum industry, represented by The Aluminum Association, published its vision document in March 1996 (Aluminum Association, 1996). A compact to undertake collaborative research to meet the vision goals was signed in October 1996. The industry developed a technology road map, which was published in June 1997 (Aluminum Association, 1997). Subsequently, a road map to address one of the industry's most pressing needs, the development of inert anode technology was published (Aluminum Association, 1998) (see Figure 4-1). The aluminum team has issued proposal solicitations for fiscal year (FY)98 and FY99 through DOE for research that addresses the needs identified in the road maps.

The chemicals industry, represented by the American Chemical Society, the American Institute of Chemical Engineers, the Chemical Manufacturers Association, the Council for Chemical Research, and the Synthetic Organic Chemical Manufacturers Association, published its vision document in December 1996 (ACS, 1996). The vision identifies four critical technology areas for the chemical industry:

• new chemical science and engineering technologies
• supply-chain management
• information systems
• manufacturing and operations

A memorandum of understanding was signed on February 26, 1997, with DOE to establish a framework to identify appropriate areas for collaborative R&D and technology demonstration. Road maps are being developed to address technology areas identified in the vision document. Road maps that pertain to catalysis (Haynes, 1997), separations (CWRT, 1998), computational fluid dynamics (LANL, 1998), and materials of construction (MTI, 1998) and draft visions for bioprocesses and computational chemistry have already been developed. The interim report from the Workshop on Process Measurement and Control: Industry Needs has also been completed. (Doyle, 1998). The chemicals team has issued proposal solicitations for FY99 to address these specific areas.

Forest products was the first IOF industry group to develop its vision in November 1994 (AF&PA, 1994). The vision was developed by a group of Chief Technology Officers (CTOs) from U.S. pulp and paper companies and was endorsed by a working group of Chief Executive Officers (CEOs) for the American Forest and Paper Association (AF&PA). Robert C. Williams, then CEO of James

River Corporation, representing the U.S. industry, signed a compact with DOE, represented by then Secretary of Energy Hazel O'Leary, in November 1994.

Six working groups were formed (one for each technology area—environmental performance, recycling, sustainable forestry, sensors and controls, energy performance, and improved capital effectiveness). Several of the groups were able to get quick starts on high-priority research because similar activities were already under way (sponsored by the Technical Association of the Pulp and Paper Industry). Each working group developed a portfolio of "Research Pathways," an example of which is shown in Figure 4-2.

Each working group has developed procedures for soliciting proposals from researchers in academia, industry, and the national laboratories and a selection process for choosing research that addresses the designated needs. The recommendations from the working groups are then passed to a committee of CTOs, who either eliminate proposals or endorse them and submit them to OIT. OIT makes the final decision as to whether or not to fund a proposal.

The glass industry published its vision in March 1996 [http://www.oit.doe.gov/glass/page9.html]

• reduce production costs by 20 percent (compared with 1995 levels)
• recycle 100 percent of all glass products
• reduce process energy use by 50 percent
• reduce air and water emissions by 20 percent
• recover and recycle 100 percent of available post-consumer glass
• achieve "six sigma"¹ quality through automation, process control, optimized glass composition and strength, and computer simulation
• create innovative products that broaden the marketplace
• increase supplier and customer relationships in raw materials, equipment, and energy savings

A compact between DOE and industry representatives—Anchor Glass Containers, Carr Lowrey Glass Company, Certainteed Corporation, Corning, Ford Motor Company, the Glass Packaging Institute, and the Society of Glass Science and Practices—was signed on April 29, 1996. An industry road map that addresses the four principal industry segments—flat glass, container glass, fiberglass, and specialty glass—has been published (Energetics, 1997).

The metalcasting industry, represented by the American Foundry Society, the Steel Founder's Society of America, and the North American Die Casting Association, published its vision in September 1995 (CMC, 1995). The Cast Metals Coalition (CMC) was established to provide direct partnership with DOE. A compact was signed with DOE in October 1995 to establish a framework to identify areas for collaborative R&D and technology demonstration. In 1998, the CMC published a technology road map, which establishes near-term, midterm, and long-term technology goals (CMC, 1998) in several key areas, including (1) products and markets, (2) materials technology, (3) manufacturing technology, (4) environmental technology, (5) human resources, and (6) industry health.

As required in the MetalCasting Competitiveness Research Act of 1990 (P.L. 101-425, 104 Stat. 915, 15 U.S.C.§5301-09), the program is managed by OIT. The CMC, through its executive board and technical committees, oversees the IOF R&D. Projects are reviewed by the CMC technical committees, which select candidate projects based on the road map and vision document. The CMC Executive Board (which includes representatives of the three industry associations, OIT, and the metalcasting industrial Oversight Panel) makes the final selections.

The mining industry is the newest industry group in the OIT program. The National Mining Association (NMA) represents the industry, which includes coal, metals, and minerals concerns. The industry vision document was published in September 1998 (NMA, 1998a), and the first of several road mapping sessions was held in October 1998, producing a road map for crosscutting (NMA, 1998b). A solicitation for R&D projects to support the identified industry needs is being developed.

Petroleum refining was one of the original IOF industries. In 1997, the group, which was organized through the American Petroleum Institute (API), disbanded. The industry group was reestablished in 1998. Efforts are under way to establish research priorities and road maps. Recent developments include the release of a report presenting benchmark energy use and environmental data for the petroleum refining industry (OIT, 1998b). API has completed the downstream industry vision, which is currently being reviewed by API committees.

OIT has been working with the steel industry for more than a decade, beginning with the Steel and Aluminum Energy Conservation and Technology

Competitiveness Act of 1988 (P.L. 100-680; 15 U.S.C. 5101 et seq.), the so-called Metals Initiative. The steel industry's IOF participation was organized through the AISI and the Steel Manufacturers Association. The industry vision document, published in 1995 (AISI, 1995), identified four areas that would be critical to competitiveness over the next 20 years: production efficiency, recycling, environmental engineering, and product development. A compact was signed with DOE in May 1995 to establish guidelines for industry-government partnerships.

A technology road map was developed and published in 1997 and updated in 1998, to establish research priorities (AISI, 1997, 1998). Much of the current research portfolio is part of the Metals Initiative. The industry has issued separate solicitations for industrial researchers (in partnerships with university and national laboratories) and for DOE national laboratories. Industrial reviewers evaluate research proposals based on the technology road map.

The committee believes that vision statements and road maps are not ends in themselves. These documents must be maintained, updated, and acted upon to achieve the technology advancement goals they describe. Industry-generated technology visions and road maps naturally address issues besides energy and waste reduction. Therefore, the long-term challenge for OIT will be to keep these documents up to date and to maintain the industries' interest in addressing all of the identified needs, not just the ones that receive OIT support. The committee believes that the vision and road map processes would probably not continue without government stimulation and financial support for the defined technology development needs.

Recommendation. OIT should maintain and update the road maps and vision statements regularly (annually or bi-annually) to reflect changes in industry objectives, technologies, and the business climate.

One of the primary motivations for the IOF strategy is to increase the participation of industry groups in the OIT program by making research more relevant to industry. A measure of industry participation used by federal agencies, including DOE, is the level of industry cost-sharing in federally funded research. Most of the solicitations for industry-specific R&D specify a cost-sharing target (e.g., aluminum industry solicitations require 30 percent cost sharing; [http://www.oit.doe.gov/IOF/aluminum/aluminum.html#solicitation]).

OIT provided the committee with a summary of the status of IOF-specific research projects and cost sharing (see Table 4-1). The current projects were divided into three categories indicative of the incomplete transition to the IOF strategy. The categories are:

• projects started before an industry vision document or technology road map was developed (pre-vision, pre-road maps)
• projects started after the release of the industry vision document but before completion of technology road maps (post-vision, pre-road maps)
• projects started after the industry vision document and road maps were released (post-vision, post-road maps)

Surprisingly, the percentage of industry cost-sharing did not increase significantly as expected under the IOF strategy. This could be because, in addition to cost-shared contracts, OIT has emphasized other mechanisms to partner with industry, including cooperative research and development agreements, R&D consortia, exchange programs, user facility agreements, and work-for-others agreements. In addition, some industry groups (e.g., forest products) have suggested that DOE focus their funding on academic research, while the industry performs other work identified in the road maps on their own (outside of the IOF program).

It will be very difficult for OIT to keep track of all types of industry research that results from the vision and road map processes, and the committee believes that the impacts of industrial research related to the industry vision documents and road maps might not be accounted for. Because OIT offers a wide range of options for industry participation, they should develop methods of tracking the status of all types of industry participation in the IOF program.

OIT divides its R&D into three categories: IOF-specific programs, crosscutting technology programs, and technology access programs. This section discusses the OIT research portfolio, including the balance of industry-specific and crosscutting research programs.

OIT's FY98 budget for the IOF program was $101.8 million, up about $16.1 million over FY97. Of the 1998 total, $49.1 million (48.2 percent) is devoted to ''crosscutting'' technologies. The balance, $52.7 million, comprised industry-specific line items related to the IOF vision statements and road maps. The crosscutting category included major projects involving co-generation, advanced materials, and continuous-fiber ceramic composites.

OIT also has a variety of technology access programs, totaling $26.3 million, and a small program ($3.0 million) on biomass turbine fuels. Finally, management and planning for all OIT activities is budgeted at $7.7 million, bringing the total FY98 budget for OIT to $138.9 million.

The IOF budget for FY99 is $166 million, an increase of $30 million over FY98 and an increase of $14 million over FY97; thus the budget has been growing. The major growth area for FY99 is in the crosscutting programs, as can be seen from Table 4-2. There was also a small (8 percent) increase in IOF-specific programs.

The recent review of progress in DOE's R&D by the President's Committee of Advisors on Science and Technology (PCAST, 1997) recommended that the FY99 budget be $185 million; the review recommended that the budget for IOF-

specific projects be increased from $53 to $65 million and the budget for crosscutting programs be increased from $49 million to $70 million.

In general, the crosscutting projects are larger than the IOF-specific projects. For example, in FY97 the forest products group had $10.8 million for 33 projects, ranging from $22,000 to $3.5 million. The aluminum industry had $5.6 million for 8 projects, ranging from $200,000 to $2.5 million.

Recommendation. OIT should perform a "portfolio analysis" to evaluate its overall research program. The analysis should include technical risk, potential payoff (in terms of energy savings and waste reduction), and time frame (near-term or long-term). The overall portfolio balance should be considered in the evaluation, as well as the prioritization of research projects; projects should be added or trimmed to balance the portfolio, as necessary.

The funding levels for IOF-specific research for each industry is shown in Table 4-3. Trends in funding reflect the industry groups' progress in developing their vision documents and road maps to establish their research priorities. For example, the 82 percent increase for industry-specific research for the aluminum industry from FY97 to FY99 reflects the industry's road maps and established priorities, which facilitated the solicitation of research proposals.

Now that most of the industry groups have finished at least preliminary road maps, OIT will have to develop a rational process for allocating limited funds among the IOF industries to support their identified needs. If the process appears to be arbitrary, the industry participants could consider the allocation process a competition among the industry groups. Allocation schemes should assess the

technical needs and priorities of each group and consider factors such as the size of the industrial community, the potential effects of the research on OIT goals, the ability of the industry to support implementation of the technology, and other potential sources of support.

Recommendation. OIT should establish a rational, transparent process for allocating funds among IOF industries and then allow them to set project directives based on their road maps, as long as the projects are consistent with OIT's mission.

Previous attempts to use input from industry representatives to identify research priorities to improve their competitiveness have been criticized in Congress as "corporate welfare." For example, DOD's Technology Reinvestment Program and the National Institute of Standards and Technology's Advanced Technology Program were criticized for inappropriately aiding commercial industry. The principal lesson to be learned from these experiences is that "it is inappropriate for the government to undertake product development without a compelling national mission" (Denman, 1996). In the past, OIT has effectively focused on applied research and technology development that furthered DOE's goals of energy efficiency and waste reduction. The committee is concerned that allowing industry groups to set program priorities could shift the focus toward near-term product development or lose sight of DOE's mission.

New industries have recently joined the IOF (agriculture in 1997 and mining in 1998). The committee believes that increasing the number of industry groups can be an effective way to expand the IOF program as long as the new industries meet the initial criteria, that is, they are large energy consumers and industrial waste producers.

Recommendation. OIT should continue to apply its criteria of energy consumption and waste generation in selecting industries for participation in the IOF program.

In Chapter 3, the committee identified four types of potential crosscutting technologies. The current program includes two of the four types: (1) existing projects that predate the IOF strategy that have been relabeled as crosscutting (e.g., AIM, CFCC [Continuous-fiber ceramic composites], and ATS [advanced turbine systems]) and (2) projects of significant interest to several IOF industries that could be more efficiently managed and leveraged if they were merged into a crosscutting program (e.g., sensors and controls). The committee believes that only the second of these types is consistent with the IOF strategy. The recommendations in Chapter 3 relate to the committee's suggested approach to managing crosscutting programs.

Crosscutting programs that predate the IOF strategy include major initiatives, such as the ATS program, which is now part of a combined heat and power global climate change initiative, and the more mature CFCC and AIM programs. Although the committee did not evaluate these programs in detail, they do not necessarily fit in with the IOF philosophy because they are not the result of the vision and road map processes. The committee recommends that these initiatives be either (1) managed separately from the IOF-specific projects or (2) re-evaluated and brought within the IOF framework.

Recommendation. To complete the transition to the IOF strategy, OIT should shift the balance of IOF-specific and crosscutting research to emphasize industry-specific work identified on industry road maps. Industry participation in the management and evaluation of crosscutting research programs should be expanded.

The committee recognizes that relying on "market pull" to define R&D objectives has inherent drawbacks. One of the drawbacks is that important research on technologies that could potentially benefit many industry groups but that are not a primary concern to any one of them is likely to go unfunded. A "market pull" strategy has no simple, self-reinforcing mechanism for identifying promising crosscutting programs.

Recommendation. OIT should adopt the following approach to managing crosscutting programs within the IOF strategy:

• Develop a consensus among the IOF industries that a certain percentage of R&D funds should be allocated for basic science and the development of crosscutting technologies.
• Using established management procedures, define and select a recommended list of basic/crosscutting technologies for development.
• Review these recommendations with the IOF industry groups, and solicit support and feedback.

The DOE national laboratories have been important in conducting research and managing interdisciplinary projects in the OIT program, especially in the legacy crosscutting technology areas (e.g., AIM, CFCC, and ATS). However, since the implementation of the IOF strategy, the national laboratories have had to reposition themselves in the OIT program by teaming with industry and responding to the needs established in the IOF road maps.

OIT was instrumental in the establishment of the Laboratory Coordinating Council (LCC) in 1995 to provide industry with information on laboratory capabilities in specific technologies and to facilitate collaborations with industry by matching laboratory capabilities with industry needs (Chum, 1997). The LCC includes representatives of the following organizations: Albany Research Center, Ames Laboratory, Argonne National Laboratory, Brookhaven National Laboratory, Federal Energy Technology Center, Idaho National Energy and Engineering Laboratory, Kansas City Plant, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, National Institute for Petroleum and Energy research, Oak Ridge National Laboratory, Pacific Northwest National Laboratory, Sandia National Laboratories, Savannah River Technology Center, and the Y-12 Plant.

The LCC, which was established to facilitate interactions between the national laboratories and the IOF industry groups, has developed separate mechanisms for interacting with each industry group and for addressing crosscutting areas. The LCC has developed matrices of laboratory competence for specific technical areas to provide industry with links to capabilities in the national laboratories and to coordinate responses to industry research initiatives among the interested laboratories.

Even with the improved coordination provided by the LCC, the national laboratories have found it difficult to align their programs with near-term, industry-specific programs. The laboratories feel that their strengths are better suited to long-term, crosscutting initiatives. The use of "market pull" strategy has shifted the emphasis toward industry-specific technology development and implementation at the expense of crosscutting technologies.

Before 1995, R&D management at OIT was "science driven" or "technology-push driven." Although this approach led to many technological successes, transitioning technology to the commercial sector was difficult to document and, in many cases, was not done. Independent, expert review panels or committees, the members of which were often distanced from the front lines where technologies were transferred from the laboratory to commercial use, provided direction and oversight.

In the transition to the IOF strategy, OIT made a commitment to increase and document the commercial impact of OIT programs provided they still met the overall goals of improving energy efficiency and reducing waste generation. To accomplish this mission, OIT recognized that its R&D management strategy had to change from a "technology push" to a "market pull" strategy. The two key questions however were (1) which markets should be served, and (2) how the market-pull could be harnessed. To answer the first question, OIT ranked industries according to their level of energy consumption and waste production, the linchpins of the IOF strategy. The second question is the subject of this section.

By soliciting IOF feedback on basic and crosscutting technologies, industry is able to exert a market pull on OIT's programs. This was a necessary, but not sufficient, condition of successful technology transfer. The commercialization of a new technology is a difficult and risky proposition even for corporations that specialize in, and depend on, commercialization. Every established materials or manufacturing company in the world has a long list of failed attempts to commercialize new technologies, and the failures usually outnumber the successes. It may be, therefore, inappropriate for a government program to measure its R&D success against the direct metrics of technology transfer and commercialization.

In the usual sense of the word, "commercialization" implies one of two possibilities. Either the embodiment of a technology must be sold directly to a market in a way that is both profitable and sustainable without corporate or government subsidies, or it must be incorporated into a component or system that is similarly sold. To be a commercial product, someone must be earning money from selling it. In most corporations, the activities required to commercialize a new technology are spread among many groups, although they are concentrated in the sales, marketing, and new business development organizations. Although these groups do not follow standard commercialization procedures, they all go through similar stages in the commercialization process:

• develop the technology
• sample it with selected "key" customers
• perform market research/develop an internal commercialization plan
• establish pilot-scale production facilities or find outside production sources
• develop promotional materials (brochures, etc.)
• "launch" the new product with sales promotions
• train internal and external sales personnel in the use and benefits of the technology

At every stage of commercialization, the proponents of a new technology must solicit, defend, and secure corporate commitment and funds to advance to the next step. For new materials technologies, the process generally takes years, often decades. Because OIT has no profit motive or profit-making capabilities, it

cannot fully participate in the commercialization process. Therefore, the final stages of the commercialization of technologies developed by OIT must be left to a third party.

Recommendation. Although successful commercialization cannot be assured, the committee believes that the following actions will improve the probability of commercial success:

• Maintain regular interactions with all critical stakeholders in the supply chain through all stages of program development, including raw material suppliers, parts makers, and systems integrators.
• Publicize the technical accomplishments of the program at popular trade meetings, and use these meetings to meet and network with technical and business people.
• Establish networks that include not just technical people, but also sales, marketing, and senior management personnel.
• Expose technical personnel to basic business principles, including elements of cost estimation, value analysis, and market research. Insist that rudimentary business plans accompany each later-stage R&D program and have these plans critically reviewed by industry stakeholders.
• Selectively subsidize and participate with third-parties in programs to demonstrate and de-bug the technology. These activities should not be confused with commercialization, however, and should be limited to cases where additional technical development is required to enable commercialization. Insertion programs should not be sustained only by government subsidies.
• Recognize that technology development is only one link, albeit an important one, in the chain of commercialization.

Recommendation. OIT should participate directly in a limited number of commercial insertion programs but only for the purpose of identifying remaining technical hurdles.

OIT has a number of technology access programs to validate and commercialize new energy-saving manufacturing technologies. These include open competition grant programs, such as the National Industrial Competitiveness through Energy, Environment, and Economics (NICE³) and the Inventions and Innovation (I&I) programs. Other programs are intended to address particular energy and environmental goals. Other technology access programs include the following:

• Motor Challenge, which was established in 1993 to identify and implement technologies to save energy in electric-motor-driven systems
• Climate Wise, a partnership program, jointly sponsored by OIT and the

  &nbsp

Environmental Protection Agency, to provide a clearinghouse for technologies that reduce greenhouse gas emissions

• Industrial Assessment Centers, a program that establishes university centers that conduct no-cost energy and environmental assessments of small and medium-sized manufacturing plants

OIT's technology access programs can provide valuable aid to businesses attempting to validate and implement industrial technologies to reduce energy use and waste generation. But all of these programs predate the IOF strategy, and their links to the IOF road maps and priorities are weak. These validation and commercialization programs should be established and planned from the onset of OIT participation to be more effectively integrated with the IOF program.

Recommendation. Validation and implementation programs, such as I&I and NICE³, should be integrated with the IOF program to improve their relevance in efforts to commercialize OIT-developed technologies identified in IOF road maps.

Representatives of industry groups participate in the management of industry-specific projects according to procedures developed by each individual group. Mechanisms have been established for developing solicitations based on industry road maps; assessing and prioritizing research proposals; and, in some cases, assessing progress and disseminating the results of ongoing projects. The committee believes that a strong industry role in the management of the OIT research portfolio is essential to the success of the IOF strategy.

In Chapter 3, the committee recommended that OIT expand the role of the IOF industry groups in the management of crosscutting research initiatives by establishing coordination groups in each technology area to develop goals and monitor the progress and results of research and by sponsoring forums to facilitate interaction between researchers and potential IOF users. The committee believes that it will be difficult to manage crosscutting initiatives in the IOF framework in a way that facilitates the development of specific performance goals based on the common needs of several industries.

The success of the OIT program will continue to be measured by the level of industry participation. Implementation of new technologies and a research agenda responsive to changing industry priorities will be the key to maintaining industrial support. OIT and the IOF representatives should continue to replenish the

program with new projects. Clear milestones should be identified for each project; the milestones should be monitored and reevaluated at yearly intervals. Projects should be terminated if they meet any of the following criteria:

• all technical goals have been achieved (project is completed)
• goals for technical progress and program expenses have not been met
• industrial support for the technology has been withdrawn

Recommendation. As part of the overall project management process, OIT should develop a mechanism for the orderly termination of (1) projects that have met their objectives and have progressed to the final stage of commercialization (market introduction), (2) projects that have not met goals for technical progress or program expenses, and (3) projects that do not have sufficient industrial interest to support demonstration, process development, and scale-up.

The committee believes that industrial experience for mid-sized to large-sized enterprises that have a mix of technology development and product development could be a guide to managing the project turnover for the OIT program. Industrial research and product development are typically managed over four to five year periods of commitment (i.e., 20 to 25 percent of projects are terminated or completed each year, and a similar number are started) (Wirth, 1996).

Recommendation. OIT should adopt guidelines for program turnover similar to those used by industrial technology-development and product-development organizations.

OIT has a number of mechanisms for communicating the status and accomplishments of its research programs, including technology workshops, publications to describe programs and partnerships and assess completed research based on energy and environmental metrics (OIT, 1997b), a detailed information site on the Worldwide Web [http://www.oit.doe.gov] , and biannual industrial energy efficiency symposia and expositions. Solicitations are accessible through the Worldwide Web and the network of the industry associations involved in the program, as well as in Commerce Business Daily.

Nevertheless, the committee believes that the OIT program could be promoted more effectively. In many cases, the OIT program is the only significant government-sponsored research program focused on process industries. Better promotion of the opportunities and broader dissemination of research results would encourage more industry participation in the program.

Recommendation. OIT should increase its attempts to promote its program by taking the following steps:

• describing technical successes in the trade literature, at technical society and industry trade meetings, in the popular press, and through other high visibility communications media
• promoting industry participation in the validation and implementation of technologies that could lead to commercialization
• describing the program approach, objectives, and levels of industry participation at high-level symposia or forums hosted by the secretary of energy to maintain the interest of industry executives in the program

Each of the many approaches to measuring the efficacy of R&D has proponents and detractors, but none is universally, or even widely, accepted. The committee recommends that OIT consider using the following metrics as a basis for comparing and selecting projects:

• potential for energy conservation
• cost/benefit ratio (i.e., risk-adjusted return on investment)
• consistency with IOF business objectives and technology road maps
• commercial potential/market value
• potential for use by more than one industrial sector (crosscutting potential)

The best metrics for measuring the efficacy of OIT research programs are likely to be some of the measures used internally by the IOF industries. R&D managers from these industries should be contacted and polled regarding their approaches to setting priorities and to measuring effectiveness. However, some "profit-based" metrics used by industry to assess the efficacy of R&D may not be appropriate for assessing government-funded research. Short-term commercial potential should not be used to direct the selection of OIT's programs.

Recommendation. OIT should develop and apply realistic metrics to provide credible assessments of the technical and commercial successes of OIT research. Metrics should include measures of energy use, waste generation, resource utilization, and economic impact. The bases of these metrics should be clear and transparent to avoid the perception of "metric inflation."

The committee believes that the IOF program, to date, has been a success. The principal tangible successes have been the creation of industry vision documents and technology road maps. Although the committee believes that the IOF strategy will make the OIT program more effective, it is too early to judge the effect of the IOF strategy on the effectiveness of DOE-sponsored research in terms of OIT's mission of reducing waste and energy consumption.

Recommendation. OIT should take the following steps to ensure that the momentum of establishing the IOF industry groups and the vision and road map processes is maintained:

• continuing to provide significant funding for research that addresses identified industry needs
• involving industry representatives in monitoring ongoing projects and evaluating planned IOF-specific and crosscutting projects.

Recommendation. OIT should continue to adhere closely to the philosophy of the IOF program (i.e., continue to work closely with industry and allow industry to guide the process and set priorities).