Naval Engineering in the 21st Century: The Science and Technology Foundation for Future Naval Fleets -- Special Report 306 (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Conclusions and Recommendations The Office of Naval Research (ONR) asked the National Research Council (NRC) to examine the state of basic and applied research in the scien- tific fields that support naval engineering and to advise it on whether ONR activities, under its National Naval Responsibility for Naval Engineering (NNR-NE) initiative, have been effective in sustaining these fields. The study committee was also to identify opportunities to enhance innovation, research, and graduate education in these fields and identify areas of scien- tific research that provide opportunities to make fundamental advances in naval ship capabilities. The committee’s conclusions and recommendations are presented in five sections: the justification and value of the NNR-NE; the state of sci- ence and technology (S&T) supporting naval engineering; the complete- ness and balance of the NNR-NE portfolio of basic and applied research; opportunities for enhancement of research and graduate and post- doctoral education, and related institutional and physical infrastructure; and, finally, the effectiveness of the NNR-NE initiative. Recommendations are addressed to the administrators of the NNR-NE initiative and of ONR. Several of the recommendations that concern scientific research oppor- tunities could be acted on within the present structure of the initiative. Recommendations for measures to increase the effectiveness of NNR-NE and to enhance innovation, research, and graduate education through changes in management processes would require action by senior ONR administrators. 174 

Conclusions and Recommendations 175 NEED FOR AND VALUE OF NNR-NE Need for Navy Support Research provides the fundamental technology and knowledge that ensure Navy success in future operations. Within the Navy’s research portfolio are basic and applied research programs that provide advances in knowledge and technology and lead to future naval capabilities. Some of the basic and applied research used by the Navy is supported by the Department of Defense (DOD), the National Science Foundation (NSF), the National Institute of Standards and Technology, and other agencies. However, some basic and applied research is so specific to Navy needs that it is sup- ported only by the Navy. Without Navy support the research would not be performed. NNR-NE was created to support such research. Navy support is necessary for basic and early applied research programs that • Are in fields critical to naval engineering that would not make progress without Navy support, • Have a long-term horizon (i.e., programs that expect to yield progress on fundamental problems over a 10- to 20-year period), and • Have potential for broad application and for discovery of knowledge that may lead to advances in naval capabilities. Navy support for NNR-NE research is critical for several reasons. Tech- nological progress is essential to security to ensure that naval superiority is maintained as the operating environment, missions, and resources avail- able to the Navy change in the future. The history of innovation in ship design and development provides examples of cases in which basic and early applied research in fields related to naval engineering was essential to the development of new naval capabilities. Moreover, the potential exists for high-payoff applications of research in progress today, even if the probability and form of applications cannot be predicted precisely. Finally, in areas where improvements in ship design and capability have been slow (e.g., innovations that aid in cost control), it is likely that lack of innovation can be attributed, in part, to past lack of support for basic research. Navy support for NNR-NE is also valuable because DOD funding of basic and applied research historically has yielded benefits beyond national 

176 Naval Engineering in the 21st Century defense and DOD is a primary funder of research and graduate education in a number of engineering disciplines of importance to the U.S. economy. Without Navy support, the critical mass of technical expertise and research talent in these fields would not be maintained, and the capability to inno- vate in naval engineering would be lost in the United States. Value of NNR-NE Conclusion 1: The NNR-NE, as defined in the 2001 ONR memoran- dum establishing it and the ONR instructions defining the NNRs in general, is a useful means of organizing ONR support of basic and applied research in the scientific and technical fields that underlie naval engineering. Assigning the NNR designation established Navy policy that the identi- fied activities are deserving of special consideration in planning and budgeting at ONR and that the activities are to be coordinated to sustain U.S. research capability to work on problems important to the Navy, maintain the supply of scientists and engineers in disciplines of unique Navy importance, and ensure that ONR can continue to provide the S&T products necessary for naval superiority. The NNR-NE designa- tion is intended to establish naval engineering as an ONR priority, define ONR objectives in naval engineering, and create a management structure for integrating a diverse group of basic and applied research programs with education and outreach activities. ONR’s naval engi- neering activities need these three elements—high priority, clear objec- tives, and effective management. ONR has designated four fields as NNRs: ocean acoustics, underwater weaponry, undersea medicine, and naval engineering. Among these fields, the need for NNR designation is arguably the greatest for naval engineer- ing. Management of research in the other three fields is simpler, because each has a relatively narrow focus and the research objectives and research community to be sustained are relatively easy to define. In contrast, naval engineering is an essentially integrative activity that must apply scientific knowledge from an expansive array of disciplines to solve multiple com- plex problems of naval ship design. Therefore, the attention to long-term planning and research coordination that the NNR process calls for is crit- ical for producing naval engineering innovation. 

Conclusions and Recommendations 177 HEALTH OF THE S&T ENTERPRISE SUPPORTING NAVAL ENGINEERING The committee examined the state of research, education, and physical and institutional infrastructure in the fields supporting naval engineering iden- tified by ONR as within the scope of NNR-NE: ship design tools; structural systems; hydromechanics and hull design; propulsors; automation, control, and system integration; and platform power and energy. Because of the breadth of the relevant scientific fields and the constraints of the study’s schedule and resources, the committee does not regard its assessments as definitive. The committee’s recommendations propose a process for monitoring the health of these fields systematically so that ONR can obtain the information it needs to guide sound research investment decisions. State of Research The committee’s conclusions address the status of the research enterprise in the scientific and technical fields supporting naval engineering as these fields are pursued in universities, government laboratories, and industry. The committee defined the health of research in a field in terms of the three kinds of research outputs intended from ONR’s S&T investments: knowl- edge, transitions, and people (ONR 2009, 4). A healthy research field was defined as one that is productive in advancing fundamental knowledge, has strong linkages to engineering practice as evidenced by the transition of discoveries to applications and by the existence of effective channels of communication between researchers and practitioners, and has posi- tive prospects as evidenced by the development and retention of talented researchers and by the attraction of new researchers and resources. Typ- ically, in a healthy research field, diverse topics are under investigation, many research methods are in use, and the resources allow ample oppor- tunity for creative research and for the pursuit of transition opportunities. The ultimate success of research depends on the availability of practition- ers who are aware of the latest scientific developments, who are proficient in the latest techniques, and who maintain close communication with the research community. Conclusion 2: Some of the fields within the NNR-NE derive strength from a breadth of related applications. These fields benefit from a 

178 Naval Engineering in the 21st Century diversity of funding sources and opportunities for cross-fertilization among communities of researchers working under different sponsor- ship. For example, vibrant research communities are devoted to com- putational fluid dynamics and to structural materials and systems. In these fields, the tasks for the NNR-NE initiative are to ensure that the Navy takes full advantage of the broad pool of researchers that could con- tribute to solving its high-priority problems and to fund basic and applied research on specific problems relevant only to Navy applications. Mech- anisms for these purposes may include better marketing of ONR support opportunities and establishment of more structured interactions with other sponsoring agencies. In other NNR-NE fields or subfields (e.g., propulsors and naval hydro- dynamics), ONR and other Navy agencies are nearly the only sources of support. If the Navy were to identify an urgent need to expand research related to naval problems in these fields, the pool of researchers qualified to work immediately on such problems and not already occupied with Navy-sponsored research would be small. ONR has great responsibility for sustaining education and the institutional infrastructure in these fields. Because of the differences among NNR-NE disciplines, ONR activities to fulfill its NNR-NE obligations need to be tailored to the status of each field. Hydrodynamics and Hull Design; Propulsors Conclusion 3a: The major supporters of hydrodynamics basic research in the United States historically have been the Navy, NSF, and the National Aeronautics and Space Administration (NASA). NSF supports a diverse and substantial program of basic and applied research in fluid mechanics, including projects that have potential applications ranging from chemical engineering to robotics to medicine, but few address hydrodynamics problems of likely relevance to naval engineering. The field of naval hydromechanics, that is, research aimed at understand- ing the physical phenomena that determine the hydrodynamic and hydro- acoustic performance of naval ships, arguably would not survive without Navy support. The move in recent years to replace experimental work with computation—in part to save costs (and time)—has not yet achieved the ultimate potential savings and has in fact created new demands for experimentation and measurements to provide the necessary validation 

Conclusions and Recommendations 179 and calibration of codes and models. Given current resources and objec- tives, the current mix of U.S. naval hydrodynamics basic research (pri- marily, the ONR program) may be the best that can be achieved to meet narrowly focused needs. However, the overall program is stretched thin and will not be able to meet unanticipated critical Navy needs. More important, it does not have sufficient depth in more basic investigations to generate the breakthrough and disruptive technologies that could redefine naval engineering in the future. The balance between computational and experimental work in hydro- dynamics must be carefully monitored. Experimental validation remains an essential step in the development of hydrodynamic models. However, experiments are costly and therefore more vulnerable during periods of budget pressure. Experimental facilities depend on funded research for their support and without use will deteriorate. Major research facilities are maintained and used at the Naval Surface Warfare Center, Carderock Division (NSWC-CD), and elsewhere, primarily at universities. Structural Systems Conclusion 3b: U.S. industry supports little naval structures research because few large commercial ships are built in the United States. Naval structures research is performed and funded in the commercial sector in such countries as Japan and Korea, where commercial shipbuilding is a major industry. Basic research in structures and structural materials (that is, research not focused on naval applications) has a broad range of potential applications and receives support from multiple public sources (including NSF and NASA) as well as private-sector sources; therefore, many structures researchers are working in the United States who could perform naval structures research if they received funding from ONR. However, the health of the field of structures research directly related to naval engineering, exclusive of ONR activities, can only be considered as poor to fair in the United States. Ship Design Tools Conclusion 3c: Little research in the United States is aimed at developing improved tools and methods for use in the early stages of the design of naval ships. 

180 Naval Engineering in the 21st Century In the early design stages, the performance requirements for the new ship must be translated into a viable design concept, and the design is defined up to the level of detail required for making cost and construction sched- ule estimates (contract design). Decisions made at the early design stages determine the basic architecture of the ship and ship systems and costs of construction and ownership. At the same time, the shipbuilding industry, with Navy support, has invested significantly in the development of tools for detail design, the stage of design that produces the plans and procedures that guide the shipyard construction workers and provides control over construction cost and schedule. These shipyard design tools are more advanced than are those in use for commercial ship design and construction, because the complexity of modern naval ships demands more sophisticated methods. The advanced shipyard design tools have potential uses throughout all stages of design. Some recent acquisition programs have applied inte- grated product and process development, an approach to ship design and construction in which the early design stages are integrated with construction planning to improve the efficiency with which performance and cost objectives are met. However, broader use of shipyard design tools and databases in this manner may be hindered because there has been little transition of the technology developed by private-sector ship- yards to Navy ship designers, many advances are regarded as proprietary, and the level of detail in associated databases is often not compatible with the early-stage analysis of alternatives and set-based design for new concepts. The NNR-NE portfolio does not include investments in detail design tools because development of these tools is not considered to be basic research. In general, research in ship design tools tends to be focused on the transition of basic research knowledge gained in multiple disciplines to design applications; hence, it is often perceived as applied research and may receive low priority in programs oriented toward basic research. Nonetheless, there are basic research opportunities associated with generic technologies such as systems engineering, multidisciplinary opti- mization, set-based design, efficiency and accuracy of solvers, physics-based modeling, and multiphysics coupling techniques. These opportunities are particularly relevant for advanced ship concepts where there is often a lack of existing rules-based methods and experimental data and existing tools 

Conclusions and Recommendations 181 have not been verified, validated, or accredited for use. Because basic re- search on ship design tools has a limited range of potential applications and receives meager support from government and private-sector sources, few researchers in the United States are predisposed to perform such research even if increased funding in the field were available from ONR. Ship design tools research is actively pursued in the commercial sector in Asia (where commercial design and shipbuilding are thriving competi- tive industries) and in Europe. The focus in these markets is on large product carriers, containerships, passenger ships, and offshore vessels and platforms, and therefore the research has limited applicability and little opportunity for transition to naval combatant ship design. However, the international design industry has produced tools with potential applica- tion to early-stage naval ship design. Automation, Control, and System Integration Conclusion 3d: Research in automation and control receives signifi- cant support from NSF and from DOD. Both agencies support basic research, and DOD is the major supporter of applied research. NASA has supported work in this area in the past. Basic and applied research in automation and control outside ONR appears to be strong in terms of funding and numbers of researchers. In general, controls, embedded systems, and automation are relatively well-funded topics in engineering research today. This activity includes research rele- vant to naval systems. The evident ONR niche in the field is application to very specific Navy requirements (e.g., robotic underwater vehicles). Sys- tem integration has fewer researchers but is funded by government agen- cies in addition to the Navy. As with automation and control, the principal Navy-specific problems appear to be application to special needs. Platform Power and Energy Conclusion 3e: Ensuring efficient transition of new power and energy technology to the designers and builders of Navy ships is an urgent concern. 

182 Naval Engineering in the 21st Century Power and energy technology is a dynamic field driven by developments in computing, telecommunications, and power electronics for industrial, consumer, and grid applications. Research and development in power sys- tems is conducted and funded by industry, the Department of Energy, NSF, and DOD. DOD, and in particular the Navy, has been among the leaders in the funding of research to support design of power systems of up to 100-MW capacity matching Navy needs. Research on land-based systems can be expected to make a major contribution to components and subsystem technologies that meet the Navy’s special power system requirements. The Navy seeks to develop power and energy systems for ships that will be equipped with electric drives and with electrically pow- ered weapons, high-power radars, and electrical components replacing hydraulic systems. Because future shipboard systems will be of small phys- ical dimensions and have power demands far exceeding the available on- board generation, the problems and possibilities for ship-based power system control and energy storage differ significantly from those for land-based systems. Each weapon and radar system will not be able to bring its own power system on board, and the future ship power system will be different from that of the past. Monitoring the Health of the S&T Fields That Support Naval Engineering Recommendation 1: To fulfill its obligation under the NNR-NE to sustain U.S. research capability to work on problems important to the Navy, ONR should carry out regular systematic assessments of the state of health of each of the research fields supporting naval engineering in the United States. ONR assessments should examine the objectives and progress of related research supported by ONR, other DOD agencies, other federal govern- ment agencies, and the private sector. The examination should include research in each field that is supported by non-Navy sources and moti- vated by the potential for applications outside naval engineering but that may constitute a pool of expertise and facilities that could be brought to bear quickly on problems important to the Navy, if support were avail- able and researchers were induced to work on naval engineering problems. 

Conclusions and Recommendations 183 Judging the relevance of current work in each field to the Navy mission should be part of the assessments. The most meaningful measure of the health of research in the fields sup- porting naval engineering is evidence of technology-driven improvement in ship performance and cost. Therefore, as part of its monitoring of the health of these research fields, ONR should evaluate U.S. and worldwide innovation in naval engineering practice. ONR’s international connections through the participation of foreign institutions in sponsored research in the NNR-NE will be valuable as a source of information about worldwide developments. The evaluation should look beyond ONR’s own programs (a) to ask whether progress in performance is being made (according to rec- ognized measures such as match to threats, cost, and survivability) and (b) to determine the sources of the technologies that allowed progress. State of Education Training of Future Researchers Conclusion 4: The U.S. graduate education establishment that con- ducts research and trains future researchers in the NNR-NE fields draws strength from the diversity of the S&T disciplines engaged. However, research centers and departments concentrating on certain specialized fields critical to naval engineering and deriving a large share of their research support from ONR (including centers and departments that perform research in hydrodynamics and in ship design methods) may be vulnerable. A decline in research support for these fields would cause these departments to diminish or disappear. Critical research capabilities would be lost, and the supply of future researchers would be interrupted. Professional Education Conclusion 5: The Navy recognizes that providing for the develop- ment of future naval engineers is essential to its mission. ONR has ini- tiated outreach programs to attract new students to the naval engineering disciplines, but they are limited in scope and resources given the needs. The recently funded Naval Engineering Education Consortium may prove to be of value in recruiting and developing students in naval systems engineering. 

184 Naval Engineering in the 21st Century The supply of engineering graduates for professional and research careers in naval engineering is constrained by citizenship requirements. Persons who are not U.S. citizens have difficulty in obtaining permanent residency even with advanced degrees and training in needed disciplines. Current initiatives are useful to the Navy in attracting new students into naval engineering and maintaining an adequate pipeline of capable researchers for the future, but the resources devoted to those initiatives are limited, and vigilance will be necessary to ensure that Navy needs continue to be met. Conclusion 6: Few graduate professional engineering programs in the United States provide multidisciplinary education focused on naval engineering problems. While more than 20 colleges or universities offer programs having some link to maritime, naval, or ocean engineering at the undergraduate level, only 12 offer a naval engineering course of study at the graduate level. Graduate programs with strong multidisciplinary components are the most promising setting for development of the knowledge and skills required to carry out the essential integrative function of naval engineering. Recommendation 2: ONR should make a special effort to encourage multidisciplinary graduate programs focused on naval engineering that train future researchers and professionals. State of Infrastructure The committee understood the reference to infrastructure in its task state- ment to mean institutional as well as physical infrastructure. Institutional infrastructure was defined as the established institutional framework of research in naval engineering—schools; university, government, and industry research laboratories; grant-making organizations such as ONR and NSF; and scientific and professional societies. Physical infrastructure was defined as the structures and equipment required to carry out research in the naval engineering–related fields, for example, towing tanks, wave basins, and cavitation flow tunnels for experimental hydrodynamics and structural test facilities. 

Conclusions and Recommendations 185 Institutional Infrastructure The committee solicited views on the state of the scientific and technical institutions critical to naval engineering from researchers; educators; the Naval Sea Systems Command (NAVSEA); and U.S. and foreign commer- cial ship designers, builders, and operators. The four participants in naval engineering S&T are the Navy, the private-sector shipbuilding and ship design industries, commercial ship operators, and the research universi- ties and other independent research organizations. The responses that the committee received from this community indicate the following: • Commercial shipbuilding is focused on efficiency and cost (one con- cern of the Navy). U.S. industry investment in offshore technology is strong, and industry demand for marine professionals is vital in sup- porting the maritime-related university infrastructure and human cap- ital pipeline. • The activity of the international research community in major ship- building countries such as Japan, Korea, and Norway (a center of the offshore industry) is isolated from U.S. interests and efforts. • U.S. universities are a rich source of expertise that potentially is applic- able to Navy problems but is not fully utilized by the Navy now. For example, there is little overlap between the list of researchers receiving support in NSF’s fluid mechanics program (NSF 2010) and NNR-NE hydromechanics research grant recipients, and ONR management expressed concern that ONR grant opportunities have not always been publicized as effectively as possible. • The U.S. government is overwhelmingly the major supporter of rele- vant research; within the U.S. government, the Navy is the largest sup- porter, and within the Navy, ONR. The ability of the naval laboratories and government R&D facilities to sup- port the naval ship systems engineering S&T infrastructure is varied. The results of the committee’s assessment indicate the following: • NAVSEA’s NSWC-CD is the primary facility conducting research and development for transitioning NNR-NE research results to naval applications. 

186 Naval Engineering in the 21st Century • NSWC-CD has been effective in supporting advanced degrees in naval engineering; in recruiting naval engineers; and in promoting science, technology, engineering, and mathematics (STEM) education. • NAVSEA’s Naval Undersea Weapons Center has relevant but limited activity in unmanned vehicles and in system integration (focused on energy sources). • The Naval Research Laboratory’s diverse mission does not emphasize investments in the NNR-NE fields. • Although NSF sponsors basic research in related areas (including fluid dynamics, structural materials, energy and power, and systems engi- neering), the projects are heterogeneous and rarely address problems critical to naval engineering progress. Similarly, the Defense Advanced Research Projects Agency (DARPA) and other DOD agencies support relevant research, but rarely with potential naval applications or specific Navy needs in mind. Physical Infrastructure Conclusion 7: The committee collected limited data on physical research infrastructure from ONR, Navy laboratories, and naval engi- neering researchers. No obvious shortfalls were identified. The exist- ing infrastructure at government, university, and private research laboratories appears to have been adequate for the needs of current Navy research programs. Improvements in computer technology and equipment have benefited research in areas such as real-time physics- based simulations. DOD’s Defense University Research Instrumentation Program, which awards competitive grants for acquisition of major equipment, has helped maintain naval engineering research facilities. Conclusion 8: Because the content of ONR’s research portfolio is strongly influenced by researcher proposals, infrastructure needs for the portfolio tend to be determined by availability. Maintaining and fund- ing test facilities are challenges. Facilities rely heavily on fees collected from users conducting government-sponsored research; therefore, if research funding is interrupted, survival of facilities is jeopardized. 

Conclusions and Recommendations 187 The committee calls the attention of ONR to the 2000 report of the NRC Committee for Naval Hydromechanics Science and Technology, which noted that these facilities require ongoing investment to update instru- mentation and strong technical support staffs to produce cutting-edge research (NRC 2000, 3). WHOLENESS OF THE NNR-NE PORTFOLIO The task statement directs the committee to “assess the wholeness of the program and, as appropriate, identify any key opportunities for the Navy to make fundamental leaps in sea platform capability and affordability.” The study also is to assess whether the technical areas included within ONR’s definition of NNR-NE adequately define its scope. The first section below presents the committee’s overall conclusions concerning the wholeness of the NNR-NE research portfolio and the defi- nition of the scientific and technical areas included within it. The second presents conclusions about the portfolio of research projects in each of the technical fields that ONR includes within its definition of NNR-NE. Overall Portfolio Relationship of the Portfolio to Needs and Objectives Conclusion 9: The wholeness of the NNR-NE portfolio can be judged only by comparing its objectives and accomplishments with the Navy’s priorities for innovation in naval engineering. Priorities should be determined through regular communication with ship designers, fleet strategic planners, and researchers in the fields allied with naval engi- neering and should be specified in a plan. Definition of the focus and expected value of NNR-NE basic research would be elements of such a plan. Planning is necessary in managing an applied research program and is not inconsistent with the spirit of basic research. The commit- tee is not aware of a plan for guiding basic and early applied research in the naval engineering–related fields that is specific enough to fulfill this need. The committee concurs with the findings of the 2005 NRC Committee on Department of Defense Basic Research that basic research in DOD should 

188 Naval Engineering in the 21st Century include studies aimed at “discovery arising from unfettered exploration” as well as “focused research in response to identified DOD technology needs” and that unless a clear understanding of the expected value of a basic research program is communicated to DOD leadership, long-term support of the research will be unlikely (NRC 2005, 3–4). In the case of NNR-NE, the importance of focused research within the basic research portfolio may be especially great. The committee concurs also with the 2005 NRC committee’s conclu- sion that “DOD should view basic research, applied research, and devel- opment as continuing activities occurring in parallel, with numerous supporting connections throughout the process” (NRC 2005, 2). There- fore, to ensure the wholeness of the NNR-NE portfolio, ONR will need to identify the implications of DOD technology needs for basic and applied research priorities. A clear correlation between needs and research empha- sis was not always evident to the committee in its examination of the NNR-NE portfolio and its review of the research needs implications of the Navy’s operational challenges. Definition of the Technical Areas Within NNR-NE In presentations and in project summaries provided to the committee, ONR management defined the scope of research within the NNR-NE to include basic and early applied research in six scientific and technical areas that support naval engineering: ship design tools; structural systems; hydromechanics and hull design; propulsors; automation, control, and system integration; and platform power and energy. The task statement indicates that “the study will assess whether these seven disciplines ade- quately define the scope of NNR-NE.” (The task statement lists hydro- mechanics and hull design as two distinct fields, refers to ship propulsion rather than propulsors, and does not refer to platform power and energy.) To respond to this part of its charge, the committee considered the prac- tical implication of including each field within the scope of NNR-NE; that is, that fields within an NNR are to receive active Navy stewardship because they are essential to the Navy and unlikely to receive support elsewhere. The committee’s conclusions concerning the definition are as follows: • Advances in all six of the areas could contribute to innovation in ship design. 

Conclusions and Recommendations 189 • The committee does not see evidence that any of the six fields is “mature” in the sense that the field is unlikely to produce future advances that would contribute to ship design and performance. • Each of the fields, if broadly defined, receives support from sources other than the Navy and has application beyond naval engineering, but the need to maintain scientific expertise in the special problems of unique importance in naval engineering justifies including each of the fields within an NNR. • Power and energy provision will be a critical design problem for future naval ships; therefore, this field should remain a part of the NNR-NE. The committee understands that ONR basic and early applied research in power and energy may not be managed in the ONR division that houses most of the NNR-NE fields. The definition of NNR-NE should not be dictated by organizational arrangements within ONR. • The major gap in the present definition is inadequate acknowledgment of the need for basic and early applied research to support the integra- tive function that is central to the practice of naval engineering. The present portfolio in automation, control, and system integration does not appear to fulfill this need, and ONR needs a new vision to guide research in these areas. Realizing the ultimate potential value (in terms of contribution to the Navy mission) of a research breakthrough in any one of the six fields in the present NNR-NE definition usually depends on advances in other fields. ONR basic and early applied research should provide an understanding of the relationships among the fields. Without an integrated multidisciplinary approach, there are likely to be omissions in basic and early applied research and incorrect projections of the course and speed of technology development; that is, capabilities will not be available when needed. Recommendation 3: ONR should retain the six fields of ship design tools; structural systems; hydromechanics and hull design; propul- sors; automation, control, and system integration; and platform power and energy in the definition of the areas of basic and applied research within NNR-NE. The definition should state that all ONR basic and early applied research in these fields is to be coordinated to meet the goals of the NNR-NE. In particular, basic and early applied research in platform power and energy should be retained in the 

190 Naval Engineering in the 21st Century definition regardless of where this activity is housed in ONR. In addition, the definition should explicitly identify multidisciplinary systems engineering as an area of basic and early applied research within NNR-NE. Recommendation 4: The Navy should dedicate an important share of its resources for naval engineering S&T to problems whose solutions are expected to have broad applicability to a range of possible future ship programs. The latter recommendation is consistent with the ONR Discovery and Invention Portfolio’s objective of providing the Navy with technology options (ONR 2009, 26) and with the long-term perspective that the NNRs are intended to take. It also is consistent with the recommendation of the 2005 NRC Committee on Department of Defense Basic Research that DOD should define basic research not as research that is designed with no specific applications relating to the Navy mission but rather as research with the potential for broad rather than specific application (NRC 2005, 1). Research Portfolio in Each NNR-NE Technical Field The committee reviewed the topics of ONR-funded projects in 2006–2009 in each of the NNR-NE fields, received presentations from ONR program officers on objectives and accomplishments in each field, and received pre- sentations from ONR-sponsored researchers. The committee did not review the content or quality of the products of individual research projects. The committee’s assessment of the portfolio was not definitive because of limitations on study resources and on the information that ONR was able to provide the committee on content and output metrics. For the same reason, recommendations concerning research priorities are offered below only in selected areas where the need appeared clear. The committee considered three aspects of the research projects in each field: intellectual quality, mission alignment, and management commit- ment and resource adequacy. On the basis of this review, the committee reached the following conclusions with regard to the six research fields: Conclusion 10a: The research portfolios in some of the fields (includ- ing power and energy and structural systems) appear to be of high 

Conclusions and Recommendations 191 intellectual quality, organized around well-defined objectives, demon- strating progress, aligned with mission needs and potential applica- tions, and adequately supported. Conclusion 10b: For other fields (including automation, control, and system integration and ship design tools), the objectives are not evident and the project portfolios appear to lack cohesion or to be too nar- rowly focused. The underlying sources of problems in the less strong portfolios may be traceable to the extent and quality of input from users and the research community in the articulation of research needs and in user evaluations of the research products. Also, in some fields (including ship design tools), relevant research at ONR that is outside the administrative definition of the NNR-NE, and regarding which the committee did not receive information, may address some apparent gaps in the NNR-NE portfolio. If ONR were to explicitly include systems engineering as a scientific area within the NNR-NE research portfolio, as Recommendation 3 calls for, it is likely that the areas of automation, control, and system integra- tion and ship design tools would gain in focus, that the potential for impact of research in these areas would become more evident, and that they would be more successful in attracting resources. The present NNR- NE definition contributes to the focus on the precise technical challenges in these areas. Under a systems engineering umbrella, it would be possi- ble to integrate all the systems modeling activities in the portfolio. Conclusion 10c: The pattern of funding large numbers of small research projects that is evident in the portfolios of several of the NNR- NE fields suggests a concern that the portfolios in these fields may not be well coordinated toward achievement of a small number of sharply defined goals. A tendency to spread available resources thinly but widely would run counter to the intent of the NNR initiative to ensure that limited resources are concentrated sufficiently to produce results in the most critical fields (Gaffney et al. 1999, 15). Awards of smaller grants allow ONR to support a larger number of aca- demic departments that provide naval engineering training. The conclusions of the committee concerning the research portfolio in each of the NNR-NE fields are presented in the subsections below. The 

192 Naval Engineering in the 21st Century final section of this chapter presents a recommendation for ONR conduct of ongoing and systematic evaluations of the portfolios. Hydromechanics and Hull Design; Propulsors Conclusion 11: The two principal themes in the portfolio of recent ONR basic and early applied research in hydromechanics and propulsors are (a) simulation-based analysis and design capabilities to augment or replace traditional physical test–based approaches and (b) targeted research to address high-priority areas in reduction of acoustic and radar signatures, nonacoustic detection, prediction of extreme motions (particularly roll and maneuvering), and prediction of wave impact loads. A major share of ship-related research is for large-scale computational fluid dynamics. The portfolio also includes a significant commitment to the conduct of prototype tests by complementary efforts at NSWC-CD. The commitment to testing seems healthy and indicates that ONR recog- nizes that progress requires a balance between experimental and compu- tational work. Propulsor modeling has a much higher profile than a decade ago, with emphasis on crash-back maneuvers. Several investigators are taking diverse approaches to this problem. Structural Systems The following are the evident objectives of the NNR-NE structures portfolio: • Developing technologies for life-cycle performance analysis and the monitoring of ship structural systems; • Understanding the behavior of novel ship structures, such as compos- ite and aluminum subsystems, during and after fire to enable modeling and predictions; • Providing a protection system or armor that can defeat several threats and meet structural and stiffness requirements; and • Facilitating use of alternative hull forms that are lighter, more surviv- able, stealthier, cheaper, easier to maintain, and longer-lived than steel or aluminum hulls. 

Conclusions and Recommendations 193 Conclusion 12: Structural systems research places strong emphasis on structural survivability from fire and explosions and on materials other than steel, such as composite and aluminum structures. Conclusion 12a: In research areas within the portfolio such as fire resis- tance of composites, blast-resistant polyurea coatings, and fully coupled fluid–structure interaction simulations, there are breakthrough oppor- tunities. The work on isogeometric analysis could lead to a breakthrough in structural and fluid–structure interaction analysis. Conclusion 12b: In the structural systems portfolio, basic research top- ics are awarded to academic institutions, and applied research topics are awarded to applied research laboratories such as NSWC-CD, the Naval Research Laboratory, and industry research organizations. The ratio of basic to applied structural systems projects is approximately 2:1; how- ever, ONR’s Future Naval Capabilities program also conducts applied structures research. Although budgets are limited, there appears to a balance between new and continuing projects. Conclusion 12c: Certain structures topics important for naval engi- neering are not in the portfolio, including coatings and fatigue life extension. These are topics of basic and early applied research in other ONR divisions not included within the NNR-NE definition. Other top- ics that are relevant to naval engineering but administered outside the division responsible for NNR-NE include bearings and lubrication. Conclusion 13: Navy plans call for building fewer new classes of ships in the future and sustaining the fleet through production of ships according to modified versions of existing designs. Existing ships will continue in service longer and be subject to modernizations to extend service life. These decisions have implications for the relative impor- tance of research on structures, design tools, and other technical areas within NNR-NE. The committee could not identify research programs in the NNR-NE portfolio that addressed this future need. Ship Design Tools Basic and early applied research on problems in total ship systems design is largely missing from the portfolio. The committee is aware that ONR 

194 Naval Engineering in the 21st Century conducts later-stage applied research (outside the scope of NNR-NE) on these problems and conducts some basic and early applied research on these problems outside the ONR Ship Systems and Engineering Research Division (which administers the NNR-NE). Recommendation 5: ONR should view total ship systems design as a legitimate topic of basic and early applied research, and all such research at ONR should be coordinated through the NNR-NE framework. The focus and objectives of the design tools portfolio were not evident to the committee. The portfolio includes topics ranging from genuinely basic research to highly applied topics. Moreover, the objectives of the portfolio, as stated in the ONR presentations to the committee, are beyond the resources available and beyond the scope of basic and early applied research. To manage the portfolio, a definition of the relationship of basic research to design tools—that is, the gaps in scientific knowledge that hinder ship design—is required. ONR did not present data to the committee on transitions achieved from NNR-NE project results to later-stage applied research or applica- tion; however, the committee is concerned that the analysis and compu- tation methods developed in basic and early applied research may not be finding their way into the software packages used in design practice. Conclusion 14: The effectiveness of ONR basic and early applied research in design tools is hindered by the lack of an adequate Navy- wide plan for research and development of design tools. Such a plan would set goals; assign responsibilities among ONR, NAVSEA, and others; and provide for coordination. Recent workshops sponsored by ONR, NAVSEA, and the DOD Com- putational Research and Engineering Acquisition Tools and Environments program and organized by the Navy’s Center for Innovation in Ship Design (CISD) have taken a step toward defining the Navy ship design process and associated tools, providing objectives for improvement, and identifying actions and research necessary to achieve these objectives. However, a for- mal connection between the results of the workshops and ONR design tools portfolio management has not been established. CISD offers an excellent environment for bringing together the people who can provide focus and objectives for design tools and should be leveraged for this purpose. 

Conclusions and Recommendations 195 Automation, Control, and System Integration Conclusion 15: The portfolio in automation, control, and system inte- gration should be growing and dynamic because the increasing com- plexity of ships is a key technical problem confronting naval engineering. However, the focus and objectives of the automation, control, and sys- tem integration portion of the portfolio were not evident to the commit- tee. The portfolio includes some highly applied projects aiming for narrow objectives, but basic research of broad potential applicability to system integration, system engineering, and system architecture appears to be absent. Assessing the NNR-NE automation, control, and system integration port- folio was difficult for the committee because it did not receive information with regard to work on these topics that is being performed in other units of ONR. Platform Power and Energy Conclusion 16: Platform power and energy was not identified in the 2001 ONR memorandum creating the NNR-NE, yet in 2006 through 2009 (the years for which research spending data were provided to the committee) research funding in this area was the largest compo- nent of the NNR-NE portfolio, with most funding for applied research projects. The research is aimed at supporting development of compo- nents and systems for providing shipboard power of very high capacity compared with historical requirements. The committee understands that the portfolio has now been relocated within ONR. Recommendation 6: To ensure continuity of component and subsys- tem technology, the Navy should pursue research and development for power and energy systems in partnership with U.S. industry. It is equally important to pay due attention to integration of the power system with the total ship system and to transition of the technology rapidly and effectively to the ship planners. The transition process should be initiated in the early conceptual design stages. Use of power electronics–based integrated power systems (IPS) to manage power and energy needs and efficiency offers great potential for enhancing 

196 Naval Engineering in the 21st Century the performance of future Navy ships. ONR has correctly defined and pur- sued a research and development plan for this technology. However, gaps in Navy planning threaten to hinder transition of the technology beyond ONR research and development. The definition of power electronics–based IPS and the design of its com- ponents, including converters, generators, energy storage systems, and design tools for more conventional ship designs and weapon system power loads, are adequately emphasized. However, there is inadequate research and development on the dynamics of future systems, where weapon load requirements may far exceed the capacity of available generation and there- fore large energy storage systems will be essential. The integration of power electronics–based IPS into overall ship design is also not adequately empha- sized. Attention to this problem is essential if future ships are to accom- modate radar and weapon systems that the Navy may wish to use. OPPORTUNITIES TO ENHANCE RESEARCH AND EDUCATION Enhancing Research Conclusion 17: Opportunities exist for offering significantly improved capabilities to the fleet through basic and applied research in the scien- tific and technical fields supporting naval engineering. Conclusion 17a: Basic research is needed on the problem of integrat- ing ship systems, and research on components will stay on a productive course only if it is tightly linked to long-term programs of research and development of total ship systems. This need is especially apparent in the areas of power and energy systems and ship design tools. Conclusion 17b: The future of naval engineering likely lies in incorpo- rating advances from younger and rapidly advancing disciplines. If it is to maintain its relevance, the NNR-NE research portfolio must reflect this trend. Recommendation 7: In planning the NNR-NE research portfolio, ONR should search for research directions and research topics by identifying both (a) emerging scientific and technological developments that hold promise for providing new capabilities or new technology options and (b) gaps in fundamental scientific and technical knowledge that are hin- 

Conclusions and Recommendations 197 dering fulfillment of needs identified by the operating Navy. The search by ONR for research direction and topics should be systematized, ade- quately funded, measured, and incentivized and should be included as part of the organization’s and its managers’ performance evaluation processes. ONR could produce a valuable list of research opportunities through regular and systematic external consultations with practicing naval engineers, the operating Navy, researchers, and other technical experts, and by documenting and publishing the research topic propos- als generated by these consultations. Research directions emanating from emerging S&T developments are often referred to as technology push, while those emanating from gaps in fundamental S&T knowledge are referred to as requirements pull. Through its workshops and commissioned papers, the committee received suggestions from researchers on opportunities presented by tech- nology and from practicing naval engineers and naval analysts on technol- ogy demands arising from Navy requirements. Future Navy requirements for S&T products will be dictated by three driving forces: the operating environment the Navy may face, the types of operations it may be expected to perform, and the Navy’s resource prospects. Examples of research needs arising from these forces are the following: • Needs arising from the future operating environment: – New technologies to reduce ship signatures and more capable radar, as means of defense against future threats – Ice-strengthened structural design and cold weather operation of ships, to prepare for an expanded Navy mission in the Arctic • Needs arising from future naval operations (e.g., to support opera- tions such as counterterrorism and irregular warfare): – More versatile platforms for inshore and special operations – Unmanned vehicles and the ships to carry and support them – Integration of complex systems into ship designs with minimum increase in complexity • Needs arising from resource prospects: – Naval engineering aspects of shipbuilding and ship construction engineering aimed at reducing procurement costs – Reduction of life-cycle costs, for example through development of life-cycle cost–benefit models, durable structures, open architecture 

198 Naval Engineering in the 21st Century features to facilitate modernization, and self-repairing and self- diagnosing systems The role of NNR-NE is to identify and fill gaps in fundamental scientific and technical knowledge that are hindering fulfillment of these needs. The following are examples of emerging developments that hold promise for providing new capabilities or new technology options for the Navy: • Physics-based modeling and simulation and computational mechanics. Advances in this area together with the advances being made in super- computers and parallel processing will greatly assist ship design. For example, development of virtual prototype designs carried out in a real- time simulation framework will allow trade-off studies to be performed quickly and efficiently; • Virtual design, testing, and evaluation capabilities for platforms systems and subsystems; • Application of power electronics–based IPS for managing shipboard power and energy needs; and • Systems engineering tools capitalizing on advances in fields such as human factors, biomechanics, and biomimicry, which may be applic- able to ship design and production problems. The preceding list illustrates needs and opportunities, but because of the limited scope of the present study, it is not systematic or comprehensive. ONR could produce a more valuable list of opportunities by systematically exploiting the same resources that the committee relied on, that is, by con- sulting with practicing naval engineers, the operating Navy, researchers, and other technical experts. Enhancing Education Conclusion 18: Outreach programs have been successful in interest- ing students in STEM education and maritime careers. ONR supports the Society of Naval Architects and Marine Engineers (SNAME) in expanding the SeaPerch program (an ocean science laboratory project for middle school students). SNAME and the American Society of Naval Engineers are well positioned to provide leadership and support for these outreach initiatives. However, their efforts are limited by the availability of volunteers and funding. 

Conclusions and Recommendations 199 Recommendation 8: ONR should embrace its role as STEM lead for the Navy and adequately fund and manage STEM activities as part of its S&T portfolio. As part of its STEM activities, ONR, in cooperation with NAVSEA, the professional societies, and industry, should consider the following activities: • Targeting populations in regions with community connections to naval engineering (e.g., local naval architecture universities, ship- builders, naval facilities), • Expanding funding and aiding professional societies and indus- try in volunteer support for collaborative outreach programs (e.g., the Junior Engineering Technical Society), and • Using NAVSEA funding of the Naval Engineering Education Cen- ter Consortium to support SeaPerch and other initiatives. NNR-NE EFFECTIVENESS The committee’s task statement provides the following direction: “The study . . . will assess the NNR-NE’s progress in the ability to: (l) provide and sustain robust research expertise in the United States working on long- term problems of importance to the Department of the Navy; (2) ensure that an adequate pipeline of new researchers, engineers, and faculty con- tinues; and (3) ensure that ONR can continue to provide superior S&T in naval architecture and marine engineering.” The committee’s evaluation was not definitive because of the limits of its own resources and because ONR does not have the information system that such an evaluation would require. Conclusions on NNR-NE effectiveness and recommendations on how ONR could establish a management process that would allow system- atic evaluation are presented below. Systematic evaluation would reveal opportunities to increase effectiveness. Overall Effectiveness The conclusions in this subsection concern the effectiveness of the NNR- NE and factors that influence its effectiveness. In outline, the committee concluded the following: • NNR-NE meets a Navy need but requires planning and stronger links to users and researchers. 

200 Naval Engineering in the 21st Century • NNR-NE is not yet recognized within or outside ONR as the focus of naval engineering basic and early applied research. • ONR does not appear to have conducted the reporting that the 2001 memorandum establishing the NNR-NE calls for. • The role of NNR-NE in the Naval S&T Strategic Plan has not been clearly defined. • ONR has not defined the practical significance of NNR designation for administration and budgeting. • Some activities called for in the 2001 memorandum or the 2010 instruc- tion have not been undertaken. • The scope of NNR-NE functions and responsibilities with respect to education and relevant research outside the Ship Systems and Engi- neering Research Division lacks clear definition. Conclusion 19: NNR-NE meets a Navy need but requires planning and stronger links to users and researchers. The research and educational activities within NNR-NE have been effec- tive in fulfilling the Navy’s basic need to sustain S&T in naval engineering– related fields. A diverse research program is supported, and significant numbers of graduate and postdoctoral students are involved. An outreach program is making efforts to attract students into the field of naval engi- neering at the kindergarten through 12th grade, undergraduate, and grad- uate levels. The physical infrastructure of laboratories and equipment, which receives important support through ONR research grants, appears to be adequate for current needs. However, the NNR-NE initiative has yet to reach its potential. In par- ticular, the vision of the 2001 NNR-NE memorandum—systematic and coordinated management of a research portfolio toward attainment of clearly defined objectives—has not been fulfilled. ONR has continued to support important basic and applied research in the designated techni- cal fields, as it did before 2001, but the NNR-NE initiative has not had visibility internally or externally, and the coordination and evaluation steps called for in the memorandum have not been conducted consis- tently. Reinvigorating the initiative by returning more closely to the letter and spirit of the 2001 memorandum would enable ONR to achieve the purposes of the initiative more reliably and efficiently. Effectiveness 

Conclusions and Recommendations 201 would be increased if ONR developed a more rigorous procedure for defining meaningful objectives for research in each of the fields within NNR-NE and measuring progress toward them and if ONR reinforced communications channels between NNR-NE managers and the broad user and research communities. Conclusion 20: NNR-NE has not yet gained recognition within or outside ONR as the focus of naval engineering research. ONR created NNR-NE as a mechanism to focus its basic and applied research and education activities in support of naval engineering and to emphasize the importance of technical progress in naval engineering to Navy missions. However, NNR-NE has never attained the intended status. The community of researchers and ONR program managers does not jus- tify or evaluate its efforts in terms of their place in the NNR-NE framework or contributions to meeting NNR-NE objectives. Marketing—outreach to the research community to help attract the best talent and ideas and outreach to sponsors and other stakeholders to ensure that the initiative remains relevant to their needs and maintains their support—is a neces- sary adjunct to the NNR-NE initiative. Conclusion 21: ONR does not appear to have conducted the reporting required by the 2001 memorandum establishing the NNR-NE. Essential to the NNR concept is that a collection of ONR activities is to be managed in a coordinated manner to reach a common objective. The 2010 NNR instruction requires that the responsible department report annually on the execution and progress of the NNR. Regular progress reporting is a necessary step toward ensuring that the elements of NNR-NE are man- aged as a unified initiative and recognized as the focal point of basic and early applied research in naval engineering. The 2001 memorandum estab- lishing the NNR-NE requires that the progress and impact of the NNR-NE be subjected to an external review every 5 years. The committee did not receive documentation of past progress reports or evaluations of the NNR-NE. Conclusion 22: The role of NNR-NE in the Naval S&T Strategic Plan has not been clearly defined. 

202 Naval Engineering in the 21st Century ONR’s 2009 Naval S&T Strategic Plan refers only briefly and generally to the NNRs. The plan states objectives for naval engineering research in such broad terms (e.g., platform survivability, stealth, efficient energy and power systems, “new and novel advanced platform design,” reduced total ownership cost of naval platforms) that the document appears to be of lim- ited use to research managers in setting priorities and balancing their pro- grams. Correspondingly, ONR has not taken the initiative to relate its NNR-NE portfolio to the Naval S&T Strategic Plan or to communicate the importance of efforts carried out under the NNR to the strategy. This is an essential step in ensuring internal understanding of the critical nature of the NNR-NE and the merits of adequately resourcing the NNR-NE initiative. Conclusion 23: ONR has not defined the practical administrative sig- nificance of NNR designation. The 2001 memorandum establishing the NNR-NE (ONR 2001) and the 2010 instruction defining the NNRs do not identify the practical conse- quences of NNR designation, that is, how designation of a portfolio of ONR activities as an NNR is to alter the management or objectives of the activities. ONR was already engaged in all or nearly all of the activities that the 2001 memorandum designated as elements of the NNR-NE when the memorandum was issued. The committee’s understanding is that, rather than initiating new programs, the memorandum served as a declaration of policy: assigning the NNR designation indicated that (a) the listed activi- ties deserve special priority in planning and budgeting at ONR because the identified S&T fields are critical to the Navy and no one else will support them and (b) management of these activities must be coordinated with the declared policy objective in mind. However, this significance of NNR des- ignation is not explicit in the ONR memorandum or instruction. Actions that ONR could incorporate in the NNR-NE initiative to pro- mote and strengthen naval engineering–related research (and which may not have been required in the absence of the NNR designation) could include periodic evaluations of research output, periodic examinations of the health of the field and of the performance of all Navy programs sup- porting the field, procedures for giving special priority to the NNR-NE fields in ONR program planning and budgeting, and management 

Conclusions and Recommendations 203 arrangements to ensure coordination of all relevant ONR activities toward achieving NNR-NE objectives. Conclusion 24: Some NNR-NE prescribed activities may not have been undertaken. The 2010 NNR instruction requires coordination of the NNRs with ONR’s Future Naval Capabilities technology transition initiatives and with DARPA. The 2001 NNR-NE memorandum requires ONR to create university–industry–laboratory consortia for fostering naval engineering S&T. The committee was not presented with information on how these requirements have been interpreted and carried out. ONR does not appear to have conducted large-scale field experiments within the NNR-NE research project portfolio, with the possible exception of certain power and energy applied research projects, or to have issued special broad agency announcements to fulfill specific objectives of the NNR-NE, as the 2001 memorandum calls for. Conclusion 25: The scope of NNR-NE functions and responsibilities lacks clear definition. The 2001 NNR-NE memorandum and the 2010 NNR instruction are imprecise with regard to how naval engineering–related basic and applied research conducted by units other than the ONR Ship Systems and Engi- neering Research Division should be coordinated with the NNR-NE and on the scope of educational activities considered to be within the NNR-NE. Conclusion 26: The committee’s assessments of the significance of the research were complicated by the lack of a full picture of ONR work related to naval engineering. Particularly in the fields of ship design tools; structures; and automation, control, and system integration, the committee understands that some relevant basic and early applied research is being conducted in ONR divisions other than Ship Systems and Engineering Research. Coordination of all relevant ONR research toward the objectives of the NNR-NE appears to be missing in the structure of the initiative. A clear definition of the scope of the educational activities that are to be considered part of the NNR-NE is lacking. The statement of task for this 

204 Naval Engineering in the 21st Century study (provided by the sponsor and accepted by NRC) identifies edu- cation of future researchers as an objective of the NNR-NE. The 2001 NNR-NE memorandum and the 2010 NNR instruction also recognize ensuring the supply of researchers as part of the NNRs. However, some provisions of the 2001 memorandum and some descriptions of the NNR-NE initiative that ONR presented to the committee indicate that the scope of NNE-NE may encompass a broader range of educational aims, including STEM education and training of professional naval engineers. ONR has been assigned primary responsibility for the Navy’s contribu- tion to the nationwide STEM initiative. It was recommended above that ONR embrace this responsibility and consider expanding some STEM activities. However, the practical significance of managing STEM as an ele- ment of the NNRs is not evident. The reports and documents listed in the 2001 ONR memorandum in support of the need for the NNR-NE frequently cite concern for the future adequacy of the workforce of practicing naval engineers. Ensuring an adequate professional engineering workforce is a primary interest of NAVSEA, because that command, directly and through its contractors, employs most engineers in the field. However, ONR research grants in naval engineering have an important indirect role in providing the profes- sional workforce. Faculty research funding is essential to the survival of naval engineering professional programs because research ensures the intellectual vibrancy of university academic programs. ONR research investments should be directed according to the value to the Navy of the scientific knowledge they produce, but the connection between research support and professional workforce supply cannot be overlooked. Recommendation 9: ONR should bring NNR-NE in line with the structure of the initiative as envisioned when it was established by tak- ing the following actions: • Recommendation 9a: ONR management should ensure that the elements and objectives of NNR-NE are communicated to researchers, program officers, and research product users and that ONR managers and grant applicants justify new activities within the scope of the NNR-NE by showing how they will contribute to the initiative’s objectives. 

Conclusions and Recommendations 205 • Recommendation 9b: ONR should develop an enterprisewide information system that will make summary information on NNR-NE research projects readily available to proposers and to ONR’s clients, by posting on its website or other means. Summary information should include an abstract, funding, and a contact for each project. Project lists would be an effective way of advertising ONR’s interests and funding availability to prospective proposers and would help ONR’s clients in the Navy and shipbuilding to stay informed of ONR research. • Recommendation 9c: ONR should use the enterprisewide informa- tion system as a management tool in assessing NNR-NE progress; tracking funding allocation trends; benchmarking performance; and communicating NNR-NE progress, achievements, and potential. • Recommendation 9d: ONR should prepare an annual report that compares activities in the NNR-NE for the year with the activi- ties required according to the 2001 memorandum and 2010 instruc- tion and that compares accomplishments with the objectives of the NNR-NE. The annual report should describe how the NNR designa- tion raised the priority and aided the coordination of ONR naval engineering activities. • Recommendation 9e: In revisions of the Naval S&T Strategic Plan, ONR should delineate the expected contributions of the NNR-NE to the plan. • Recommendation 9f: To fulfill the requirement of the 2001 mem- orandum for creation of consortia to foster naval engineering S&T, ONR should consider adoption of the alternative organizational models proposed by the 2002 NRC Committee on Options for Naval Engineering Cooperative Research (TRB 2002). • Recommendation 9g: ONR should revise the definition of NNR-NE, specifying educational responsibilities and require- ments for coordination of naval engineering–related basic and applied research outside the Ship Systems and Engineering Research Division. The definition should specify that all relevant research be coordinated through the NNR-NE, regardless of its location in the ONR organization. Requirements in the 2001 memoran- dum that have not proved useful should be dropped from the definition. 

206 Naval Engineering in the 21st Century Increasing NNR-NE Effectiveness Framework for Research Portfolio Management High-performance research organizations standardize portfolio manage- ment processes on the basis of a series of information search, decision- making, performance, and evaluation tasks. The processes are outlined in Figure 5-1. Planning the NNR-NE includes articulating the research mis- sion, aligning the research agenda with the mission, and identifying researchers and research opportunities. Execution includes supporting and funding research, graduate education, and associated infrastructure and tracking performance indicators. Assessment includes measuring outcomes, benchmarking performance and evaluating results, providing feedback, establishing continuous improvement processes, and publish- ing lessons learned and best practices. Key to high-performance portfolio management processes are the assessment, benchmarking, and continu- ous process improvement activities that align incentives with desired per- formance (Eccles 1991; Eccles and Pyburn 1992; Brown 1996; Melnyk et al. 2004; Reugg 2007; Newell and Simon 1971; Simon 1996; Tan and Platts 2003; Tan et al. 2004). Feedback Planning Execution Assessment • Articulate research • Support and fund • Measure outcomes mission research • Evaluate results • Align research agenda • Support and fund • Benchmark with mission graduate education performance • Identify researchers • Support and fund • Assess portfolio impact and research associated and contribution to opportunities infrastructure discovery and invention • Establish key • Gather performance • Provide feedback performance indicators data • Communicate lessons learned • Align incentives with research mission and research agenda • Establish continuous process improvement program FIGURE 5-1 Framework for research portfolio management. 

Conclusions and Recommendations 207 This management procedure is consistent with the practices that Con- gress required all executive agencies to follow in the Government Perfor- mance and Results Act of 1993 (GPRA) and is related to the Office of Management and Budget’s Program Assessment Rating Tool, a parallel initiative to improve executive agency efficiency. GPRA requires agencies to develop performance plans and to measure and report on progress toward goals defined in the plans (GAO 2010, 2–5). ONR’s procedures in the NNR-NE for establishing a research agenda, identifying performers, supporting research, measuring outcomes, and evaluating results are relatively informal. Practices appear to vary by field at the discretion of the program officer. As a consequence, the initiative appears to lack a consistent and rigorous process to define and track performance indicators, assess performance, benchmark outcomes, and achieve continuous improvement. Conclusion 27a: ONR collects information on a variety of metrics that could be helpful in evaluating progress toward objectives, incentivizing performance, and improving the organization over time. However, it was not clear to the committee that these metrics are linked to a set of measurable objectives for the S&T enterprise in NNR-NE. The commit- tee also could not determine whether any NNR-NE goals or objectives were tied to strategic plans at the department or agency level. The committee was unable to identify an NNR-NE strategic plan that establishes priorities and identifies measurable objectives, an annual performance plan, or performance reports. Conclusion 27b: The committee could not identify a process by which NNR-NE mission area needs and research strategies are prior- itized or a systematic process by which research funds are allocated. Instead, it appears that NNR-NE program officers fund research pro- jects and principal investigators as opportunities arise, without an enterprisewide evaluation process that prioritizes and evaluates research project merit in a consistent manner across the NNR. Conclusion 28: The committee did not find evidence that NNR-NE measures or achieves balance in its research portfolio, despite its stated balance goal. The committee found no metrics to measure or establish balance in the portfolio. The lack of a metric leads to ques- tions about how such a portfolio could be balanced or demonstrate balance. 

208 Naval Engineering in the 21st Century Recommendation 10: ONR should establish an enterprisewide strate- gic planning and assessment process to develop a strategic plan for NNR-NE, link the plan to guiding goals and objectives, communicate those goals and objectives clearly throughout the naval research com- munity, and evaluate and incentivize NNR-NE performance against the strategic plan and objectives. The NNR-NE strategic planning and assessment process should encompass all facets of the NNR-NE mission and should include the following elements: • A process to articulate and prioritize NNR-NE mission area needs and research priorities on an annual and continuing basis. Priori- ties could be established by following a balanced scorecard or other methodology; the priorities should guide annual and long- term research program funding allocations in a transparent and consistent manner across the NNR. • A process for NNR-NE research fund allocation that is aligned with the articulated mission area needs and priorities so that resource allo- cation decisions are guided by a transparent, enterprisewide eval- uation process that prioritizes and evaluates research project merit in a consistent manner across the NNR. • Metrics for measuring the activities of research needs identifica- tion, resource allocation, research management performance, and continuous process improvement. • A continuous process improvement activity that utilizes the metrics to assess research portfolio management activities and alignment with Navy needs and to evaluate and report annually on organiza- tional progress over time. • An enterprisewide communication system to promulgate lessons learned, best practices, and organizational heuristics associated with the NNR-NE strategic planning and assessment process. • A research portfolio management procedure for the NNR-NE as a framework to guide planning and information collection, research administration, and assessment of performance and outcomes. The procedure should follow recognized standards for research portfolio management, including performance benchmarking. The goal of instituting the procedure should be to establish a culture of continu- ous process improvement. 

Conclusions and Recommendations 209 The enterprisewide strategic planning and assessment process should include the following: • A process to develop NNR-NE strategic priorities with respect to con- nectivity with the wider naval engineering community as well as to communication with stakeholders, technical advisory groups, the user community, and the broader research community. The process should include adoption of one or more of the cooperative research mod- els reviewed in the 2002 NRC Committee on Options for Naval Engineering Cooperative Research report; • A process to identify NNR-NE priorities associated with human capital and organizational development; and • Metrics associated with connectivity with the naval engineering com- munity and human capital and organizational development. Recommendation 11: ONR should identify, utilize, and periodically reassess metrics to measure NNR-NE portfolio balance, in line with ONR’s stated goals and articulated mission needs. Once established, these metrics should be incorporated into ONR’s enterprisewide assess- ment and continuous process improvement program. Recommendation 12: As input to the identification of performers, to enhance systematic dissemination of Navy mission and needs, and to improve communications between ONR and operational Navy units, in managing NNR-NE, ONR should utilize mission capability managers who are responsible for understanding specific end-to-end Navy mis- sions (e.g., antisubmarine warfare–antisurface warfare). All program officers should justify their projects in terms of NNR-NE goals, and ONR management should ensure that all aspects of the goals are attended to. Necessary elements of the recommended process include maintenance of a team of talented and experienced managers, including managers responsible for acting as technology interpreters (as defined below); an internal review process for administrative accountability; arrangements for vertical and horizontal integration of the NNR-NE research portfolio with other related research and development within and beyond ONR; and peer review (as proposed below) for quality control, relevance, and accountability. 

210 Naval Engineering in the 21st Century Measuring the Output of NNR-NE ONR uses three groups of metrics as indices of the output of its invest- ments in its Discovery and Invention activities, including NNR-NE: papers published, paper citations, and patents (as measures of new knowledge produced); basic and applied research results that lead to Innovative Naval Prototype or Future Naval Capabilities projects and basic research results that lead to applied research projects (as measures of transitions of results toward application); and numbers of graduate students supported, participants completing degrees, and participants joining Navy laboratories (as measures of contribution to the research workforce). These metrics provide useful information. Trends will indi- cate whether research is being completed and students are being trained at rates consistent with experience. The transitions measures may indi- cate the strength of NNR-NE’s linkage to ONR’s later-stage research and development activities. However, the present metrics fall short of adequate measures of ONR’s investment in NNRs. The transitions metrics are problematic for a basic research program because a basis for determining an acceptable transition “batting average” is lacking. Basic research should be expected to have a low frequency of direct payoffs but often very high value when there is a payoff. Also, basic research can lead to innovation by paths that are indi- rect and difficult to observe. Papers published and students supported are limited in value as metrics because they correlate with funding levels regardless of the value of output. ONR should systematically monitor the state of the S&T fields that sup- port naval engineering, in the United States and internationally. The indi- cators produced by this monitoring would be metrics of the impact of NNR-NE, because the NNR-NE initiative is intended to ensure the long- term health of these fields. Recommendation 13: As part of the research portfolio management process for NNR-NE, ONR should develop a set of research perfor- mance metrics that assess the contribution of its investments to dis- covery and innovation. In addition to the traditional numbers of publications, patents, and citations, ONR should develop metrics of portfolio impact, discovery, and innovation. These metrics should be inputs to investment decisions in managing the NNR-NE. They also 

Conclusions and Recommendations 211 should be used to increase visibility and understanding of the impor- tance of ONR naval engineering research investments. Research successes identified through the metrics should be publicized and com- municated broadly, and research excellence should be incentivized and celebrated in order to raise the visibility of high-quality research and raise the standard of research quality. Recommendation 14: Because of the importance and complexity of emerging problems in naval engineering S&T, along with increasing demands for integrative and interdisciplinary research across all tech- nological disciplines (NRC 1999), ONR should consider, as part of its continuous process improvement and assessment practices, adopting integrative and interdisciplinary metrics of performance in and across each of the NNR-NE functional areas. Planning and assessment of a program of basic and early applied research present special problems. Individual basic research projects are inherently high-risk, and the social benefit of the information a project produces may only appear after a long time and through a difficult-to-trace sequence of events. Because of this difficulty, the results of planning and assessment exercises will be imperfect, and determination of useful procedures will require trial and error initially. However, a major share of the basic research within NNR-NE falls into the category of basic research defined by the 2005 NRC Committee on Department of Defense Basic Research as “focused research in response to identified DOD technology needs” rather than in the category of studies aimed at “discovery arising from unfettered exploration” (NRC 2005, 3). That is, practical problems that the research may help solve are recog- nized. When the ultimate goal is defined, all research should have a spec- ified relationship to the goal. The 2005 NRC committee also observed that if research managers insist that the value of basic research cannot be mea- sured, institutional support for research will be undermined. Although the value of individual projects is difficult to isolate, the value of basic research can be measured at least cumulatively and retrospectively (NRC 1999, 22), that is, by showing how realized gains in ship performance depended ulti- mately on basic research. The recommended monitoring of the state of the S&T fields that support naval engineering would provide such a retro- spective measure of research impact. 

212 Naval Engineering in the 21st Century NNR-NE integrative metrics could include, for example, the number of interdisciplinary projects, the number of interdisciplinary publications, impact measures of research conducted within and outside the primary disciplines, citations and funding received outside the primary disciplines, and the numbers of publications and citations within disciplines. Such metrics would encourage program officers and principal investigators to consider and adopt interdisciplinary perspectives in research projects and encourage program officers to look for opportunities for collabora- tion across naval engineering S&T and across ONR to address critical naval research priorities. For applied research, appropriate metrics would relate to technology transition into Navy research and development projects at the Budget Activity 3 level and above. Management of basic research depends primarily on the competence of the program officers, ONR’s staff scientists. ONR’s expectation is that its program officers “have the appropriate technical expertise and scien- tific credibility to administer awards and recognize quality—in the mar- ketplace of science and technology, they are the Navy’s ultimate smart buyers” (Gaffney et al. 1999, 13). The program officer is required to have “the ability . . . to recognize a promising line of research even before it has been summoned by a formally declared requirement” (Gaffney et al. 1999, 15). The committee’s observation is that ONR has such talented pro- gram officers overseeing the NNR-NE research portfolio. However, staff with such skills always are in short supply. Formal processes for planning and for selecting research investments are necessary as a backup to the judgment of the program officers, for quality control and management oversight, and to ensure that progress in a field is not disrupted when a tal- ented program officer is not available. Staff time and funds for overhead activities, such as planning and eval- uation, must be expended efficiently. Procedures should be kept simple. Once ongoing collection of the necessary data is established, the burden of periodic reporting and review should be minimized. Peer Review ONR’s performer evaluation process, including that for its NNR-NE port- folio, differs from that of some other government research sponsors in that 

Conclusions and Recommendations 213 it does not include an evaluation of its basic research proposals by exter- nal peer reviewers. External review of proposals can be a valuable tool for government agencies that fund basic research, whose impact on future capabilities systems may not become apparent for decades. Organizations that use external scientific peer review for most or all of the basic research they fund include NSF, the National Institutes of Health, and the Office of Research and Evaluation of the National Institute of Justice (NAPA 2009, 67). Within DOD, the Air Force Office of Scientific Research employs a peer-review process using review panels that typically include two review- ers from other DOD offices and one from outside DOD. Conclusion 29: External peer review (that is, review by technical experts from outside ONR) throughout the research project selec- tion process offers the opportunity to strengthen project selection and to obtain the advice and counsel of technical experts, NAVSEA technical authorities, and industry practitioners who are the ulti- mate recipients of the developed technology, while maintaining the ONR program officer’s independence in making decisions for his or her program. Recommendation 15: ONR should establish a process for NNR-NE (and potentially other programs) in which the program officer assembles a small group of Navy laboratory technical experts (e.g., from NSWC-CD) and NAVSEA technical authorities (who also serve as industry surrogates) to review, assess, and rank relevant proposals received in response to ONR broad agency announcements. The pro- gram officer then would be responsible for considering these recom- mendations and selecting projects. The midproject external review that ONR already conducts would be carried out by this panel with the addi- tion of external reviewers according to the requirements of the present midproject review procedure. The proposal review panel would not remove ultimate responsibility from the program officer. Instead, the panel process would create a dialogue and open lines of communica- tion among ONR and the key Navy constituencies. Management of the NNR-NE relies primarily on the ONR program officers in the selection of projects and project investigators. Review 

214 Naval Engineering in the 21st Century of research project proposals and investigators before selection does not involve formal external peer review or other formal consultative pro- cedures. Peer review of proposals tends to sustain competition, avoid parochialism, and enhance communication within a research field. Review by appropriately constituted expert panels would ensure that selection of NNR-NE research projects resulted in a portfolio reflecting both the collective judgment of the research community and the views within the Navy commands with regard to needed technology or strate- gies. Introducing a wider spectrum of inputs to decisions on the direction of research would speed the application of research results to new tech- nologies of value to users in the design and shipbuilding communities. The proposed review panels would implement the function of the tech- nology interpreter (described below) in the operation of the NNR-NE. The program officer and the expert panel members would have explicit responsibilities for fostering the technology transition process. Frequent communication would inform the program officer of technologies that the technical authorities need and want and would inform technical authorities of new technologies as they emerge and mature. Technology Interpreter ONR today defines the scope of the NNR-NE in terms of six discrete sci- entific and technical areas that support naval engineering (structural sys- tems; hydromechanics and hull design; propulsors; automation, control, and system integration; platform power and energy; and ship design tools). However, the discipline of naval engineering is essentially integrative. The problem of the naval engineer is to apply the capabilities provided by scientific knowledge in all these areas to the design, construction, and operation of naval ships that satisfy mission requirements, respecting the constraints imposed by human factors and cost considerations. In recognition of the central importance of integration in naval engi- neering, ONR should establish a formal role for a technology interpreter in the NNR-NE to provide an institutional focus and mechanism for inte- grating research, discovery, and innovation in naval engineering. The tech- nology interpreter would be responsible for working with NNR-NE’s clients to specify the technology and research implications of their perfor- mance requirements and for working with NNR-NE researchers to ensure 

Conclusions and Recommendations 215 that their proposals and projects are informed by understanding of the interests of the operating Navy and the constraints of the naval vessel environment. Recommendation 16: To improve communication of operational requirements and the transitioning of technology to naval ships, ONR should implement the concept of a technology interpreter in the NNR-NE. The task of the technology interpreter would be to assist the technology transition process. The recommended peer-review panels would implement the concept of a technology interpreter in the program officer and technical authority communities. Frequent communication between these communities would inform the program officer of tech- nologies that the technical authorities need and want and inform the technical authorities of new technologies as they emerge and mature. In addition to the review panels, personnel dedicated to improving communications and execution could significantly improve NNR-NE integration with Navy missions, needs, and operational requirements. The technology interpreter role could be implemented in a variety of ways—as the responsibility of an individual within an ONR department or division tasked with technology interpretation, advocacy, and connec- tion responsibilities; as responsibilities of existing program officers who were encouraged to pursue technology integration within and across their disciplines; or through advisory or consultative arrangements, for example, through peer review or other interactions with contributors outside ONR. Maintaining Connections Across the Wider Naval Engineering Community Maintaining connections among the wider naval ship systems engineering community means bridging the valleys that naturally exist between the naval research, design, manufacturing, and operational communities and the commercial and offshore communities. While these communities all share a bond relating to the environments in which they operate, the systems that they build, and the manner in which they are deployed, an innate separation is reinforced by regulations, cultures, values, motiva- tions, and behaviors. 

216 Naval Engineering in the 21st Century A critical aspect of developing human capital and revitalizing naval engineering is to enable the people who make up that community. Enabling naval engineers requires the following actions: • Providing naval engineering education; • Providing naval engineering training to keep the workforce up to date; • Providing naval engineering mentoring in and outside the workplace, including activities with and through professional technical societies; • Developing tools, including collection of supporting data and support for verification, validation, and accreditation activities; • Developing ship design processes, including those for continuous pro- cess improvement and technology transition; and • Developing documentation, including specifications, standards, hand- books, and rules. The committee did not find evidence that these activities are part of NNR-NE planning, operations, or performance monitoring activities. Conclusion 30: Connectivity, communication, and human resource and organizational development are important to the success of the naval engineering enterprise. However, the committee was unable to find evidence that NNR-NE strategic planning makes use of measures of connectivity, communication effectiveness, human capital, or organizational development. Recommendation 17: To maintain connectivity among the wider naval engineering community, NNR-NE should utilize the concept of technology interpreter or otherwise establish integrative and connective responsibilities within ONR management and should continue to support, participate in, and incentivize its ongoing connectivity and communication activities, including conferences, workshops, and seminars, and the activities of ONR Global. ONR should consider adopting additional connectivity and communica- tion activities, including brown bag seminars, scholarly exchange events, and rotation and refreshment opportunities for NNR-NE program officers. The latter should include research sabbaticals at Navy laboratories and academic research institutions and in oper- ational Navy settings. 

Conclusions and Recommendations 217 Recommendation 18: ONR should incorporate human capital and organizational development goals and objectives as explicit responsibil- ities of NNR-NE during its enterprisewide strategic planning and assessment activities. Integrating Naval Engineering S&T The committee found several examples of interdisciplinary and integrative research in the NNR-NE portfolio. The commissioned papers and work- shops provided additional evidence of signature integrative and inter- disciplinary naval engineering projects, such as the integrated composite mast and a number of materials, hydrodynamics, and ship structures pro- grams. However, the committee concluded that these efforts were the out- growth of individual program officers or industry representatives who, for personal or professional reasons, engaged in interdisciplinary research and played a key role in developing such programs, rather than the outgrowth of systematic ONR processes that fostered, nurtured, encouraged, or incentivized interdisciplinary or integrative research. Recommendation 19: As part of its enterprisewide strategic planning process, ONR should establish a culture of interdisciplinary and inte- grative research within and around the NNR-NE S&T enterprise and should establish processes that foster, nurture, encourage, and incen- tivize interdisciplinary or integrative research. The NNR-NE interdis- ciplinary and integrative research objectives should be established as part of the NNR-NE strategic planning processes and should include assessment, benchmarking, and continuous process improvement components. Developing Human Capital and Revitalizing Naval Ship Systems Engineering Developing a robust naval engineering pipeline is critical to the devel- opment of a robust naval engineering enterprise. NNR-NE efforts in naval engineering S&T workforce development have been sporadic and inadequately supported to date. ONR has also been designated the lead agency for STEM efforts for the Department of the Navy; however, such efforts are considered auxiliary rather than core and critical functional responsibilities. 

218 Naval Engineering in the 21st Century Recommendation 20: ONR should reinvigorate its efforts in develop- ing the 21st century naval engineering workforce, including improve- ment of outreach activities to underrepresented groups. ONR’s lead role for STEM activities should be strengthened and incorporated into its strategic planning processes, and performance metrics for work- force development and STEM achievements should be identified, measured, incentivized, and included in ONR’s assessment, bench- marking, and continuous process improvement activities. REFERENCES Abbreviations GAO Government Accountability Office NAPA National Academy of Public Administration NRC National Research Council NSF National Science Foundation ONR Office of Naval Research TRB Transportation Research Board Brown, M. 1996. Keeping Score: Using the Right Metrics for World Class Performance. Amer- ican Management Association, Washington, D.C. Eccles, R. 1991. The Performance Measurement Manifesto. Harvard Business Review, Jan.–Feb., pp. 131–137. Eccles, R., and P. Pyburn. 1992. Creating a Comprehensive System to Measure Perfor- mance. Management Accounting, Oct. Gaffney, P., F. E. Saalfeld, and J. F. Petrik. 1999. Science and Technology from an Invest- ment Point of View: How ONR Handles Department of the Navy’s Portfolio. Public Management, Sept.–Oct., pp. 12–17. GAO. 2010. Streamlining Government: Opportunities Exist to Strengthen OMB’s Approach to Improving Efficiency. May. Melnyk, S. A., D. M. Stewart, and M. Swink. 2004. Metric and Performance Measurement in Operations Management: Dealing with the Metrics Maze. Journal of Operations Management, Vol. 22, pp. 16–29. NAPA. 2009. Department of Homeland Security Science and Technology Directorate: Devel- oping Technology to Protect America. Newell, A., and H. A. Simon. 1971. Human Problem Solving. Prentice Hall, Englewood Cliffs, N.J. NRC. 1999. Evaluating Federal Research Programs: Research and the Government Perfor- mance and Results Act. National Academy Press, Washington, D.C. 

Conclusions and Recommendations 219 NRC. 2000. An Assessment of Naval Hydromechanics Science and Technology. National Academy Press, Washington, D.C. NRC. 2005. Assessment of Department of Defense Basic Research. National Academies Press, Washington, D.C. NSF. 2010. Fluid Dynamics. Last updated Nov. 29. http://www.nsf.gov/funding/pgm_ summ.jsp?pims_id=13365. ONR. 2001. Memorandum: National Naval Program for Naval Engineering. Oct. 22. ONR. 2009. Naval S&T Strategic Plan: Defining the Strategic Direction for Tomorrow. Feb. http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA499909&Location=U2&doc= GetTRDoc.pdf. Reugg, R. 2007. Quantitative Portfolio Evaluation of U.S. Federal Research and Develop- ment Programs. Science and Public Policy, Dec., pp. 723–730. Simon, H. A. 1996. The Sciences of the Artificial. Massachusetts Institute of Technology Press, Cambridge. Tan, K., and K. Platts. 2003. Linking Objectives to Action Plans: A Decision Support Approach Based on Cause–Effect Linkages. Decision Sciences, Vol. 34, No. 3, pp. 569–593. Tan, K., K. Platts, and J. Nobel. 2004. Building Performance Through In-Process Mea- surement: Toward an “Indicative” Scorecard for Business Excellence. International Journal of Productivity and Performance Management, Vol. 53, No. 3, pp. 233–244. TRB. 2002. Special Report 266: Naval Engineering: Alternative Approaches for Organizing Cooperative Research. National Academies, Washington, D.C.