Design in the New Millennium: Advanced Engineering Environments: Phase 2 (2000)

In the Phase 1 report, the committee identified a number of barriers that will slow or limit the implementation of AEE systems. In this chapter, the committee discusses ways to overcome long-term barriers in five key areas:

• integration of tools, systems, data, and people

• knowledge management

• organizational culture

• education and training

• management and economics

Efforts to overcome technical barriers to integration and knowledge management should be redirected to take full advantage of technologies and applications being developed for the Internet of the future. Overcoming cultural inertia in the engineering enterprise and inciting a revolution in design, business, and educational methodologies will be difficult. Currently, however, many organizations either ignore AEEs altogether or have delegated internal responsibility for AEEs to technology researchers and developers who do not have the authority or resources to influence the overall organization. As a result, not enough action has been taken to address important barriers to the implementation of AEEs, especially in the areas of organizational culture, education and training, and management and economics.

Like AEEs, U.S. railroads in the early 1800s had to overcome problems associated with technological, economic, and cultural barriers (see Box 4-1). Nevertheless, railroad executives persisted through decades of moderate economic success to bring revolutionary change to the country. The commitment to a vision of a connected continent—much more than incremental advances in technology—was the key to success.

One of the most daunting, long-term barriers to establishing AEEs is the integration and portability of software tools for design and development across (1) disparate operating systems, distribution networks, and programming languages and (2) governmental and corporate cultures. From the technological perspective, crossing this barrier will not be as easy as selecting a standard rail gauge. AEEs of the future will require general solutions to interoperability (i.e., the ability of various systems to work together in a meaningful and coherent fashion) and composability (i.e., the ability to build new systems using components designed for existing systems) (NRC, 1997b).

The current state of practice is typified by a proliferation of nonuniform software tools written by engineers working in isolation to solve discipline-specific problems, by tools that are monolithic rather than modularized in structure, and by special-purpose tools created by individual organizations for their own use.

For some companies, limiting access to critical, proprietary software is an important factor in creating and maintaining a competitive advantage. Many software programs also use proprietary file formats for processing and storing data, which inhibits data exchange. Consequently, a product or process designed with one software tool may be very difficult to work on using another tool. In some cases, old data are ignored because they cannot be accessed by new software tools. Interoperability problems are exacerbated by software vendors who do not consider interoperability to be

in their best interest. The same is true of the lack of standards for software interfaces, files, and even basic data definitions. Software vendors' experience has shown them that proprietary solutions make money; they have no other experience.

The proliferation of individual tools is not a problem, per se. Allowing (or even encouraging) experts to update old tools and create new ones is often necessary to advance the state of the art in individual disciplines. However, without interoperability, innovation is slowed and advances in tools may be counterproductive.

Overcoming the tool and data interoperability barrier in the next 15 years will require major research into software composability and interoperability, as well as significant cultural and educational changes. Composability would facilitate the development of AEE systems with more robust, reusable components and flexible structures that can evolve

FIGURE 4-1 The lasting effect of legacy systems: a modern train passing through a surviving section of an ancient Roman aqueduct. Source: Von Hagen, 1967.

as technologies, users, and their organizations evolve. Reusable software modules would eliminate the need for each organization to develop the same tools. By packaging software for easy reuse, composability would also diminish the problems of monolithic software tools and systems.

Increasing the use of “open-source” guidelines (i.e., each software program's source code would be openly available via the Internet, with changes coordinated through on-line source-code control systems) is a promising approach for developing and implementing composable software (Raymond, 1999). To address proprietary concerns associated with competition-sensitive design and development software, open-source guidelines could be used for infrastructure software that supports interoperability and composability functions, while limiting access to competition-sensitive functions. The concept behind open-source code is that many people and organizations will continually examine and improve the code, increasing its reliability. Also, many engineers prefer open-source technology because they believe that they can correct problems more readily than with proprietary source codes that are owned and managed by individual corporations.

The committee believes that the current trend toward using the Internet as a universal medium should be expanded to search for general, Internet-based solutions to complex tool interoperability issues. Current ad hoc interoperability mechanisms tend to be governed either by the sharing of data files formatted in proprietary formats or by government mandates regarding the use of languages (such as Ada) and architectures (such as High Level Architecture). Government mandates may improve interoperability within a niche market controlled by the government, but they can also result in policies that isolate that market from the larger software community and unnecessarily hinder the use of more efficient software (NRC, 1997a). Converting data between different proprietary file formats is problematic, at best, but history provides little hope for establishing a universal standard; the diversity of software vendors, applications, languages, operating systems, and architectures, and the rate at which all of these factors change, is simply too great for any single standard to address. Therefore, basic research on interoperability should be supported in the flow of open Internet computing, open standards, industry-wide consortia, and other processes that have served the Internet so well. The same technological advances that would enable disparate tools and applications to interact would also overcome disparities between different versions of the same software tool.

New telecommunications and collaborative capabilities of future Internet technologies and applications will enable users in the same or separate locations to engage in interdependent, cooperative activities using a common computerbased environment. The nature and extent of these collaborations will depend, in part, on the extent to which AEE developers have been able to integrate their technologies with the Internet. The new, federally funded Information Technology Research Initiative (which grew out of the proposed research initiative, Information Technology for the Twenty-First Century) will create opportunities for information technology research related to AEEs that would be compatible with the Internet of the future.

The federal government's budget for fiscal year 2000 provides $236 million for the new Information Technology Research Initiative divided among five agencies, as shown in Figure 4-2 (NCO, 1999). The National Science Foundation (NSF) is the lead agency for this initiative, which is focused on long-term, fundamental research on information technology. The initiative is also intended to develop advanced computing technologies to meet the needs of science, engineering, and the nation as a whole and to evaluate the economic and social impact of the information revolution (for additional information, see CRA, 2000).

Finding 4-1. Interoperability and composability problems are a major barrier to realizing the AEE vision. The understanding of and technology base for developing interoperable and composable software architectures need to be improved.

Recommendation 4-1. The federal government should support basic research on the interoperability and composability

FIGURE 4-2 Fiscal year 2000 funding for the Information Technology Research Initiative. Source: NCO, 1999.

of component software architectures in the context of open Internet computing to increase software reliability and encourage the widespread use of promising solutions. Efforts to resolve interoperability and composability problems should investigate approaches, such as open-source guidelines, for bringing together software designed for diverse applications (e.g., mechanical, electrical, software, and biomedical systems).

Recommendation 4-2. Government, industry, and academia should seek consensus on interoperability standards.

Legacy systems are difficult to integrate with advanced tools that support AEE capabilities. Moving from today's monolithic codes to open-source, interoperable, and composable tool sets that will work on the Internet of tomorrow will require in-depth knowledge of the legacy systems that are retained. These tools must either be rewritten in accordance with composability and interoperability requirements for AEE systems or encapsulated in an interoperable software “shell. ” The latter option is illustrated by the growth of middleware, programs that act as intermediaries between sets of incompatible software tools.

Integrating the multiple hardware platforms (computers, hardware, databases, and operating systems) on which AEEs rely is a major software challenge. For example, many different operating systems are currently in use: the Macintosh operating system, Windows operating systems (Windows NT, 2000, 1998, 1995, ...), Unix and its cousins (Irix, Solaris, HPUX, Linux,...), and Java and its extensions. Assuming that the transition to Internet computing (i.e., writing application programs in Java so they work on the Internet) continues and that basic research in interoperability and composability proceeds, fewer choices for operating systems are likely to be available in 15 years. Java was invented as a proprietary code but is becoming more like an open-source code. Three proprietary versions of Java are now available, and Sun Microsystems is trying to establish a Java standards organization under the auspices of the European Computer Manufacturers Association. The committee believes that, in the future, an open-source successor to Java is likely to be dominant, becoming the primary interface with the underlying operating systems embedded in the hardware of individual users. The underlying operating systems will be much simpler than current operating systems and will probably have achieved prominence by acclamation and adoption, rather than by government mandate or corporate control.

Finding 4-2. Engineering tools and systems have been developed on a variety of incompatible operating systems and with a variety of programming languages. This situation is changing as more advanced tools and systems are being developed for Internet deployment.

This section addresses technical issues associated with organizational interoperability. The cultural barriers associated with organizational interoperability are discussed in the section on “Organizational Culture” later in this chapter.

AEEs should facilitate interoperability not just among different tools, but also among different organizations. For example, a significant fraction of the products of large manufacturing companies, such as the automotive and aerospace industries, is supplied by manufacturers of components and subassemblies. In 1999, The Boeing Company, with $58 billion in sales, had 29,000 suppliers worldwide, who represented about 50 percent of Boeing's total product cost. As the global economy continues to evolve, market forces will continue to exert pressure on product availability, quality, and cost. Prime manufacturers striving to meet market demands will be analyzing and streamlining their total production systems. Industry penetration of digital CAD, CAE, and CAM tools, which began some 15 years ago, will continue to expand. Business systems are also becoming more responsive to corporate requirements for higher speed and better accuracy.

Changes experienced by prime manufacturers are being felt throughout the supply chains of various industries. Because some 50 percent of product cost is represented in

the supplier base, pressure will continue for improvements in quality, cost, and speed at the supplier level. Boeing recently announced a plan to rate suppliers on their ability to produce and deliver quality components and subassemblies on schedule. In addition, Boeing suppliers must show plans for continuing improvements in cost and quality. Boeing also plans to reduce the overall number of suppliers.

The automotive industry has taken similar actions. In many cases, suppliers are being required to work only from digital data sets. For major components and subassemblies, suppliers are often required to provide a digital model and operational simulation to the prime manufacturer. In the near future, this information will be a requirement for suppliers who bid on new products or wish to maintain their current positions in the supplier base. As the capabilities of CAE, CAD, and CAM tools and Web-based delivery capability increase, the industrial supplier base will need to keep pace with their customers' requirements to remain in business.

Prime manufacturers will continue to focus on product innovation, quality, speed, and cost. The science of enterprise modeling and management will improve and include the supply chain because of its significant contribution to the product and the well-being of the corporation. AEEs will provide connectivity throughout the market development, design, manufacture, and support processes of manufacturing industries and their customers.

The current exponential growth in the rate at which new information is becoming available is likely to continue during the 15-year window examined in this report. To handle this flood of data, technologies are being developed to extract, manipulate, integrate, and display information culled from huge amounts of heterogeneous data. The capability of the Internet to display visual information is rapidly improving, as are the speed and reliability of data communication technologies. Industry is energetically tackling the challenges of accessing and displaying available data. Security systems for controlling access to data will also be necessary. AEEs must provide effective knowledge management in this constantly changing environment.

The scientific and engineering communities continue to generate data (empirical and analytical) in increasing quantities. The productivity of data-generating systems has long exceeded the capacity of communications systems used for dissemination and display. The World Wide Web (created, in part, to address the problem of access to a growing body of knowledge) seemed to provide the answer to the inability of printed media to keep pace with the volume of new knowledge. Unfortunately, the World Wide Web still depends largely on text to communicate knowledge. Large quantities of images are also used on the World Wide Web, but only recently has a reasonable level of standardization for coloration of images become accepted. Much work remains to be done on image-attribute associations, animations, video, and audio delivered via the Web.

Two key questions must be answered to resolve the fundamental problem posed by the information explosion:

• How can someone intelligently and quickly locate necessary information?

• How can someone comprehend the amount of information generated in even a simple search?

The first question is in the process of being answered, at least in part, by the development of search engines with higher levels of intelligence. Engines will soon be available that can accept the typically fuzzy ways in which humans articulate their queries. In addition, new types of engines will greatly improve the ability to search intranets within organizations, as well as the Internet and other large data repositories. For example, industry is developing a high-speed parallel search engine that can deliver integrated results from variously formatted sources (Restivo, 2000). Organizations could use these search engines to locate applicable parts, subsystems, and software modules from old products for reuse on new products and otherwise facilitate the exchange of information among organizational elements that may have little knowledge of each other's operations, capabilities, or products. In addition, new software is being constructed to extract and manipulate meaningful information from huge data collections without complicating the human-computer interface. In fact, data mining is now the focus of at least one journal, Data Mining and Knowledge Discovery (Kluwer, 2000) and international conferences (e.g., the Sixth International Conference on Knowledge Discovery and Data Mining; for additional information, see ACM, 2000). Thus, without any investment by NASA or the aerospace industry, search engine and data extraction technology will continue to evolve rapidly. At most, the aerospace engineering community will have to ensure that the scope of search engine capabilities meets the community's needs.

Visualization of scientific or engineering data is making an important contribution to the problem of data understanding. This nascent field is exploring new methods of representing data, investigating interactions among data sets, and displaying data with the goal of enabling knowledgeable experts to comprehend and manipulate large quantities of data effectively.

Rapid advances in data visualization are already being made in some applications. For example, a four-dimensional, interactive model has been developed that incorporates diverse geospatial and other types of data into a virtual earth through which users can interactively move in space and time. Users can also inject data, which are visually fused

with existing data to simulate current and future states and comparisons between states (Gordon, 2000).

The most likely means of improving existing capabilities for data visualization is within the framework of multimodal display and interaction. This work will extend the term visualization to include characterization of nonvisual displays and interaction modes (e.g., audio and haptic representations, displays, and interactions).

In the long term, the government could enhance the management of large amounts of information by conducting basic research in several areas, including multidimensional data visualization (i.e., visualization of data that contains more than four dimensions) and multisensory display and interaction. These areas are too risky to attract large investments by industry, but successful research could lead to new applications that greatly improve human capabilities for comprehending large, complex data sets.

Recommendation 4-3. Research and development by the federal government on the visualization of engineering and scientific data should focus on long-term goals that go beyond those of ongoing research and development by industry.

Communicating large amounts of engineering data quickly and reliably requires hardware and software infrastructures that today are neither uniform or ubiquitous. Meeting the data communication needs of an engineering team is difficult, especially if the team is dispersed geographically and organizationally. The AEE vision requires that data be accessible, in quantity, from any location and that interaction with that data be instantaneous in human terms. This presupposes that all engineers have desktop access to high-bandwidth, low-latency networks.

Data transmission rates for user interfaces and throughput limits on the total system for the current Internet impede the exchange of large amounts of data or detailed images. However, industry and government are investing billions of dollars in high-capacity data transmission technologies and systems, and transmission rates will increase greatly in the next few years. For example, the bandwidth available for transmitting data through a fiber-optic cable is growing at a Moore's Law rate,¹ as is the amount of data that can be transmitted using a given amount of bandwidth. Thus, the total amount of data that can be transmitted via a fiber-optic cable is doubling every nine months. Assuming that AEE requirements will not be significantly larger than the commercial applications that will drive the deployment of the Internet of the future and other new data transmission systems, data transmission will not be a significant constraint on the deployment of future AEEs.

Improving latency may be more challenging. A latency of less than about 100 milliseconds is required to create a three-dimensional, networked virtual world without losing the illusion of presence. Speed-of-light limitations impose a latency of at least 8.25 milliseconds per time zone, which is then increased by latency in the responsiveness of sensors, processors, transmission equipment, displays, and systems (Singhal and Zyda, 1999).

The need for increased Internet capacity is illustrated by the rate of growth of Internet users and hosts. As of February 2000, there were about 275 million Internet users worldwide, about half of them in the United States (Nua Ltd., 2000). Also as of February 2000, more than 72 million host computers were connected to the Internet, and the annual rate of growth was 63 percent. At that rate, 100 million hosts will be on line by the last quarter of 2000, and 1 billion hosts will be on line by 2005. About 60 percent of hosts are in the United States (NGI, 2000).

The availability of low-latency, high-bandwidth networks will be increased by information technology research and development programs, such as the Internet-2, the Next Generation Internet (NGI), and the very high performance Backbone Network Service (vBNS). The purpose of Internet-2 is to develop and deploy advanced, network-based applications and network services to enable a new generation of Internet services and applications. Started in October 1996 by 34 U.S. research universities, Internet-2 is now a collaborative research effort being carried out by a consortium of more than 170 universities and coordinated by the University Corporation for Advanced Internet Development. Internet-2 is also supported by partnerships with telecommunications, networking, and computer companies (for additional information, see Internet-2, 2000a).

Several federal agencies, including the U.S. Department of Defense, the U.S. Department of Energy, NASA, and the National Institutes of Health, are supporting the NGI program to develop advanced networking technologies and applications. System capabilities will be demonstrated on testbeds that are 100 to 1,000 times faster than the current Internet. The NGI program involves many of the same technologies as Internet-2 and funds some Internet-2 activities, but NGI is focused more on the needs of the sponsoring agencies (for additional information, see Internet-2, 2000b).

The vBNS is being developed by the NSF in cooperation with MCIWorldCom to provide high-bandwidth networking for research applications and to develop technology and applications for the Internet of the future. The vBNS provides high-speed interconnections among the NSF super-computing centers and more than 100 universities and other research organizations, but it is not being used for general Internet traffic. The vBNS offers high data throughput (more than 490 megabits per second) and low latency (an average of less than 100 milliseconds coast to coast). Ongoing

developments have significantly increased system capability since the project began in 1995, and current program goals include transmission speeds of more than 2.2 gigabits per second (for additional information, see vBNS, 2000).

Finding 4-3. Advanced Internet technologies and applications are likely to provide the universal, high-bandwidth, low-latency communications network necessary to meet most communications needs for AEEs.

Recommendation 4-4. Research, development, and engineering organizations in government, industry, and academia should ensure that technical staff and students have access to advanced data communications networks as those systems become available.

An ideal computer security system would protect organizations from unauthorized use or modification of data or systems while providing easy access to authorized users. Both qualities are important for AEEs, which often involve the sharing of sensitive data among distributed engineering teams. Compromised security could reduce productivity (e.g., if access to AEEs is disrupted by denial-of-service attacks), result in defective products (e.g., if corrupted data is used for product development), or harm a company's competitive position or national security (e.g., if proprietary or classified information is compromised).

Many organizations, such as NASA and large aerospace companies, are attractive targets for computer hackers. Security measures, such as firewalls, intrusion detection systems, data encryption, decoupling of computer systems used for different functions, and security education can be used to limit access by unauthorized users, detect intruders who gain access, minimize problems caused by authorized users (inadvertently or by design), and respond appropriately.

Designing and implementing security systems for complex systems can be challenging, especially if multiple organizations are involved. As the number of points at which security can be compromised increases, administration and oversight of the security system becomes more difficult.

Security measures also create significant barriers to data access by authorized users. For example, the data systems that NASA uses in mission operations are not directly interconnected with data systems used in engineering design, even in the same NASA center.

A central aspect of the committee's vision for AEEs is ubiquitous access of the entire engineering team to relevant data, and a sophisticated system for managing access control is essential. To limit the potential for compromising sensitive data or altering data without authorization, individuals should only have access to information necessary for their jobs. Access controls must not be too rigorous or cumbersome, however, because the entire engineering process can be disrupted if data are not available or if significant delays or complex processes are involved in accessing data. Thus, no matter how well conceived an AEE might be, successful implementation will require resolving critical data security problems in a way that balances the need for robust security with ease of use for the engineer. Specific approaches for enhancing the security of distributed computing networks are described in a recent report, Trust in Cyberspace (NRC, 1999).

Successful AEEs cannot be burdensomely complex; they must improve productivity by simplifying processes and reducing worker stress. Potential barriers associated with organizational culture and human behavior include the following:

• resistance to change because of fear of the unknown, aversion to risk, and hesitancy to criticize past practices or those who established and maintained them

• organizational cultures that impede innovation and participatory design through excessive oversight and the lack of effective reward systems

• differences in how individuals work efficiently

• disruptions caused by new technologies that do not work as expected

• discipline-specific jargon that impedes communication or leads to miscommunication

• difficulties associated with creating and maintaining collaboration within teams and across an organization or group of organizations

• simulator sickness,² information overload, and other shortcomings of human-machine interface technologies, which are not maturing fast enough to counteract the effects of increasing complexity in projects, processes, tools, and systems

• difficulties in applying lessons learned from other organizations or projects to the task at hand

Overcoming these barriers will require social and behavioral changes because of (1) the increasing scale and scope of collaborative activities within and across institutions, (2) the increasing complexity and informational density of advanced engineering projects, and (3) the increasing interdependencies among engineers and other team members and between them and their computer-embedded work processes.

Historically, as this committee's Phase 1 report underscores, not enough attention has been paid to the psychological, behavioral, and social aspects of the user environment for advanced engineering. AEEs, however, are expected to intensify the interrelationships between the social and technical dimensions of tasks, making it especially important to overcome the kinds of cultural barriers listed above. The committee identified four steps for overcoming these barriers:

• Create, maintain, and dynamically reconfigure collaborations.

• Cope effectively with complexity.

• Institute participatory design.

• Establish a culture of innovation.

The increase in collaborative activities is being driven by economic and competitive pressures that are unlikely to diminish. The design team for a very complex project may be so large that it becomes uneconomical or impractical for an individual company to increase its workforce enough to staff the team using only its own employees. For example, Sun Microsystems in the “Silicon Valley” south of San Francisco, California, cannot hire enough qualified people locally to support major new projects. As a result, it breaks up design tasks for complex chips so that remote teams can collaborate in the design of a single chip. Separating chip development into discrete tasks that can be easily integrated at the end of the project is technically very difficult. The use of remote teams can also create organizational and management problems because of extremely limited personal contacts or personal relationships among teams members. Videoconferences and teleconferences can be used to supplement face-to-face meetings, but video and audio technologies cannot yet effectively substitute for face-to-face meetings. Exchanging written summaries of meeting outcomes after remote meetings is essential, especially when participants speak different languages. The written summaries help identify misunderstandings so they can be resolved quickly.

In distributed environments that depend on more—and more diverse—collaborations to accomplish their missions, the emergence of new norms and rules of interaction must support team-building, as well as communications and task coordination in both synchronous and asynchronous activities. The dynamics of engineering teams can increase the pace of change, as each member of the team strives to contribute as much as possible to project success. Also, by working with experts in other disciplines, each person develops a better feel for how her or his work fits in with work in other disciplines. AEEs should improve interactions, for example, by improving the sense of place and point of view in virtual reality environments, by speeding some classes of asynchronous events to yield near real-time experience, by improving the representations of locations and actions of humans and other active objects, and by adjudicating efforts by multiple parties to manipulate the same object simultaneously. AEEs will have to provide a range of options between today's avatars³ and full holographic representations.

In addition, advances in interoperability should make it easier for groups with different tools to work in collaborative distributed teams. The greatest incentive to working in these teams, however, may be offered by AEEs that can provide experiences that go “beyond being there” (Dewan, 1999). In other words, when distributed networked technologies are used simply as a less expensive alternative to face-to-face meetings, they are likely to be viewed as inferior interaction media. However, when they enable interactions that could not be carried out in real environments (e.g., interactions with very small objects, very numerous objects, very distant or otherwise inaccessible objects, or observations of phenomena at normally inaccessible scales), AEEs will become the only—or at least the only desirable—way of carrying out highly advanced projects or missions.

AEEs will be able to accommodate complex tasks and tools, along with complex information about the status of ongoing actions and outcomes generated by both people and computers. AEEs should enable users to cope with these complexities using interfaces and presentations that can be easily understood and managed. This will require improved representation techniques, including better visualization of engineering and scientific data (as discussed earlier) and better methods of giving users a sense of their place in the environment, the place of other actors and activities, and the status of ongoing processes. The goal is to make it easy for team members to develop and maintain general situational awareness while providing them with task-specific information.

AEEs should also support users by actively informing them of ongoing events. For instance, AEEs should offer intelligent dynamic help to users (coaching or expert advisor systems). They should also incorporate “smarter” knowledge repositories that will improve the retrieval of relevant information, help users formulate options, or guide their next steps.

As illustrated in the vignette in the prologue, innovative scheduling can also help reduce the intensity and stress associated with complex multiperson tasks. Synchronous, distributed, collaborative tasks in existing environments often heighten stress, but they may also encourage innovation, partly because they bring “tacit” knowledge to the surface and partly because they create opportunistic synergies from “loose ties” (Granovetter, 1978). Having all members of a

team in the same location (physically or in a virtual environment) enables them to participate in multiple, simultaneous conversations and move from one conversation to another, as needed. If a team leader is present to coordinate the action, productivity is greatly enhanced compared to one-on-one conversations (or point-to-point communications). Future research should explore methods of achieving the same benefits using asynchronous collaborations.

Constructing the kinds of distributed collaborative work environments envisioned as AEEs will require participatory design and development. That is, very close cooperation between engineering design team members and information and computer scientists will be necessary to generate the tools, models, data sets, interfaces, communications media, and application systems incorporated in AEEs. No other approach would bring together the varied types of expertise that AEEs must embody. Participatory methods that engage users throughout the design process have the added benefit of making users much more knowledgeable about the technologies that support their tasks and, therefore, much more effective at troubleshooting when breakdowns occur.

Participatory design methods, however, will face most of the same problems as other types of multidisciplinary collaborative projects. The work of designing AEEs will never be finished because continued technical advances will stimulate continued transfer of new technologies to AEEs. But even now, workers could begin to learn about participatory design techniques in relation to the digital technologies that support today's engineering environments.

As suggested above, AEEs will be characterized by continuing technological change. At the same time, the successful implementation of AEEs will require dramatic changes in the social order of many organizations. Aversion to risk and resistance to change are well-known barriers to the implementation of new processes. Risks associated with greater dependency on new technologies can be alleviated in at least two ways. First, AEEs should incorporate high levels of redundancy in critical technologies (both hardware and software). Second, the ability to conduct multiple iterations of solutions very rapidly or try out great numbers of design options should encourage bold exploration and increase confidence in the ultimate decisions, while still reducing cycle time. In addition, enlightened management can reinforce a culture of innovation by establishing policies and practices that reward innovation on the part of teams and individuals. In the long term, however, working in an environment with world-class technologies and teams of innovative colleagues from diverse disciplines may be considered an important reward in itself.

Highly capable AEEs will lead to a drastic reduction in the use of physical models and prototypes in favor of digital objects (e.g., simulations, scientific representations, avatars, and virtual objects). But little is known about the roles that physical artifacts play in supporting collaborative design processes. Research suggests that artifacts are bearers of shared “tacit” understanding, but not enough is known about these aspects of physical artifacts to replicate them virtually. Therefore, further research is needed.

AEEs will enable rapidly reconfigurable teams, with members moving fluidly from project to project as their special skills are needed. Individuals may have the opportunity to work on concurrent project teams with different schedules and varying task demands. Not all individuals will thrive in this kind of environment; some are likely to prefer working alone in a concentrated way over long periods on single projects. Not much is known about how to prepare people for a transition to the new work structures that AEEs will enable. As discussed below, changes in the educational system should be designed, in part, to provide preparatory experiences. In addition, AEE technologies themselves should be designed to support the rapid and intuitive acquisition of situational awareness for people transitioning between tasks, teams, and projects. Moreover, AEEs should be designed to support heterogeneous work styles (as well as heterogeneous data, models, tools, and languages). This may require trade-offs between individual worker satisfaction and the efficiency of the total enterprise because, in some cases, work processes that maximize individual productivity will not optimize overall productivity.

AEEs are expected to overcome geographic barriers to shared, distributed work, but temporal constraints will probably be much harder to overcome. In fact, even current synchronous collaborative environments can create satisfying, yet very intense work experiences that are difficult to sustain over long periods of time. Thus, even though AEEs may reduce cycle time and improve productivity, they are unlikely to be perceived by workers as a step forward unless they also improve the workplace atmosphere, for example, by reducing stress and time pressures and helping to build social relationships. Therefore, new research is needed to address the psychological and temporal dimensions of engineering design work in synchronous and asynchronous distributed collaborative activities.

Recommendation 4-5. The government and academia should conduct research to improve understanding of the following topics:

• the role of physical artifacts in supporting collaborative design processes and how that role can be fulfilled when physical artifacts are replaced by simulations, virtual objects, avatars, and other nonphysical artifacts

• methods for designing AEE systems that accommodate workers with a variety of work styles and improve the new work environment (e.g., by improving situational awareness for workers transitioning between tasks, teams, and projects)

• the psychological and temporal dimensions of engineering design work in synchronous, distributed collaborative activities, especially if team members are located in multiple time zones and work for organizations with different cultures and business goals

“Economic development is a prerequisite for growth and opportunity, research is a prerequisite for development, and education is the foundation of research” (Moniz, 2000). Barriers associated with education and training generally fall into two categories: (1) undergraduate and graduate education and (2) continuing education and training to sustain and improve the skills of engineers and scientists throughout their careers.

Four forms of scholarship are recognized in today's educational environment: education, discovery, integration, and application (Boyer, 1990). Undergraduate education primarily focuses on the scholarship of education, while graduate education focuses on the scholarship of discovery. The scholarship of integration and application are largely undeveloped in the academic community. With the development of AEEs, these important forms of scholarship could be developed by combining research opportunities and professional development at the undergraduate and graduate levels.

Current engineering programs typically require that undergraduate engineers take only one course in introductory programming. However, as nearly all forms of engineering increase their reliance on computer-based methodologies, courses on computer programming and methods will become increasingly important. In addition, designing systems with a high degree of composability and interoperability will require that engineers both understand the importance of these features and include computer scientists with this expertise on tool and model development teams.

Interdepartmental cooperation is essential for universities to address these challenges, but university engineering and computer science departments have historically operated independently. Even the accreditation boards have been separate: the Accreditation Board for Engineering and Technology for engineering and the Computing Sciences Accreditation Board for computer science.

Every demand for additional course work places additional pressure on already overcrowded engineering curricula. Unless the efficiency of the educational process is improved, each new course requirement results in the removal or compression of other essential courses. Degrading the core curriculum, however, would jeopardize the ability of students to learn the basic principles behind the increasingly sophisticated engineering tools and software models at their disposal. Furthermore, meeting all of the needs of industry and government will require improvements in both computer science programs and traditional engineering programs. Many of the barriers described in this chapter will be especially difficult to overcome as long as the current shortage of information technology workers persists.

Fortunately, changes in the educational process are already under way. In the past decade, undergraduate engineering programs have increasingly emphasized design, manufacturing, communication, and teamwork. In addition, the Computing Sciences Accreditation Board has agreed to integrate its accreditation commission into the Accreditation Board for Engineering and Technology (CSAB, 2000). Under pressure from industry and academia, the accreditation process, which is governed by the accreditation boards and professional societies, is focusing more on educational outcomes (i.e., preparation for professional practice) than credit hours (seat time). Under the new approach, educational institutions and programs have redefined their missions and objectives in terms of their ability to meet the needs of constituencies with an emphasis on outcomes. For example, students are generally required to complete a capstone design project or other culminating experience that incorporates the knowledge and skills acquired in earlier course work; engineering standards; and real-world constraints, such as economics, the environment, sustainability, manufacturability, ethics, worker health and safety, and social issues. AEEs would directly improve the ability of students to complete complex capstone projects.

Other positive developments in the educational process include the following:

• Professional societies, such as the American Institute of Aeronautics and Astronautics, the Society of Automotive Engineers, and the American Society of Mechanical Engineers, are providing opportunities for student teams to compete in design-build-test competitions involving, for example, solar cars, remotely controlled aircraft, and human-powered vehicles.

• NASA is providing opportunities for student teams to compete for the chance to conduct experiments on board the KC-135 microgravity research aircraft.

• The Industry, University, and Government Roundtable for Enhancing Engineering Education is exploring options for reforming the educational process so that engineering graduates can meet the challenges of future business environments and professional standards. The roundtable is also concerned with improving knowledge management, enhancing the engineering profession, and improving continuing education (IUGREEE, 2000).

• The NSF's Engineering Education and Centers Division is improving engineering research and education by sponsoring eight engineering education coalitions, which involve approximately 55 universities. These coalitions are investigating issues such as multi-disciplinary pedagogical models and integrated approaches to undergraduate engineering and manufacturing education (NSF, 2000b). The NSF has also established engineering research centers at 33 universities to create educational environments that integrate engineering and research for undergraduate and graduate students. The centers focus on next-generation advances in complex engineered systems and expose students to industrial practices (NSF, 2000a).

Although the advent of AEEs will increase educational requirements and the use of advanced tools could obscure students' awareness of basic principles, AEEs could also enhance the educational process by teaching students in virtual environments. For example, currently available training technologies enable students to interact within a three-dimensional, virtual reality to explore basic concepts, such as the forces created by electrical charges, as well as complex problems, such as the impact of orbital mechanics on the field of view of instruments on an orbiting platform. By providing opportunities to combine professional development with research more effectively, AEEs could stimulate interest in graduate engineering education by increasing its perceived value and relevance to industry.

Despite the progress that has already been made, however, much more needs to be done to incorporate AEE technologies into undergraduate and graduate curricula. Based on the experience of committee members, the committee believes that the following barriers remain at many universities:

• a university culture that rarely fosters or rewards interdisciplinary academic careers

• insufficient appreciation by faculty of the benefits of emerging AEE techniques and tools, which are too often perceived as producing little more than interesting graphics

• limited time and resources for faculty to develop interdisciplinary programs⁴

• the view by government and industry that academia is a minor player in implementing AEEs in the engineering enterprise

• a lack of proven methods for preparing students and working engineers for the new work structures and processes that AEEs will enable

Overcoming these barriers will require investments of time, effort, and money. In addition, demonstrations of new methods will be necessary to convince a skeptical university audience that AEEs can (1) help students develop critical thinking skills and (2) improve educational efficiency so much that AEE-related education can be integrated into curricula without having to eliminate existing courses.

AEEs could also be used to integrate schools of engineering with the liberal arts and sciences. Just as the functionality of AEEs is not limited to traditional engineering tasks, access to AEEs should not be limited to engineering students. Synergies can arise when different communities come together in a constructive way (NRC, 1997b).

Just as individual corporations will require a champion to implement AEEs (see Recommendation 8 from the Phase 1 report, which is reprinted in Appendix B of this report), individual engineering schools will need an influential AEE champion to make AEEs an important part of the university environment and sustain support for AEEs and related interdisciplinary programs. A champion will be especially important at schools where individual departments do not embrace AEEs. At many universities, strong internal leadership, along with external pressure from accrediting organizations and industrial engineering organizations, will be needed to encourage faculty to accept AEEs, support the scholarship of integration, and modify undergraduate and graduate curricula accordingly.

In a broader context, funding for long-term research will be necessary to support interdisciplinary research associated with AEEs. Because publishing research results is a sign of academic competence, journals dedicated to interdisciplinary research must be available to publish research results. The federal government can facilitate change by funding research and, with industry, by including academia in a national partnership for fostering AEEs (see Recommendations 1, 2, and 12 in Appendix B). The involvement of the federal government will be essential for supporting long-term research whose expected benefits are likely to (1) be realized beyond the investment horizons of industry; (2) be so generic that the benefits cannot be contained and applied in a way that gives the sponsoring company a proprietary, competitive advantage; or (3) be of such high risk that it is impossible to make a solid financial justification for conducting the research.

Continuing education and training does not lead to an undergraduate or graduate degree and may be conducted at government or industry sites, specialized training facilities, or universities. Continuing education and training is needed for teaching newly hired workers to use their employers' engineering and design systems and to refresh the skills of existing staff, especially when systems are updated.

To make the most effective use of new AEE technologies and systems, the training and education program for the existing labor force and future workers must be upgraded. Special efforts will be needed to prevent engineering skills and knowledge from atrophying in the presence of AEEs that automatically perform increasingly sophisticated engineering tasks. Just as overreliance on complex engineering tools and software models can prevent students from developing a fundamental understanding of important engineering and scientific principles, overreliance on AEE systems and technologies may prevent design teams from developing a fundamental understanding of their products and missions. As a result, teams could unwittingly use tools on tasks for which they have not been validated. For example, several years ago a manufacturer of large, complex products tried to use analytical models to reduce the need for destructive testing. When test engineers attempted to qualify the models, however, the analytical results repeatedly failed to match actual test results. As costs and frustrations increased, the testing manager asked the modelers to predict what would happen if a simple component, such as a pipe with a square cross section, were bent in half. The model had to be revised several times before it accurately predicted how the walls of the pipe would deform during the bending process. Starting with that success, the models were refined with increasingly complex test articles until they could accurately predict the performance of the entire product. The same models could then be used to predict the performance of new products, as long as the models were updated to handle new structural components. This example illustrates the importance of educating the engineering workforce to understand the capabilities and limitations of their AEE technologies and systems. Continuing education and training will be crucial to prevent existing knowledge from degrading or becoming outdated as experienced personnel leave and new tools are adopted.

AEEs designed for complex engineering applications will generally have sophisticated simulation and modeling capabilities and, as a result, will be technically capable of functioning as sophisticated, automated training devices for new users. However, education and training requirements should be explicitly addressed during the development of AEEs to ensure that adequate training features are included in the final product.

For most AEEs, training and education will be a secondary function used to train new users. Training simulators, however, use AEE technologies and systems to teach users some other activity, such as flying an aircraft. Training simulators provide many benefits. For example, advanced training simulators benefit the U.S. Air Force in the following ways:

• reducing the use of aircraft, which are expensive to operate and are reaching end of life faster than they are being replaced

• eliminating unrealistic restraints, such as minimum altitudes, that are imposed on real-world training because of safety concerns

• reducing costs

• increasing training opportunities

• reducing air crew deployments for training⁵

AEEs will be used to create increasingly sophisticated training environments that combine real and virtual environments for individual and group training locally and in distributed training environments. In fact, the Air Force has already used transportable simulators to conduct distributed mission training exercises with pilots at several remote sites. Trainee evaluations indicate that simulator training is more effective than real-world aircraft training in some ways, but less effective in others. In general, simulators were judged to be less effective for tasks involving visual fidelity but more effective for tasks involving decision making and complex team interactions (e.g., encounters involving several aircraft on each side). Why? Primarily because Air Force combat pilots rarely have the opportunity to train with large numbers of aircraft in the real world. Distributed mission training also seems to be especially effective for intensive training (because it provides many training exercises in a few days), for enabling pilots to fly with and against disparate aircraft, and for breaking down organizational barriers (by making it easy for pilots in different squadrons to train together) (Andrews, 1999).

Finding 4-4. Research funding, interdepartmental cooperation, and organizational support for interdisciplinary programs has traditionally been difficult to obtain from the government or academia, largely because funding agencies have usually set narrow limits on the types of projects they are willing to support.

Recommendation 4-6. Accrediting organizations, industrial organizations, and professional societies should continue to advocate greater use of AEE technologies and systems in the academic environment at both the undergraduate and graduate levels.

Recommendation 4-7. Universities should appoint AEE champions to provide strong, long-term leadership for

implementing AEE technologies and systems; establish the innovative, interdisciplinary educational programs and faculty needed to take full advantage of the capabilities of AEEs; increase the emphasis in undergraduate and graduate education on the scholarship of integration and application; and develop curricula with a stronger foundation in software development, including component software architecture, composability, and interoperability.

Recommendation 4-8. Initial and continuing education should include strategies for (1) maintaining scientific and engineering understanding of processes and tasks that will be done automatically by AEE technologies and (2) training people for the transition from conventional working environments and processes to the pace and structure of working as members of multiple, concurrent, rapidly reconfigurable teams.

Recommendation 4-9. AEE research and development should consider training and education requirements for undergraduate, graduate, and continuing education. The lessons learned from the development of advanced training simulators that incorporate AEE technologies and systems should be used to improve the training and educational capabilities of AEEs focused on other applications, including university education.

The new and different types of work teams that may be necessary for effective implementation of AEEs will, in some cases, highlight shortcomings in organizational structure, leadership, and management. AEEs will also accelerate design and engineering processes and reduce cycle times. Done properly, this will reduce costs and improve competitiveness. Done poorly, however, AEEs may simply speed the way to failure. Correcting organizational and process problems is a prerequisite for—not a result of—the successful implementation of AEEs. AEEs will not eliminate the need for leading, motivating, and inspiring employees, who will still be the most important part of the enterprise. This section describes economic considerations and management solutions for implementing AEEs, particularly with regard to start-up costs, uncertainties, and metrics.

Some commercial organizations are turning to AEE technologies to reduce costs and cycle times, increase customer satisfaction, and improve competitiveness. Government organizations are turning to AEE technologies to meet the needs of some challenging and costly missions. The importance of AEEs, however, goes far beyond the immediate goals of reducing the costs and schedules of current projects. The committee believes that the long-term viability of many companies will increasingly depend on their willingness and ability to embrace an AEE vision that looks beyond short-term gains. In fact, the expense and trouble of making the cultural and technological changes to implement AEE technologies often cannot be justified economically in the short term.

The economic risks of implementing AEEs are often high, but the committee believes that the risk of not implementing AEEs is increasing. In many applications, AEE technologies are already reducing costs, increasing customer satisfaction, and enabling new missions. Organizations that rely on traditional methods may be confronted by mission and product challenges they cannot meet, while AEE-enhanced competitors may be able to move forward. The committee believes that the increasing benefits of AEE technologies will ultimately promote their widespread use despite current barriers.

The growth of electronic commerce illustrates the importance of adapting to new methods. Electronic commerce is expanding every day to support the rapid, accurate transfer of funds among business, industry, government, and financial institutions. The growth of electronic commerce on the Internet is expected to continue as more consumers look to their PCs to purchase goods and services, although the profitability of many Internet-based businesses remains uncertain. For example, Amazon.com has been very successful in terms of market share, but it has yet to earn a profit.

All forms of electronic commerce are expected to continue to grow rapidly over the next decade, aided by technologies that ensure the security of transactions. These advances will reduce reliance on paper records and enable contracts to be negotiated, executed, and administered almost entirely in the virtual realm. The federal government has already established regional centers for electronic commerce to help small and midsize companies make the transition to electronic commerce.

The following start-up costs are associated with AEE technologies and systems:

• acquiring software licenses and hardware

• translating legacy data

• implementing effective product data management

• training for start-up and proficiency maintenance

• validating simulation models and analyses

• updating or replacing software and hardware

• overcoming hardware and software incompatibilities throughout the enterprise

• establishing and verifying metrics

These costs can be extraordinarily high, and the cost-time curve for projects developed with AEEs are usually very

FIGURE 4-3 Comparison of notional cost-time curves for projects using traditional processes and AEEs.

different from traditionally managed projects (see Figure 4-3 ). High start-up costs increase economic risk because the project may fail or the AEE infrastructure may turn out to be poorly suited to the project, potentially leaving the company with an expensive white elephant. Despite the expectation of long-term benefits, high start-up costs can also be a significant barrier to the approval of AEE projects, especially for companies with limited capital, government agencies with declining budgets, or organizations using AEEs for the first time.

Uncertainties about the impact of AEEs on costs and benefits over the life of a project also increase risk. Early attempts by industry or government organizations to implement AEE technologies have had limited success in the short term. The economic uncertainties and concerns of some AEE pioneers can be summarized as follows:

• poor near-term returns, although economic returns have been much better in the long term as technologies mature and expertise in their use grows

• high cost uncertainties associated with overcoming cultural barriers, gaining acceptance across the enterprise, and maintaining AEE systems in a rapidly evolving technological environment

• lack of experience with nontraditional cost-time curves for new product development and acquisition

• difficulty in separating the impact of AEEs from other efforts to lower costs, reduce cycle time, etc.

• uncertainty about the amount and timing of returns on investment, financially and in terms of improved quality, customer satisfaction, or mission effectiveness, for different products and processes

• difficulty of certifying that AEE simulations can reliably replace physical validation tests, especially for complex new projects and one-of-a-kind missions

• difficulty of identifying and analyzing low-frequency events that may be caused by cost-saving modifications to existing products and processes⁶

Uncertainties about the costs and benefits of using AEEs will persist until a set of metrics is available to quantify improvements. Precise measurements using traditional metrics, such as those listed in Table 4-1 , require well-defined data collected over long periods under carefully controlled conditions for similar programs or functions with and without AEEs. This type of data is rarely available. Most products and processes experience many changes over time, and many variables are at work when AEE technologies are being implemented. Isolating the effects of individual factors on cost or quality is usually impossible, especially if the benefits fall under different functions or time frames. For example, if the use of AEEs results in new designs that are more tolerant of variations in the manufacturing process, the primary benefits would not be in reduced engineering costs or even in lower production costs, but in higher quality and lower product returns. Managers need a comprehensive set of accurate, short-term, cost-justified metrics for predicting the effects of implementing AEEs.

Metrics related to risk are important because risk reduction is the focus of many technology development and demonstration programs and because perceived risk is a key parameter in management decision making. Different stakeholders may have different perceptions of the specific or overall risks associated with a particular product or mission, and risk estimates are not precise. Rather, levels of risk are estimates that are best characterized by probability distributions.

Affordability, in terms of a cost-to-benefits ratio, is another important metric, because AEEs will affect both the costs and performance of products and missions. To be comprehensive, affordability metrics should capture as many of the costs and benefits as possible, including intangible benefits, such as employee morale and customer goodwill. Although these factors are hard to pin down, they can be tracked through staff turnover rates, repeat customer business, and other indicators.

One approach for evaluating the effect of AEEs would be to establish project tracers in a large project. As described in the success stories in Chapter 2, Concepts ETI, Inc., has been able to measure the impact of AEEs on cost and time to market for a relatively narrow range of products (i.e., pumps, compressors, and other types of turbomachinery). The techniques used by Concepts ETI are not yet suited for tracking the impact of AEEs on large projects. However, the managers of a large project could take advantage of the lessons learned by Concepts ETI by designating specific project tasks that reflect the progress and level of success of the overall mission or product.

Project tracers should relate to established, semistandardized products or processes for which improvements can be measured in traditional terms. Project tracers should be limited in scope; rather than encompassing the entire project, they should provide a statistical sampling across the large project. For example, an automobile manufacturer might monitor the number of engineering change orders, manufacturing labor hours, and defect rate for left-front body fenders as an indication of how AEEs are affecting the design and development for other large sheet metal parts. Project tracers would have to be carefully selected to build confidence among managers and other decision makers that the tracers accurately indicate the effect of AEEs on the overall project.

Problems associated with economic uncertainties, high start-up costs, the lack of accurate metrics, and other economic and management barriers noted in the Phase 1 report (see Table B-1 in this report) add to risk. Setbacks and failures may occur as managerial, cultural, technological approaches to the implementation of AEEs evolve. Costs are likely to increase in the near term, and the return on investment may seem questionable. Nevertheless, although the economic risks of employing AEEs are high, the risk of continuing to rely on traditional methods is even higher. Organizations that do not move forward are likely to remain on the sidelines as others reap the rewards of an effective AEE culture and technology base. As described in the success stories in Chapter 2, AEE pioneers are achieving increasingly ambitious goals, reducing costs, increasing customer satisfaction, and enabling new missions. As demonstrated by Concepts ETI, even small design and manufacturing companies can take advantage of AEE technologies, if they have sufficient technological expertise. AEE technologies that can be easily adapted to a variety of applications would facilitate the use of AEEs by other small companies with less technological expertise.

Advanced technologies can help organizations overcome economic and management barriers. As discussed above, Internet-based AEE technologies, systems based on open architectures, and other technological advances can improve the interoperability and composability of software, as well as strengthen engineering education programs. These changes will produce a more capable information technology workforce and improve the interoperability of tools used by prime manufacturers and their suppliers. AEE technologies will also improve data transmission among users of tools created by different organizations, increase information security, and lower cultural barriers. Nevertheless, approaches for implementing AEEs that are limited to technological changes will not succeed. Management priorities must be realigned to accomplish the following tasks:

• Fund sustained, interdisciplinary research and development, which are critical to the future of AEEs.

• Nurture collaborative approaches to complex problems.

• Reduce the amount of specialized training required to operate ngineering systems and make that training readily available.

• Reward workers for participating in both short-term and long-term collaborations.

• Efficiently implement complex teaming arrangements.

• Quickly assess complex problems and produce easily understandable information, instead of overloading workers and managers with enigmatic data.

• Balance users' needs for easy access and high security.

• Maintain corporate knowledge of products and processes regardless of staff turnover, which is especially high in the information technology workforce.

Creating a work environment that accomplishes these tasks will require (1) innovative management to ensure that complex AEE systems are applied in an orderly way; (2) a flexible management approach that rewards team collaboration and is willing to accept risks to reach important long-term goals; and (3) a strong organizational commitment that includes buy-in by key managers. Commitment is essential

to sustain changes long enough to work through start-up problems and assess long-term benefits. Strong leadership is needed to maintain momentum and focus, especially for electronically linked project teams that span continents, time zones, and diverse cultures or corporate philosophies. Striking an appropriate balance between the tight management control possible with traditional teams and the freedom associated with loosely regulated cooperative efforts (such as development of the Internet or the Linux operating system) may be difficult. However, balance is essential for realizing the benefits of the dynamic interactions that AEEs enable. At the same time, management must retain enough focus to achieve the goals of specific products or missions.

Finding 4-5. The costs of implementing AEEs may not be justifiable in the short term. However, pioneers in the use of AEE technologies are realizing economic benefits. As AEE technologies become more common and sophisticated, the long-term viability of most commercial design and manufacturing companies and other complex, technical enterprises will increasingly depend on their ability to implement AEE technologies and systems of increasing sophistication.

Recommendation 4-10. Because of the technical and social complexities involved in applying AEEs, efforts to implement them should include the following:

• processes for taking advantage of the lessons learned by AEE pioneers, especially with regard to the reduction of implementation costs, uncertainties, and risks

• realistic goals for the economic payoffs of implementing AEEs, especially the time needed to realize a positive return on investment

• innovative and determined management that is willing to accept risks; appoint a “champion” with broad, interdisciplinary authority; persevere despite temporary setbacks; and accept uncertainty in assessments of the cost and benefit of implementing AEE technologies

• Web-based AEE technologies with open architectures and improved interoperability and composability to reduce implementation costs

ACM (Association for Computing Machinery) . 2000 . Home page for the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , Boston, Mass. , August 20–23, 2000 . Available on line at: http://www.acm.org/sigs/sigkdd/kdd2000/ May 9, 2000 .

Andrews, Dee . 1999 . Simulated Battlefields . Briefing by Dee Andrews, Air Force Research Laboratory, to the Committee on Advanced Engineering Environments , RAND Corporation , Santa Monica, California , August 18, 1999 .

Boyer, Ernest . 1990 . Scholarship Reconsidered: Priorities of the Professorate . Princeton, N.J. : Princeton University Press .

CRA (Computing Research Advocacy) . 2000 . CRA Government Affairs home page . Available on line at: http://cra.org/Policy/it2.html and http://www.nsf.gov/od/lpa/news/media/99/fsnsf_it.htm May 9, 2000 .

CSAB (Computing Sciences Accreditation Board) . 2000 . CSAC (Computing Sciences Accreditation Board) Integration into ABET (Accreditation Board for Engineering and Technology) . Available on line at: http://www.csab.org/ July 5, 2000 .

Derrick, Samuel . 1930 . Centennial History of the South Carolina Railroad . Columbia, S.C. : The State Company .

Dewan, Prasun . 1999 . Research in Collaboration Environments: A Technical Perspective . Briefing by Prasun Dewan, University of North Carolina, to the Committee on Advanced Engineering Environments , RAND Corporation , Santa Monica, California , August 18, 1999 .

Goldberg, Steve . 1999 . Team Training in Virtual Environments . Briefing by Steve Goldberg, Army Research Institute, to the Committee on Advanced Engineering Environments , RAND Corporation , Santa Monica, California , August 18, 1999 .

Gordon, Daniel . 2000 . Diverse Geospatial Information Integration . Briefing by Daniel Gordon, Autometric, Inc., to Data for Science and Society, the Second National Conference on Scientific and Technical Data , National Academy of Sciences , Washington, D.C. , March 13–14, 2000 . Proceedings to be published on line at: http://www4.nationalacademies.org/oia/codata.nsf/web/meetings?OpenDocument

Granovetter, Mark . 1978 . The strength of weak ties . American Journal of Sociology 78(6) :1360–1380 .

Harris, Kenneth , ed. 1998 . Jane's World Railways , 1998–1999 . London, U.K. : Jane's Yearbooks .

Internet-2 . 2000a . Internet-2 home page . Available on line at: http://www.internet2.org/ May 9, 2000 .

Internet-2 . 2000b . Internet2 and the Next Generation Internet (NGI) . Available on line at: http://www.internet2.edu/html/ngi.html#about_ngi/ May 9, 2000 .

IUGREEE (Industry, University, and Government Roundtable for Enhancing Engineering Education) . IUGREEE home page . Available on line at: http://www.eng.iastate.edu/iugreee/ July 13, 2000 .

Kluwer Academic Publishers . 2000 . Data Mining and Knowledge Discovery: Journal home page . Available on line at: http://www.wkap.nl/journalhome.htm/1384-5810 May 9, 2000.

Margary, Ivan . 1973 . Roman Roads in Britain . London, U.K. : John Baker, Ltd .

Moniz, Ernest . 1999 . Shaping the Research Agenda , presented at the Federal Government/Research University Workshop, Purdue University, November 4, 1999 . Available on line at: http://www.cic.uiuc.edu/OSTP/OSTPreport.htm March 30, 2000 .

NASA (National Aeronautics and Space Administration) . 1988 . NSTS (National Space Transportation System) 1988 News Reference Manual . Solid Rocket Boosters . Kennedy Space Center, Fla. : National Aeronautics and Space Administration . Available on line at: http://www.ksc.nasa.gov/shuttle/technology/sts-newsref/srb.html#srb May 5, 2000 .

NCO (National Coordination Office for Computing, Information, and Communications) . 1999 . NCO News . Special Edition–SC 99 . November 1999 . Arlington, Va. : National Coordination Office for Computing, Information, and Communications . Available on line at: http://www.hpcc.gov/highlights/index.html#testimony-1mar00 March 19, 2000 .

NGI (Center for Next Generation Internet) . 2000 . Internet Trends . Available on line at: http://www.ngi.org/trends.htm May 9, 2000 .

NRC (National Research Council) . 1997a . Ada and Beyond: Software Policies for the Department of Defense . Washington, D.C. : National Academy Press . Available on line at: http://books.nap.edu/catalog/5463.html June 23, 2000 .

NRC . 1997b . Modeling and Simulation: Linking Entertainment and Defense . Washington, D.C. : National Academy Press . Available on line at: http://books.nap.edu/catalog/5830.html June 2, 2000 .

NRC . 1999 . Trust in Cyperspace . Washington, D.C. : National Academy Press . Available on line at: http://www.nap.edu/catalog/6161.html June 13, 2000 .

NSF (National Science Foundation) . 2000a . Engineering Research Centers Program

. Available on line at: http://www.eng.nsf.gov/eec/erc.htm March 30, 2000 .

NSF . 2000b . National Science Foundation Engineering Education Coalitions . Available on line at: http://www.smete.org/coalitions/ March 30, 2000 .

Nua, Ltd . 2000 . Nua Internet Surveys home page . Available on line at: http://www.nua.ie/surveys/how_many_online/index.html May 9, 2000 .

Raymond, Eric . 1999 . The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary . Sebastopol, Calif. : O'Reilly and Associates .

Restivo, James . 2000 . Experience with Metadata on the Internet . Briefing by James Restivo, Blue Angel Technologies, to Data for Science and Society, the Second National Conference on Scientific and Technical Data, National Academy of Sciences, Washington, D.C., March 13–14, 2000. Proceedings to be published on line at: http://www4.nationalacademies.org/oia/codata.nsf/web/meetings?OpenDocument

Shupe, Timothy . Personal communication from Timothy Shupe , senior logistics analyst, Thiokol Propulsion Division of Cordant Technologies, to Douglas H. Bennett, research associate , Aeronautics and Space Engineering Board, National Research Council , January 4, 2000 .

Singhal, Sandeep , and Michael Zyda . 1999 . Networked Virtual Environments: Design and Implementation . New York, N.Y. : ACM Press Books .

Vance, James . 1995 . The North American Railroad: Its Origins, Evolution, and Geography . Baltimore, Md. : Johns Hopkins University Press .

vBNS (very high performance Backbone Network Service) . 2000 . vBNS home page . Available on line at: http://www.vbns.net:8080/ May 9, 2000 .

Von Hagen, Victor . 1967 . The Roads That Led to Rome . London, U.K. : Weidenfeld and Nicolson, Ltd .