4
Technology Transition and Program Management: Bridging the Gap Between Research and Impact

Researchers, subject-matter experts, research program managers, and operational line managers participating in the workshops convened by the committee all pointed to the challenge of achieving actual impact on mission performance from research programs. Even under ideal conditions—when the research results are peer-recognized to be of high quality, when the user agency has a solid understanding of operational requirements, and when there is strong institutional incentive on both sides to accomplish the transition—success in bridging the gap between the creation of promising research results and the realization of effective impact may nonetheless be elusive. This chapter examines how systematic approaches can help research managers and user organizations collaborate to bridge the gap between conception and use.

The gap-bridging metaphor should not be taken to imply that the path from concept to product is linear or predictable. Indeed, many major IT innovations have followed paths with duration, complexity, and diversity of players rarely foreseen by the original innovators. This phenomenon was well illustrated in the 1995 CSTB report on the High Performance Computing and Communications Initiative,1 which offered a

1  

Computer Science and Telecommunications Board (CSTB), National Research Council (NRC). 1995. Evolving the High Performance Computing and Communications Initiative to Support the Nation’s Information Infrastructure. National Academy Press, Washington, D.C.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 88
Information Technology Research, Innovation, and E-Government 4 Technology Transition and Program Management: Bridging the Gap Between Research and Impact Researchers, subject-matter experts, research program managers, and operational line managers participating in the workshops convened by the committee all pointed to the challenge of achieving actual impact on mission performance from research programs. Even under ideal conditions—when the research results are peer-recognized to be of high quality, when the user agency has a solid understanding of operational requirements, and when there is strong institutional incentive on both sides to accomplish the transition—success in bridging the gap between the creation of promising research results and the realization of effective impact may nonetheless be elusive. This chapter examines how systematic approaches can help research managers and user organizations collaborate to bridge the gap between conception and use. The gap-bridging metaphor should not be taken to imply that the path from concept to product is linear or predictable. Indeed, many major IT innovations have followed paths with duration, complexity, and diversity of players rarely foreseen by the original innovators. This phenomenon was well illustrated in the 1995 CSTB report on the High Performance Computing and Communications Initiative,1 which offered a 1   Computer Science and Telecommunications Board (CSTB), National Research Council (NRC). 1995. Evolving the High Performance Computing and Communications Initiative to Support the Nation’s Information Infrastructure. National Academy Press, Washington, D.C.

OCR for page 88
Information Technology Research, Innovation, and E-Government number of examples of well-established multibillion-dollar IT businesses emerging only after many years of research and development, with shifting roles of government-sponsored research, industry research, and industry development. Business areas considered in that report, nearly all of which continue to remain pivotal in the IT economy, include graphics and windows, redundant arrays of inexpensive disks (RAID), very-large-scale integrated circuit (VLSI) design, and reduced instruction set computing (RISC) processors (for an updated display of the links between research and major IT industries, see Figure 4.1). In each case, there was sustained government research participation, including both formative exploratory research and more focused attention to particular technical challenges. (Of course, not all research has this sort of outcome—some successful research has far-reaching socioeconomic or mission impact but nonetheless does not lead to billion-dollar industries furnishing products or services.) From workshop discussions and other interactions of committee members with research-program managers, it is abundantly clear that government agencies that sponsor major IT research programs—such as NSF, DARPA and military service laboratories, NASA, DOE, and the National Institutes of Health (NIH)—have all evolved diverse cultures of program-management practice. Programs are managed according to management models and practices that are embedded in organizational culture and that may be highly evolved. For example, the DOD has a rich structure of technology-transition activities involving diverse contractors that enable it to hasten the maturing of critical defense technologies along the path from laboratory to operational deployment. This structure can be understood, for example, as a systematic addressing of the many risk issues (see the section “Dimensions of Risk,” below) that arise along this path. There are significant differences in management style and approach among the various mission agencies and programs. (Box 4.1 describes two agency programs in more detail.) In traditional basic research programs, research teams often operate independently of any particular end user. The NSF’s current Digital Government program, however, represents an important first step in having researchers collaborate directly with potential end users. This kind of direct engagement enables researchers to understand requirements better, validate concepts earlier, and accelerate transition into practice. It also enables potential users to anticipate—and influence—emerging technologies. But regardless of whether an end user is present or not, transition issues must be explicitly addressed, and obtaining an impact in practice requires careful and flexible management by many parties. The teaming structure of the NSF Digital Government program can accelerate this process, but it does not replace it. The right combination of careful manage-

OCR for page 88
Information Technology Research, Innovation, and E-Government FIGURE 4.1 Examples of government-sponsored IT research and development in the creation of commercial products and industries. SOURCE: 2002 update by the Computer Science and Telecommunications Board of a figure originally published in Computer Science and Telecommunications Board, National Research

OCR for page 88
Information Technology Research, Innovation, and E-Government Council, 1995, Evolving the High Performance Computing and Communications Initiative to Support the Nation’s Information Infrastructure, National Academy Press, Washington, D.C.

OCR for page 88
Information Technology Research, Innovation, and E-Government Box 4.1 Research Cultures at DARPA and NSF Defense Advanced Research Projects Agency For many years there has been tacit acceptance within the R&D community of DARPA’s primary national role in certain IT areas critical to DOD in the long term. These include packet-switched networking, distributed computing, machine intelligence, software-reliability technology, and computer security. In each of these areas, program managers crafted engagements linking diverse participants in the research community, industry, and DOD according to principles that had been evolved through practice over several decades. This national leadership role enabled DOD to command the attention of the computer science research community, and indeed many computer scientists in universities have come to describe their research in military terms such as survivability, decision support, and situation awareness. Since the early 1990s, the strategic environment has been changing rapidly, prompted by the emergence of the private sector Internet, increasingly widespread computer-security challenges, greater military emphasis on asymmetric and coalition warfighting, broadening of the base of strong IT research universities, and a shift in the locus of innovation for many information technologies. Research and acquisition managers have attempted to adapt their strategies in response to these changes (with varying degrees of success). National Science Foundation At NSF, the principal criteria for research support are scientific quality and long-term socioeconomic impact. Perhaps unique to NSF is the emphasis on development of scientific foundations and a long time horizon, though other agencies, such as the DOD’s Office of Naval Research and the Army Research Office, also accept a long time horizon when they consider a potential mission impact. In recent years, NSF has increased its emphasis on broad socioeconomic impact yet further, coinciding with the acknowledgment in the policy and legislative community of the emerging pivotal role of information technology in the national infrastructure and in society broadly. This is also a reflection of the increased attention to measuring the effectiveness of government programs. At the same time that it seeks to better identify the socioeconomic impacts of the research it sponsors, NSF continues to recognize that breakthroughs in “curiosity-driven” basic research in fundamental areas can have unexpected impacts in practice, and it maintains a broad portfolio of basic research in information technology subjects. ment and favorable and even fortuitous circumstances may be needed in the operating environment of the recipient organization and in the markets for particular underlying technologies. Complicating this, the time scales can be very long—a decade or more in many cases—possibly well beyond the life span of individual research projects. The NSF’s Digital Government program illustrates that there is a natu-

OCR for page 88
Information Technology Research, Innovation, and E-Government ral alliance between researchers and end users. Both groups seek to identify new ways to address fundamental challenges in capability by stimulating new concepts and accelerating their definition and development. Researchers gain access to challenging problems and a real-world context in which to explore them. End users who participate in this process gain insights into technology opportunities and thus are better able to influence their future and accelerate the introduction of new ideas into their production systems. Despite this alliance, however, the transition from new concepts and prototypes to production systems and market acceptance can be lengthy and difficult. Below are examples of questions that might be raised by a potential technology adopter in considering a promising new information technology. These questions, which apply regardless of whether the technology has been developed within the government, at a university, or by a commercial contractor, illustrate how the challenges of transition go beyond identifying functional requirements and solving technical problems: Can the technology be effectively integrated into existing operations and procedures? Can it be packaged to interconnect with related systems? Does it comply with the interface standards that are adopted within the organization or by the integrators responsible for other major systems used by the organization? Will it be a cost-effective way of meeting requirements—in terms of both initial acquisition and total life-cycle costs? Are there fundamental difficulties in adjusting the human interfaces in a concept-demonstration system to conform better with organizational culture and practices? Can the new technology scale up appropriately to expected and potential operational levels? What are the dimensions of scaling, and how can testing and evaluation be conducted to understand the limitations of capacity and performance of the concept? What re-engineering is needed with respect to the prototype concept-demonstration system? Are certain capacity limits expediently “hard-wired” into the prototype, for example, in order to simplify implementation? What are the dimensions of dependability and robustness that must be considered in adopting the prototype system? Will it respond appropriately, for example, to operator errors or misconfiguration? Can the prototype system handle the full range of categories of inputs, or does it handle only a representative set? Are there hidden scaling challenges? Has the prototype been designed to support possible trajectories of evolution in the requirements it is addressing, in the systems environ-

OCR for page 88
Information Technology Research, Innovation, and E-Government ment in which it is situated, in the underlying technologies, and in the culture and mission of the organization in which it is used? Who will be available to respond to questions and provide technical support when problems arise or when adaptation is needed? What assurances can be provided to enable an organizational commitment to this new technology? From the standpoint of the researchers exploring a new algorithm or systems concept, these questions may appear to be significantly less interesting than the more fundamental issues related to overall feasibility of the technical approach. This amounts to a division of both expertise and responsibility: researchers and their sponsors may focus their initial effort on the overall “concept” or the new algorithmic (or other) idea. But part of the price of success is that once these core conceptual issues are addressed, a diversity of other kinds of expertise must be applied to address adoption-related challenges such as those enumerated above. A successful transition process must address a wide range of concerns beyond the “merely” technical—including economic and market issues, intellectual property and nonappropriability, standards, the “tipping” of the market in favor of a particular product or standard, organizational issues, and engineering and usability issues. At the committee’s workshops, in the extensive discussions about the development, evaluation, and transition of new technologies to practice, participants particularly noted the risks and challenges of scaling and integrating new capabilities. But perhaps the most important lesson learned from those discussions is that transition cannot occur without broad knowledge of the technological and operational environments, including the extent to which those environments are affected by changing market conditions, standards, and other external forces. STRATEGIES AND MODELS FOR PROGRAM MANAGEMENT How can researchers, research managers, acquisition managers, and users develop successful strategies in this complex and changing environment? The sections below present an approach that is based on three principal elements: Consideration and analysis of past programs, including both positive and negative experiences; Use of program-management models, which facilitate comparative consideration of alternative approaches; and Identification of proven program-management strategies by which

OCR for page 88
Information Technology Research, Innovation, and E-Government management decisions can maximize the opportunity for impact with acceptable risk and time horizons. This chaper discusses two models2 in conventional use in program management. Also enumerated are several examples of strategies employed by program managers in implementing programs to achieve particular kinds of goals. (The emphasis here is on long-term programs seeking a focused impact; approaches to managing purely exploratory programs are not within the scope of this document.) Models play an informal but essential role in developing and managing successful long-term research programs. Little real data on research-program management exist, which means that managers generally plan and decide largely on the basis of personal experience and organizational culture. The models identified here are intended to organize and express concepts that are generally already present in the culture of program management in various agencies and organizations. The following two models are widely used, often in a tacit way, by research-program managers in articulating, developing, and managing their programs. The supply chain, or technology food chain, is an identification of stakeholders—and the relationships between them—involved in transforming a new technical idea into operational reality (see Box 4.2). Because this model is useful in understanding the respective roles and interests of stakeholders in the process of developing and delivering capability, it can help guide a program manager in identifying potential partners in an innovation process. The dimensions of risk are categories of engineering or market strategy issues that a program manager must address as a new technology or concept moves through the supply chain from an initial innovative 2   Generally speaking, a model is a conceptual structure used to understand and predict phenomena that may be difficult to observe and influence directly. Models are abstract, in the sense that they omit details in order to facilitate expression and understanding of aspects of the phenomenon. Good models are expressive, in that they capture knowledge that cannot easily be gained by experience except perhaps at great expense or danger or over long periods of time. The validity of a model is the extent to which it has predictive power in the domains for which it is designed. Engineers, managers, and others may use a diversity of models in a design process in order to understand the various aspects of a problem, the design issues, and the potential consequences of particular design decisions. Qualitative and informal models, such as those summarized here, may be less predictive than a validated scientific model. The validation of these informal management models is necessarily informally based, and indeed all variables may not be identified. But for many disciplines of endeavor, including research-program management, this is the best that can be achieved with present levels of understanding.

OCR for page 88
Information Technology Research, Innovation, and E-Government Box 4.2 The Innovation Pipeline It is well understood that innovation does not follow a neat, linear progression from research laboratory to delivered product; rather, innovation involves complex flows of ideas and people among the various organizational participants. Nonetheless, it is often useful to view IT research, development, and application as constituting an innovation pipeline that includes the following: Academic and industry researchers develop new concepts and technologies. Exploratory efforts made in partially realistic settings by industry, academia, and government agencies co-develop concepts and technologies to identify promising new solutions that combine innovations in technology and business practices. Vendors and tool suppliers develop software and hardware components and provide them to system integrators or to the agencies directly. System integrators, vertical suppliers, and vendors respond to requests for proposals (RFPs) and build (and sometimes operate) systems for government agencies. Government agencies develop and modify programs, issue RFPs for information systems, select vendors, procure IT systems and services from them, and operate IT systems. This is a normative model for incorporating new technologies and practices into government systems and for improving services. As is discussed in this chapter, the interests and incentives of the actors in this process are not in reality always well aligned, which leads to a set of challenges for mission agencies and research-program managers. idea to deployment in systems. Engineering issues include, for example, the scalability of a component or system, usability and adoption issues, and robustness of internal interfaces. Market strategy issues include, for example, whether to focus on adaptation of existing components to provide some new capability or to stimulate a new component market that packages the capability. The model for budget management used in the DOD for research, development, test, and evaluation addresses the dimensions of risk explicitly by categorizing funds used for programs (e.g., 6.1, 6.2, 6.3a, and 6.5) according to the character of the risk being addressed. The following sections use these two models to elucidate a number of program-management strategies and tactics.

OCR for page 88
Information Technology Research, Innovation, and E-Government LEVERAGE IN THE SUPPLY CHAIN MODEL Government IT systems are acquired in several different ways. If off-the-shelf components (see Box 4.3) such as operating systems, network components, and office-productivity tools can be used, they are obtained from IT vendors. If custom engineering is necessary (to fill in where off-the-shelf components are lacking), contracts are typically let with large commercial systems integrators. Integrators themselves often have incentives to incorporate off-the-shelf components into systems, thereby limiting the extent of new engineering. Larger mission agencies such as DOD, DOE, NASA, and NIH do some of this custom work in their own, internal engineering organizations and affiliated laboratories. In some cases, the custom engineering can be replaced by using vertically tailored products from domain specialists (for example, geographic information systems). Finally, in many government agencies, “seat management” approaches are used in order to provide federal workers with a base level of common, largely off-the-shelf services (desktop computer, office automation and other basic software, network connectivity, and support).3 These provide both a core set of “productivity” tools and a common platform for custom and vertical applications. In any case, the collaboration to develop new agency systems involves more than the immediate integrators and suppliers, because these organizations themselves obtain components and technologies from external sources. Each of the acquisition paths noted above involves a wide range of players—including hardware and software component vendors, systems integrators, niche technology providers, vertical-solution developers, government research-program managers, government acquisition-program offices, acquisition regulators, standards organizations, and other nongovernment customers—that have diverse capabilities and interests. Most integrators maintain significant relationships with vendors in order to assure a future supply of critical technology components that can address likely needs. A mission control system developed for NASA Space Science, for example, may include off-the-shelf commercial hardware in the form of workstations, servers, and network appliances, and commercial software components such as operating systems, servers, database systems, file systems, graphical toolkits, third-party domain-specific (or “vertical”) tools, and networking components. An integrator will develop custom software components and “glue code” to provide the aggregate functionality and enable a flexible integration of components. 3   See <http://www.gsa.gov/seatmanagement/>.

OCR for page 88
Information Technology Research, Innovation, and E-Government Box 4.3 Off-the-shelf Components Very few systems now being developed are entirely custom-made. As their dominance of the IT arena diminished, federal agencies recognized that they could no longer afford to have vendors design and build systems “from scratch.” This led to the notion of purchasing systems fabricated from commercial hardware and software—so-called commercial off-the-shelf (COTS) technology. The goal with COTS is to employ standard, widely used hardware and software products wherever possible so as to decrease costs and increase the likelihood that systems will be interoperable with other systems. The COTS strategy must grapple with a basic tension: even as government seeks through COTS to obtain a greater degree of flexibility that enables future vendor choice, vendors have an incentive to maximize the proprietary content of a system in order to increase the likelihood of future sales. Agencies thus face the dual issue of selecting an appropriate framework/architecture and managing the consequences of that commitment. In other words, once an organization has made decisions about overall system design, this constrains future decisions and limits which software products and vendor product lines will be compatible. Recent legislation and regulation have created strong incentives for acquisition program managers to aggressively incorporate off-the-shelf (OTS) components as well as to support iterative acquisition models, prototyping, product lines, and other processes. Such incentives are appropriate in an environment of rapid technological change, shifting requirements, uncertainties in the life cycle, and demanding interoperation requirements. Use of OTS technologies can improve affordability, reduce training requirements, improve dependability and interoperability, and enable the acquiring organization to “ride the curves” of performance growth (e.g., Moore’s law) over the life span of the system. OTS components can be from commercial sources (COTS), government sources (GOTS), or open sources (no acronym yet). On the other hand, the acquisition organization may find that it must sacrifice some control in the engineering process in order to accommodate vendor products. In particular, adoption of OTS components may force slight shifts in functionality that must be accommodated in requirements engineering and validation. In addition, there can be significant risk in verification and validation, since the acquiring organization may not have full visibility into the design or even functionality of the component. The acquiring organization also does not control the trajectory of growth for OTS components, and may indeed have little opportunity to influence vendors to accommodate special needs, given the breadth of their customer communities. Achieving a suitable balance in the adoption of OTS components can be a significant challenge, especially because the benefits of adopting OTS components may be harder to measure (though they may be greater) than the benefits of increased control.

OCR for page 88
Information Technology Research, Innovation, and E-Government sures. Indeed, one management strategy often used is to challenge researchers to identify appropriate measures of their own progress as part of the research effort, and to apply those measures on an ongoing basis to their own and other work. This can include, of course, retrospective analysis of similar projects in order to identify correlates with various aspects of success. Benchmarking. In the absence of direct measures, especially in problem spaces of unknown dimensionality, benchmark data sets are useful surrogates. This approach has been successfully used in areas such as high-performance computing, speech recognition, image understanding, and text management. Well-designed benchmark data sets can help reveal the particular elements of capability within a set of “competing” technologies. Different approaches to a problem—speech recognition, for example—may excel in different respects, such as the handling of ambient noise, use of free microphones, speaker independence, extent of system training, and limits on vocabulary and grammatical complexity. This identification assists potential downstream adopters because it enables them to identify relevant evaluation criteria better. Testbed. Many research programs focused on achieving realism or scale-up along various dimensions create testbed facilities for conducting larger-scale experiments. “Scale,” here, can refer, for example, to the size of a database, the number and skills of users, the extent of distribution or interconnectivity, robustness, performance, and other factors. Further, by creating a facility shared by multiple research efforts, costs can also be shared and experimental results more easily compared and evaluated. In addition, potential users can participate in the testbed definition and evaluation in order to ensure realism. This approach is widely used in major programs (e.g., DARPA Gigabit Networking and DOD Joint Warrior Interoperability Demonstration programs). The principal risk associated with the approach is defining the testbed project in a way that achieves realism while enabling research issues to be addressed effectively. Gentle-slope adoptability. One risk-mitigation approach is to design programs that offer a “gentle slope” with regard to return on investment—that is, increments of effort in adopting a technology yield increments of impact. This approach (named by Michael Dertouzos) can enable researchers working with early adopters both to make midcourse corrections and to collaborate with the adopters in defining evaluation criteria.8 The resulting “early validation” can help reduce evaluation risk even when objective measures are unavailable. 8   The idea first appeared in an unpublished DARPA study in which Dertouzos wrote, “The gentle-slope systems will have a few key properties. First and foremost, they will give incrementally more useful results for incrementally greater effort.”

OCR for page 88
Information Technology Research, Innovation, and E-Government Solution-Concept Risk Another risk issue faced by managers of a problem-directed research program is the extent to which the program should commit to particular solution approaches. Such commitments could include choices for system architecture (e.g., business-rule framework in a three-level architecture), measurement frameworks, technical approaches (e.g., neural networks versus other decision frameworks), or infrastructural components (e.g., operating-system choice). Solution-concept risk is an issue, for example, in areas such as improvement of information security, text-search capability, capability to integrate databases, or support for distributed teamwork in some operational domain. Underconstraint of solution approach can lead to solutions that are incompatible with the target environment. Overconstraint, on the other hand, can exclude development of innovative out-of-the-box approaches—such as RAID storage architecture to complement development of larger and denser disks—that may offer superior capability. When early projects were initiated in parallel computing, for example, a diversity of architectural approaches were considered, which yielded a number of different options that permit a number of different classes of problems to be tackled today. One example of solution-concept risk in the domain of digital government is the development of technologies to model and remediate unwanted linking of databases (e.g., to prevent revealing identities of medical-research subjects). Solution approaches could include, for example, development of large-scale meta-models of released information, data-dithering techniques, and creation of surrogates for zip codes as geographic identifiers. Premature commitment to any one of these could increase risk. Computer-security research faces a similar challenge—it can be easy for a program to overcommit to particular threat models or security architectures and thereby be deflected from critical technical areas and a diverse portfolio. Strategies for addressing solution-concept risk include these: Mixed strategy. This approach for exploratory programs is to adopt a diverse portfolio of technical approaches. While it is an obvious response, how to manage it for success is not so obvious, primarily because of evaluation risk. That is, a more constrained strategy may reduce within-program evaluation risk but increase the overall evaluation risk with respect to the intended impact. Iceberg model. Diversity enables individual efforts to accept greater risk while not substantially increasing overall program risk. If there is an expectation that not all research efforts will yield impact—if local failures can be tolerated—then greater risks can be taken, leading potentially to greater levels of innovation. A supporting organizational culture is pre-

OCR for page 88
Information Technology Research, Innovation, and E-Government requisite to success in adopting this model. DARPA and NSF, for example, have followed this model in certain critical technical areas, with considerable success. These agencies understand that too-aggressive management at this stage could inhibit the exploratory nature of this activity and prematurely eliminate promising approaches. This approach may be adopted by researchers as well as funding agencies: a principal investigator may consider many possible approaches to a problem before selecting a subset for more focused effort. This approach can be applied iteratively, following a “progressive deepening” strategy. Avoid overmanagement. Project managers may be tempted to constrain solution concepts prematurely in order to obtain a more linear program execution model with predictable milestones. This optimization in favor of near-term predictability may actually increase overall solution concept risk because it compresses or eliminates the exploratory phases critical to the invention or identification of new solution concepts. Unless a solution concept is clearly understood in advance, program managers need to balance visibility and predictability (and hence measurability) with the fostering of exploratory and creative activity. Problem-Concept Risk Often the perceived gaps in capability are not the actual gaps. An organization may identify as a technology challenge what is in fact a challenge in process, organizational structure, or technological-legacy management. That is, risk is associated with the correct identification of requirements to be addressed (validation). For example, when designing a system to facilitate information sharing in an organization, the real challenge may in fact derive from organizational culture rather than from technological impediment. Or, an effective solution may need to address organizational culture factors and technological factors. Sometimes the difficulty is not in requirements elicitation but rather in fluidity of requirements. Consider, for example, the design of command-and-control systems for use in crisis management or military operations. There are important interactions between the capabilities of available communication and collaboration technologies and the processes and physical organization of command posts. If aggressive innovation is sought in command-and-control capability, constraining the technical focus to address communications capabilities within existing organizational and physical structures may fail to deliver sufficiently innovative results and may indeed reinforce older, less-effective practices. On the other hand, an overly broad focus may not yield meaningful results. Identifying the right scope of research focus is thus pivotal in effective program design.

OCR for page 88
Information Technology Research, Innovation, and E-Government Examples of areas of potential problem-concept risk in digital-government programs include new command-and-control capabilities for crisis management, statistical-analysis support for nonexpert users of government statistical data, and database interoperation support for rapidly linking geographical databases to support crisis response. Strategies for addressing problem-concept risk include these: Wizard of Oz. Even when a research challenge is appropriately scoped, there can be significant difficulties in designing a program that enables rapid and graceful coevolution of technology and its context of use. In particular, naÏve iterative approaches can yield, at best, long validation cycles and, at worst, divergence from goals. The so-called Wizard of Oz approach involves the creation of system mock-ups that do not necessarily embody real capability but that can nonetheless enable potential adopters to experiment with concepts of operation. The early validation received from experiences with the mock-ups can reduce problem-concept risk and focus technology development efforts. The mock-ups also help researchers communicate with potential adopters by providing a concrete instantiation of concepts; where appropriate, they may also serve as surrogates for formal specification of requirements or technological capabilities. The Wizard of Oz tactic thus also addresses risks related to evaluation, integration, and usability. Double Helix. As potential users gain experience with new capabilities, the problem concept may shift in response to that experience. For example, the use of collaborative-filtering technologies has had significant effects on concepts of operation for electronic-commerce sites. In the Double Helix approach, a coevolution of operational concepts and technology concepts occurs. Close collaboration of technical researchers and subject-matter experts may, after several iterations, yield simultaneous innovation in both operational concept and underlying technology. Also, validation can be enhanced through ongoing empirical assessment of such coevolution. This approach has been applied with some success in the DARPA Command Post of the Future (CPOF) program, particularly in identifying and evaluating new approaches to visualization and collaboration in midechelon military command posts. Integration and Adoption Risk Can the technologies developed within a program be effectively integrated into an intended systems or organizational context? Answering this question requires consideration of issues ranging from interoperation to usability. For example, wearable computer systems used in systems-maintenance applications need to be integrated with evolving practices

OCR for page 88
Information Technology Research, Innovation, and E-Government and systems for documentation management and maintenance reporting. This could be called “business process adaptation risk.” While certain adoption risks must be addressed early in a research effort, addressing too many of them too early can multiply risk and impede accomplishment. Integration risk is a significant factor for many (and perhaps most) digital-government research projects. Many new capabilities must be integrated with evolving commercial components as well as with custom-integrated government IT solutions. In other words, both the target environment and the base technological infrastructure are rapidly moving targets. Many of the strategies mentioned above are used to address adoption risks. In addition, the following specific strategies apply: Pipeline. The pipeline model, often used in long-lived research programs focused on strategic challenges, embraces a portfolio of research efforts that span a range of levels of technological maturity. Achieving that broad span may entail working with multiple stages of the innovation supply chain—basic researchers, exploratory-development teams, companies developing new products, and companies integrating systems, among others. This approach was used with success in the DARPA Strategic Computing program in the 1980s. The program followed a “pyramid” model that encompassed simultaneous work on generic enabling technologies, multipurpose applications infrastructure, and aggressive experimental applications. Producer-consumer connection. One of the features of the multiagency High Performance Computing and Communications program (HPCC) of the 1990s was the direct connection it established between computational scientists and engineers and the developers of advanced high-performance computing and communications systems. The connection enabled system developers to get early validation and feedback from users (see the subsection “Evaluation Risk,” above), and it enabled the users to gain experience and assess scalability issues for emerging platforms. The program accomplished this by seeding the market—“buying down risk” on both sides, by assisting users in acquiring aggressive systems and assisting developers in maturing concepts so that they could sustain external evaluation. This is analogous to, but not the same as, the alignment of end users and innovators in mission programs, including NSF’s Digital Government program. The distinction is that the producer-consumer model is focused on building links in the supply chain, while the NSF Digital Government program emphasizes directly connecting the producer and consumer. Producer-consumer connection issues also arise when organiza-

OCR for page 88
Information Technology Research, Innovation, and E-Government Box 4.5 Using a Clearinghouse to Assimilate New IT in Crisis Response How can the introduction and use of new technologies in the immediate response to a crisis be managed so as to maximize their availability and better use the talents of emergency staff? One idea that emerged from the committee’s consideration of crisis management is to attach a clearinghouse function to emergency operations centers that are generally established to respond to crisis situations. A clearinghouse would help systematize the introduction of technology into crisis response activities (an effort that is frequently haphazard today) and help promote effective use of information technology by technical specialists and emergency managers during response to and recovery from disasters. The clearinghouse would provide a single point of contact for easy exchange of information among technology companies, researchers, emergency managers, and practitioners and provide a check-in and check-out point for all technology vendors and researchers at the incident scene. It would also help direct vendors, researchers, and observers away from overburdened local government officials and toward the specific areas of need. The clearinghouse would allow for a more coherent and methodical investigation of all disaster impacts, the gathering of “perishable” data, and the tracking of all field investigations. It would help ensure that all areas had been thoroughly explored for resource needs. Further, it would facilitate documentation of findings and observations and provide all investigators with updated information on damage, through daily briefings and reports. The clearinghouse should be as close to the affected area as possible, while still providing the necessary space and support services. It should be located so that it affords access to the emergency operations center, although physical proximity may not be as important as electronic connectivity. tions seek to rapidly acquire technology to support ad hoc needs, as occurs in crisis response (see Box 4.5). Common architecture. Interoperation risk can be reduced considerably by adopting common architectural frameworks and interface specifications. When the commonalities are well chosen, system- or component-specific interoperation challenges are replaced by framework compliance. Widely adopted frameworks—for example, mainstream commercial application programming interfaces (APIs)—can provide significant inter-operation benefit in mission systems. Conversely, poor choices in frameworks can result in exclusion of mainstream components, high costs of integration and, when framework technical characteristics have not been fully evaluated, high risks both at the component and system levels. Because of network externalities, frameworks must be evaluated on the basis of technical criteria, market acceptance, and potential trajectory.9 The 9   These economic effects are discussed in depth in Carl Shapiro and Hal R. Varian, 1998, Information Rules: A Strategic Guide to the Network Economy, Harvard Business School Press, Cambridge, Mass.

OCR for page 88
Information Technology Research, Innovation, and E-Government point is that the system architecture can itself be a source of risk, which is appropriate when architectural exploration is a program focus. Framework building. Introducing a new protocol or architectural framework shifts emphasis in a research program from particular realizations of capability to the service interface through which it is delivered. This approach was used in the early days of the Arpanet in order to ensure that issues of scaling and heterogeneity were addressed from the outset. It is a risky approach, however, because protocols, interfaces, and other commonalities only have impact if they are widely adopted. In order to drive down the risks of adoption, specific tactics are used. For example, both the Internet Engineering Task Force and the World Wide Web Consortium generally require reference implementations to exist before a commonality can be considered for potential adoption. Introduction of service interfaces into existing prototype systems can be a successful strategy to reduce the costs and risks of entry for new participants. This was, for example, the goal of the Defense Modeling and Simulation Office (DMSO) when it introduced the HLA protocol in the early days of the Defense Simulation program. The risks of this effort were considered acceptable because existing prototypes provided evidence of value and capability, enabling a more direct focus on interoperation and scaling up. In the early days of the International Organization for Standardization’s Open Systems Interconnect (OSI) network architecture effort, uptake was slowed by the absence of compelling evidence of feasibility and value. Scaffolding. Interoperation risk at the system level can be addressed through the creation of scaffolded components. In scaffolding, a common technique in component-oriented system design, missing components are “stubbed out” with relatively trivial placeholder components that have limited functionality. This enables partial early validation of compliance with internal protocols and interfaces (APIs). Moore’s Law Risk Several years of effort may be required to develop prototypes and evaluation systems that embody a new IT concept. Few IT implementations are insensitive to the performance of underlying technologies. Improvements in processor speed, network capacity, storage capacity, manufacturability and economies of scale, and infrastructural reliability can enable significant new approaches, for example, to capability, architecture, and usability. Researchers can anticipate the improvements likely to occur over the lifetime of a research program by extrapolating infrastructural capability according to Moore’s law and its analogs. If the matured concept is deployed 3 years after program initiation, for instance, the conventionally deployed platforms at that future time could be more than

OCR for page 88
Information Technology Research, Innovation, and E-Government four times as powerful as those generally used when the research program was initiated. To enable researchers to anticipate these changes, research-program managers sometimes employ a tactic of “living in the future,” in which unusually high-performance platforms are deployed to researchers in order to enable them to gain experience with the targeted future platforms. This is sometimes also called “leading the receiver,” because of the analogy to a football quarterback throwing the ball to the expected location of the wide receiver, even though the receiver may be far from that location when the throw is initiated. The tactic has been applied successfully in programs for computational science, information visualization, and applications for high-performance networking. In digital government, this approach could be used, for example, to experiment with the use by census takers of low-cost wireless handheld devices (now emerging in the marketplace), with speech interfaces for use in crisis management, or with very-high-performance network links available in rapidly deployed field settings for crisis management. Reliability and Usability Risks Many IT applications require high levels of assurance of system availability, of data confidentiality and integrity, of design compliance to laws or regulations, or of correctness of design and implementation with respect to safety or functional properties. This is particularly true for mission-specific government applications, and across a wide spectrum of government sectors ranging from health care, social security, and statistical data to aerospace, defense, and energy. The experience of highly reliable systems development is that users must sacrifice capability, flexibility, interoperation, and performance in order to achieve high levels of reliability—which includes fault tolerance, security, robustness in the face of component failure, and many other attributes. With improvements in reliability practices, the extent of this sacrifice can be reduced. Conversely, requirements for reliability can force program managers to scale back ambitions for capability, performance, and even usability. Usability poses similar challenges. All Internet users have had the frustrating experience of attempting to extract information or conduct transactions through poorly designed Web sites. In applications in which government information and transactions are made available to citizens, the need to provide broad access to a widely diverse population creates enormous challenges in achieving usable designs. Usability challenges are also faced in systems designed for internal use—for example, crisis management applications intended for use by professionals who may be in a state of stress when interacting with the system.

OCR for page 88
Information Technology Research, Innovation, and E-Government When designing a research program that is addressing a particular capability challenge, an important early decision is when and how to take on usability issues. There is growing evidence, for example, that the overall architecture of a system can influence the ability to create usable human interfaces.10 In addition, poor usability may deter potential adopters even when the underlying technology has considerable merit. Early attention to these issues may not be enough, however. Certain product attributes, especially those related to reliability and usability, are multifaceted issues that must be considered at every stage of the engineering process. Reliability in particular must be addressed during system conceptualization, design, development, evaluation, integration, and evolution. Planning Risks This section on “Dimensions of Risk” has illustrated and partially enumerated the dimensions of risk that might be addressed in a mission-focused research program and some of the strategies and tactics conventionally used to address them. Of course, these and other kinds of risks may be present to different degrees, depending on the particular research challenges being addressed. Regardless, it is clear that successful program designs incorporate, if only tacitly, an identification of the critical risk elements, indicating which ones are the primary drivers and which others can be safely deferred for later consideration. A full identification of risks does not imply that they should all be addressed in the scope of a program—rather, it can help a manager understand, and later communicate, how transition steps might be taken by others once results start to emerge from the program. In addition to identification of the risks, the sequencing of risks to be considered can also be a risk issue: an overambitious program design that takes on too many risk dimensions at once effectively multiplies overall program risk and lessens the chance of overall success. This challenge of sequencing the risks may best be addressed by drawing on experience in research, engineering, evaluation, and operational settings, where the full range of risk issues can be best identified. Questions include how the risk dimensions interact, and which risk issues can be addressed in parallel and which must be done sequentially. 10   Leonard J. Bass and Bonnie E. John. 2001. “Supporting Usability Through Software Architecture,” IEEE Computer 34(10):113-115.

OCR for page 88
Information Technology Research, Innovation, and E-Government SUMMARY While there can be no recipes for success in program management, this chapter suggests some models and principles that may be usefully applied by a program manager in seeking innovation in government applications. Most importantly, a program manager may benefit from understanding the challenge of achieving a particular innovation in the framework of the two principal models identified in this chapter—the supply chain model and the risk-identification model. This understanding may be developed and applied in a process that includes, for example, steps such as the following: Identify the desired innovation and the stakeholders in successfully achieving it. Identify the horizon (time available to achieve the operational impact) and the overall risk tolerance for the effort. Map out the full set of participants in the supply chain that links operational end users with innovators. This includes, for example, understanding the readiness of the potential participants and the natural opportunities for collaboration. Identify the principal risk issues at each stage and consider appropriate strategies for addressing those risks. Identify critical points of leverage in the supply chain. Understand the timing considerations and other factors regarding the opportunities so identified. Take action according to the understanding gained in the previous steps in order to stimulate appropriate participants in the supply chain to achieve the innovation goals. Example actions: Invest to stimulate exploration or buy down risk, stimulate collaboration, and facilitate new standards.

OCR for page 88
Information Technology Research, Innovation, and E-Government This page in the original is blank.