Part III
The Future: Scenarios, Conclusions, and Recommendations



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 275
Human-System Integration in the System Development Process: A New Look Part III The Future: Scenarios, Conclusions, and Recommendations

OCR for page 275
Human-System Integration in the System Development Process: A New Look This page intentionally left blank.

OCR for page 275
Human-System Integration in the System Development Process: A New Look 9 Scenarios for the Future In this report, we have outlined a systems engineering view of the life cycle of development activities, a human-system integration (HSI) view, and an overview of how the two should be fit together seamlessly. We have emphasized that these development processes should be risk-driven, iterative, incrementally growing, and providing a basis for agreement among all stakeholders. There are a variety of tools and methodologies that the systems analyst can apply to meet these challenges; and new and revised tools are constantly under development. We have summarized the kinds of methods, tools, and shared representations that are available to support the HSI development process. Although our summary is not exhaustive, we think it is representative of the state of the art. We have indicated where there are gaps in the currently available methodologies and some needed new tools and methodologies. There are tools and methods for investigating and documenting the task requirements and the context of use. Contemporary forms of simulation and virtual environments can support rapid prototyping, visualization, and human-in-the-loop testing. Human performance models and related analytic tools are often used, sometimes in conjunction with engineering models, to evaluate alternative designs early, eliminate impractical alternatives, and to narrow the choices and set parameter bounds on alternatives to be tested. Product and usability evaluation methodologies are widely used. In this chapter we advance the clock 5 to 10 years into the future to envision new directions for how, with the addition of new supporting technology, the HSI discipline, including this collection of HSI system development tools and processes, could play out. In the following sections we present

OCR for page 275
Human-System Integration in the System Development Process: A New Look scenarios for the future. The first describes the bases for an integrated methodology. The second focuses on HSI-led system development and the need for the development of a formal HSI discipline. The third scenario suggests a set of knowledge-based planning aids that would support HSI activities in the larger systems engineering context. The fourth scenario features a new perspective on active user participation in system design. AN INTEGRATED METHODOLOGY We think that there are many advantages to streamlining and supporting the HSI process with advanced technology in the ways proposed here. It can reduce the development risk, cycle time, and cost by ensuring that products developed early can be expanded and reused throughout the development cycle. It can support the visualization of how the system will function and be used before it is fully committed to hardware and software, leading to fewer unforeseen difficulties and required retrofits. Finally, it should contribute to the creation of systems that can continue to evolve as experience with their operation is accumulated. We think that such an integrative methodology is becoming feasible because of continued advances in semantic web technology (Shadbolt, Berners-Lee, and Hall, 2006), virtual environment technology, simulation, modeling and gaming technology, multimedia technology, and collaboration technology. Here we provide an overview of such a methodology in terms of four of the main HSI activity categories in Figure 2-3 and Table 2-1: Defining opportunities and requirements. Defining the context of use. Designing solutions. Evaluating. As Figure 2-3 illustrates, these activity categories are not to be confused with life-cycle phases. They comprise the steps that are taken repeatedly, in some cases concurrently, and iteratively as the life-cycle phases and milestones are met. However, when defining an integrative methodology, it is these activities that provide the basis for integration that can result in cost savings and more efficient development as a design is formalized, extended in depth, implemented, and tested throughout the life cycle. Generating a Baseline Most system developments are undertaken to replace and improve an existing system or set of procedures. It is important to begin the process

OCR for page 275
Human-System Integration in the System Development Process: A New Look by ensuring that there is documentation supporting the understanding and performance of the prior implementation. Such baselines often do not exist and have to be produced. The more quantitative these baselines can be the better. They are important in order to understand the basis for improvement and to develop a quantitative business case for undertaking the new development. Defining Opportunities and Requirements and Defining the Context of Use We take these two categories together because they are the most open-ended of the HSI development activities and are similar in terms of the methods used and representations that result. It is here that exploration and evaluation of high-level opportunities take place. These activities require information collection and representation. As we project these processes into the future, we would not propose to change the methods of information collection from those described in Part II, with the caveat of incorporating the new developments and improvements suggested there and in our recommendations. The integration emphasis in these activities focuses on incorporating the documentation of the baseline, when it is relevant, and improved representation of the results. Their goal in the future should be to produce shared representations or artifacts that are linked associatively to each other. It is to be expected that, early in the process, these representations will be incomplete. As the initial steps in designing solutions are undertaken, there will be much iteration of the initial representations. Also, it should be noted that the choice, scope, and completeness of these representations will be in scale with the size and complexity of the enterprise. Below are some examples intended to survey the alternatives applicable to the most complex project: Personas representative of the potential individual users of the system. Goal/task decompositions reflecting the activities required. A catalog of information required to accomplish these activities. A description of the anticipated work environment that ultimately can be populated with product or workstation descriptions. Scenarios representative of the domain and activities to be performed. Situations in which the current system does not fully meet user requirements. Time lines or Gantt charts visualizing the potential sequences of overlapping activities implied by the scenarios. A risk analysis identifying potential development risks.

OCR for page 275
Human-System Integration in the System Development Process: A New Look A risk analysis identifying HSI risks, including safety risks and potential for human error. Stakeholder success criteria. The business case for undertaking the development. System requirements specifications (at varying levels of detail) derived from the information gathering and representation activities. There may be other intermediate representations that are produced prior to these, such as storyboards, artifacts from the field, workshop reports, etc., but ultimately those artifacts will be used to produce the representation on which integration will be based. In the early phases, these will be static descriptions represented in a set of associative databases. They must be interlinked because an important feature of the representations should be that a stakeholder could ask questions and trace audit trails through them. For example, the information requirements should be linked to the goal decomposition, the Gantt chart, or to a scenario so that one could ask where and when that information is needed, or a requirement could be traced to the source that generated it. Having these representations in interactive form makes it possible for stakeholders to study, explore, and review the state of the development in more depth so that they do not have to rely solely on the presentations at the milestone reviews. This early phase of investigation and analysis provides a crucial moment of flexibility, in which new ideas can be explored and compared at low cost to the project and its stakeholders. Project teams can engage in various types of what-if analyses, assuming for example the consequences of using certain types of new technologies or exploring the consequences of potential new threats. The interlinkage of descriptions should include the ability for any stakeholder to make annotations and recommendations, which can then be analyzed by the team when it is time to move from exploration to stakeholder commitment. Design Solutions As design is initiated and alternative function allocations between human and system considered, the representations described above will continue to be enriched and, in some cases transformed into more quantitative representations. Priorities for which activities to consider first should be based on the risk analyses suggesting where the greatest uncertainties and HSI risks lie. System components will be enumerated and prototypes of the user interfaces will be sketched out as facades, with the functionality only implied. Implications of the tentative design solutions may be explored by high-level simulation before committing resources to a particular solution.

OCR for page 275
Human-System Integration in the System Development Process: A New Look At this point, the beginnings of a formal system simulation that will embody the growing richness of the system representation should be kicked off. The previously static descriptive scenarios will become executable in the context of the simulation so that the operational concepts can be envisioned as a part of the system representation. The Gantt charts can become time-based and synchronized with the scenarios guided by GOMS (goals-operators-methods-selection rules) analysis. The personas may be implemented as human performance models of those roles. The simple facades will become working prototypes, but much of the system backing it up may still be scripted. At this stage, it becomes possible to postulate alternative system designs that can be quantitatively evaluated, either in a modeling framework or as human-in-the-loop simulations. Gradually, as the design is committed, the scripted modules will be fleshed out in hardware and software and those modules substituted for the scripted versions. In the prototyping languages of today and tomorrow, it should be possible to move seamlessly from early prototypes to production-quality software. The goal is that at each stage there will artifacts that represent the current state of development that may be examined and used by relevant stakeholders. These artifacts become the basis for visualization of the operational concepts and how they might play out. Evaluation Evaluation is ongoing throughout the development life cycle, with peaks at the incremental commitment milestones, as illustrated in Figure 2-3. As the modeling and human-in-the-loop simulation efforts progress, a measurement module is added that makes it possible to generate performance measures appropriate to the current state of system development. At different stages in development, the measurement may consist of video recording of simulated or real interactions, keystroke-level monitoring of model users or real users’ activity, eye-movement recording, and higher level derivation of human and system performance measures. As mentioned in Part II, the ability to coordinate, interleave, and annotate these data records will also be important. Early in the development, the formative evaluations may be nothing more than written critiques produced by various stakeholders—or user-informed critiques from the participatory design or contextual design traditions—but when simulations become available, then more systematic and quantitative evaluation becomes possible. Model results must be validated with human-in-the-loop simulations. As detailed design and implementation are completed, the simulation transitions to actual system hardware and software, and evaluation of actual system components in use is undertaken. The evaluation culminates in a formal, summative evaluation—field test and evaluation in the case of the military; early deployment

OCR for page 275
Human-System Integration in the System Development Process: A New Look to a restricted number of field sites in the case of commercial software. Evaluation reports become shared representations that are useful at each life-cycle milestone and are linked with the configurations tested. The Meaning of Integration There are several senses in which this postulated development process is integrated. First, it is integrated in the sense that the products of each activity are manifest in representations that may be shared across the development community. Second, it is integrated in the sense that each product builds on the reusable components of previous ones. Common threads are provided by storyboards, use cases, scenarios, timelines, models, and system simulations. Documents, such as the business case, are elaborated, not reinitiated from scratch. Third, it is integrated in the sense that achieving the goals described requires, even demands, the cooperation of many stakeholders serving as an integrated team. Finally, the successful resulting design will accomplish much of system integration before implementation begins, and the result will represent a system that is truly responsive to the needs of its users, the ultimate goal of human-system integration. HSI-Led System Development Currently human-system integration is viewed as a support discipline, when it is engaged at all. This scenario for the future envisions it as the lead discipline in the system development life cycle. Current development practice tends to be dominated by the technical disciplines that are most salient for the particular system being built: software for information-processing-intensive systems, and various electrical, mechanical, and physical sciences for systems heavily dominated by electronics, structures, or sensors, respectively. In these instances, it is often the case that technical performance overrides human factors and operational considerations, these being considered secondary to, for example, sensor system optimization. Indeed, the system development process can become dominated by technical functionality that is later unused or, worse, gets in the way of the task at hand—that is, generates risk. The ultimate goal of our vision for HSI-led systems is that an HSI professional with a system engineering background and training will be responsible for overall program management for new, complex systems, especially systems in which people play a significant role. The program manager with an HSI background and experience will speak the language of developers and understands their constraints, while also being properly attuned to business case issues, such as schedule and resources. At the same time, the specialist will ensure that human-system integration is appropriately ad-

OCR for page 275
Human-System Integration in the System Development Process: A New Look dressed by the HSI specialty team. This assignment will lead to the proper balance for ensuring that systems meet (satisfice) stakeholder requirements, especially operational stakeholders, while delivering a product within the schedule and budget constraints. We have emphasized the ways in which the hierarchical decomposition of the work domain should take precedence over the engineering decomposition of functional modules, because work domain factors are the ultimate contributors to operational system effectiveness and success. It is the HSI professional who has the broadest perspective on these factors. Such a person, when also endowed with systems engineering training and expertise, becomes the strongest candidate for program manager. The HSI specialist–program manager will provide a leadership and management culture that understands, embraces, and promotes the importance of human-system integration in system development. There is no need for education or salesmanship on the part of human-system integration, because, with an HSI-knowledgeable program manager, the culture sees it as integral to good design as well as cost-effective. Human-system integration becomes the glue that pulls all the system components together in a way that emphasizes human use. This leads to a supportive environment and appropriate levels of resources to carry out the HSI functions. Human-system integration is viewed as an important component of overall risk reduction in complex development, and the specialty is always an integral element of system development from the earliest stages. As the program manager for complex systems involving people, the HSI specialist-systems engineer would lead the program management team. Cross-functional and multidisciplinary interaction is critical to the success of large programs and can be accomplished through use of integrated product teams (IPTs), as advocated by Rouse (2005). The program manager’s assignment of resources will be based on risk-opportunity analyses, but separate IPTs would be established to coordinate the most critical risks. One such team, if warranted, would be an HSI IPT. Some large projects are already using an HSI IPT. This team is responsible for the aspects of system concept definition involving end-users, further defining requirements associated with the concept, communicating those requirements in appropriate shared representations to affiliated IPTs, such as software development or structural design, and working with those teams to develop specifications for aspects of the system affecting end-users, such as displays, operational processes, and communications. The HSI IPT would also have representation from individuals representing planning for operations support, such as manpower and personnel domains and training developers. A typical IPT structure is shown in Figure 9-1. A key element of the implementation of the HSI IPT process is the application of the various methods described in this book. During the

OCR for page 275
Human-System Integration in the System Development Process: A New Look FIGURE 9-1 IPT structure for HSI-led system development. early phases of development, methods for defining context of use and requirements are applied. Design methods are used to develop solutions, and evaluation approaches are applied to characterize performance. During each phase, the HSI IPT produces appropriate shared representations, such as display concepts and behavior specifications, facility drawings, and process descriptions, to communicate design-relevant information to other specialties. Developing Human-System Integration as a Discipline The committee envisions a new educational perspective on the specialties associated with HSI design and implementation, perhaps eventually leading to a new engineering discipline. As described in Chapter 2, the committee uses the following definition of human-system integration: A comprehensive management and technical program that focuses on the integration of human considerations into the system acquisition and development process to enhance human-system design, reduce life-cycle ownership cost, and optimize total system performance. Furthermore, a key element of the HSI approach is the coordination and integration of the HSI domains at each system life-cycle phase. The vision of human-system integration as a discipline will require new educational programs that cover the HSI disciplines but also include training in systems engineering. It will also provide linking interfaces to such disciplines as computer science, software engineering, and acquisition management, rather than create additional wedges with these functions. Many

OCR for page 275
Human-System Integration in the System Development Process: A New Look current academic programs have certain components of human-system integration. The Naval Postgraduate School is in the process of initiating such a curriculum, but no other known programs have all the necessary components and focus on their integration. The traditional recruiting ground for HSI personnel has been the academic discipline of experimental psychology, reflecting the origins of the field. Industrial engineering programs often have an ergonomics and human factors specialty. More recently, usability professionals have been developed from the academic tradition of information sciences and technical writing. Although these types of background serve important functions in human-system integration as a support discipline, they are to be too narrowly focused to integrate effectively with other engineering personnel or program management constraints. Similarly, traditional systems engineering without HSI perspectives does not, by itself, meet the needs of this new discipline. This perspective asserts that human-system integration is fundamentally an engineering discipline. It can emerge as a recognized discipline in its own right, within an engineering program supported by the appropriate academic curricula and programs. We think that a market study would demonstrate that there is demand for this kind of HSI professional. This demand would presumably be derived from an increasing recognition among acquisition, program, and project managers of the important role of humans in systems and that effective human-system integration can significantly reduce risk. We envision meeting these increased demands through HSI courses and curricula. At the undergraduate level, this content would be likely to be covered to track with a chapter in a text. At the graduate level, assuming that demand can be demonstrated, masters and Ph.D. programs will emerge. The domain lends itself particularly well to satellite campus or distance-learning technologies to support the part-time student working professionally in industry or government. Similar integrative academic programs have begun to emerge to meet integration demands for cognitive science and social science, in the new “schools for information,” and more broadly in programs that grant a combination degree in human-computer interaction and business. It is expected that these educational programs will convince prospective students that successful careers can be pursued through the study of human-system integration. Career ladders in both industry and government will be created to legitimize human-system integration and to emphasize HSI knowledgeability in promotion criteria. Workshops and continuing education programs for working professionals will emerge, including programs for making non-HSI people HSI knowledgeable. The definition of the domain and the currency of methods and tools will need to be maintained to have long-term success and impact as well. Kleiner and Booher (2003)

OCR for page 275
Human-System Integration in the System Development Process: A New Look provide some initial thinking on levels of HSI competency for different functional assignments ranging from entry level to HSI manager. They discuss both the core competencies needed at all career levels and the specific knowledge, skills, and aptitudes needed for each specific level. Since human-system integration is a project-oriented discipline, it would be beneficial in the future to create a “practicum” environment out of existing, complex projects in which undergraduate students who are on a work-study program or graduate students could have available applied experiences. In addition to developing processes by which HSI projects can make use of graduate and undergraduate interns and assistants, it is envisioned that there would be opportunities for which federal HSI specialists could be involved in interagency or industry projects. Such assignments would be rewarded and recognized and should not be perceived as detrimental to career development. Finally, our vision includes HSI tracks at professional conferences and special editions of relevant journals. KNOWLEDGE-BASED PLANNING FOR HUMAN-SYSTEM INTEGRATION Many complex system development efforts begin with a core team of managers and systems engineers who may know that getting the HSI aspects right is important, but who have little knowledge of which HSI techniques work best in different situations, or of when such HSI techniques are no longer cost-effective. In helping such managers and systems engineers, another scenario for what may be achievable in the next 10 years with sustained investment in HSI support technology is the development, usage, and growth of a family of domain-specific tools for helping projects to assess their risks and to suggest what HSI skills, methods, and tools they would need to identify, analyze, prioritize, and mitigate HSI risks. Here is an example future scenario of the use of such a capability in the domains of command and control (C2) for defense or emergency services. An IPT consisting of operational stakeholders representing the major C2 functions of observation, orientation, decision, and action management, as well as development stakeholders representing human-system integration, hardware engineering, software engineering, and C2 system acquisition management functions is convened for the purpose of a scoping and planning project to develop a new C2 system. As part of their team building, scoping, and planning activity, they interact with a C2-domain, knowledge-based planning aid for an HSI tool.

OCR for page 275
Human-System Integration in the System Development Process: A New Look The tool input requires the IPT to provide a set of project descriptors addressing the project and system as a whole and its C2 functions. The system or project specifications would include such descriptors as: Size in terms of the number of people, information sources, and assets in need of C2. Organizational complexity in terms of the number of independently managed organizations involved in providing the services being commanded and controlled, as well as the degree of interorganizational coupling involved in providing the services. Precedents for this team in terms of the past history of developing similar systems, of having the organizations work together, of C2 development experience of the organizations, and the need for new C2 doctrine, organization, training, material, logistics, personnel, and facilities. Criticality in terms of the risk to human life and the value of the assets at stake. Technical and human factors complexity of the functions involved in providing the C2 services and of the need for such additional system functions as security, instant response, rapid adaptation to change, and degraded mode operation. Available expertise among organizations for system engineering, developing, and acquiring similar C2 systems. Drawing on its knowledge base of related successful and unsuccessful C2 development projects, the C2 knowledge-based tool provides the IPT with the following: A summary of the most significant acquisition and operational risks needing to be managed. Recommended development timelines and staffing profiles. Necessary levels of system acquisition, human-system integration, hardware engineering, software engineering, and C2 subject matter expert staffing required during the system life-cycle phases. The likely most relevant methods and tools to be used during the various phases, along the lines of Appendix Table 3-A1. These tool capabilities would enable the IPT to perform sensitivity analyses of differences in tool inputs in order to better identify, avoid, and manage risks; to avoid the late rework and project overruns; and to deliver more cost-effective C2 system performance.

OCR for page 275
Human-System Integration in the System Development Process: A New Look USER PARTICIPATION Our state-of-the-art review has emphasized the importance of grounding design in a deep understanding of work domain activities and the context of use. We have also argued for the importance of including domain practitioners who are the intended users of the system as active partners in the design endeavor. While we have argued for the importance of these activities to successful design, we acknowledge that many of the current approaches for analysis of context of use can be time and labor intensive, require expertise to employ, and produce results that are not always packaged in a way that can readily be assimilated in the system development process. These factors combine to slow their adoption and limit their effectiveness. A related consideration is that user activities and context of use are not fixed elements that can be captured once and for all. The activities that people engage in and the physical, social, and organizational environment in which they take place are constantly evolving. It is important to develop efficient techniques that can dynamically capture changes in work context and requirements and to create systems that can be readily adapted to meet changing demands (e.g., Woods and Dekker, 2000; Hoffman and Elm, 2006; Roth et al., 2006). These points highlight the importance of developing new approaches to capturing user activities and context of use in ways that are less obtrusive, less resource intensive, more continuous, and more readily assimilated into the system development (and update) process. Approaches to Capturing User Input In Chapter 6 we pointed to some promising directions for streamlining the capture and analysis of context of use knowledge, such as event data analysis methods that are intended to collect information on context of use unobtrusively. In this section we point to an emerging confluence of activities and technologies that promise to help end-users learn more about their activities, reflect on their actions, and provide useful contributions to the system development and evolution processes. In the past, system designers often assumed that users received their technologies in a finished state then went on to use those technologies as intended by the designers. Numerous studies have now shown that users often have to modify the technology or its usage extensively (see, e.g., Bikson and Eveland, 1996; Dourish, 2001, 2003; Muller et al., 2003; Pipek, 2005; more broadly, see Darrah, 1995; Eglash et al., 2004). In military terms, the practice of “field modification” is another example of users’ needs to change and reinvent the technologies that they receive. Developments in Web 2.0 have accelerated this process (O’Reilly, 2005).

OCR for page 275
Human-System Integration in the System Development Process: A New Look In the new networks, it is common to interface one application or service with another, to create new functionalities and new value propositions. Each application provides a standardized interface (typically XML) to other applications, and new services can be created through simple interfaces among these existing applications (making a “call” between applications, similar to a subroutine call in a conventional program architecture). The standardization of data formats among these services allows very rapid prototyping and testing of new service concepts, and these integrations can lead to user experiences that appear to be entirely new concepts and functionalities. Each such web site or module uses these standardized formats to offer “services” that can be called from other web sites or modules—hence the more formal description as “service-oriented architectures” or SOAs (Erl, 2005; SOA Technical Committee, 2006). We describe five classes of new services here. The first class of such services are the examples of combining list-based advertising entries from one system with map-based visualizations from a second system, using standardized address data representations as the common service-calling protocol, to provide interactive geographic summaries of opportunities that change dynamically with new textual entries to the original list.1 These quickly assembled services have been called “mash-ups” to emphasize that they have been constructed by bringing together two different data sets. These technology-centric developments have enabled new forms of shared usage and collaboration-at-a-distance, often on a massive scale, and often involving users who have no knowledge of one another other than through these new systems and forms of collaboration. These developments have been generally described as “social software” (Allen, 2004; IBM, n.d.; Teton and Allen, 2007; see also Chi et al., 2007). The remaining four classes of new services fall into this general area. A second class of such services provides awareness services in the form of “feeds” of information via the RSS protocol.2 Each feed is provided in the form of updates on a specified page at a web site—a “weblog” or “blog.” These blogs can be read and aggregated by a user via one of many “feedreaders,” leading to increasingly integrated lists of updates from selected web sites. Commercial uses range from financial awareness to competitive intelligence. Military uses could include situational awareness. A third class of such services involves the collection within a web site of shared references (e.g., “bookmarks”) to entities at other web sites, in which each reference includes keyword descriptors called “tags” (Golder 1 See http://www.craigslist.org. 2 See http://www.rssprotocol.com, http://www.rssboard.org/rss-specification.

OCR for page 275
Human-System Integration in the System Development Process: A New Look and Huberman, 2006). These references may refer to documents,3 pictures,4 recorded music, and many other types of data and are created independently by thousands of users, and each such reference is generally shared with all other users of the original web site. Searches can thus be conducted by tag or by user, resulting in a powerful and low-maintenance alternative to complex directories or organizational taxonomies (classification schemes). Significantly, people have begun to aggregate these emergent “folksonomies” (i.e., bottom-up, user-co-constructed alternatives to taxonomies) across web sites and services, and there is a trend toward linking selected types of references to commercial sites (e.g., user-constructed references to books at LibraryThing are often linked to book product descriptions at Amazon.com). A fourth class of such services is much more person-oriented and involves the posting of information by a user about herself or himself.5 Some of the information may be relatively static, while some of the information may be frequently updated, including in the form of a blog (see above). In addition, information about each person may be aggregated from other web sites through mash-up or SOA-based technologies. A fifth class of such services involves the co-creation of knowledge resources by many users, with the expectation that the knowledge will be accessed by many more users—a group-constructed encyclopedia, of which Wikipedia6 is the most well known of many instances. The pace of development using these new technologies is so swift that there are web sites dedicated to providing daily updates about the status of various Web 2.0 experiments, beta tests, and business propositions.7 Key characteristics of these developments are the reuse of technologies and services for new offerings, the diffuse and bottom-up nature of both the development effort, and the data accumulation through the contributions and negotiations of thousands of users. Users are rapidly becoming designers and data providers in these new web services. In a related trend, networked technologies have empowered people to “cache” their lives. Users—especially young users—are integrating text, video, photos, and audio to produce moment-by-moment descriptions of their daily activities, using commonly available end-user web technologies. These young users are beginning to enter the civilian and military workforce and are bringing their familiarity and expectations of these technologies 3 See http://del.icio.us/. 4 See http://www.flickr.com/. 5 See http://www.facebook.com. 6 See http://www.wikipedia.org. 7 See http://www.momb.socio-kybernetics.net/, the Museum of Modern Betas.

OCR for page 275
Human-System Integration in the System Development Process: A New Look into work cultures. These technologies are likely to be transformed for self-reflection in most any situation. The tools enable things such as auto-uploading, tagging by association, dynamic views of tag clouds and—crucially—the “mash-up” technologies of Web 2.0 to integrate these diverse media into coherent new services. In the future, additional information will be gathered from sensors in objects in the world and digital tagging of locations in the environment. We see these tools as a means to end-user empowerment in much the same way that desktop publishing on personal computers transformed business communication in the workplace. Recently, Bradley Horwitz, vice president of Yahoo!’s product strategy group, explained that only a small number of people needs to leverage the tools in order for the resulting information to become useful for the masses (see also discussions of the “long tail,”—a statistical analysis of the influence of a small number of high-frequency contributors on a much larger community of low-frequency contributors and readers—Anderson, 2006).8 We suspect that the same will be true in this context. Not all users will have to be actively reflecting on their activities and environment, but those who do will help positively transform the environment for everyone. McKinsey describes this as a new model of knowledge production, access, and distribution. He goes on to suggest that communities, not individuals, become the sources of innovation in a world of open-source approaches to knowledge development (Davis and Stephenson, 2006). In this vein, the following vision of the future is presented. Systems Engineering for User Participation In 5 to 10 years, HSI professionals are still focused on identifying HSI needs, translating their findings into opportunities, developing prototypes, requirements, and ultimately designing solutions that respond to those needs. But the world has fundamentally changed and systems engineering and HSI professionals have anticipated these changes and are working in new ways. Some activities look the same, but others look radically different. HSI professionals are gathering their information with some of the methods they used in the past (such as cognitive task analysis or through observation) but they also have new ways to uncover opportunities, understand users and their contexts, and define solutions—they are constantly sensing and responding and have the skills to create not only solutions—but also wholly new ways of doing things from the data they are collecting and through the collaborative efforts of the real end-users. 8 See also http://www.thelongtail.com.

OCR for page 275
Human-System Integration in the System Development Process: A New Look So what is different? Data about the users, the environment, and objects in the environment are being continuously collected in real time and then re-presented for users to comment and learn from. Nearly every object in the user’s system has been “spime”-enabled. A spime is a currently theoretical object with embedded sensing and responding capabilities that enable tracking through space and time (Sterling, 2004). The geospatial web has enabled the environment to constantly update people with location-based services and location-aware applications. Information will be integrated with the historical, cultural, and other relevant information of the specific place or setting or from smart dust distributed among places (Liebhold, 2004). This technological development is a logical maturation of some of the technologies reviewed in sections about event data analysis in previous chapters. At the same time, end-users are becoming increasingly sophisticated producers and distributors of interlinked media. Collections of users have the ability to form reporting communities. Best practices in one community can easily be shared across the network. Events produce dynamic blogs, too, in which participants can see the activity of people, objects, and the interaction with the environment as it happens. The histories are then mined to look for patterns that can be used to build models, and then the models are trained to make better predictions about future events. Users are monitoring and contributing to their own “moblogs” that compile information about their activities into views that are meaningful to them. People post to these blogs through mobile cellular or other input devices. The algorithms are constantly updated and made better through the analysis of the behavior of real people in real settings and the commentary users provide through their blogging activities. Why are the users participating? There are several reasons. It is easy to comment and re-present information. Wearable technologies have made “in the action” collection automatic or nearly so. Another reason is that there are widgets embedded in the interfaces that enable users to build and create their own ways to track data based on what they think they really need. As individuals become recognized by their community for their contributions—we can anticipate more and more participation in much the same way bloggers today do. Eventually the “best of the best” contribute to design efforts and trainers—of the systems, the other users, and the developers. In the meantime, HSI professionals have also been able to study what has happened even without users’ annotations. They have studied how different people have appropriated the technologies in the moment. They have watched, in real-time, how people have built new applications out of old technology, what used to be a work-around is now a “work as.” HSI staffs for their part are now building the tools that others formerly built for themselves. For a system to remain work-centered over time, it

OCR for page 275
Human-System Integration in the System Development Process: A New Look must not only support the elements of work identified at the design stage, but it must also be able to accommodate elements that the initial design did not appropriately capture and be adaptable to meet the changing nature of the work (Roth et al., 2006). Systems need to explicitly incorporate mechanisms to enable users to adapt the system to evolving requirements. The development of these modular systems place even greater demands on HSI professionals. Similar calls have been made in the computer-human interaction community to move toward end-user development systems (Fischer et al., 2004). The goal of end-user development is to develop tools to enable end-users to adapt and further develop applications to meet evolving requirements. It has its roots in early calls to enable users to create customizations, extensions, and applications so as to address unanticipated requirements (Mackay, 1990; Nardi, 1993). Fischer and his colleagues (2004) have argued for the importance of developing meta-design approaches that create open systems that can be modified by their users and evolve over time. End-user development systems range from systems that provide for modest user modifiability to systems that have end-user programming features (e.g., open source code). These new evolvable, work-centered systems are consistent with a growing recognition in the sociotechnical literature that software system requirements should not be viewed as fixed but rather as emergent over time as changes arise in the context of work (Floyd, 1987; Truex, Baskerville, and Klein, 1999; Scacchi, 2004). As Truex et al. (1999) have argued, this implies a need for ongoing analysis, negotiated requirements among system stakeholders, and an ongoing investment in software maintenance activities. Another change that can be anticipated is that HSI personnel become the experts in issue tracking and resolution as systems are now never finished—but more dynamically evolving—and in continuous beta mode. Their experience in requirements gathering and documentation has been successfully leveraged in this new role of keeper of the desired future state of the systems. Together, these trends have sharply reduced a number of system integration risks. The greater participation by end-users in the design of systems has led to technologies that are finally ready for use as delivered to end users. From a military perspective, ready for use translates into reduced training and user assistance requirements, faster learning, more effective use, and fewer accidents. From a consumer products perspective, ready for use translates into reductions in use errors or other problems with unanticipated uses. From a business-to-business perspective, ready for use translates into immediate return on investment and reduced total cost of ownership. A second area of risk reduction occurs because of the richer, more

OCR for page 275
Human-System Integration in the System Development Process: A New Look immediate, and more broadly based sources of data. Spimes promise to provide nearly instantaneous awareness of changing conditions, and the data-participative trends of Web 2.0 bring many users’ knowledge to bear on collaborative problems (e.g., the “wisdom of crowds”). The third area of risk reduction occurs in the rarer cases in which a system is delivered that does not meet the users’ requirements—or in cases in which the users’ environment has changed so quickly (due to changing threats or new business challenges) that the original design has been made obsolete by changing conditions. In these cases, the abilities of users to modify and enhance the technologies (e.g., through the mash-up capabilities in Web 2.0 technologies) allow users to make rapid changes that can provide new functionality to an obsolete technology so that it remains a worthwhile investment or a valuable part of defensive or offensive capability. In this chapter we have envisioned a future in which knowledge acquisition will no longer be a laborious manual process but will instead leverage the collective knowledge that naturally emerges as domain practitioners act in the world, reflecting on their own practices and on the ability of their tools to support their work, engage in collaborative knowledge sharing, and appropriate and adapt their software tools to accommodate dynamically changing needs. Already today we see evidence of users embracing new technologies to share experiences and lessons learned and build shared knowledge bases (e.g., specialized blogs, discussion groups, and tag-based sharing cites have emerged in multiple domains, including military groups). We also see evidence in virtually every domain of users creatively extending and adapting software tools (e.g., creating new visualizations, local databases, and home-grown software support systems) to meet the constantly changing demands of work. We think this is an important positive trend that needs to be fostered and facilitated through design methods that acknowledge and accommodate evolving requirements, as well as software systems that are designed with expectation of user appropriation and adaptation. As Hoffman and Elm (2006) have pointed out, there is a need to rethink the assumption that system requirements can be fixed in a world that is not fixed (see also Floyd, 1987). To them, “‘Requirements creep’ is not a nasty thing to eradicate, but an empirical inevitability to accommodate and understand empirically” (Hoffman and Elm, 2006, p. 76). Having sketched out the broad vision, we want to acknowledge that there are technical challenges to be overcome and to diffuse some potential misconceptions. First, we want to make clear that, while we envision that knowledge of user practices and use contexts will naturally emerge and that software systems will be appropriated and adapted, we do not intend to suggest that explicit analysis of users and context of use will no longer be needed. Nor do we mean to suggest that users will evolve their own software so that explicit systems analysis and software design will no lon-

OCR for page 275
Human-System Integration in the System Development Process: A New Look ger be required. Human factors analysts will still be needed to synthesize and interpret the domain knowledge gleaned; they will simply be able to do their job more efficiently and comprehensively than has been possible in the past. Similarly, systems, software, and hardware engineers will still need to analyze requirements and architect solutions, but with more explicit awareness of the need to develop solutions that accommodate change. This is especially true in safety-critical domains, such as the military and the transportation and health care industries and systems of systems more generally, in which explicit consideration of unanticipated side effects and risk consequences of design decisions are critical. Finally, we want to make clear that our vision of a more automated means of collecting information on user goals, needs, and activities is not intended as a substitute for including users as explicit stakeholders and equal partners in the design endeavor. Effective design will continue to require active dialogue and discovery among a variety of stakeholders, including users, human factors specialists, systems engineers, and software developers.