3
Assert DoD Architectural Leadership for Innovative Systems

The increasing complexity and scale of software systems demand that the Department of Defense (DoD) play an active role in developing and iterating systems architecture throughout the project lifecycle. This chapter characterizes the special role of architecture in software producibility, describes its particular challenges, and discusses how the DoD can strengthen its architectural leadership in software development when so much of its software development is conducted by contractors working at arm’s length from DoD mission stakeholders.

SOFTWARE ARCHITECTURE AND ITS CRITICAL ROLE IN PRODUCIBILITY

Software architecture is conventionally defined as “the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationship among them.”1,2 Just as in physical systems, architectural commitments comprise more than structural connections among components of a system. The commitments also encompass decisions regarding the principal domain abstractions to be represented in the software and how they will be represented and acted upon. The commitments also include expectations regarding performance, security, and other behavioral characteristics of the constituent components of a system, such that an overall architectural model can facilitate prediction of significant quality-related characteristics of a system that is consistent with the architectural model.

Architecture represents the earliest and often most important design decisions—those that are the hardest to change and the most critical to get right. Architecture makes it possible to structure requirements based on an understanding of what is actually possible from an engineering standpoint—and what is infeasible in the present state of technology. It provides a mechanism for communications among the stakeholders, including the infrastructure providers, and managers of other systems with requirements for interoperation. It is also the first design artifact that addresses the so-called non-functional attributes, such as performance, modifiability, reliability, and security that in turn drive the ultimate quality and capability of the system. Architecture is an important enabler of reuse and the key to

1

Len Bass, Paul Clements, and Rick Kazman, 2003, Software Architecture in Practice, 2nd Ed, Boston: Addison-Wesley.

2

There are other definitions (see http://www.sei.cmu.edu/architecture/published_definitions.html#Modern) but the principles are consistent among them.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 68
3 Assert DoD Architectural Leadership for Innovative Systems The increasing complexity and scale of software systems demand that the Department of Defense (DoD) play an active role in developing and iterating systems architecture throughout the project life - cycle. This chapter characterizes the special role of architecture in software producibility, describes its particular challenges, and discusses how the DoD can strengthen its architectural leadership in software development when so much of its software development is conducted by contractors working at arm’s length from DoD mission stakeholders. SOFTWARE ARCHITECTURE AND ITS CRITICAL ROLE IN PRODUCIBILITY Software architecture is conventionally defined as “the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relation - ship among them.”1,2 Just as in physical systems, architectural commitments comprise more than struc - tural connections among components of a system. The commitments also encompass decisions regarding the principal domain abstractions to be represented in the software and how they will be represented and acted upon. The commitments also include expectations regarding performance, security, and other behavioral characteristics of the constituent components of a system, such that an overall architectural model can facilitate prediction of significant quality-related characteristics of a system that is consistent with the architectural model. Architecture represents the earliest and often most important design decisions—those that are the hardest to change and the most critical to get right. Architecture makes it possible to structure require - ments based on an understanding of what is actually possible from an engineering standpoint—and what is infeasible in the present state of technology. It provides a mechanism for communications among the stakeholders, including the infrastructure providers, and managers of other systems with require - ments for interoperation. It is also the first design artifact that addresses the so-called non-functional attributes, such as performance, modifiability, reliability, and security that in turn drive the ultimate quality and capability of the system. Architecture is an important enabler of reuse and the key to 1 Len Bass, Paul Clements, and Rick Kazman, 2003, Software Architecture in Practice, 2nd Ed, Boston: Addison-Wesley. 2 There are other definitions (see http://www.sei.cmu.edu/architecture/published_definitions.html#Modern) but the prin - ciples are consistent among them. 

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS system evolution, enabling management of future uncertainty. In this regard, architecture is the primary determiner of modularity and thus the nature and degree to which multiple design decisions can be decoupled from each other. Thus, when there are areas of likely or potential change, whether it be in system functionality, performance, infrastructure, or other areas, architecture decisions can be made to encapsulate them and so increase the extent to which the overall engineering activity is insulated from the uncertainties associated with these localized changes. 3 Attention to the architecture is not limited to just the design and coding phases of software. Integrity of the architecture is maintained, often with supporting code analysis tools, throughout the software system lifecycle. This is done because a single software change at any stage, including maintenance in the latter stages of a system lifecycle, can violate the key architectural decision parameters essential for acceptable system behavior, for future evolution and enhancement, and for assurance.4 During construc- tion of a system, the architectural perspective is essential to assessing progress and risks, and the ability to make decisions and tradeoffs among various alternatives. As systems scale up, the extent of effort that must be devoted to architecture also scales up, and slightly more steeply so that a greater percentage of overall effort is devoted to architectural consid - erations.5 These include design, tradeoff analysis with respect to quality attributes from requirements, identification and analysis of precedent and related ecosystems, etc. It is noted by Boehm and Turner 6 that risk and precedent drive the balance between practices appropriate for precedented systems (i.e., “plan-driven methods”) and practices appropriate for innovative systems (i.e., “agile methods”). As noted in Chapter 1, larger-scale systems most often must include both kinds of practices, especially when architectural design successfully localizes or encapsulates innovative elements and maximizes use of precedented ecosystems and infrastructures. (For further discussion of architecture, see Box 3.1.) As also noted in Chapter 1, precedented systems are those systems whose capabilities and attributes are highly similar to those that have been produced before and therefore do not require significant soft - ware innovation. In these cases, from the standpoint of engineering risks, the most critical precedents are not of requirements, but of architecture—whenever possible, the software architecture should be well understood and derived from an analysis of previous instances of the architecture. The analysis should strongly influence the development of the software architecture for the new system, as should an understanding of the likely evolution of the involved ecosystems—and incremental evolution is characteristic of successful ecosystems. Major weapon and command-and-control systems may typically 3 The nature of modularity and its value to business outcomes are explored in Alan MacCormack, John Rusnak and Carliss Baldwin, 2007, “The Impact of Component Modularity on Design Evolution: Evidence from the Software Industry,” Harard Busi- ness School Technology & Operations Mgt. Unit, Research Paper No. 08-038. Available at SSRN http://ssrn.com/abstract=1071720. Last accessed August 20, 2010; and Carliss Baldwin and Kim B. Clark, 2000 , Design Rules, Volume , The Power of Modularity, Cambridge, MA: MIT Press. 4 One of the first studies of the consistency of modeled architectural intent and as-built reality in a very-large-scale code base was undertaken by Gail C. Murphy, 1996, “Architecture for Evolution,” in Alexander L. Wolf, Anthony Finkelstein, George Spanou - dakis, and Laura Vidal, eds., Proceedings of nd International Software Architecture Workshop (ISAW’), San Francisco: ACM, pp. 83-86. Follow-up work is reported in Martin P. Robillard and Gail C. Murphy, 2003, “FEAT: A Tool for Locating, Describing, and Analyzing Concerns in Source Code,” Demonstration Session, Proceedings of the th International Conference on Software Engineering (ICSE’0). Portland, OR, May 2003, pp. 822-823. 5 See Barry Boehm, Ricardo Valerdi, Eric Honour, 2008, “The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems,” Systems Engineering (11)3:221-234; Mark W. Maier and Eberhardt Rechtin, 2000, The Art of Systems Architecting, 2nd Ed., Boca Raton: CRC Press; Manuel E. Sosa, Steven D. Eppinger, Craig M. Rowles, 2004, “The Misalignment of Product Architecture and Organizational Structure in Complex Product Development,” Management Science 50(12):1674-1689; Alan MacCormack, John Rusnak, and Carliss Y. Baldwin, 2006, “Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code,” Management Science 52(7):1015-1030; and Manuel E. Sosa, Jürgen Mihm, and Tyson Browning, 2009, “Can We Predict the Generation of Bugs? Software Architecture and Quality in Open-Source Development,” INSEAD Working Paper 2009/45/TOM. 6 Barry Boehm and Richard Turner, 2003, “Using Risk to Balance Agile and Plan-Driven Methods,” Computer 36(6):57-66. Avail- able online at http://faculty.salisbury.edu/~xswang/Research/Papers/SERelated/Agile/r6057.pdf. Last accessed August 2 0, 2010.

OCR for page 68
0 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box 3.1 Software Architecture Software system architecture is conventionally defined as the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationship among them. In essence, architecture is the set of organizing principles, both structural and semantic, that constrain the ways by which software elements of a system interact. A well-conceived soft- ware architecture can facilitate separation among work assignments allocated to software developers be- cause it defines key elements of the “contracts” that regulate the intended connections among software elements. Architectural commitments for a software system have implications throughout the lifecycle of the system. Architectural commitments can predict critical quality attributes related to performance, security, reliability, and other “non-functional” properties. As a consequence, architectural decisions are among the most important first decisions program managers and stakeholders make yet the most difficult to change or correct. The committee’s definition of software architectures includes more than just structural considerations relating to which components may interact with which other components through what kinds of “con- nectors.”1 Semantic architectural commitments may relate to protocols, data representation invariants, exceptional flow of control, timing properties and deadlines, concurrency and threading, pattern compli- ance, and other attributes. This combination of structural and semantic commitments is of benefit when 1 David Garlan, 2003, “Formal Modeling and Analysis of Software Architecture: Components, Connectors, and Events” in Formal Methods for Software Architectures, Pp. 1-24 in Formal Methods for Software Architectures, Marco Bernardo and Paola Inverardi, eds., Berlin: Springer Publishers. contain many innovative elements, as do some DoD business systems. Nevertheless, there are very often large spans of functionality that are precedented. This means the overall system architecture is likely to include a mix of precedented and innovative structures. In modern systems, the concept of “architecture” has broadened to include not only structural com - mitments (as noted in the definition at the start of this chapter), but also other design commitments that constrain and guide subsidiary design decisions, particularly decisions regarding how components of the system are meant to interact with other components. Architecture commitments open or close opportunities for future evolution and enhancement. In other words, it is risky if not impossible to evalu- ate different alternatives or different vendors without an understanding of the architectures implicit in what they propose. In particular, modern software ecosystems have a set of architectural commitments at their core. Web services, for example, are structured in conventional ways with well-defined software interfaces among the components and, additionally, a number of “rules of the road” regarding what are appro - priate interactions across those interfaces (examples shown in Chapter 1). Another example is modern data-intensive computing, such as done extensively at companies such as Google and Yahoo!. Much of this data-intensive computing activity shares a single relatively simple system architecture concept, called MapReduce, which is designed to address the challenges of distributed computing with enor- mous amounts of data. The MapReduce concept has turned out to be applicable to a very wide range of problems, with the result that there is now a growing community of users for a “big data” ecosystem focused around a set of open-source tools.7 Despite the technical simplicity of the MapReduce meta- 7 Hadoop, Zookeeper, HDFS and MapReduce at Apache are explained further online at http://hadoop.apache.org/.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS it is transparent—that is, susceptible to modeling, analysis, and management of consistency with code. This enables software developers to more readily predict behaviors and interactions among existing and proposed system elements. Software architecture also has implications for how users interact with software systems, in the sense that architectural commitments can regulate the responsiveness of a system to user redirection due to shifts in operational needs. In some cases, a systems concept may be amenable to well-established architectural concepts and design ideas. The resulting system implementations are considered precedented systems because they rely on best-practice architectural designs or styles.2 Often these designs are manifest in established software ecosystems that have emerged around similar system concepts, as in the case of web services implemen- tations. In other cases, the proposed functionality and/or its associated quality attributes are sufficiently novel and complex that already established architectural concepts may not fully suffice. In these cases, new system architectures must be developed and validated in order to implement the proposed system concepts. This is very frequently the case in the DoD due to its unique mission and the reality of continued aggressive growth in functional and quality requirements. This report focuses primarily on the conception, design, and development of these innovative or “unprecedented” systems. Although well-matched architecture is not a guarantee of success in system development projects, many project failures have been associated with inappropriate, late-breaking, or poorly articulated software architectures. Applying architectural design best practice can help decrease failure risks for both develop- ment projects and also (because of influence on quality attributes) for system operations. 2 Mary Shaw and David Garlan, 1996, Software Architecture: Perspectives on an Emerging Discipline, Upper Saddle River, NJ: Prentice Hall. phor, its ecosystems include highly complex infrastructure to support reliability and robustness in the face of intermittent failures of individual components in large-scale data centers, such as commodity processors and disk drives. This enables data centers to be established with larger numbers of cheap commodity components rather than the more expensive options of highly reliable components, or of overall task-level checkpointing, etc.8 This ecosystem shares architectural characteristics with Google’s internal MapReduce ecosystem.9 A consequence is that architectural decision making for any particular software development project is profoundly influenced by knowledge of related ecosystems, of systems and hardware infrastructure, of available frameworks and libraries, and of previous experience with similar systems and projects. Small changes to architectural requirements can open or close opportunities to exploit rich existing ecosystems, greatly reducing both cost and risk. 8 Such was the experience of the Aegis High Performance Distributed Computing Program HiPer-D project to replace expensive “mission-critical” single processors with a network of multiple affordable processors that can be made more survivable using techniques of distributed computing. For more information, see, for example, L. R. Welch, Binoy Ravindran, Robert D. Harrison, Leslie Madden, Michael W. Masters, and Wayne Mills, 1996, “Challenges in Engineering Distributed Shipboard Control Systems,” in Proceedings of Work-in-Progress Session of the IEEE Real-Time Systems Symposium , December 1996, available at http://citese- erx.ist.psu.edu/viewdoc/download?doi=10.1.1.42.4454&rep=rep1&type=pdf. Last accessed August 20, 2010. See also Karen F. O’Donoghue and David T. Marlow, 1998, “The Application of NTP to Navy Platforms,” th Annual Precise Time and Time Interal (PTTI) Meeting, Long Beach, CA, December 1997. 9 Jeffrey Dean and Sanjay Ghemawat, 2004, “MapReduce: Simplified Data Processing on Large Clusters,” OSDI’04: Sixth Sym- posium on Operating System Design and Implementation, San Francisco, CA, December 2004. Available online at http://labs.google. com/papers/mapreduce.html. Last accessed August 20, 2010.

OCR for page 68
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Consider, for example, choices made by developers of a software system that involves complex interactive graphical user interaction (GUI) to visualize results stored in a large database, such as results of modeling and simulation. The architecture of this software system will be influenced by available choices for the database and the GUI frameworks. The architecture of the overall application will be guided by several factors: (1) framework design constraints implicit in the GUI frameworks available, (2) database interfaces offered by potential vendors, and (3) prior experience in developing interac - tive visualization systems for modeling and simulation. The architectural decisions are thus informed by experience, by knowledge of the principal ecosystems and conventional structures used for GUI frameworks and databases, by particular refinements of those conventional structures associated with candidate software components, and by infrastructure commitments. Architecture can often be reused across similar products, encouraging development of supporting tools to facilitate its reuse. In a product-line approach, a family of related systems shares architectural elements and components. This kind of approach offers not only significant cost advantages, but also promotes faster time to market, ability to reuse infrastructure, and, perhaps most importantly, signifi - cantly reduced engineering risk (i.e., uncertainties and consequences related to the process of making, acting on, and validating design commitments during the engineering process). Indeed, sound architec - tural decisions, often even more than requirements decisions, are a principal enabler of product lines, of flexibility in evolution and maintenance, of ability to exploit the rapid growth in hardware infrastructure (processors, storage, communications), and of ability to build effectively on the increasingly powerful base of software infrastructure (operating systems, databases, GUI frameworks, etc). SOFTWARE ARCHITECTURE IN INDUSTRY Industry leaders in the development of software pay considerable attention to the software architec - ture as an essential, not an accidental, set of design decisions. Throughout a system’s lifecycle, a managed software architecture provides critical evidence that a software-reliant system is capable of meeting its business and mission goals. Respected companies such as Microsoft, Intuit, IBM, J.P. Morgan, Bosch, and Siemens engage in rigorous training and certification processes for their architects. In the case of Microsoft, software architects are drawn from the most senior and accomplished technical people in the company and have the responsibility for investigating architectural alterna - tives, designing a system’s architecture, and ensuring that the resulting software product adheres to the desired architecture. At one time, the Windows team had an architecture review board composed of senior technical contributors and tasked with reviewing all groups’ potential changes and extensions to Windows. IBM has evolved corporate standards for technical roles and technical career paths for software professionals including architects. These senior positions are achieved through years of apprenticeship, a track record of accomplishment, and approval by a certification board composed of technical peers. Titles such as senior IT architect, distinguished engineer, and IBM fellow reflect highly influential roles with “executive” standing within IBM. In addition to staffing and standards, architecture, or rather architectural reuse, plays an important role in the development of many successful commercial ecosystems and product lines. To take just one example of a product line, Apple’s recently introduced iPad tablet reuses in large measure the software architecture from the iPhone. The iPhone, in turn, reuses many key architectural and infrastructural elements from systems as early as the NexT computer, which was introduced more than two decades ago. Many product-line commonalities exist across time, in the sense of architecture designed to antici - pate evolution of capability, institutional growth in capacity, and other “natural” steps of growth. The generality of a suitable systems architecture is significant for the DoD because many of its systems persist for several decades. But there is a danger in misconstructing the concept of “generality” in architecture. It is well estab -

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS lished in software practice—and a core tenant of agile techniques—that excessive structural generality at the outset of a project can create enormous inefficiencies and engineering risks, slowing down developers who must provision that generality despite the fact that it may not be needed downstream. The gener- ality also creates added engineering risk, because it complicates the natural progression of change and evolution that is characteristic of incremental development and long-lived software systems generally. There is thus a balance that must be struck. The best “generality” may be in the form of well-crafted “seed” commitments at the outset of a project that can: (1) establish architectural direction, (2) yield the intended benefits in quality attributes and flexibility, and (3) afford engineering leaders sufficient intel - lectual control and flexibility to enable ongoing refactoring as required over a systems lifecycle. Other product-line commonalities exist within subsystems or in components of larger systems—that is, “product line” is not just meaningful at the level of overall systems. As DoD systems scale up and interconnect, the architectural commonalities that comprise “product lines” could focus increasingly on designs for the common protocols that support interlinking and for the many sets of data formats and representations that are needed to permit information flow among DoD and other systems. The com - mittee characterize these as “architectural” commonalities because they serve in the role described at the outset of this chapter—as key design commitments that support quality attributes and that constrain implementation choices. Finding 3-1: Industry leaders attend to software architecture as a first-order decision, and many follow a product-line strategy based on commitment to the most essential common software architectural elements and ecosystem structures. Note that this finding focuses on the most essential commitments as comprising initial architectural decisions—more is not necessarily better. ARCHITECTURAL PROBLEMS AS A SOURCE OF SOFTWARE PROBLEMS Whether or not it is explicitly identified and managed, every software system has an architecture, and larger systems have multiple levels of architecture definition, addressing design choices regarding subsystems and components. Although having a well-matched architecture is not a guarantee of suc - cess, software systems that are not based on well-formulated software architectures are more likely to exhibit the kind of software horror stories too often experienced in DoD acquisitions with respect to project risk.10 At the product level, with respect to systems risk, these are the systems with communi - cations bottlenecks, systems that hang up or crash, systems that have difficulty re-synchronizing after disconnect, systems with database access that is sluggish or unpredictable, and systems that users judge as overly complex. These horror stories also occur in previously well-performing legacy systems after a maintenance release, often because code changes violate the architecture, which in many cases was not explicit and therefore not managed. At the process level, these are the systems that are unable to exploit established and evolving ecosystem infrastructure and improving software component capabilities, that cannot readily interoperate and federate with other systems, that defy effective quality evaluation practices, and that iterate in development without convergence (so-called “death spirals”). Perhaps most signifi - cantly, these are the systems whose engineering risks and uncertainties most often fail to resolve and 10 See Daniel L. Dvorak, 2009, “NASA Study on Flight Software Complexity,” Technical Report, AIAA Infotech@Aerospace Confer- ence, April 6-9, 2009, Seattle: American Institute of Aeronautics and Astronautics, Inc.; see also J. Elm, D. Goldenson, K. Emam, K. Donatelli, and A. Neisa, NDIA SE Effectiveness Committee, 2008, A Surey of Systems Engineering Effectieness, Software Engineer- ing Institute, Carnegie Mellon University, CMU/SEI-2008-SR-034, available online at http://www.sei.cmu.edu/reports/08sr034. pdf, last accessed August 20, 2010. See also NRC, Daniel Jackson, Martyn Thomas, and Lynette I. Millett, eds. Software for Depend- able Systems: Sufficient Eidence? Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog. php?record_id=11923. Last accessed August 10, 2010.

OCR for page 68
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE so are transformed into system and project risks (Chapter 2). Recent Studies by the Defense Science Board (DSB) and NASA confirm that software architectural issues are identified as a systemic cause of software problems.11,12 THE DOD ExPERIENCE WITH ARCHITECTURE-BASED DEVELOPMENT The DoD experience with long-term software acquisition programs has provided strong evidence for the value of software architecture. The Air Force Command Center Processing and Display System- Replacement (CCPDS-R) program13 demonstrated the benefits of architecture-based development and the value of supporting tools, even when developed for a single program. 14 In addition, the 1994 DSB15 study observed that for many DoD systems, both the functionality and the non-functional attribute requirements were similar to previously acquired systems, and could be subjected to the kind of analysis that the Air Force Electronic Systems Division (ESD)16 performed on Command Centers. In a demonstration program called PRISM (Portable Reusable Integrated Software Module), ESD analyzed the Command Centers that it had helped acquire over the preceding decade. Based on that analysis, a common architecture was developed that could be tailored to accommodate unique requirements so that ESD could support a product line for a class of command centers. 17 ESD also recognized that such common architecture could be supported by a collection of tools, including code generation, that as a suite significantly reduced the risk, cost, and time required to acquire a new command center. In this manner, success in innovation in early designs yields explicit precedent, in the form of architectural successes, that can reduce at-the-margin costs and risks for adding new com - mand centers and incremental new functionalities. A similar demonstration effort was supported by the Air Force Aeronautical Systems Division (ASD) for development of a prototype architecture for aircraft simulators.18 That demonstration showed that the Air Force was able to acquire simulators for different 11 Defense Science Board (DSB), 2009, Report of the Defense Science Board Task Force Department of Defense Policies and Procedures for the Acquisition of Information Technology, Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. Available online at https://www.stk.com/downloads/resources/download/risk-and-cost-reduction/2009-04- IT_Acquisition_New.pdf. Last accessed August 20, 2010. 12 Daniel L. Dvorak, ed., March 2009, “NASA Study on Flight Software Complexity,” Jet Propulsion Laboratory California Institute of Technology, Pasadena, CA. Available online at http://oceexternal.nasa.gov/OCE_LIB/pdf/1021608main_FSWC_ Final_Report.pdf. Last accessed August 20, 2010. 13 “The Air Force’s Command Center Processing and Display System Replacement (CCPDSR) program provides another reuse variation. TRW, the prime contractor, took software developed and funded under the CCPDSR contract, and updated and re - worked the product using internal funds, with the intention of selling it commercially. TRW was successful and has since licensed it, under the acronym UNAS (Universal Network Architecture Services), to both Digital Equipment Corporation and Rational. … Clearly, this reuse occurred through TRW’s initiative, and has been commercially successful. Other applications which may benefit from work done under CCPDSR include ATCCS and the Air Force’s Systems Software and Design Center.” Quoted from Unisys Corporation, March 1991, US4 - Current FAR and Budget/Finance Requirements, Reston, VA. Available online at http://www.dtic. mil/cgi-bin/GetTRDoc?AD=ADA240917&Location=U2&doc=GetTRDoc.pdf. Last accessed August 20, 2010. 14 Walker E. Royce, 1998, Software Project Management: A Unified Framework , Reading: Addison Wesley. 15 DSB, 1994, Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially , Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics.Available online at http://www.dod.gov/pubs/foi/ reading_room/859.pdf. Last accessed August 20, 2010. 16 ESD and ASD were part of the Air Force Systems Command (AFSC) until 1992 when the Air Force Materiel Command was established. 17 Randall W. Lichota, Robert L. Vesprini, Bruce Swanson, 1997, “PRISM Product Examination Process for Component Based Development,” th International Symposium on Assessment of Software Tools (SAST ‘), Pittsburgh, PA, June 3-5, p. 61. 18 William K. McQuay, 1997, “Air Force Modeling and Simulation Trends: Modeling and Simulation Makes Possible the Unaf - fordable.” Program Manager Magazine, September-October 1997: 128-132. Available online at http://www.dau.mil/pubscats/ PubsCats/PM/articles97/ms_usaf.pdf. Last accessed August 20, 2010. See also William K. McQuay, 1996, “Modeling and simu- lation trends and J-MASS technology,” Proceedings of the IEEE  National Aerospace and Electronics Conference (NAECON),May 20-23,1996 , Dayton OH, pp. 579-584. Available online at http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=517707&i snumber=11050. Last accessed August 20, 2010.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS aircraft using the same base architecture at significantly lower cost and on an accelerated schedule, even in parallel with the aircraft development. Similarly, tools were developed to support the effort. These demonstration projects were originally conducted in the late 1980s and promised to put the DoD in a leadership position with respect to software architecture-driven acquisition. Based on these promising demonstration efforts and similar experience among industry leaders, the DSB recommended that the DoD institute an architecture-driven acquisition strategy for software-intensive systems. 19 The DSB also recommended that other application areas be similarly analyzed and common software archi - tectures developed to establish the basis of a product line. Unfortunately, these demonstration programs were not continued. This was largely for the reason that the DSB has cited as a primary inhibitor—namely that each program is autonomous and believes that its requirements are somehow unique, and thus conformance with precedent could be perceived as excessive compromise. The reality is that such conformance is a principal pattern of successful com - mercial innovation, even for the most aggressive projects. The study recommended that the Program Executive Offices (PEOs) be given funding and staffing to perform such analyses and develop or acquire appropriate software architectures. That advice has been repeated by the DSB in other studies. The ben - efits of such approaches are proven, significant, and well documented by industry case studies across multiple domains.20 One of the challenges to success is the capability of the PEO organization to make the technical case for a set of architectural decisions that constrain the decision space of program offices and primes. These constraints provide broad advantage across a family of systems and for particular system development efforts, with respect to precedent and risk. Chapter 2 considers the nature of engineering risk and the considerable benefits of reducing it by following precedented architectural pathways. Unfortunately, those benefits (or architectural reuse) may not be easily measured at the outset, while at the same time designers and developers may have concrete complaints over the architectural constraints imposed (also at the outset) so that those downstream benefits can be realized. Additionally, contractors (and govern - ment program managers) may not always be offered appropriate incentives. The natural inclination is to develop new architectures and infrastructural elements rather than compromise some autonomy, accept modest near-term engineering risk (in exchange for mitigation of major long-term engineering risk), and implement a bias toward adopting existing infrastructure, product-line architectures, and other ecosystem models.21 Another trend within the DoD is to find ways to accelerate fielding of new capabilities. One idea beginning to get traction is to focus on getting 80 percent of the requirement fulfilled in significantly less time and cost.22 One enabler of rapid deployment with reduced risk is to base the development on known software architectures that are appropriate to the application or to develop architectures such that they support incremental releases while still providing persistent quality behavior. Although there are intrinsic incentives in the commercial product space, as noted above, those incentives may not always be present in system acquisition efforts, unless there is appropriate planning at the outset. 19 DSB, 1994, Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially , Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, Washington, DC. Available online at http://www. dod.gov/pubs/foi/reading_room/859.pdf. Last accessed August 20, 2010. 20 “Catalog of Software Product Lines” Sofware Engineering Institute.edu http://www.sei.cmu.edu/productlines/plp_catalog. html. Last accessed August 20, 2010 21 Several of the ideas mentioned here and in Chapter 2 are supported in Section 804 of the 2010 National Defense Authorization Act. National Defense Authorization Act (NDAA) of Fiscal Year 2010, Pub. L. no. 111-84, 111 Congress, (2009). Available online at http://www.wifcon.com/dodauth10/dod10_804.htm. Last accessed August 1 0, 2010. 22 DSB found that “Program management does not encourage 80% solution for 20% cost.” On p. 23 in DSB, June 1994, Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially , Washington, DC: Office of the Under Secretary of Defense for Acquisition and Technology. Available online at http://www.dod.gov/pubs/foi/reading_room/859.pdf. Last accessed August 10, 2010.

OCR for page 68
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Finding 3-2: The technology for definition and management of software architecture is sufficiently mature, with widespread adoption in industry. These approaches are ready for adoption by the DoD, assuming that a framework of incentives can be created in acquisition and development efforts. Finding 3-3: The DoD would benefit from explicit attention to software architecture and industry best practice, including (1) formalizing career paths and role descriptions for software architects, (2) identifying ways that DoD-aligned software architects can provide objective advice (see Chapter 2), and (3) enhancing organizational structures to support effective architectural leadership. This finding applies to both precedented and innovative ecosystems. Architecture is increasing in importance due to the broadening role of software in systems, the increasing interlinking of sys - tems, and the growing role of ecosystems and the consequent growth in access to common software infrastructure. Despite the organizational presence of PEOs, the committee has unfortunately not seen evidence that the DoD has moved toward an overall acquisition strategy for innovative software-intensive sys - tems in which software architecture has a principal role. As noted above, such approaches can make sense for several possible reasons: First, where software requirements for multiple systems are similar, software architectural commitments enable product-line strategies, with the benefits not only of reuse of common infrastructure, but also of reduced engineering risk because the reuse is planned. A bias toward commonality across similar systems is the means by which new software ecosystems are created (Chapter 1). The resulting benefits in cost can be very significant—sometimes an order-of-magnitude reduction. A second rationale for early architectural commitment is planning for interoperation. Indeed, many of the identified post-deployment difficulties with interoperation and platform evolution are symptoms of insufficient planning with respect to requirements and architecture. A third benefit, very important for management of costs and engineering risk in long-lived defense systems, is planning for flexibility—architecture commitments effectively define and encapsulate areas where change is antici - pated, or not (more on this aspect below). Admittedly, at the time of the initial 1994 DSB recommendation,23 software architecture was not as well understood, and supporting practices and technology not as well developed, as they are today. In the ensuing 10 years, a significant body of work has been amassed that validates these recommendations and codifies best practices. Books have been written about software architecture, software architecture training is available, universities offer courses, software architecture assessment methodologies are available, and code validation tools are available to verify consistency with the architecture. Finding 3-4: Several DoD programs are using software architecture-driven acquisition with success - ful results. There are programs that followed an acquisition strategy driven by early commitments regarding software architecture and that illustrate the benefits that would be obtained from a pervasive commit - ment to an architecture-driven approach. For example, the Army Integrated Battle Command System (IBCS), the Air Force Joint Mission Planning System (JMPS) and the Navy Common Link Integration Processing (CLIP) Program had architecture-driven approaches written into the request for proposals (RFP) and contract language. Software architecture played a major role in the RFP and source-selection activities in the Navy DDG-1000 Program. The Army Warfighter Information Network-Tactical (WIN-T) has applied an architecture-centric approach in two different acquisition phases after the contract was 23 DSB, November 1994, Summer Study on Information Architecture for the Battlefield , Washington, DC: Office of the Under Secre- tary of Defense for Acquisition, Technology. and Logistics. Available online at http://www.dtic.mil/cgi-bin/GetTRDoc?AD=AD A286745&Location=U2&doc=GetTRDoc.pdf. Last accessed August 20, 2010.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS in place and has conducted both software architecture and system architecture evaluations using the Architecture Tradeoff Analysis Method (ATAM).24 The Air Force GPS-OCx Program is also pursuing an architecture-driven approach. Recognizing the importance of the software architecture and providing for contractual means to ensure appropriate software architecture practices and artifacts have provided early, objective evidence about software struc- ture and system behavior. Moreover the Air Force has established baselines against which to manage software development over the lifecycle. Such efforts, when supported with appropriate analysis tools, identify risks early in the lifecycle and, when such risks are mitigated, significantly reduce overall program risk. The Army’s Strategic Software Improvement Program (ASSIP), which seeks to improve the way in which the Army acquires software-intensive systems, established an architecture initiative to increase the focus on software architecture as part of major systems acquisitions and to develop organic capabil - ity within the Army for architecture-centric practices.25 One study of 12 major systems indicated suc- cessful use of architecture practices, in particular architecture evaluation, to increase understanding of system requirements, design decisions, and risks.26 On average, 25 significant risks are found during a software architecture evaluation. For example, a recent use of the ATAM coupled with an Architecture Analysis and Design Language (AADL)27 uncovered previously undiagnosed design flaws in Apache’s new runtime system.28 Although it is difficult to place a quantitative value on early-detected technical risks, in a study performed by the National Defense Industrial Association (NDIA) in 2006-2007, which surveyed 64 programs and projects that had defense contractors, 46 percent of the projects that had higher product (software) architecture capability delivered the best project performance. In comparison, only 11 percent of the projects that had the least architecture capability delivered the best project performance. 29 This study concluded that the early phases of systems engineering, which include software architecture development, have the most impact. When an architecture-driven approach is taken to establish a software product line, the quantitative impact is well documented. At this juncture, there are both government and defense industry examples including Army Training Support Center (ATSC); Advanced Multiplex Test System (AMTS); Army’s Common Avionics Architecture System (CAAS) Product Line;30 Textron Overwatch Intelligence Center Software Product Line;31 the Live, Virtual, Constructive Integrating Architecture (LVCIA) product lines 24 “Software Architecture and Tradeoff Analysis Method,” available online at http://www.sei.cmu.edu/architecture/consulting/ systematam/index.cfm. Last accessed February 20, 2010. 25 Mark Kasunic, 2004, Army Strategic Software Improement Program (ASSIP) Surey of Army Acquisition Managers, Technical Report, CMU/SEI-2004-TR-003, Pittsburgh, PA: Carnegie Mellon University/SEI. Available online at http://www.sei.cmu.edu/library/ abstracts/reports/04tr003.cfm. Last accessed August 20, 2010. 26 Robert Nord, John K. Bergey, Stephen Blanchette, Jr., and Mark H. Klein, April 2009, Impact of Army Ealuations, Pittsburgh, PA: Carnegie Mellon University. Available online at http://www.sei.cmu.edu/library/abstracts/reports/09sr007.cfm. Last ac - cessed August 20, 2010. 27 “Architecture Analysis and Design Language,” available online at http://www.sei.cmu.edu/dependability/tools/aadl/ index.cfm. Last accessed August 20, 2010. 28 Peter H. Feiler and Dionisio de Niz, 2008, ASSIP Study of Real-Time Safety-Critical Embedded Software-Intensie System, Engineer- ing Practices, Special Report, CMU/SEI-2008-SR-001, Pittsburgh, PA: Carnegie Mellon University/SEI. Available online at http:// www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA480129. Last accessed August 20, 2010. 29 Joseph P. Elm, Dennis Goldenson, Khaled El Emam, Nichole Donitelli, Angelica Neisa, and NDIA SE Effectiveness Com - mittee, 2008, Surey of Systems Engineering Effectieness— Initial Results, A, Special Report, CMU/SEI-2008-SR-034, Pittsburgh, PA: Carnegie Mellon University/SEI. Available online at http://www.sei.cmu.edu/library/abstracts/reports/08sr034.cfm. Last ac - cessed August 20, 2010. 30 Paul C. Clements and John K. Bergey, 2005, The U.S. Army’s Common Aionics Architecture System (CAAS) Product Line: A Case Study, Technical Report, CMU/SEI-2005-TR-019, Pittsburgh, PA: Carnegie Mellon University/SEI. Available online at http://www. sei.cmu.edu/library/abstracts/reports/05tr019.cfm. Last accessed August 20, 2010. 31 Paul Jensen, 2009, “Experiences with Software Product Line Development,” CrossTalk 22(1):11-14.

OCR for page 68
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE at Army PEO/STRI; and BAE’s Diamond software product line.32 As the successful case studies indicate, the government benefits through reduced engineering risk, reduced development and maintenance costs, decreased time to field, increased system agility, and improved system quality. The development of new ecosystems centered around the derived architectures also fuels the competitiveness of U.S. defense software industries. There is the possibility that the focal point for such cross-program leverage could be through the PEO structure. But the committee perceives that the necessary authority and budget flexibility do not exist. Nonetheless, occasionally PEOs are able to accomplish such leverage. SUPPORTING TECHNOLOGY AND RESEARCH NEEDS A number of tools are emerging or maturing that can assist in assessing potential architectural design decisions. One of the most fundamental is the tradeoff analysis of diverse quality attributes in require - ments with architectural models. An example of a systematized process for conducting this tradeoff analysis, noted above, is ATAM.33 Another example is the Google File System (GFS) and the tradeoffs in its design among scale, cost, and reliability.34 In addition to this and similar process-based approaches, there are a number of significant technical enablers of architectural design. Examples of these include: • Systems instrumentation and profiling. Techniques to collect data from running systems can give very significant insights into the behavior and structure of the system, and provide inputs for recon - structing the architecture “as-implemented.” A principal feature of modern adaptive and self-healing architectures (sometimes called “autonomic systems”) is a pervasive approach to instrumentation within a system, including at enterprise scale. The resulting instrumentation data can be used in real-time to support monitoring for security, dynamic balancing of resource usage, and reassignment of tasks in the event of local errors and failures within a large system. Additionally, the data can be used forensically to diagnose performance, reliability, and security issues. Diverse techniques can be used to analyze the data in real time, in near-real time, and forensically. These techniques range from simple rule-based pattern matching to machine-learning technologies and data-mining techniques. The multi-purpose nature of instrumentation data has the added benefit of facilitating a “return on investment (ROI) case” for inserting the instrumentation and support for storage and analysis of the resulting database. • Interface specification models and tools. The specification and enforcement of protocols of interaction among heterogeneous software components include not only language-support “API” specifications, but also many additional constraints or “rules of the road” regarding protocols for interaction, precondi - tions on inputs, state constraints on objects, roles for threads, and so on. As we improve our ability to specify these constraints more completely, we become better able to separate the processes of developing separate components and assuring their compatibility. On the other hand, modern framework APIs are much more complex than the simpler library application programming interfaces (APIs) and protocol definitions of earlier systems. These are pervasive in web services, GUI development, AJAx rich clients, enterprise resource planning (ERP) systems, mobile frameworks, and many other areas. The advance 32 John K. Bergey, Sholom Cohen, Patrick Donohoe, Matthew J. Fisher, Lawrence G. Jones, and Reed Little, 2009, Software Product Lines: Report of the 00 U.S. Army Software Product Line Workshop, Technical Report, CMU/SEI-2009-TR-012 Pittsburgh, PA: Carnegie Mellon University/SEI. Available at http://www.sei.cmu.edu/reports/09tr012.pdf. Last accessed August 20, 2010. 33 Rick Kazman, Mark H. Klein, and Paul C. Clements, 2000, ATAM: Method for Architecture Ealuation, Technical Report, CMU/ SEI-2000-TR-004, Pittsburgh, PA: Carnegie Mellon University/SEI. Available at http://www.sei.cmu.edu/library/abstracts/ reports/00tr004.cfm. Last accessed August 20, 2010. 34 For visualization of the GFS architecture and more discussion see Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, 2003, “The Google File System,” ACM SIGOPS Operating Systems 37(5):29-43. Available online at http://labs.google.com/papers/ gfs-sosp2003.pdf. Last accessed August 20, 2010.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS of the underlying software technology both increases this complexity (as the power of frameworks increases) and also increases our ability to handle it using interface specification techniques. • Change impact analysis. Artifacts such as modularity analyses and dependency matrices can be used to identify interdependencies among design decisions relating to separate components of a system. 35 When there are design decisions that are expected to change over time, architectural structures can be developed to “contain” or encapsulate those particular design decisions in an implementation such that anticipated subsequent changes influence relatively few other components. Techniques related to dependency matrices are used to assess interdependencies and, based on models of potential changes to design decisions, assess the extent of impact of potential changes. When expectations regarding change rates are known early in the process, architectures can be structured to risks associated with change and to localize the cost impact of subsequent changes. Looking down the road, a combination of change impact analysis and interfaces modeling can facilitate a more incremental approach to verification of new changes. • Architectural suitability measures. In addition to considerations of Conway’s Law, which observes the relationship between system structure and organizational/sourcing structure in large systems, there are also internal measures of quality for architectural designs. The most significant measures are (1) coupling among components, (2) cohesion within components, and (3) correspondence of changeability in requirements elements with encapsulation in architectural elements. The exact character of the mea - sures depends highly on the specific nature of architectural models and code manifestations selected by a project. Nonetheless, these overall measures can be significant indicators of the success of an architect in achieving modularity goals. • Cross-cutting analyses. Mechanisms such as architectural specification techniques, code-quality specification fragments, compile-time “pragmas,” and other declarations can be used to check that dependencies disallowed in the architecture are not added to the code. These techniques can enable developers and architects to monitor the consistency of an evolving code base with architectural intent (which may also be evolving, albeit at a slower pace). There are particular families of techniques, such as aspect-oriented technologies, that, roughly speaking, facilitate better linkages and traceability between models and code. Although there are many potential perspectives on the code base for a software system, certain cross-cutting considerations, although completely precise and evident in models, cannot readily be made evident in code in a way that supports monitoring and analysis. Aspect-oriented and related technologies can provide a means to accomplish this for many kinds of models. • Assisted code generation. Tools that support model-driven engineering approaches can be used to assist developers in deriving implementations from models. In some cases, source code is generated directly from high-level problem-focused specifications. In other cases, code templates and frameworks are used, which can be filled dynamically or at load time. This concept of domain-specific languages (DSLs) is directly analogous to the generation of intermediate or machine code by compilers (e.g., the Common Intermediate Language and associated infrastructure in Microsoft’s .NET, or the Java Virtual Machine bytecode language in the Java ecosystem from Oracle/Sun). In particular, if generated code is edited, then there is no longer an assurance (derived from trusting a compiler) of conformance with the high-level intent. Over the long haul, techniques for code generation are an intermediate step from rou - tinized design toward configurable automation of capabilities and conventionalized interfaces in larger systems. Indeed, there is no significant operational difference between fully automatic code generation and configurable components—both lose traceability when results are directly modified, and both pose similar challenges to verification. 35 Carliss Baldwin and Kim Clark, 1999, Design Rules: The Power of Modularity Volume , Cambridge, MA: MIT Press. See also Carliss Baldwin and Kim Clark, 2007, “Modularity in the Design of Complex Engineering Systems,” pp. 175-205 in Complex Engi- neered Systems: Science Meets Technology, Dan Braha, Ali A. Minai, and Yaneer Bar-Yam, eds. Berlin: Springer Berlin/Heidelberg.

OCR for page 68
0 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Many of these techniques are sufficiently well established that they can be adopted by conservative and prudent program managers. Unfortunately, the up-front costs of adapting and using such tech - niques present an apparent budgetary risk that most program managers will not accept. In addition, in some cases these tools need to be matured and adapted for DoD systems, and that level of investment is beyond the capability of individual program managers. The DoD needs a systematic investment in identifying and maturing such tools and a means of encouraging their use. However, the extremely rapid pace of technology development mandates constant attention to the advancement of these techniques to ensure that capabilities continue to be applicable, especially in response to the enhanced architectural needs of more complex future systems. In particular, the DoD needs to conduct the research necessary to understand the software architectural issues for DoD appli - cations (Chapter 5). As previously observed, such research is unlikely to be supported at the program level. Also as previously mentioned, neither will it emerge from industry research efforts, which are primarily product driven. Architecture leadership is well established in commercial industry, as noted. 36 It is also a specialty in aerospace firms. Contractual and business incentives may sometimes conflict with goals in advanc - ing underlying technological enablers for architectural leadership, in advancing reuse, and in creating architecture-enabled mechanisms to support interlinking (interoperation across systems). In open-source projects, architectural innovation is less of a primary focus. The established successful open-source proj - ects such as Linux, Apache, and Firefox have historically tended to build on precedented architectural concepts. Recommendation 3-1: Initiate a targeted research program to provide software architects with better tools and techniques for DoD systems. Chapter 5 lays out a broad range of research that the DoD must conduct to understand the software architectural issues for DoD applications. But there are significant near-term opportunities for the DoD to invest in a systematic way to improve practice. Specifically, each PEO could be appropriated a budget to support: • Identification and analysis of existing software architectures and ecosystems for the application areas for which the PEO is responsible; • Evaluation of the common features of those architectures leading to the definition of a product- line approach for those systems and of common architectural elements and data models across systems; and • Development of improved architecture-based practices for future development. 36 Industry examples can be seen in Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels, 2007, “Dynamo: Amazon’s Highly Available Key-value Store,” st ACM SIGOPS Symposium on Operating Systems Principles, W.A. Stevenson, ed., ACM, pp. 205-220. Also see an example of interface specifications—WSDL (Web Service Description Language, available online at http://www. w3.org/TR/wsdl and cutting analysis tools available at http://research.microsoft.com/apps/pubs/default.aspx?id=70226, and http://www.coverity.com/). Last accessed August 20, 2010.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS STRENGTHENING DOD CAPABILITIES WITH RESPECT TO ARCHITECTURE The committee also notes that segments of the defense industry are capable of supporting software architecture-driven acquisition and product-line strategy.37,38,39,40 Despite a lack of financial incentives from the DoD, a number of defense industry contractors have developed a cadre of software architects, supported by training and certification processes. They state that they expect it will enable them to do a better job for programs they win and will better position them to win similar programs in the future. Also within the defense industry are corporate software product-line initiatives and successful product- line efforts that support DoD acquisitions. This implies that segments of the defense industry are well positioned to support such a move on the part of the DoD. If the DoD decides to pursue a software architecture-driven acquisition strategy followed by a product-line strategy for systems with consistent base requirements, the segments of the defense industry are equipped to respond. Further, if the DoD backs up the move to a software architecture-driven acquisition strategy with training of software architects, and supports research to develop and improve related supporting technology, the DoD could assert leadership. (See Chapter 2 for a discussion of the current state of DoD software expertise and options for increasing expertise.) The Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology has recently issued a mandate that each program be staffed with a software architect. 41 This is an important example of the kind of organic capability the DoD can and should develop. These architects will neither develop software architectures nor design or implement code. However, they will be trained in software architecture practices and how that relates to acquisition of software-reliant systems in the Army. The skill set they will need includes: understanding how to evaluate software architectures, having analy - ses available to understand which architectural decisions will be appropriate for their requirements, employing tools to ensure that code conforms to the architecture, and building on experience to manage integrity of the software architecture during system evolution. An investment by the DoD in practices and research that will support this Army initiative can help it succeed. There are significant challenges to achieving success with an architecture-led model. These derive from the difficulty of structuring incentives and allocating/sharing risks among the key stakeholders, including the development contractor, the DoD program management organization, the ultimate opera - tional users, and managers of related systems that might be potentially interlinked. The committee identifies here several of the challenges. One of the early issues that must be addressed involves the question of how the architecture is defined, as well as who owns the architecture. The options that have been considered42 include the DoD separately contracting for the development and/or selection of the architecture and supporting tools through a competition, and then selecting the best bid. An issue with this approach is the sharing of risk and responsibility among the architecture developer, the prime, and the DoD regarding architectural decisions. This is a significant challenge when the DoD seeks to impose architectural constraints to enhance product-line opportunities, foster interoperation, and manage the development of ultra-scale systems where multiple prime contractors are involved. 37 Lisa Brownsword, Paul C. Clements, 1996, A Case Study in Successful Product Line Deelopment, Technical Report, , CMU/SEI- 96-TR-016, Pittsburgh, PA: Carnegie Mellon University. Available online at http://www.sei.cmu.edu/library/abstracts/reports/ 96tr016.cfm. Last accessed August 20, 2010. 38 David C. Sharp, 1999, “Avionics Product Line Software Architecture Flow Policies,” Proceedings of the th IEEE/AIAA Digital Aionics Systems Conference (DASC), St. Louis, MO. 39 David Kaslow, 2000, “Architecture Based Design Applied to a Remote Sensing Satellite Planner,” INCOSE 2000, Minneapolis, MN. Available online at http://www.incose.org/delvalley/3_4_3_abd.pdf. Last accessed August 20, 2010. 40 Bert Schneider, Dale Anglin, Erik Baumgarten, John Dinh, and Mark Hall, 2008, “Raytheon Reference Architecture (RA): Enabling Timely & Affordable Customer Solutions,” th ICCRTS: C for Complex Endeaors. 41 See memo from LTG N. R. Thompson dated May 26, 2009, and referenced in Bill Pollak, 2010, “Software Architects: Are You Losing Ground If You Are Not Credentialed?” Saturn Network Blog, February 12, 2010. Available online at http://saturnnetwork. wordpress.com/tag/architecture-tradeoff-analysis-method/. Last accessed August 20, 2010. 42 Adapted from John K. Bergey and Wolfhart B. Goethert, 2001, Deeloping a Product Line Acquisitions Strategy for a DoD Orga - nization: A Case Study, SEI Technical Report, Pittsburgh, PA: Carnegie Mellon University.

OCR for page 68
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE One idea is for contractors, or bidders, to first contribute to the architecture in the early phases of an incremental development, and then help to implement it—with contract incentives for the imple - mentation phase tied to the actual success of the architecture. This can be challenging in a competi - tive environment where contractors may derive differential advantage from particular architectural commitments—and where those commitments may conflict with other goals, such as interlinking of systems. Another challenge is to develop a cadre of architects who have suitable engineering experience—and have ongoing engineering engagement, to assure they maintain currency in ecosystems, programming technologies, tools, and architecture-related assurance issues. This is related, also, to the expertise issue elaborated in Chapter 2. Competent software architects are rare. They cannot be created through short- term training programs but must be grown through extensive experience with associated ecosystems. Further, successful architects must be involved in design and implementation so that they do not pro - mote obsolete technology and practice. They will exhibit creativity in those things that they specify to ensure a workable architecture emerges. Additionally, they will avoid over-specification where it will inhibit creative opportunities for the developers. The arm’s-length relationship of the DoD with its contractor-based development teams creates challenges in structuring contractual relationships that facilitate the free exchange of information and feedback between developers and other stakeholders—and in supporting the kind of adjustment and refactoring that is required to achieve early validation for crucial architectural commitments. It may be difficult for some stakeholders to ascertain which kinds of architectural commitments are most essential to success of a particular project. Simple certification of evolving industry-standard ecosystems and APIs, although important to an assurance process, does not necessarily constitute archi - tectural commitment of the kind that is the principal subject of this chapter. Finally, focus on any architecture should not be viewed as an invitation for the creation of compli - ance-focused bureaucracy. Competent managers must, on an ongoing basis, assess the choice of archi - tectural constraints for the intended benefits to quality attributes and functional capabilities. Although architects will use standard ecosystems, protocols, and interfaces in defining an architecture, these choices are not equivalent to selection of standards. In developing a product-line strategy, the PEOs need to consider the fact that the know-how associ - ated with the architecture, its constituent ecosystems, and the diverse related technical resources and tools are all critical factors in a selection. They must also factor into the strategy the cost of tool devel - opment and training. Alternatively, they could provide financial incentives for contractors to develop those capabilities independently. Recommendation 3-2: This committee reiterates the past Defense Science Board recommendations that the DoD follow an architecture-driven acquisition strategy, and, where appropriate, use the software architecture as the basis for a product-line approach and for larger-scale systems potentially involving multiple lead contractors. Recommendation 3-3: The DoD should enhance existing practices to afford better distinctions between precedented portions of systems and innovative portions of systems, wherein architectures are developed both to encapsulate the innovative elements and to afford maximum opportunity to build on experience and existing ecosystems for precedented elements. These overall architectures, and particularly the innovative elements, should be subject to early and continuous validation, especially in systems that have requirements for interoperation. The foregoing discussion, findings, and recommendations apply to both precedented and innovative (unprecedented) DoD systems and subsystems. As the DoD considers development of larger systems, including systems of systems, where some of the capabilities are precedented and others are innova - tive, the acquisition challenges become more difficult. In those cases, the development of software

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS architecture will be even more significant, in some cases requiring architectural innovation. Indeed, in many cases of large-scale interconnected systems (variously called ultra-scale systems, systems of sys - tems, net-centric systems, etc.), architectural considerations may even be a driver of choices regarding system functionality. That is, once the “scope” of functional capability is identified, then architectural commitments may need to be contemplated before details are worked out regarding those functional capabilities. In this respect, quality attributes or “non-functional” requirements (reliability, scalability, security, performance, etc.) and anticipated interoperation requirements both dominate, because, from a technological perspective, their potential for fulfillment is most directly predicted by architectural decisions. Further, if the architecture effectively supports a defined “scope” of functional capabilities, then many of the details regarding the particular manifestations of those capabilities can be worked out later in the process—when the engineering (and potentially operational) uncertainties regarding the downstream impacts of particular possible choices are reduced. 43 Finding 3-5: In systems with innovative functional or quality requirements, benefit is derived from an early focus on the most essential architectural commitments and quality attributes, with deferred commitment to specifics of functional characteristics. This approach can reduce the overall uncer- tainty of the engineering process and yield better outcomes. These more complex and interconnected systems consist of multiple components interacting, and include functionality that may cut across multiple traditional defense functional areas. Future Combat Systems (FCS) is one such example. This means that building on existing, proven architectures can reduce the amount of innovation required and risk sustained in a project of this sort. Because the power prom - ised by these systems comes at a significant price in complexity (e.g., the multitude of sensors, weapons, and battle command centers), a greater focus is needed on engineering risk when planning the sequence of engineering commitments. In these cases, the biggest lever on engineering risk—and enabler of scale, interoperation, and other critical non-functional requirements—is very often architecture. Moreover, in many of these cases, the overall architecture is a composite of diverse precedented ecosystems structures combined with encapsulated innovative/unprecedented elements whose architecture, in the purely local context, may have associated design risks. In this manner, a massive global uncertainty is replaced by localized uncertainties, which, from a systems engineering perspective, means a consequent reduction in overall systems engineering risk. With prototyping and instrumentation—the software analogs of the modeling and simulation prac - tices pervasive in the development of physical systems—the engineering risks can be discharged through early validation, thus reducing the overall project risk even for highly innovative projects. As noted above, architecture is very often the fulcrum of potential for evolution and complexity. For example, how can architectures be developed and validated to support the kind of local autonomy necessary for diverse kinds of vehicles to navigate effectively over mixed terrain? How can software and systems architectures be evolved, for example, as algorithms and machine-learning capabilities improve? Moreover, by specifying interfaces for which testing or measurement is possible, by defining reusable components, and by separating critical from non-critical parts of the system, architecture plays an essential role in localizing uncertainties regarding assurance and thus reducing overall risk related to assurance. An interesting case study of architecture and evolution in an environment with multiple competing organizations is the architecture of large-scale web systems. This has been a topic of intense interest to startup and established companies for 15 years. One of the key early architectural ideas related to servers was the use of scalable networks of PC workstations and “shared-nothing computation.” This 43 Barry Boehm, Ricardo Valerdi, Eric Honour, 2008, “The ROI of Systems Engineering: Some Quantitative Results for Software- Intensive Systems,” Systems Engineering 11(3):221-234.

OCR for page 68
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE originated in academic research,44 and in turn was rapidly developed and proven in startup companies such as Inktomi, Google, and others. The academic research approach is quickly evolving into standard “best practice” supported by web development tools and web servers. Google took several big jumps beyond this architecture to address issues related to much larger scale and the need for new computational abstractions suited to this new category of high-performance data-intensive computations. It shared its ideas through an influential series of papers on the Google File System (GFS)45 and the MapReduce computational model.46 The MapReduce model, although seem- ingly quite radical, builds on ideas from functional programming that have their heritage in the 1960s. The architecture of these systems was replicated and adapted by others including Amazon and Yahoo!. Indeed, Yahoo! and the Apache Software Foundation collaborated to host the open-source Hadoop system, now widely adopted for applications that go well beyond text analysis.47 Many of the infrastruc- ture-level ideas also fed the development of cloud-computing architectures such as those supported by Amazon, Google, Microsoft, and others.48 The point of all this is that each player incrementally adapted an existing, mostly proven architecture into a new model, with the result that the end-state (the key computational abstractions and their delivery in modern distributed data centers) appears singularly revolutionary, despite the reality of its evolutionary development. Recommendation 3-4: The DoD should learn from commercial experience and, in addition, sponsor diverse areas of technical research to help reduce the engineering risk in architecting systems that include unprecedented functional and quality attributes. Some specific areas of research focus are given in the inventory of topics above, and these are fur- ther elaborated in Chapter 5, which offers specific recommendations for addressing the complexity that the DoD will face. A theme that cuts across many of these topics is the idea of developing modeling and simulation tools suitable to informing architectural decisions, analogous to the modeling and simulation done for physical elements of many different kinds of DoD systems. This creates the possibility of a try- before-buy approach to key architectural decisions, wherein architectural concepts are modeled using tools, and analysis and testing can be done to assess scalability, performance, robustness, and resiliency to failures and attacks. In other words, the techniques offer a kind of “early validation” whereby engi - neering risks can be discharged earlier and with lower cost than if the uncertainties persisted until later implementation and test phases of development. By modeling and simulation, the committee means something broader than the current theory and practice of testing and analysis of software code, which focus on conformance of program behavior to specified behavior. The goal is to augment this with tests and analyses that provide information at the earliest possible stages of the process to support evaluation and validation of architecture concepts, interface and framework definitions, and other architectural elements. A second theme, also noted in the topic inventory, is to develop audit and instrumentation tools to provide early data once architectures are designed and initially populated. 44 Armando Fox, Steven D. Gribble, Yatin Chawathe, Eric A. Brewer, and Paul Gauthier, 1997, “Cluster-Based Scalable Network Services, Proceedings of SOSP ‘, St. Malo, France, October 1997. Available online at http://www.cs.berkeley.edu/~brewer/ papers/TACC-sosp.pdf. Last accessed August 20, 2010. 45 For more information, see Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, 2003, “The Google File System,” ACM SIGOPS Operating Systems Reiew 37(5):29-43. Available online at http://portal.acm.org/citation.cfm?id=1165389.945450&coll= GUIDE&dl=GUIDE&CFID=90089376&CFTOKEN=82606234. Last accessed August 20, 2010. 46 For more information, see Jeffrey Dean and Sanjay Ghemawat, 2008, “MapReduce: Simplified Data Processing on Large Clusters,” Communications of the ACM 51(1):107-113. 47 For more information, see http://hadoop.apache.org/. Last accessed August 20, 2010. 48 Werner Vogels, 2007, “Enterprise Scale and Beyond,” presented at Meeting #2 Adancing Software-Intensie Systems Produc- ibility, January 17-18, 2007, Washington, DC. A summary of briefers’ presentations and the workshop discussion can be found in NRC, 2007, Summary of a Workshop on Software Intensie Systems and Uncertainty at Scale, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=11936. Last accessed August 20, 2010.

OCR for page 68
 ASSERT DOD ARCHITECTURAL LEADERSHIP FOR INNOVATIVE SYSTEMS A third theme, relating to the idea of precedent and architectural risk management, is to develop and analyze a family of precedented “scalable architectural patterns” that could provide a well-understood infrastructure of building blocks out of which ultra-large-scale architectures could be designed. This could facilitate the use of multiple suppliers at the architectural and component levels. Additionally, if tools are in place that can support more aggressive restructuring, then a more principled approach can be taken to architectural design that includes iterative development, currently very difficult at the architectural level. This could also enable constructive response even to relatively late-breaking news regarding the consequences of early architectural commitments. Some combination of all these approaches will likely be necessary.