5
Reinvigorate DoD Software Engineering Research

In this chapter, the committee summarizes and recommends seven technology research areas as critical to the advancement of defense software productibility. These seven areas were identified by the committee on the basis of the following considerations:

  • Priorities identified from the analysis reported in the foregoing chapters: architecture, incremental process, measurement, and assurance. This builds on extensive interviews with leaders from the DoD and industry and research regarding both challenges and potential opportunities for the DoD.

  • Areas of potential technology and practice that might not otherwise develop sufficiently rapidly without direct investment from the DoD. Although other agencies are investing in areas related to software producibility, the focus and approach to investment do not sufficiently address the priorities as identified above.

  • Potential for a fleshed-out program proposal to satisfy research management “feasibility” criteria such as the Heilmeier questions (see Box 5.1), which identify a set of “tests” for research program proposal1—that is, areas where investment most likely leads to a return that benefits the DoD.

  • Areas not sufficiently addressed by other major federal research sponsors, including the Networking and Information Technology Research and Development (NITRD) agencies.

Prefacing this summary of areas recommended for future research investment is an exploration of the role of academic research in software producibility and a discussion of the impacts of past investments. The chapter also includes a brief discussion regarding effective practice for research program management to maximize impact while managing overall programmatic risk.2

1

There are many versions of the questions; one such version can be found in Box 5.1.

2

Indeed, there is a parallel between programmatic risk in the development of innovative software and programmatic risk in research program management. More important, perhaps, is the analogy between engineering risk in innovative software development and management risk in research program management. Several kinds of research management risk and various approaches to management risk mitigation are identified in Chapter 4 of National Research Council (NRC), 2002, Information Technology Research, Innovation, and E-Government, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=10355. Last accessed August 20, 2010.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 112
5 Reinvigorate DoD Software Engineering Research In this chapter, the committee summarizes and recommends seven technology research areas as critical to the advancement of defense software productibility. These seven areas were identified by the committee on the basis of the following considerations: • Priorities identified from the analysis reported in the foregoing chapters: architecture, incremental process, measurement, and assurance. This builds on extensive interviews with leaders from the DoD and industry and research regarding both challenges and potential opportunities for the DoD. • Areas of potential technology and practice that might not otherwise develop sufficiently rapidly without direct investment from the DoD. Although other agencies are investing in areas related to soft - ware producibility, the focus and approach to investment do not sufficiently address the priorities as identified above. • Potential for a fleshed-out program proposal to satisfy research management “feasibility” crite - ria such as the Heilmeier questions (see Box 5.1), which identify a set of “tests” for research program proposal1—that is, areas where investment most likely leads to a return that benefits the DoD. • Areas not sufficiently addressed by other major federal research sponsors, including the Network- ing and Information Technology Research and Development (NITRD) agencies. Prefacing this summary of areas recommended for future research investment is an exploration of the role of academic research in software producibility and a discussion of the impacts of past invest - ments. The chapter also includes a brief discussion regarding effective practice for research program management to maximize impact while managing overall programmatic risk. 2 1 There are many versions of the questions; one such version can be found in Box 5.1. 2 Indeed, there is a parallel between programmatic risk in the development of innovative software and programmatic risk in research program management. More important, perhaps, is the analogy between engineering risk in innovative software development and management risk in research program management. Several kinds of research management risk and various approaches to management risk mitigation are identified in Chapter 4 of National Research Council (NRC), 2002, Information Technology Research, Innoation, and E-Goernment, Washington, DC: National Academies Press. Available online at http://www. nap.edu/catalog.php?record_id=10355. Last accessed August 20, 2010. 

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH THE ROLE OF ACADEMIC RESEARCH IN SOFTWARE PRODUCIBILITY The academic research community—along with a small number of industry research groups—has traditionally addressed many of the core technical problems related to software producibility. The academic value proposition has several direct components: The first is workforce. University gradu- ates are the core of the engineering workforce. The most talented and highly trained graduates—those who contribute to innovation in a primary way—tend to come from PhD programs. More generally, the research community generates a steady supply of people—graduates at all levels—educated at the frontiers of current knowledge in important areas of specialization. The economics of these programs depend on externally funded research projects. That is, unlike bachelor’s and master’s enrollments, the production of PhD graduates by universities is in direct proportion to sponsored research. It is perhaps too obvious to point this out, but cleared individuals with top technical qualifications are most likely to be graduates of U.S. universities. The second component is new knowledge. The style of computer science and software research, histori- cally, has focused on the creation of scientific understanding that is both fundamental and applicable. This is in keeping with the “boundlessness” of software as described in Chapter 1.3 Although industry plays a limited role in performing research relevant to fundamental open problems, there is no institution in the United States other than the research community, located primarily at universities, that focuses on broad and often non-appropriable advancements to knowledge that are directly relevant to practice. Indeed, major corporate labs that have historically supported non-appropriable and open-publication research as a significant part of their overall portfolios (such as Bell Labs and xerox PARC) have been restructured or scaled back in recent years. This scaling back of private-sector research is due to numer- ous factors, including a loss by many players of safe monopoly status, analogous to that which enabled Bell Labs to thrive. This creates greater internal pressure on laboratory managers to create measurable return on investment (ROI) cases for research projects. This is particularly challenging for software producibility research, which is often focused on creating new measures of “return” rather than on incremental advances according to readily measurable criteria. This increases the significance of the role of academic research, government laboratories, and federally funded research and developments centers (FFRDCs). This is not to say that major research effort in software producibility is not underway in industry. At Microsoft and IBM, particularly, there is aggressive and forward-looking work in this area that is having significant influence across the industry. Academic research and development (R&D) is also a major generator of entrepreneurial activity in information technology (IT).4 The small companies in that sector have an important role in developing and market testing new ideas. The infrastructure to support these ventures is an important differentiator of the U.S. innovation system. This infrastructure includes university intellectual property and people supported by university R&D projects. These companies may sometimes disrupt the comfortable market structures of incumbent firms, but arguably not in the same way as do competition or foreign innova - tion. Regardless, weak incumbents tend to fall by the wayside when there is any disruption. Strong incumbents become stronger. This constant disruption is a characteristic of the more than half-century of IT innovation. It is essential that the DoD itself be effective as a strong incumbent that is capable of gaining strength through disruptive innovations, rather than being a victim (see below). The intelligence community’s Disruptive Technology Office (DTO, now part of Intelligence Advanced Projects Research Agency5) can be presumed to have been founded upon this model. A third area of value provided by university-based R&D (and industrial lab R&D as well) is surprise reduction. Computing technology is continuing to experience very rapid change, at a rate that has been 3 This is the fundamental yet eventually useful knowledge in what Donald Stokes has called Pasteur’s Quadrant. See Donald E. Stokes, 1997, Pasteur’s Quadrant—Basic Science and Technological Innoation, Washington, DC: Brookings Institution Press. 4 The committee uses “information technology” or “IT” to refer to the full range of computing and information technology areas in the scope of the NITRD multi-agency coordination activity (see http://www.nitrd.gov/ Last accessed August 20, 2010). 5 See http://www.iarpa.gov/. Last accessed August 20, 2010.

OCR for page 112
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box 5.1 Heilmeier Criteria When George Heilmeier was DARPA director in the mid 1970s he developed a set of pithy questions to ask research program managers when they proposed new program ideas. That set of questions has persist- ed, and it continues to be applied in various forms by research managers everywhere. Here is a composite rendering of the questions, along with some commentary regarding research program management. 1. What are you trying to do? Explain objectives using no jargon. The scope of the project must be defined, as well as the key stakeholders in the outcome. The purpose of “no jargon,” in part, is to assure that the scope and value can be described in ways that individuals outside the immediate field can appreciate the context and value of what is proposed. 2. How is it done today? What are the limits of current practice? This is an accounting of the baseline state, the value it delivers, the limits on what can be done in the present configuration, and, to some extent, the pain experience as a consequence of those limits. 3. What’s new in your approach? Why do you think it will be successful? Often the novelty is less in the form of a dramatically “new idea,” but rather in the convergence of existing ideas and new developments elsewhere in the field. A cynical view of “cloud computing,” for example, is that it is a delivery on the dream of “utility computing” articulated in the early 1960s at the dawn of the era of timesharing. Cloud, of course, takes this idea many steps forward in scalability, capability, and other ways. In other words, it is less impor- tant that the idea be “novel,” but rather timely, potentially game changing, and feasible. Feasibility, in this context, does not mean free of risk, but rather that the dependencies on infrastructure and other elements of the package are realistic. Feasibility also means that there are potential research performers who have the means and motive to engage on the topic. For academic research, this means the ability to build a capable team of PhD students, engineering staff as required, potential transition partners, collaborators at other institutions, etc. 4. If you’re successful, what difference will it make? To whom? This is an identification of stakeholders, and in addition an indication of potential pathways from research results to impact. For many research projects related to computing and software, those pathways can be complex. These complexities are discussed in the undiminished for several decades and perhaps is accelerating because of a now-global involvement in advancing IT. Given the rapid change intrinsic to IT, the research community (in academia and in indus - try, especially start-up companies) serves not only as a source of solutions to the hardest problems, a source of new concepts and ideas, and a source of trained people with high levels of expertise, but also as a bellwether, in the sense that it anticipates and provides early warning of important technological changes. For software, the potential for surprise is heightened by a combination of the rapid growth of globalization, the concurrent movement up the value chain of places to which R&D has been outsourced, and the explicit investments from national governments and the European Union in advancing national technological capability. Given the role of externalities in IT economics, it is not unreasonable to expect the innovation center of gravity to change rapidly in many key areas, which could shift control in criti - cal areas of the technology ecosystems described above. This is already happening in several areas of IT infrastructure, such as chip manufacturing. In this sense, the research community has a critical role in defense-critical areas that are experiencing rapid change. A consequence of this role is the availability of top talent to address critical software-related defense problems as they arise. The fourth component of the academic R&D value proposition is non-appropriable inention, as described in Chapter 1. This is one of the several forms of innovation carried out by the university

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH NRC “tire tracks” reports.1 For software, the path often connects the research results to the DoD through the development of commercial capabilities, where private investment takes a promising research idea and matures it to the point that it can be adopted by development teams. This adoption could be by software development teams in defense contractors or it could be by development teams creating commercial products or services. For example, the reliability of DoD desktop computers undeniably was improved, quite dramatically, as a result of the improvements made by Microsoft to the process of development and evaluation for device driver code enabled by the SLAM tool (described elsewhere in this chapter), which in turn were enabled by research sponsorship from DARPA and NSF. In addition to defining the impact, there is value in understanding not only those stakeholders who will benefit, but also those who may be disrupted in other ways. 5. What are the risks and the payoffs? This is not only an accounting of the familiar “risk/reward” model, but also an indication of what are the principal uncertainties, how (and when) they might be mitigated, and what are the rewards for success in resolving those uncertainties.2 6. How much will it cost? How long will it take? An important question is whether there are specific cost thresholds. For certain physics experiments, for example, either the apparatus can be built, or not. But for other kinds of research there may be more of a “gentle slope” of payoff as a function of level of effort. The answer to the questions of cost and schedule, therefore, should not only be specific numbers, but also, in many cases, should provide a description of a function that maps resources to results. 7. What are the midterm and final “exams” to assess progress? It is essential that there be ways to assess progress, not only at the end of a project, but also at milestones along the way. (This is analogous to the idea of “early validation” of requirements, architecture, design, etc., as a way to reduce engineering risk in software.) In many research areas, quantitative measures of progress are lacking or, indeed, their formula- tion is itself the subject of research. For this reason, in some challenging research areas the identification 1 See National Research Council (NRC), 1995, Evolving the High Performance Computing and Communications Initiative, Washington, DC: National Academy Press; and NRC, 2003, Innovation in Information Technology, Washington, DC: National Academies Press. 2 An inventory of “engineering” risks related to research program management is in the NRC report on E-Govern- ment National Research Council, 2002, Information Technology Research, Innovation, and E-Government, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=10355. Last accessed August 3, 2010. research community. In a market economy, with internal ROI cases prerequisite for R&D investment inside firms, this is a role most appropriate to universities and similar institutions—of course firms often carry out or sponsor such innovation for a variety of reasons, but it is not their core purpose. For IT in particular, such R&D is essential to national competitiveness and to increases in market-wide value. Although the openness of university research is sometimes considered a negative factor with respect to the advancement of technology for national security, it is also the case that universities have unique incentives, unlike industry, to advance the discipline even when the hard-won results are non-appropri - able or difficult to fully appropriate. As noted above, it is evident from the history of the field that the advancement of IT and software producibility disproportionately depends on this kind of technology advance. Of course, universities also create an enormous body of appropriable intellectual property that has the potential to be transitioned into practice. Finding 5-1: Academic research and development continues to be the principal means for develop - ing the most highly skilled members of the software workforce, including those who will train the next generation of leaders, and for stimulating the entrepreneurial activity that leads to disruptive innovation in the information technology industry. Both academic and industry labs are creating

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE the fundamental advances in knowledge that are needed to drive innovation leadership in new technologies and to advance software technologies that are broadly applicable across industry and the DoD supply chain. DoD Influence on Academic R&D The overall directions and priorities for sponsored research that leads to university-originated invention are greatly influenced by funding levels and agency priorities. For example, the Defense Advanced Research Project Agency’s (DARPA’s) deliberately strong relationship with the IT research community, which began in the 1960s and endured for nearly 40 years, has had profound influence on IT research priorities, the overall culture of computer science research, and the massive economic and national outcomes. This is documented in multiple NRC reports relating to the innovation pipeline for IT, which trace the origins of a broad set of specific IT innovations, each of which has led to a multi- billion dollar market.6 Data available from NITRD and other sources indicate that there has been a significant reduction in federally sponsored research related to software producibility as well as to high-confidence software and systems (see Box 1.5). Furthermore, it is the committee’s impression that in recent years many of the researchers in these areas have moved into other fields or scaled down their research efforts as a result of, among other things, the DoD’s having shifted funding away from software-related R&D, apparently on the assumption that industry can address the problems without government intervention. As stated previously, industry generally has less incentive to produce the fundamental advances in knowledge that enable disruptive advances in practice, building on fundamental advances but less often creating them. The impact of R&D cutbacks generally (excluding health-related R&D) has been noted by the top officers of major IT firms that depend on a flow of innovation and talent. Academic R&D, Looking Forward There are some challenges to proceeding with a new program for academic R&D related to soft - ware-intensive systems producibility. These challenges relate generally to saliency, realism, and risk. University researchers and faculty tend to be aware of broadly needed advances, but they do not always have adequate visibility into the full range of issues created by leading demands for large-scale, complex industrial and military systems. This awareness is hindered by many things, including national security classification, restricted research constraints, professional connectivity, and cost, in the sense of time and effort required to move up the learning curve. In a different domain, DARPA took a positive step in this regard by initiating the DARPA Computer Science Study Group, wherein junior faculty are given clearances and so are able to gain direct exposure to military challenge problems. Several specific DoD programs have undertaken similar efforts to give faculty a domain exposure, often with great success. One example from the 1990s is the Command and Control University (C2U) created by the Command Post of the Future (CPOF) program, which not only gave researchers access to military challenges, but also led to collaborations yielding new innovation in system concepts. 7 With respect to ensuring that researchers have access to problems at scale, companies such as Google and Yahoo!, and national laboratories such as Los Alamos, have developed collaborative programs to expose faculty and graduate students to high-performance computing systems, large datasets, and the software approaches being taken with those systems. These companies, like the DoD, have worked out 6 See NRC, 2003, Innoation in Information Technology, Washington, DC: National Academies Press. Available online at http:// www.nap.edu/catalog.php?record_id=10795. Last accessed August 20, 2010. Also see the predecessor report NRC, 1995, Eol- ing the High Performance Computing and Communications Initiatie, Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog.php?record_id=4948. Last accessed August 20, 2010. 7 The committee understands that prototype systems from this program are now deployed in Iraq.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH a level of exposure that enables researchers to engage productively without compromising core intel - lectual property. The DoD has a track record of success in this regard as well. For software producibility research, a different kind of access is needed. Certainly, the success of large-scale production-quality open source has afforded researchers great opportunity not only to experi- ment with large code bases, but also to undertake longitudinal and organizational analyses of larger projects. This has been enabled by the sophistication of the tools—code management systems, defect databases, designs and models, test cases. These projects are comparable in scale and functionality to commercial software and have greatly assisted the software engineering community in its research. Additionally, commercial firms are affording researchers greater access to proprietary code bases for experimentation and analysis. An early and significant example is work by Gail Murphy in which she assessed consistency of an as-built code base with architectural intent. 8 She studied both an open source project and a proprietary project. If security and commercial ownership issues could be resolved (perhaps by clearing selected researchers), members of the research community would benefit greatly from access to DoD-related artifacts, including surrogates and “sanitized” artifacts that omit critical algorithms and/or data. Regardless of access, the committee recommends improved data collection to support analysis (see Recommendation 2-2). INVESTING IN RESEARCH IN SOFTWARE PRODUCIBILITY The Impact of Past Investments Software development has changed and, for the most part, improved considerably during the past several decades. Software systems have grown in size and complexity and are now an integrated com - ponent of every aspect of our society, including finance, transportation, communication, and health care. Since the 1960s, Moore’s Law has correctly predicted the tremendous growth in the number of transistors on chips and, generally speaking, the extent of hardware-delivered computing power. An analogous growth has occurred in the size and power of software systems if machine-level instructions, rather than transistors, are the measure of growth.9,10,11 Today’s systems are built using high-level languages and numerous software library components, developed using sophisticated tools and frameworks, and executed with powerful runtime support capabilities. Research in software engineering, programming technologies, and other areas of computer science has been a catalyst for many of these advances. Nearly all of this research was undertaken at research universities as part of federal programs led by DARPA, the National Science Foundation (NSF), and the Service basic (category 6.1) research programs of the Office of Naval Research, Air Force Office of Science Research, and Army Research Office. Three illustrations of the impact of federal sponsorship (in academia and industry) that is specifi - cally related to software engineering are presented in Box 5.2. These illustrations, drawn from a study undertaken by Osterweil et al.,12 complement the analyses of the NRC reports cited above relating to research impacts on practice and on the IT economy. 8 Gail Murphy, 1995, “Software Reflexion Models: Bridging the Gap Between Source and High-level Models,” Proceedings of the Third ACM SIGSOFT Symposium on Foundations of Software Engineering , Washington, DC, October 10-13, pp. 18-28. 9 Barry Boehm, 1999, “Managing Software Productivity and Reuse,” IEEE Computer September, 32(9):111-113. 10 Mary Shaw, 2002, “The Tyranny of Transistors: What Counts about Software?” Proceedings of the Fourth Workshop on Econom- ics-Drien Software Engineering Research, IEEE Computer Society, Orlando, FL, May 19-25, pp. 49-51. 11 Barry Boehm, 2006, “A View of 20th and 21st Century Software engineering,” Proceedings of the th International Conference on Software Engineering, ACM, Shanghai, China, May 20-28, pp. 12-29. 12 Leon J. Osterweil, Carlo Ghezzi, Jeff Kramer, Alexander L. Wolf, 2008, “Determining the Impact of Software Engineering Research on Practice,” IEEE Computer 41(3):39-49.

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Challenges and Opportunities for Investment Notwithstanding the enormous payoffs from past investments in software research, making the case for future research investments in software producibility faces a number of challenges, rooted largely in the nature of software development as a field of study. All scientific research fields face challenges in justifying future investments, but the unique characteristics of software and the dynamics of knowl - edge creation in software producibility create particular challenges for this field. There are, however, opportunities based on developments in the technology, in the overall environment of practice, and in the improvement of scientific practice. These challenges and opportunities influence the application of the criteria summarized at the outset of this chapter. Below are a few examples of influences, both positive and negative: • Maturation of the discipline. Many researchers will agree that, as a discipline, software engineering research has matured considerably in the past decade. This is a consequence of both improved research methods and improved circumstances. The circumstances include a vast improvement in access to large bodies of code, both through large-scale open-source projects and through improvement in researcher access to proprietary code bases. An additional circumstance is the emergence of highly capable tools, including source-code management systems, development environments, analysis frameworks, etc., that afford researchers opportunity to conduct experiments at meaningful levels of scale. The effect is that it is more often possible for software engineering researchers to give satisfactory responses to the Heilmeier questions (see Box 5.1). At the same time, software engineering practice remains behind the state of the art in research. As discussed in Chapter 1, software development remains more akin to a craft than to an engineering discipline, in which the productivity and trustworthiness of system development rest on fundamental and well-validated principles, practices, and technologies. And it is still the case that even sanitized representative software artifacts are not available for academic analysis in many defense areas. • Diffusion pathways and timescale. Many of the results of software research are broadly applicable and provide for enabling technologies and methods useful in a range of specific application domains. Breadth of applicability is valued in research, but it is also double-edged from the standpoint of spon - sors. First, there is a greater chance that results may diffuse to adversaries as well as to collaborators. Second, there is a commons problem: because the benefits are broad, no particular stakeholder can justify the investments needed to produce them. Thus, for example, DoD Service R&D programs tend to focus much more on Service-specific technologies than on common-benefit software technology. Twenty years ago, the Service Laboratories played a significant part in maturing and transitioning software produc - ibility technology, but the “tragedy of the commons” has virtually dried up this key channel. Moreover, advances in software producibility ery often are enabling adances rather than being adances of immediate use in particular products. Better techniques for identifying, diagnosing, and repairing software faults, for example, enable production of better systems but are not directly used in particular software prod - ucts. The value of such advances is thus often hard to quantify precisely for any single advance, or from the perspective of any single program. Yet when integrated over longer periods of time and in terms of impacts on many engineering products, the benefits of the stream of advances emerging from software research are very clear (as summarized above). In the case of defense software producibility, there are clear drivers of defense software “leading demand,” and there are ways that the DoD can invest in and realize benefits earlier and more effectively than can potential adversaries. Moreover the DoD remains a major beneficiary of the longer-term production of software producibility knowledge. • Noelty of ideas. It is noted earlier in this chapter that cloud computing, taken broadly, is really a manifestation of a half-century-old idea of “utility computing” that has just now become feasible due to the positions of the various exponential curves that model processor, storage, and communications capabilities and costs—as well as enabling engineering, management, and business innovation. This account is a bit simplistic, obviously, but it makes an essential point: The specific novelty of an idea

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH Box 5.2 Three Examples of the Impact of Past Investments 1. From bug detection to lightweight formal methods. Bugs have plagued software since before there were computers,1 and researchers have been actively working on developing tools to help detect and prevent errors in software systems for at least half a century. Early compilers focused on syntactic er- rors and simple debugging support, but soon tools were developed to detect more complex semantic errors. Simple definition-reference bug detection techniques2,3 were followed by more sophisticated approaches.4,5,6 Programming languages such as Ada, Java, and C# incorporated some of these concepts directly into the language, and thus, for example, checked for array indexes being out of bounds dur- ing compilation and added runtime checking only when necessary. This work laid the foundation for a range of model checking and program analysis tools that are now emerging at companies like Micro- soft and Google as these companies increase their concern for secure, high-quality systems. Systems such as Microsoft’s SLAM and PreFAST are based upon the research advances funded by the federal government. For example, a report on SLAM states, “The project used and extended ideas from sym- bolic model checking, program analysis and theorem proving.”7 Those ideas emerged from academic research performed years earlier related to model checking and binary decision diagrams, and indeed Edmund Clarke won the 2008 Turing Award for his work on model checking. The authors of this tool, which has been is credited with significantly reducing the incidence of “blue screen” system crashes, were awarded the 2009 Microsoft Engineering Excellence, a success that represents the culmination of federally funded research from the 1970s through the 1990s. Early research on software testing advocated for coverage measures, such as statement and branch coverage, and tools were developed for symbolically executing paths in programs and automatically generating test cases to satisfy such measures8,9,10 The storage and speed of the machines at that time made this approach impractical, but advances in hardware combined with continued research advances in lightweight reasoning engines and higher-level languages have now made coverage monitoring a 1 Letters from Ada Lovelace to Charles Babbage discussing programming errors are mentioned in Grady Booch and Doug Bryan, 1993, Software Engineering with ADA, 3rd Ed., Boston: Addison-Wesley Professional. Also see Grace Murray Hopper’s note in the log for the Aiken Mark II in 1947 in Grace Murray Hopper, 1981, “The First Bug,” Annals of the History of Computing 3(3):285-286, 1981. 2 Leon J. Osterweil and Lloyd D. Fosdick, 1976, “Some Experience with DAVE: A Fortran Program Analyzer,” in Proceedings of the National Computer Conference and Exposition, ACM, New York, NY, June 7-10, pp. 909-915. 3 Barabara G. Ryder, 1974, “The pfort Verifier,” Software: Practice and Experience 4(4):359-377. 4 Kurt M. Olender and Leon J. Osterweil, 1990,"Cecil: A Sequencing Constraint Language for Automatic Static Analysis Generation," IEEE Transactions on Software Engineering 16(3):268-280. 5 Edmund M. Clarke and E. Allen Emerson, 1981, “Synthesis of Synchronization Skeletons for Branching Time Temporal Logic.” Pp. 52-71 in Logic of Programs: Workshop Lecture Notes in Computer Science 131, Berlin: Springer. 6 Gerard J. Holzmann, 1997, “The Model Checker SPIN,” IEEE Transactions on Software Engineering 23(5): 279-295. 7 Thomas Ball, Byron Cook, Vladimir Levin, and Sriram K. Rajamani, 2004, “SLAM and Static Driver Verifier: Technology Transfer of Formal Methods Inside Microsoft,” Lecture Notes in Computer Science (LNCS) 2999:1-20; Eerke A. Boiten, John, Derrick, and Graeme, Smith, eds., 2004, Fourth International Conference on Integrated Formal Methods (IFM 2004), Canterbury, Kent, England, April 4-7. Researchers at Microsoft have stated that the majority of the “blue screen of death” errors evident in the 1990s were attributed to problems that could have been prevented with this analysis tool. 8 Lori A. Clarke, 1976, “A Program Testing System,” in Proceedings of the 976 ACM Annual Conference, ACM, Houston, TX, October 20-22, pp. 488-491. 9 James C. King, 1975, ”A New Approach to Program Testing,” in Proceedings of the International Conference on Reliable Software, ACM, Los Angeles, CA, April 21-23, pp. 228-233. 10 Robert S. Boyer, Bernard Elspas, and Karl N. Levitt, 1975, “SELECT—A Formal System for Testing and Debug- ging Programs by Symbolic Execution,” in Proceedings of the International Conference on Reliable Software, ACM, Los Angeles, CA, April 21-23, pp. 234-245. continued

OCR for page 112
0 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Box 5.2 continued common industrial practice, along with sophisticated support for test case generation.11 Similarly, current trends in testing, such as the “Test First” approach widely adopted by agile software development teams (and tools such as the JUnit unit-testing framework) owe their foundation to early work on test case descriptions and automated execution,12,13 again based on U.S. government-funded research. Programming language development has also been strongly influenced by work in analysis of software systems, as noted above with Ada and its support for type safety and automated bounds checking. Although Ada was not a broad commercial success for various political, programmatic, and social-economic reasons, it is recognized as the direct ancestor of Java, which is widely adopted partly because of its embodiment of lessons from type theory, program analysis, and programming environment research. These lessons enabled Java to support richly capable libraries and software frameworks (as noted in Chapter 1). The C# language from Microsoft builds on similar foundations, and all three languages provide a stronger foundation for writing secure and high-quality code. 2. From development environments to software architectures to domain-specific frameworks. The success of Java is also partly due to the recognition of the importance of an interactive development environment (IDE). Early development environments were language centric, such as Interlisp14 from 1981, but continued government-supported research, such as Field15 and Arcadia,16 advocated for looser interaction models and broad support for interoperability. This led to work on common data interchange models,17the forerunners to XML and all its variants, multi-language virtual machine models such as the Java Virtual Machine, and common interoperability protocols, such as Java’s remote method invocation (RMI) and certain features of Microsoft’s .NET framework. These advances, combined with the language principles, enabled the develop- ment of modern integrated development environments (IDEs) such as Microsoft’s Visual Studio and Eclipse, originally developed by IBM but later released to open source. As software systems continued to grow in size and complexity, software engineering research broad- ened from algorithm and data structure design to include software architecture issues18,19 and the recog- 11 Dorota Huizinga and Adam Kolawa, 2007, Automated Defect Prevention: Best Practices in Software Management, Hoboken, NJ: Wiley-IEEE Computer Society Press. 12 Phyllis G. Frankl, Richard G. Hamlet, Bev Littlewood, and Lorenzo Strigini, 1998, “Evaluating Testing Methods by Delivered Reliability,” IEEE Transactions on Software Engineering 24(8):586-601. 13 Phyllis G. Frankl, Richard G. Hamlet, Bev Littlewood, and Lorenzo Strigini, 1997, “Choosing a Testing Method to Deliver Reliability,” in Proceedings of the 9th International Conference on Software Engineering, ACM, Boston, MA, May 17- 23, 1997, pp. 68-78. 14 Warren Teitelman and Larry Masinter, 1981,“The Interlisp Programming Environment,” Computer 14(4):25-33. 15 Steven P. Reiss, 1990, “Connecting Tools Using Message Passing in the Field Environment,” IEEE Software 7(4):57-66. 16 Richard N. Taylor, Frank C. Belz, Lori A. Clarke, Leon Osterweil, Richard W. Selby, Jack C. Wileden, Alexander L. Wolf, and Michael Young, 1989, “Foundations for the Arcadia Environment Architecture,” ACM SIGSOFT Software Engineer- ing Notes 24(2):1-13. 17 David Alex Lamb, 1987, “IDL: Sharing Intermediate Representations,” ACM Transactions on Programming Languages and Systems 9(3):297-318. 18 Mary Shaw and David Garlan, 1996, Software Architecture Perspectives on an Emerging Discipline, Upper Saddle River, NJ: Prentice Hall. 19 Dewayne E. Perry and Alexander L. Wolf, 1992, “Foundations for the Study of Software Architecture,” ACM SIGSOFT Software Engineering Notes 17(4):40-52.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH nition of common styles and design patterns.20 Work in software architecture was also an enabler of the development of high-level frameworks, such as Service Oriented Architectures21 electronic enterprise sys- tems, the backbone of current e-business. Commercial architecture standards such as REST22 derive from government-supported software architecture research. 3. From the waterfall to an agile compromise. Early software developers often viewed themselves as inde- pendent artisans because they worked individually or in very small groups. The reality of the complexity of the systems that were being developed, the long-term duration, the vast resources required, and the large percentage of unsuccessful projects, led to the realization that large software system development needed to be supported by a carefully managed process. Early process models, and particularly the waterfall mod- el,23 were developed as organizing frameworks to help organize the considerable pre-implementation and modeling work, and within them were identified the major software development phases. The actual flow from phase to phase was sometimes interpreted overly simplistically, leading to process models (e.g., the DoD 2167A standard) that are now considered cumbersome and overly rigid. Software leaders in academia and industry, such as Belady, Lehman, Mills, Boehm, and others, argued for more reasoned development models that incorporated risk assessment and incremental, evolutionary development.24,25,26 These models contained the seeds of the iterative ideas that now are nearly ubiquitously adopted by small development teams throughout industry. These were documented in the case of Microsoft by Cusumano and Selby27 and in the now extensive literature of small-team iterative methods under rubrics such as extreme, agile, scrum, TSP, and others. These methods are quite aggressively driving the development of tools to better support team activity including coordination across teams to support larger projects. Concepts including code refactoring, short development sprints, and continuous integration are now accepted practices. However, most agile practices have serious assurance and scalability problems,28 and need to be used selectively in large mission-critical systems or systems of systems. 20 Martin Fowler, 2002, Patterns of Enterprise Application Architecture, Boston: Addison-Wesley Longman Publishing. 21 Michael Bell, 2008, “Introduction to Service-Oriented Modeling,” Service-Oriented Modeling: Service Analysis, De- sign, and Architecture, Hoboken, NJ: Wiley & Sons. 22 Roy T. Fielding and Richard N. Taylor, 2002, “Principled Design of the Modern Web Architecture,” ACM Transactions on Internet Technology 2(2):115-150. 23    Winston W. Royce, 1970, “Managing the Development of Large Software Systems: Concepts and Techniques,” Technical Papers of Western Electronic Show and Convention (WesCon), August 25-28, Los Angeles, CA. 24 Laszlo Belady and Meir M. Lehman, 1985, Program Evolution Processes of Software Change, London, UK: Academic Press. 25 Barry Boehm, 1986, “A Spiral Model of Software Development and Enhancement,” ACM SIGSOFT Software Engineer- ing Notes 11(4):14-24. 26 Harlan Mills, 1991, “Cleanroom Engineering,” American Programmer, May, pp. 31-37. 27 Michael A. Cusumano and Richard W. Selby, 1995, Microsoft Secrets: How the World’s Most Powerful Software Com- pany Creates Technology, Shapes Markets, and Manages People, New York: Harper Collins Business. 28 Barry Boehm and Richard Turner, 2004, Balancing Agility and Discipline, Boston: Addison-Wesley.

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE may matter much less than the timeliness of the idea and the readiness of the environment to address it in a successful way. Many old ideas, considered as failing concepts, resurfaced years later at the “right time” and made a significant difference. In other words, the key question is not so much, What are the new ideas? but rather, What are the ideas whose time has come? • Measurement of effectieness and performance. The challenges of software measurement as discussed in the previous chapters—with respect to process measures, architecture evaluation, evidence to support assurance, and overall extent of system capability—apply also to software engineering research. We lack, for example, good ways to measure the impact of any specific research result on software quality, which stems in part from the lack of good measures of software quality. Without reliable, validated measures it is hard to quantify the impact of innovations in software producibility, even those that are widely credited with improving quality, such as the introduction of strong typing into programming languages or traceability in software-development databases. This is analogous to the productivity para- dox, recently resolved.13 Because software is an enabling technology—a building material rather than a built structure—it may not fit with research program management models that focus on production of artifacts with immediately, clearly, and decisively measurable value. • Timescale for impact. Frequently, it is only after a significant research investment has been made and proof of concept demonstrated that industry has stepped in to transition a new concept into a commercial or in-house product. Also, there are many novel products/services that result from mul - tiple, independent research results, none of which is decisive in isolation, but which when creatively combined lead to breakthroughs. Although it may appear that a new development emerged overnight, further inspection usually reveals decades of breakthroughs and incremental advances and insights, primarily funded from federal grants, before a new approach becomes commonly accepted and widely available. CSTB’s 2003 report Innoation in Information Technology reinforces this point. It states, “One of the most important messages … is the long, unpredictable incubation period—requiring steady work and funding—between initial exploration and commercial deployment. Starting a project that requires considerable time often seems risky, but the payoff from successes justifies backing researchers who have vision.” AREAS FOR FUTURE RESEARCH INVESTMENT In this section, the committee identifies seven areas for potential future research investment and, for each area, a set of specific topics that the committee identifies as both promising and especially rel - evant to defense software producibility. These selections are made on the basis of the criteria outlined at the beginning of this chapter. The descriptions summarize scope, challenges, ideas, and pathways to impact. But, obviously, these descriptions are not (even summary) program plans—the development of program plans from technical descriptions requires consideration of the various program management risk issues,14 development of management processes and plans on the basis of the risk identification, identification of collaborating stakeholders, and other program management functions. In the develop - ment of program plans, choices must be made regarding scale of the research endeavor and the extent of prototype engineering, field validation, and other activities that are required to assess the value of emerging research results. In some areas, a larger number of smaller projects may be most effective, while in other areas more experimental engineering is required and the research goals may be best addressed 13 This is analogous to the so-called “productivity paradox,” according to which economists struggled to account for the pro - ductivity benefits that accrued from investments made by firms in IT. The productivity improvements due to IT are now identi - fied, but for a long time there was speculation regarding whether the issue was productivity or the ability to measure particular influences on productivity. (This issue is also taken up in Chapter 1.) 14An inventory of risk issues for research program management appears in Chapter 4 of NRC, 2002, Information Technology Re- search, Innoation, and E-Goernment, Washington, DC: National Academy Press. Available online at http://www.nap.edu/catalog. php?record_id=10355. Last accessed August 20, 2010.

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE architectural models. This is a deeply technical challenge, as are many of the other challenges related to assurance. Goal .: Enhance the Portfolio of Preentie Methods to Achiee Assurance In addition to the primarily evaluative techniques, interventions in development activities can greatly enhance the potential for accreting, on an ongoing basis in development, a body of evidence in support of assurance cases. For example, assurance considerations affect architecture modeling, require - ments-related models, traceability and team information management tooling, programming language design, the design of runtime infrastructure, and many other areas. If research in this area is successful, the difference it will make will be evident in two ways. First, less work will produce higher assurance, in the form of stronger claims with respect to critical quality attributes, and, second, the balancing of effort in evidence production will shift from acceptance evalu - ation toward development, thus reducing engineering risk with respect to assurance. A wide range of technical ideas have emerged over the years in support of this concept, and this has also influenced language design. A crude way to think about this is that an existing language, together with additional specification information (e.g., types) and analysis capability (e.g., a type checker), can lead naturally to the next-generation programming of language in which the specification information becomes intrinsic and the analysis capability is integrated with the compiler, loader, and runtime system. Additional specific challenges include the following: • Preventive methods also include ideas building on the concept of “proof-carrying code” or more generally “evidence-carrying code.”22 • A significant enabler of preventive techniques in development activity is the adoption of processes and practices that enhance assurance. Examples include the Lipner/Howard Security Development Lifecycle and Gary McGraw’s process.23 These processes can continue to be enhanced and refined as new practices, tools, and languages emerge. • Architectural building blocks can be enhanced to facilitate instrumentation and logging in systems to support real-time, near-real-time, and forensic checking of consistency with models. It is important to note that not all significant attributes can be checked in this way, although sometimes modifications to architecture can expand the scope of what can be checked dynamically. • Develop architectures for containment such as sandboxing, process separation, virtual machines, and abstract machines. There is great opportunity to rethink basic concepts in systems software support, with a focus on achieving the simplifications that can lead to greater assurances regarding regulation of control and data flows among major components. The success of restricted ecosystems such as those evident on iPhones and other restricted platforms suggests the possibility of progress in this area. • Employ development techniques including co-development of software, selective specifications (for functional and quality attributes), and evidence of verification (consistency) of the software code with the specifications and associated models. Different techniques apply to different properties— what may be workable for particular quality attributes may not be useful for functional, performance, or deadline properties. Most of these techniques rely on some use of explicit specifications. A goal is to reduce the extent of specification required, ultimately to fragmentary specifications that enable design - ers and developers to distinguish what is an intended truth from what may be an accidental truth. The intended truth may be a design commitment that can be relied upon. The accidental truth may be a consequence of a particular algorithm or infrastructure choice that needs to be subject to revision as 22 George C. Necula and Peter Lee, 1998, Safe, Untrusted Agents Using Proof-Carrying Code. Lecture Notes in Computer Science— Mobile Agents and Security, London, UK: Springer-Verlag, pp. 61-91. 23 See Michael Howard and Steve Lipner, 2006, The Security Deelopment Lifecycle, Redmond, WA: Microsoft Press; also Grady McGraw, Software Security: Building Security In, Boston: Addison-Wesley.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH technology evolves. This co-development approach is intended to facilitate incremental and iterative development practices because it simultaneously creates software and assurance-related evidence. • The reality of enriched and diversified supply chains for software systems suggests that pervasive acceptance of preventive methods may not be fully achievable. For this reason, it is important to also address the challenge of improving a posteriori methods, including not only evaluative techniques, but also other approaches based on obfuscation and dynamic techniques. • Develop and use programming languages that enhance assurance. The experience of software developers is that language shifts occur at unpredictable times and for unpredictable reasons. None - theless, these shifts are generally extensively influenced by research. For example, Ada95, Java, and C# were all influenced by the same set of ideas regarding types, safe storage management, concurrency, name space management, access management, and many other languages elements. The emerging generation of domain-specific languages and dynamic languages is now well established, providing developers with greater flexibility in development practice but also less safety than the established languages. Research work could be accelerated to augment these languages with features that preserve the usability and flexibility while enhancing the potential for assurance. Area 3. Process Support and Economic Models for Assurance and Adaptability Chapters 2 and 4 both address issues related to process and assurance and suggest the following as research goals. Goal .: Enhance Process Support for Both Agile and Assured Software Deelopment This includes both product and process architectures based on identifying the parts of the product and process most needing agility or assurance, and organizing the architectures around them. For prod - ucts, one way to do this is by encapsulating the major sources of change into modules to be handled by agile methods.24 Examples of such sources of change are user interfaces, interfaces to independently evolving systems, or device drivers. For projects, one way to do this is to partition evolutionary devel - opment around stabilized high-assurance development increments, while a parallel team is handling the change traffic and developing the specifications and plans for the next increment. It also includes further improvements in information management for teams and larger develop - ment organizations. Areas of focus could beneficially include improved traceability particularly for formal and “semi-formal” information items, integration of models and analyses and simulation, and measurement support to facilitate iteration and evaluation (e.g., to dynamically identify and adapt to new sources of rapid change). Goal .: Address Supply-Chain Challenges and Opportunities As supply chains are enriched and diversified, there is an increasing potential benefit from tools that can manage a joint corpus of information and whose content and sharing is regulated according to a contractual relationship. Enhancements of this kind can better support evidence production by pro - ducers to accelerate client acceptance evaluation. The enhancements can also better support intertwined iterations. Such tools need to be reinforced by contractual provisions enabling visibility and measur- ability of development and risk management plans and progress vs. plans, both along a supply chain, and up and down the subcontractor chains. 24David Parnas, 1978, “Designing Software for Ease of Extension and Contraction,” Proceedings of the rd International Conference on Software Engineering, IEEE, Atlanta, GA, May 10-12, pp. 264-277.

OCR for page 112
0 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE Goal .: Facilitate Application of Economic Principles to Decision Making An additional area of potential significance is the development of applicable economic models to guide decision making, for example, related to the interplay of architecture choices, component and ecosystems choices, supply-chain choices, and attributes of cost, risk, schedule, and quality. As dis - cussed in Chapter 2, prioritizing features to enable time-certain development provides a strong proxy for economic value-based management and decision making. Goal .4: Deelop and Apply Policy Guidance and Infrastructure for Conducting Eidence-Based DoD Milestone Reiews As also discussed in Chapter 2, this task includes establishing the evidence of solution feasibility as a first-class deliverable, reviewing evidence-development plans, and tracking evidence development progress vs. plans via earned value management. It also requires research into which classes of process- focused evidence development (models, simulations, prototypes, benchmarks, exercises, instrumenta - tion, etc.) are best suited for which classes of system elements. Goal .: Enhance Process Support for Integrated Definition and Deelopment of System Hardware, Software, and Human Factors Requirements and Architectural Solutions Too often, system architectures are driven by hardware relationships that overly constrain software and human factors solutions. Examples of approaches are “soft systems engineering,” systems architect- ing, co-evolution, incremental iterative development (IID) models based on spiral development, and Brooks’s design processes and patterns.25 Area 4. Requirements The challenges for requirements are, in many respects, similar to those of architecture. How to achieve early validation? How to express the information that is gathered from stakeholders concerning both functional requirements and quality attributes? How to achieve traceability and model consistency that effectively links requirements with architecture and assurance? As noted in the previous chapters, requirements are only occasionally fully established at the outset of the development of an innovative software system. More often, there are early constraints on quality attributes, definitions of the overall scope of function and interlinking, and a few other “shall” or “must- have” constraints. Many of the other elements that eventually become manifest as features or quality attributes are in fact the result of early iterations with stakeholders, and many of these are informed by the improved understanding of both the technological and operational environments as they evolve. In other words, requirements engineering is an ongoing activity throughout development. For long- lived systems, as noted in the 2006 Software Engineering Institute (SEI) report Ultra-Large-Scale Systems, requirements engineering is ongoing throughout the lifetime of the system. 25 Soft systems engineering (see Peter Checkland, 1981, Systems Thinking, Systems Practice, Hoboken, NJ: Wiley); systems architecting (see Eberhardt Rechtin, 1991, Systems Architecting: Creating & Building Complex Systems, Englewood Cliffs, NJ: Prentice Hall), co-evolution (see Mary Lou Maher, Josiah Poon, and Sylvie Boulanger, 1996, “Formalizing Design Explora - tion as Co-evolution: A Combined Gene Approach,” pp. 3-30 in Adances in Formal Design Methods for CAD, John S. Gero and Fay Sudweeks, eds., London, UK: Chapman and Hall), the incremental commitment model upgrade of spiral development (NRC, Richard W. Pew and Anne S. Mavor, eds., 2007, Human-System Integration in the System Deelopment Process: A New Look, Washington, DC: National Academies Press, available online at http://books.nap.edu/catalog.php?record_id=11893), and Brooks’s design processes and patterns (see Fred Brooks, 2010, The Design of Design: Essays from a Computer Scientist, New York, NY: Addison-Wesley).

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH Goal 4.: Expressie Models and Supporting Tools A feature of modern requirements methodology is the capture of scenarios and use cases and the expression of these using effective but mostly informal notations. For agile or feature-driven develop - ments, attention is addressed to the granularity of featuring, so to speak, because that becomes both the basis of priority setting (“above the line”) and the metric of progress (“velocity” or “burn down”). For innovative large systems, there is more focus on capturing the model in a sufficiently precise form to support progress measurement and acceptance evaluation. Regardless of the approach, however, there are common core technical challenges, which are to improve our ability to express requirements-related models (in the sense of unified modeling language (UML) scenarios and use cases), to reason about those models (in the sense of the Massachusetts Institute of Technology’s Alloy), and to facilitate traceability with respect to architecture and implementation (correspondence measures). Requirements engineering is fundamentally about the transition from informal human expression to explicit structured represen - tations. Any incremental improvement in formality that doesn’t compromise overall expressiveness or efficiency of operation has the potential to make a big difference with respect to these goals. Related to this goal is the development of improved domain-specific models and methods that per- tain to critical defense domains such as control systems, command and control, large-scale information management, and many others. Goal 4.: Support Traceability and Early Validation Traceability is more readily achieved when the feature-driven model is adopted, but this is not always readily applicable to defense systems. Research on requirements expression will result in improvements to models, tooling, and early validation practices (e.g., prototyping and simulation). As part of this effort, it is essential to also address traceability issues, because these have a profound influ - ence on assurance and validation generally. Goal 4.: Process Support for Stakeholder Engagement and Model Deelopment Stakeholders in large projects may come from diverse perspectives and may have diverse interests. The requirements can often appear to be a negotiation among stakeholders regarding the significance of various functional features and quality attributes. This creates a challenge of avoiding both over-com - mitment (e.g., through negotiation) to particular characteristics as well as under-commitment. What modeling mechanisms, processes, and tools can be developed to assist stakeholders in identifying goals and models, and in managing not just what is committed to, but also how much commitment is made? This is particularly critical in incremental and iterative development projects. Area 5. Language, Modeling, Coding, and Tools As noted in the previous chapters, programming languages and associated capabilities have a considerable influence on the major factors identified in this report—architecture, assurance, process, and measurement. For example, programming language improvements have influence on the ability of architects to achieve goals related to system structure and modularity. More generally, programming languages are the medium by which human developers convey their intent both to the computer and to other developers. As such, a programming language both constrains what a developer can say and at the same time encourages particular styles of expression. As noted in the previous chapters, programming language design has considerable influence on the major factors identified in this report—architecture, assurance, process, and measurement. For example, programming language improvements have influ - ence on the ability of architects to enforce goals related to system structure and modularity. Modularity is much more than a matter of control and data flow. There are abstractions related to objects, types, and

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE actions that are increasingly supported in modern languages, and these enable developers to express domain concepts more directly in program text. They also enable architects to render their abstract models for system modular structures more directly into the explicit structure of programs. Modularity is much more than a matter of control and data flow; there are abstractions related to objects, types, and actions that are increasingly supported in modern languages, and these not only enable developers to express domain concepts more directly in program text 26 but also enable architects to render their abstract models for system modular structures more directly into the explicit structure of programs. Goal .: Enhance the Expressieness of Programming Language to Address Current and Emerging Challenges Despite many declarations of success, the evolution of programming languages continues, and it is driven by strong demand for improvements from developers seeking greater levels of expressiveness (e.g., through direct expression of concepts such as higher-order functions,27 deterministic parallelism, atomicity, data permissions, and so on), improved ability to support particular domains (either through direct expression as intrinsics or through the ability to provision appropriate libraries and frameworks), improved flexibility for developers (e.g., dynamic languages that compromise less static checking for a more rapid iterative development model, but with more risk of unwanted runtime errors), improved a priori assurance (e.g., through the simultaneous development of code, specifications, and associated proofs for assurance), and improved access to scalable performance (e.g., through intrinsics such as Generate/MapReduce that support data-intensive computing in microprocessor clusters). Goal .: Enhance Ability to Exploit Modern Concurrency, Including Shared Memory Multicore and Scalable Distributed Memory For the past 30 years, as a consequence of the steady improvements in processor design, software developers have been given a doubling in performance every year and a half, adding up to a million- fold improvement in three decades. Over that period, the same code ran faster on the new chips. In the past few years, processor clock speeds have topped out; there are now multiple processors on a chip, and chip designers continue to provide the expected performance improvement, but only in a potential way and accessible only to those software developers who can harness the power of multiple proces - sors. Suddenly, everything has to be “done by committee”—by multiple threads of control coordinating together to get the work done. It is said that Moore’s Law has given way to Amdahl’s Law. To make matters more difficult, the ability of multiple threads to access shared state in memory does not scale 26 I nthe early days of Fortran, for example, the only data types in the language were numbers and arrays of various dimension - alities. Any program that manipulated textual data, for example, needed to encode the text characters, textual strings, and any overarching paragraph and document structure very explicitly into numbers and arrays. A person reading program text would see only numerical and array operations, because that was the limit of what could be expressed in the notation. This meant that programmers needed to keep track, in their heads or in documentation, of the nature of this representational encoding. It also meant that testers and evaluators needed to assess programs through this (hopefully) same layer of interpretation. With modern languages (including more modern Fortran versions), these structures can be much more directly expressed—characters and strings are intrinsic in nearly all modern languages. This is a simple illustrative example, but the point remains: There are concepts and structures in domains significant to defense that, in modern languages, must be addressed through similar representational machinations. This is a part of the “endless value spiral” argument of Chapter 1, and it explains why we should not expect any plateau in the evolution of programming languages, models, and other problem-relevant expressive notations. Indeed, it is why language names such as “Fortran” and “Ada” have the staying power of strong brands, even when the specific languages to which they refer are evolving quite rapidly (for example, Ada83 to Ada95 and thence to Ada 2005). 27An example is Microsoft’s F#, which builds on two decades of work on advanced functional languages such as Standard ML and Haskell. Another example is Sun’s Fortress language, which builds on a combined heritage of functional programming, deterministic parallelism, and numerical computation.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH up—it must eventually be supplanted by distributed models, with information shared using message passing. This hybrid approach, combined with a distributed approach to scalable storage, is the reality of many modern high-performance data centers.28 Software developers, language designers, and tool developers are still struggling to figure out how to harness the concurrency in a way that works well for software development. What are the correct abstractions? What are suitable concepts for data structures? How can assurance be achieved when programs operate in non-deterministic fashion? This provisioning of modern computing power is a major challenge for language designers and tool designers. Goal .: Enhance Deeloper Productiity for New Deelopment and Eolution As noted above, languages enhanced with models and tools often merge into new languages that incorporate the model concepts directly in the language design. But there is a growing suite of tool capa - bilities that are conceptually separate from language, and the delivery of these capabilities is a significant influence on developer and team productivity and on software producibility generally. Modern tools such as the open source Eclipse (created by IBM29) and Microsoft’s Visual Studio environment for “man- aged code” provide rich features to support application development generally. They also have tailored support for development within certain ecosystems, such as the Visual Studio support for web applica - tions developed within the Microsoft Asp.NET framework. Individual developer tools are often linked with team capabilities, which include configuration management of code and related artifacts, defect and issue tracking and linking, build and test support, and management of team measures and processes. This linkage greatly empowers small teams and, increasingly, larger development organizations. Area 6. Cyber-Physical Systems DoD systems are increasingly operating in large-scale, network-centric configurations that take input from many remote sensors and provide geographically dispersed operators with the ability to interact with the collected information and to control remote effectors. In circumstances where the presence of humans in the loop is too expensive or their responses are too slow, these so-called cyber-physical systems must respond autonomously and flexibly to both anticipated and unanticipated combinations of events during execution. Moreover, cyber-physical systems are increasingly being networked to form long-lived systems of systems—and even ultra-large-scale systems 30—that must run unobtrusively and largely autonomously, shielding operators from unnecessary details (but keeping them apprised so they can react during emergencies), while simultaneously communicating and responding to mission-critical information at heretofore infeasible rates. Cyber-physical systems are increasingly critical in defense applications of all kinds and at all levels of scale, including distributed resource management in shipboard defense systems, coordinating groups of unmanned air vehicles, and controlling low-power sensors in tactical urban environments. These are systems with very close linkage of hardware sensors and effectors with software control. They are often structured as control systems, but also can involve multiple complex interacting control systems, such as in deconflicting multiple call-for-fire requests in a crowded battlespace consisting of joint services and coalition partners. One critical area of concern is the creation and validation of the cyber-physical stack. For example, 28 These issues are the focus of a forthcoming report from the National Research Council, The Future of Computing Performance: Game Oer or Next Leel?, Samuel Fuller and Lynette Millett, eds., Washington, DC: National Academies Press, forthcoming. 29 Siobhan O’Mahony, Fernando Cela Diaz, and Evan Mamas, 2005, “IBM and Eclipse (A),” Harvard Business School Case 906007, Cambridge, MA: Harvard University Press. 30 Software Engineering Institute, 2006, Ultra-Large-Scale Systems: The Software Challenge of the Future , Pittsburgh, PA: Carnegie Mellon University. Available online at http://www.sei.cmu.edu/library/assets/ULS_Book20062.pdf. Last accessed August 2 0, 2010.

OCR for page 112
4 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE how to evolve the development of distributed real-time and/or embedded systems from a cottage craft that does not generally yield scalable or readily assurance solutions to a more robust approach guided by model-integrated computing, domain-specific languages and analysis tools, and control-theoretic adaptation techniques. This is a significant challenge for language and platform design, ecosystem development, tool design, and practices.31 Another challenge facing the DoD is how to routinize and automate more of the development of embedded cyber-physical control systems. There are particular challenges related to scalability, predict - ability, and evolvability of systems. Assurance is also a major issue, and it is exacerbated by the lack of ability to reliably connect models with code and with the various components in the current generation of embedded software stack, which are typically optimized for time/space utilization and predictability, rather than ease of understanding, analysis, composition, scalability, and validation. Yet another challenge facing DoD cyber-physical systems—particularly net-centric cyber-physical systems—is how to handle variability and control adaptively and robustly. Cyber-physical systems today often work well as long as they receive all the resources for which they were designed in a timely fashion, but fail completely under the slightest anomaly. There is little flexibility in their behavior, that is, most of the adaptation is pushed to end users or administrators. Instead of hard failure or indefinite waiting, what net-centric cyber-physical systems require is either reconfiguration to reacquire the needed resources automatically or graceful degradation if they are not available. Goal .: Accelerate Ecosystem Deelopment for Cyber-Physical Systems Today, it is too often the case that substantial effort expended to develop cyber-physical systems focuses on either (1) building ad hoc solutions based on tedious and error-prone low-level platforms and tools or (2) cobbling together functionality missing in off-the-shelf real-time and embedded operating systems and middleware. As a result, subsequent composition and validation of these ad hoc capabilities is either infeasible or prohibitively expensive. One reason why redevelopment persists is that it is still often relatively easy to pull together a minimalist ad hoc solution, which remains largely invisible to all except the developers and testers. Unfortunately, this approach yields brittle, error-prone systems and substantial recurring downstream ownership costs, particularly for complex and long-lived network- centric DoD systems and larger-scale systems-of-systems. One of the most immediate goals is therefore to accelerate ecosystem development for cyber-physi - cal systems. There has been considerable exploration of this area in a multi-agency setting under the auspices of the NITRD coordination activity (see Box 1.5), and there are benefits to linking it with other efforts related to software producibility. There are opportunities to exploit and advance modern language concepts, innovative operating system and middleware ideas, scheduling and resource management techniques, and code generation capabilities. Achieving this goal will require new cyber-physical system software architectures whose component functional and quality-of-service (QoS) properties can be expressed with sufficient precision (e.g., via the use of model-integrated computing techniques and domain-specific languages and tools) that they can be predictably assembled with each other, leaving less lower-level complexity for application devel - opers to address and thereby reducing system development and ownership costs. In particular, cyber- physical system ecosystems must not simply build better device drivers, operating system schedulers, 31 Thischallenge was discussed in the committee’s 2007 workshop report. See NRC, 2007, Summary of a Workshop on Software- Intensie Systems and Uncertainty at Scale, Washington, DC: National Academies Press. Available online at http://www.nap.edu/ catalog.php?record_id=11936. Last accessed August 20, 2010. It has also been explored in the NRC report Software for Dependable Systems: Sufficient Eidence? See NRC, Daniel Jackson, Martyn Thomas, and Lynette I. Millett, eds., 2007, Software for Dependable Systems, Sufficient Eidence? Washington, DC: National Academies Press. Available online at http://www.nap.edu/catalog. php?record_id=11923. Last accessed August 20, 2010. It has been the subject of a series of workshops under the auspices of the NITRD HCSS area sponsored by NSF and other agencies. For more information see http://www.nitrd.gov/subcommittee/hcss. aspx. Last accessed August 20, 2010.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH or middleware brokers in isolation, but rather integrate these capabilities together and deliver them to applications in ways that enable them to achieve fine-grained tradeoffs between key QoS properties, such as throughout, latency, jitter, scalability, security, and reliability. Key R&D breakthroughs needed to meet the goal of accelerating ecosystem development for cyber- physical systems involve devising new languages and platforms that enable users and operators to clearly understand the QoS requirements and usage patterns of software components so it becomes possible to analyze whether or not these requirements are being (or even can be) met and to aggregate these requirements, making it possible to form decisions, policies, and mechanisms that can support effective global management in net-centric environments. Meeting these needs will require flexibility on the parts of both the application components and the cyber-physical system infrastructure ecosystem used throughout DoD systems. Goal .: Deelop Architectures and Software Frameworks to Support Embedded Applications Embedded cyber-physical systems can operate robustly in harsh environments through careful coor- dination of a complex network of sensors and effectors. Given the increasing complexity of emerging DoD embedded cyber-physical systems, such fine-tuned coordination is ordinarily a nearly impossible task, both conceptually and as a software engineering undertaking. Model-based software development uses models of a system to capture and track system requirements, automatically generate code, and semi-automatically provide tests or proofs of correctness. Models can also be used to build validation proofs or test suites for the generated code. Model-based software development removes much of the need for fine-tuned coordination, by allowing programmers to read and set the evolution of abstract state variables hidden within the physical system. For example, a program might state, “produce 10.3 seconds of 35% thrust,” rather than specify the details of actuating and sensing the hardware (e.g., “signal controller 1 to open valve 12,” and “check pressure and acceleration to confirm that valve 12 is open”). Hence a model-based program constitutes a high-level specification of intended state evolutions. To execute a model-based program an interpreter could use a model of a controlled plant to continuously deduce the plant’s state from observations and to generate control actions that move the plant to specified states. Achieving the goal of model-based embedded software development requires new expressive languages for specifying intended state evolutions and plant behavior, automated execution methods for performing all aspects of fine-grained coordination, and software architectures and frameworks for pervasive/immersive sensor networks. Key R&D breakthroughs needed to meet the goal of developing architectures and software frameworks to support embedded applications include closing the consis - tency gap between model and code, preserving structural design features in code, translating informal requirements into formal requirements, tracing requirements into implementation, integrating dispa - rately modeled submodels, and enriching formalisms that support QoS properties, as well as techniques that support rapid reconfiguration and reliability with unreliable components. Goal .: Deelop and Validate Technologies That Support Both Variability and Control in Net-Centric Cyber-Physical Systems As DoD cyber-physical systems become increasingly interconnected to form net-centric systems of systems it is becoming clear that (1) different levels of service are possible and desirable under different conditions and costs and (2) the level of service in one property must be coordinated with and/or traded off against the level of service in others to achieve the intended mission results. To date, little work has focused on techniques for controlling and trading off the overall behavior of these integrated net-centric cyber-physical systems. Another key goal is therefore to develop and validate new technologies that support both variability and control in net-centric cyber-physical systems. Achieving this goal will require devising new adaptive and reflective software technologies, recog -

OCR for page 112
 CRITICAL CODE: SOFTWARE PRODUCIBILITY FOR DEFENSE nizing that not all requirements can be met all of the time, yet still ensuring predictable and controllable end-to-end behavior. In adaptive software technologies, the functional and QoS-related properties of cyber-physical software can be modified either statically (e.g., to reduce footprint, leverage capabilities that exist in specific platforms, enable functional subsetting, and minimize hardware/software infra - structure dependencies) or dynamically (e.g., to optimize system responses to changing environments or requirements, such as changing component interconnections, power levels, CPU/network bandwidth, latency/jitter, and dependability needs). Reflective software technologies go further to permit automated examination of the capabilities they offer and automated adjustment to optimize those capabilities. Reflective techniques make the internal organization of systems—as well as the mechanisms used in their construction—both visible and manipulable for application and infrastructure programs to inspect and modify at runtime. Reflec - tive technologies thus support more advanced adaptive behavior and more dynamic strategies keyed to current circumstances, that is, necessary software adaptations can be performed autonomously based on conditions within the system, in the system’s environment, or in system QoS policies defined by operators. Key R&D breakthroughs needed to meet the goal of developing and validating adaptive and reflec - tive software for net-centric cyber-physical systems involve investigating ways to make such modifica - tions dependably (e.g., while meeting stringent—often conflicting—end-to-end QoS requirements) while simultaneously ensuring that the system functional requirements are met. Area 7. Human-Systems Integration It is significant that most large-scale complex enterprise systems include fallible humans as con - stituent elements, but there has been a lack of design practices, including architecture concepts and development processes, that account for the ways in which humans integrate into systems as partici - pants. Human-systems integration (HSI) is about much more than the colors of pixels and the design of graphical user integration frameworks. The presence of humans in a system, such as pilots in an airplane, fundamentally affects the design and architecture of that system. This issue was the subject of a separate NRC report32 and is not elaborated upon here except to emphasize some of its software-related recommendations: • Conduct a research program with the goal of revolutionizing the role of end users in designing the system they will use. • Conduct research to understand the factors that contribute to system resilience, the role of people in resilient systems, and how to design more resilient systems. • Refine and coordinate the definition of a systems development process that concurrently engi - neers the system’s hardware, software, and human factors aspects, and accommodates the emergence of HSI requirements, such as the incremental commitment model. • Research and develop shared representations to facilitate communication across different disci - plines and lifecycle phases. • Research and develop improved methods and testbeds for systems-of-systems HSI. • Research and develop improved methods and tools for integrating incompatible legacy and external-system user interfaces. 32 See NRC, 2007, Human-System Integration in the System Deelopment Process: A New Look, Washington, DC: National Academies Press, Available online at http://www.nap.edu/openbook.php?record_id=11893. Last accessed August 20, 2010.

OCR for page 112
 REINVIGORATE DOD SOFTWARE ENGINEERING RESEARCH Summary Findings and Recommendations Finding 5-2: Technology has a significant role in enabling modern incremental and iterative software development practices at levels of scale ranging from small teams to large distributed development organizations. Recommendation 5-1: The DoD should take immediate action to reinvigorate its investment in soft - ware producibility research. This investment should be undertaken through a diversity of research programs across the DoD and should include academia, industry labs, and collaborations. Recommendation 5-2: The DoD should take action to undertake DoD-sponsored research programs in the following areas identified as critical to the advancement of defense software producibility: (1) architecture modeling and architectural analysis; (2) assurance: validation, verification, analysis of design and code; (3) process support and economic models for assurance and adaptability; (4) requirements; (5) language, modeling, coding, and tools; (6) cyber-physical systems; and (7) human- systems integration.

OCR for page 112