National Academies Press: OpenBook

Ada and Beyond: Software Policies for the Department of Defense (1997)

Chapter: D Detailed Comparisons of Ada and Other Third-Generation Programming Languages

« Previous: C Glossary
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Appendix D

Detailed Comparisons of Ada and Other Third-Generation Programming Languages

This appendix presents details of studies, summarized in Chapter 2 in the sections titled "Analyses of Language Features" and "Comparisons of Empirical Data," that have analyzed the technical features of programming languages and have used empirical project data to compare languages' relative strength with respect to desired characteristics. Given the unscientific nature of most of the data, no strong conclusions are warranted. But in a general sense, most experience with Ada supports the conclusion that it benefits DOD warfighting applications; few data were found to refute such a conclusion, although the absence of such data may reflect a lack of any organized effort to promote the use of languages other than Ada in warfighting systems.

EVALUATIONS OF LANGUAGE FEATURES

A 1985 Federal Aviation Administration (FAA) study compared Ada with four other languages by conducting expert evaluations based on 48 technical language features arranged in six categories, and combining the evaluation results with the results of performance benchmark test results (IBM, 1985). A follow-on study was conducted by the Software Engineering Institute (SEI) in 1991 using the same expert evaluation methodology as the original study to compare Ada 83 with C++, but without the benchmark tests (Weiderman, 1991). The studies, the results of which are presented in Table D.1, compared the languages against the maximum possible scores for each of the six categories, which can be summarized as follows:

  • Capability—facets of the implementation language relevant to programming or software engineering;
  • Efficiency—factors relevant to optimization of generated code and run-time utilization of resources;
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.1 Language Functionality Attributes

 

 

FAA/IBM Study

 

 

 

SEI Study

Language

Attributes

Maximum

Score

Ada 83

C

Pascal

Jovial

Fortran

Ada

83

C++

Capability

16.7

16.1

9.6

10.4

7.6

3.9

15.3

11.3

Efficiency

16.4

8.0

11.8

10.8

11.0

11.1

10.7

10.9

Availability/ Reliability

22.6

21.5

11.6

14.5

15.6

10.3

19.1

12.6

Maintainability/

Extensibility

17.4

14.0

10.2

12.2

6.8

8.3

13.6

11.4

Life-cycle cost

11.3

8.2

7.4

7.8

4.9

5.2

8.4

8.0

Risk

15.6

8.8

8.9

7.6

9.6

8.2

11.7

9.8

Total

100.0

76.6

59.5

63.3

55.5

47.0

78.8

64.0

 

SOURCES: Federal Aviation Administration (FAA)/IBM data from IBM (1985); Software Engineering Institute (SEI) data from Weiderman (1991).

  • Availability/Reliability—factors influencing day-to-day safety of operational systems;
  • Maintainability/Extensibility—factors influencing long-term viability of operational systems;
  • Life-cycle cost—elements of cost associated with or affected by the implementation language; and
  • Risk—areas of uncertainty or concern associated with or affected by the implementation language.

Ada scored the highest overall in both studies. It scored the highest in each individual category, except for efficiency, where it was ranked last, and risk, where it was ranked first in the SEI study and third in the FAA study.

PROJECT EVALUATION DATA

In comparisons of Ada with other programming languages in actual development projects, empirical data on characteristics such as expressibility, maintainability, defect rates, reusability, and cost factors are of particular interest. Since Ada has been used in relatively large programs, a larger proportion of data has been collected for Ada than for contemporaneous commercial programming languages like C and C++.

Much of the data are subjective in nature, and care must be taken in drawing conclusions from such studies. For instance, the data in Table D.2 are confounded by differences in the expressive power of a source line of code in different programming languages. One way of normalizing uses tables of source lines of code per function point (Jones, 1995). However, as shown in Table D.2, these ratios have wide variability. Lubashevsky (1996) reports variations in source lines of code per function point exceeding factors of 2 for C and 6 for C++. Finally, there are differences in expressiveness for the same language across different application domains.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.2 Source Lines of Code per Function Point

Language

Low

Mean

High

Ada 83

60

71

80

C

60

128

170

C++

30

53

125

 

SOURCE: Data from Jones (1995).

Open-Source Data

The Verdix Ada Development System (VADS) data (Zeigler, 1995) covers a period of approximately 10 years of development and enhancement of VADS compilers, debuggers, builder tools, and run-time support systems using both C and Ada. Table D.3 summarizes the comparative C and Ada experience data over this period.

Zeigler analyzed potential confounding effects of relative C and Ada software complexity, personnel capability, and learning curve effects and found that these factors did not cause any significant bias. The study thus provides a strong case that Ada was significantly better than C, in terms of both life-cycle cost and freedom from defects, for this large (over 1 million lines of both Ada and C) project in the compiler and tools area. However, the study can only be suggestive rather than definitive about the applicability of this result to other domains and other teams (the Verdix teams were both composed of high-capability, low-turnover personnel).

Since 1976, NASA's Software Engineering Laboratory (SEL) has been carefully collecting software data on non-embedded, non-real-time satellite flight dynamics software. Figure D.1 (McGarry et al., 1994) summarizes SEL's history of development error rates over this period. The data show a significant general decrease in error rates over the period, due to such techniques as code reading, systematic testing, and Cleanroom techniques. The data from 1988 to 1994 also show significantly lower error rates for Ada than for Fortran, measured per 1,000 lines of delivered code, or KDLOC (an average of 2.1 defects/KDLOC for Ada vs. 4.8 defects/KDLOC for Fortran). SEL's analyses for Ada initially concluded that owing to reuse, Ada projects experienced significant cost and schedule reductions compared to Fortran projects. Subsequently, however, the Ada object-oriented reuse approach was applied to Fortran projects, resulting in comparable gains.

Proprietary Data

This section summarizes proprietary data from Reifer (1996) and Jones (1994) that compare Ada to other programming languages with respect to cost, delivered defects, reliability, and productivity. Because the source data and detailed analysis are unavailable, these results are difficult to assess.

Cost

Data on cost, reliability, and maintainability have been compiled in a proprietary database of outcomes of 190 software projects 3 years old or less (Reifer, 1996). These projects were written in Ada 83, Ada 95, C, C++, and other third-generation programming languages (3GLs, such as Cobol or Fortran) and were compared to a norm for each domain, drawn from a database of over 1,500 projects 7 years old or less.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.3 Verdix Ada Development System Development Data

Characteristic

C

Ada

Script Files

Other Files

Totals

Total lines

1,925,523

1,883,751

117,964

604,078

4,531,316

SLOCa

1,508,695

1,272,771

117,964

604,078

3,503,508

Files

6,057

9,385

2,815

3,653

21,910

Updates

47,775

34,516

12,963

12,189

107,443

New features

26,483

23,031

5,594

6,145

61,253

Fixes/feature

.52

.25

.82

.17

.41

Fixes/KSLOCb

9.21

4.59

39.02

1.75

7.25

Cost/SLOC

$10.52

$6.62

$15.38

$3.72

$8.10

Defects/KSLOC

.676

.096

n/a

n/a

.355

a SLOC: Source lines of code.

b KSLOC: 1,000 SLOC.

SOURCE: Zeigler (1995).

Reifer's project cost data, shown in Table D.4, indicate that, in general, Ada, C, and C++ programs were less expensive per delivered source line of code (SLOC) than other 3GL programs or the domain norm for military applications, although C and C++ programs were less expensive than Ada programs in all military domains except airborne and spaceborne weapons systems. In commercial domains, the development cost of C and C++ code was again less than or equivalent to the cost of Ada, and Ada was roughly equivalent in cost to other 3GLs and the domain norms. However, the wide variations in expressive power of a line of code in Ada, C, and C++ (see Table D.2) tend to overshadow the cost/SLOC differences shown in Table D.4.

Another study used telecommunications systems to evaluate object-oriented programming languages (Jones, 1994). The study used as a baseline a private branch exchange switch software application of 1,500 function points in size, compared across several telecommunications companies. Based on these data, Table D.5 shows the cost of developing the project in different languages, in dollars per function point. The results indicate that, in this domain, coding in C++ was less expensive than coding in Ada, while using C was more expensive.

Delivered Defects

Reifer's (1996) data, shown in Table D.6, indicate that in the 190 software projects represented, Ada code had fewer delivered defects than C and C++ code, which, in turn, had fewer defects than other 3GLs. In command and control and telecommunications applications, the difference between Ada and C/C++ was smaller than the difference between C/C++ and other 3GLs and the norm. In commercial products, information systems, and weapons systems, Ada code clearly had fewer defects than code in C, C++, and other 3GLs.

Based on Jones's (1994) data, Table D.7 indicates that, for telecommunications applications, the reliability in delivered defects per function point was roughly equivalent for C++ and Ada, and higher for C code.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

FIGURE D.1 NASA-SEL development error rates. KDLOC: 1,000 delivered lines of code. (Reprinted by permission from McGarry et al. (1994). Copyright 1994 by Carnegie Mellon University.)

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.4 Cost Data by Language (dollars per delivered source line of code)

Application Domain

Ada 83

Ada 95

C

C++

3GL

Norm

Command & Control

Commercial

50

n/a

40

35

50

45

Military

75

n/a

75

70

100

80

Commercial Products

35

30

25

30

40

40

Information Systems

Commercial

n/a

n/a

25

25

30

30

Military

30

35

25

25

40

35

Telecommunications

Commercial

55

n/a

40

45

50

50

Military

60

n/a

50

50

90

75

Weapons Systems

Airborne & Spaceborne

150

n/a

175

n/a

250

200

Ground-based

80

n/a

65

50

100

75

 

SOURCE: Reprinted from Reifer (1996).

Table D.5 Telecommunications Project Costs by Language (dollars per function point)

 

Assembly

C

CHILL

Pascal

Ada 83 Ada 9Xa

C++

Smalltalk

Costb

5,547

2,966

2,260

1,993

1,760

1,533

1,180

1,007

a Ada 9X figure based on simulation.

b At a monthly salary rate of $10,000.

SOURCE: Data from Jones (1994).

The fairly consistent factor-of-10,000 relationship between the data on cost per function point shown in Table D.5 and the data on defects per function point shown in Table D.7 suggests that Jones's cost and defect data are highly correlated to some third factor. This third factor appears to be the mean SLOC/function point ratio from Jones (1995), shown in Table D.2. Comparing these ratios across C, Ada, and C++ indicates that they are approximately, although not exactly, equal, as shown in Table D.8.

Reliability

Data from Reifer (1996) in Table D.9 indicate that the incidence of failure for Ada programs was lower than that with C, C++, and other 3GLs. The outcome of this reliability comparison is similar to the defect comparison; Ada was superior to C and C++, which were better than other 3GLs.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.6 Delivered Defect Data by Language (post-delivery errors per KSLOCa)

Application Domain

Ada 83

Ada 95

C

C++

3GL

Norm

Command & Control

Commercial

1.5

n/a

1.9

1.5

2.7

2.5

Military

0.7

n/a

1.0

1.2

2.3

2.0

Commercial Products

2.8

3.5

5.0

3.0

4.5

4.0

Information Systems

Commercial

4.0

n/a

7.0

5.1

7.0

7.0

Military

3.0

n/a

6.0

4.0

6.0

6.0

Telecommunications

Commercial

1.6

n/a

2.0

1.7

3.3

3.1

Military

1.0

n/a

1.5

1.2

2.7

2.5

Weapons Systems

Airborne & Spaceborne

0.3

n/a

0.8

0.6

1.0

1.0

Ground-based

0.5

n/a

0.8

0.7

1.0

1.0

a KSLOC: 1,000 source lines of code.

SOURCE: Reprinted from Reifer (1996).

Table D.7 Defects for Telecommunications Applications (delivered defects per function point)

 

Assembly

C

CHILL

Pascal

Ada 83

Ada 9Xa

C++

Smalltalk

Defects

0.52

0.29

0.23

0.20

0.17

0.15

0.14

0.13

a Ada 9X figure based on simulation.

SOURCE: Data from Jones (1994).

Table D.8 Correlation in Function Point (FP) Measures

 

SLOCa/FP (Jones, 1995)

Dollars/FP (Jones, 1994)

Defects/FP (Jones, 1994)

C/Ada

1.80

1.69

1.71

C++/Ada

0.75

0.67

0.82

a SLOC: Source lines of code.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

Table D.9 Reliability Data by Language (weeks until next major repair incidence)

Application Domain

Ada 83

Ada 95

C

C++

3GL

Norm

Command & Control

Commercial

3.0

n/a

2.5

n/a

2.0

2.5

Military

4.0

n/a

3.0

n/a

2.0

2.5

Commercial Products

1.0

n/a

0.4

1.0

0.4

0.5

Information Systems

Commercial

1.0

n/a

0.5

0.6

0.4

0.5

Military

0.8

n/a

0.5

n/a

0.4

0.5

Telecommunications

Commercial

3.0

n/a

1.0

2.0

1.5

1.8

Military

4.0

n/a

2.0

3.0

2.0

2.0

Weapons Systems

Airborne & Spaceborne

8.0

n/a

3.0

n/a

2.5

2.5

Ground-based

6.0

n/a

3.0

n/a

2.0

2.5

 

SOURCE: Reprinted from Reifer (1996).

Productivity

As described in Chapter 2 in the section titled "Software Engineering Process and Architecture," one of the key relationships in software development productivity is the relationship of development effort to program size that is, the value of the "process exponent." A MITRE study (Giallombardo, 1992) found substantial improvements in productivity for large embedded military software developments provided by Ada compared with other 3GLs. The study found that effort, as measured by staff-months required per 1,000 equivalent delivered source instructions (KEDSI; which counts terminal semicolons rather than source lines of code), increased linearly for the large Ada projects (process exponent close to 1). For large non-Ada projects, effort increased more than linearly; the increase in effort was also greater than that predicted by the COCOMO model, which has a process exponent of 1.2 for embedded or very large systems (Boehm and Royce, 1988). The differences were clearest for system sizes greater than 100 KEDSI (which the MITRE study used as a definition of "large").

In a study of telecommunications applications, Jones (1994) compared the productivity of several 3GLs, using function points per staff-month as the metric. The results, presented in Table D.10, indicate that, in this domain, C++ developments had a higher productivity rate than those using Ada, while developments using C were less productive than those using Ada.

Table D.10 Productivity for Telecommunications Projects by Language (function points per staff-month)

 

Assembly

C

CHILL

Pascal

Ada 83

Ada 9Xa

C++

Smalltalk

Productivity

1.80

3.37

4.42

5.01

5.68

6.52

8.47

9.99

a Ada 9X figure based on simulation.

SOURCE: Data from Jones (1994).

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×

CONCLUSION

In summary, based on the results of currently available empirical data and feature analysis comparisons, a conclusion that Ada is superior, with respect to availability/reliability and a lower incidence of defects, appears warranted. The evidence is not strong enough to assert Ada's superiority in the cost area. However, there is some positive evidence in the cost area, and when it is combined with the anecdotal conclusions favoring Ada (described in "Anecdotal Experience from Projects" in Chapter 2) and the lack of solid evidence indicating less expensive custom software development in other languages, a case can be made that using Ada provides cost savings in building custom software, particularly for real-time, high-assurance warfighting applications.

Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 92
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 93
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 94
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 95
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 96
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 97
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 98
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 99
Suggested Citation:"D Detailed Comparisons of Ada and Other Third-Generation Programming Languages." National Research Council. 1997. Ada and Beyond: Software Policies for the Department of Defense. Washington, DC: The National Academies Press. doi: 10.17226/5463.
×
Page 100
Next: E Briefings and Position Papers Received by the Committee »
Ada and Beyond: Software Policies for the Department of Defense Get This Book
×
Buy Paperback | $45.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Ada programming language was created by the U.S. Department of Defense (DOD) nearly two decades ago to provide a general-purpose programming language for defense and commercial use, but has evolved into a niche solution for safety-critical systems, primarily in defense applications. Ada and Beyond presents an approach for the DOD to move beyond the debate over its policy that requires the use of Ada for all new software development.

It describes the importance of the software engineering process and recommends to DOD mechanisms for more effective review of software development and improved collection of data on software project outcomes. The volume also analyzes the technical, empirical, and business cases for using Ada and other programming languages, makes recommendations regarding the appropriate conditions under which DOD should continue to require the use of Ada, and details activities that require funding by DOD in order for Ada to remain a viable programming language.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!