methodology, typically using students as the experimental subjects. Industrial experiments fall somewhere between case studies and academic experiments. Because of the expense and difficulty of performing extensive controlled experiments on software, case studies are often resorted to. The ideal situation is to be able to take advantage of real-world industrial operations while having as much control as is feasible. Much of the present work in this area is at best anecdotal and would benefit greatly from more rigorous statistical advice and control. The panel foresees an opportunity for innovative work on combining information (see below) from relatively disparate experiences.
Conducting statistically valid software experiments is challenging for several reasons:
The software production process is often chaotic and uncontrolled (i.e., immature);
Human variability is a complicating factor; and
Industrial experiments are very costly and therefore must produce something useful.
Many variables in the software production process are not well understood and are difficult to control for. For software engineering experiments, the factors of interest include the following:
"People" factors: number, level, organization, process experience;
Problem factors: application domain, constraints, susceptibility to change;
Process factors: life cycle model, methods, tools, programming language;
Product factors: deliverables, system size, system reliability, portability; and
Resource factors: target and development machines, calendar time, budget, existing software, and so on.
Each of these characteristics must be modeled or controls done for the experiment to be valid.
Human variability is particularly challenging, given that the difference in quality and productivity between the best and worst programmers may be 20 to 1. For example, in an experiment comparing batch versus interactive computing, Sackman (1970) observed differences in ability of up to 28 to 1 in programmers performing the same task. This variation can overwhelm the effects of a change in methodology that may account for a 10% to 15% difference in quality or productivity.
The human factor is so strongly integrated with every aspect of the subjective discipline of software engineering that it alone is the prime driver of issues to be addressed. The human factor creates issues in the process, the product, and the user environment. Measurements of the objects (the product and the process) are obscured when qualified by the attributes (ambiguous requirements and productivity issues are key examples). Recognizing and characterizing the human attributes within the context of the software process are key to understanding how to include them in system and statistical models.
The capabilities of individuals strongly influence the metrics collected throughout the software production process. Capabilities include experience, intelligence, familiarity with the application domain, ability to communicate with others, ability to envision the problem spatially, and ability to verbally describe that spatial understanding. Although not scientifically founded, anecdotal information supports the incidence of these capabilities (Curtis, 1988).