and mouse movements. For a hardware system, the inputs can include the environment of use and the actions of friendly and enemy soldiers. A few of the design flaws are assumed to be easy to find in that many inputs are likely to expose them, while many of the flaws are assumed to be relatively difficult to find in that very few inputs will disclose their presence. That is, assuming that inputs are selected uniformly from the space of all possible inputs, a few design flaws will be discovered with high probability, and many more will be discovered with low probability.
Some mathematical details follow. The system is assumed to have N faults. The assumption is also made that the waiting time to discovery of fault i (i = 1, ... , N) can be modeled using independent, exponential random variables Z1, Z2, ... ZN with respective failure rates λi. (Here fault i means the fault with label i, not the ith fault discovered.) The results of the testing are the first k waiting times (or cycles of operation) between the discovery of successive faults (again, not faults with successive labels), which are denoted D1,D2, ... , Dk. Conditional on the unobservable fault labels, the distribution of the Di’s is that of independent, exponential random variables with decreasing failure rates (decreasing since, of course, each time a fault is discovered, the system becomes more reliable). This conditional distribution is used to derive the unconditional marginal distribution of the first k Di’s, which in turn can be used to derive useful estimates concerning system reliability.
While the mathematics underlying inference for this model are complicated given that faults identified previously have an impact on the probability of discovering future faults, Scholz has derived the maximum-likelihood estimates of the residual failure rate at each stage in the fault discovery process using tools from the field of isotonic regression. Scholz has also provided upper bounds for confidence intervals for the residual failure rate. In other words, Scholz’s method estimates and bounds system reliability at each stage of the fault discovery process.
The Department of Defense collects considerable information on the performance of its systems while in development—especially test results— as well as when fielded. However, since test results are currently collected mainly to support decisions on whether to promote systems to the next stage in the milestone process, test data are often not saved and archived in a manner that facilitates their further use. Further, while data on field