In producing critical software the primary worries are minimizing bugs in the software and ensuring reasonable behavior when nonmalicious users do unexpected things or when unexpected combinations of external events occur. Producing critical software presents the same problems as producing production-quality software, but because the cost of failure is higher, the standards must be higher. In producing critical software the goal is to decrease risk, not to decrease cost.
Secure software is critical software that needs to be resistant to attack. Producing it presents the same problems as does producing critical software, plus some others. One of the key problems is analyzing the kinds of attacks that the software must be designed to resist. The level and kind of threat have a significant impact on how difficult the software is to produce. Issues to consider include the following:
To what do potential attackers have access? The spectrum ranges from the keyboard of an automated teller machine to the object code of an operational system.
Who are the attackers and what resources do they have? The spectrum ranges from a bored graduate student, to a malicious insider, to a knowledgeable, well-funded, highly motivated organization (e.g., a private or national intelligence-gathering organization).
How much and what has to be protected?
In addition, the developers of secure software cannot adopt the various probabilistic measures of quality that developers of other software often can. For many applications, it is quite reasonable to tolerate a flaw that is rarely exposed and to assume that its having occurred once does not increase the likelihood that it will occur again (Gray, 1987; Adams, 1984). It is also reasonable to assume that logically independent failures will be statistically independent and not happen in concert. In contrast, a security vulnerability, once discovered, will be rapidly disseminated among a community of attackers and can be expected to be exploited on a regular basis until it is fixed.
In principle, software can be secure without being production quality. The most obvious problem is that software that fails frequently will result in denial of service. Such software also opens the door to less obvious security breaches. A perpetrator of an intelligence-grade attack (see Appendix E, "High-grade Threats") wants to avoid alerting the administrators of the target system while conducting an attack; a system with numerous low-level vulnerabilities provides a rich source of false alarms and diversions that can be used to cover up the actual attack or to provide windows of opportunity (e.g., when the system is recovering from a crash) for the subversion of hardware or software.