. "Appendix D: The Life Cycle of Technology, Systems, and Programs." Protecting Individual Privacy in the Struggle Against Terrorists: A Framework for Program Assessment. Washington, DC: The National Academies Press, 2008.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Protecting Individual Privacy in the Struggle Against Terrorists: A Framework for Program Assessment
use in evaluating fingerprint and other biometric detection algorithms may serve as a useful model.1 The program should proceed beyond this phase only after demonstration that a sound experimental basis exists in a laboratory setting; measures of effectiveness and measures of performance will likely require refinement during this phase of program development.
System development and demonstration. During this phase, the program should be field-tested—subjected to the equivalent of human subject trials in the drug development process or operational test and evaluation (OT&E) in traditional technology development programs. The test environment must mimic real-world conditions as nearly as possible, and so both the simulation environment and requisite data sets must be designed and implemented with appropriate oversight. If it is necessary, for example, to use real-world data, then the test regime must provide appropriate protections to guard against inappropriate use of either the data or the results. During this phase of testing, the various elements of question 3 of the effectiveness criteria summary in Section 2.5.1 should be addressed (field-tested? tested to take into account real-world conditions? successful in predicting historical events? experimental successes replicated?), as should questions 4, 6, and 7 (scalability, capability for integration with relevant systems and tools, robustness in the field and against countermeasures). In addition, the development team should respond to questions 8 and 9 (guarantees regarding appropriateness and reliability of data, provision of appropriate data stewardship).
Also, given the class of programs under consideration in this report, a requirement for IV&V is needed at this phase of the life cycle. The IV&V process should review results from prior phases of testing and address the inquiries in question 10 (objectivity). Measures of effectiveness and measures of performance should be finalized for use in ongoing monitoring of the program if it is subsequently operationally deployed.
Operational deployment. The final gate prior to operational deployment is an agency-level review of all items delineated in the summary of criteria for evaluating consistency with laws and values in Section 2.5.2, assurance that an ongoing monitoring process is in place, and definition of the conditions for operational deployment (e.g., threshold values for key measures). This review process should ensure that compliance is documented and reviewed in accordance with question 12 of the effectiveness criteria summary in Section 2.5.1.
