National Academies Press: OpenBook

Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version (2008)

Chapter: Appendix E Role of Test and Evaluation in Department of Defense Acquisition

« Previous: Appendix D Summary of Committee Meetings
Suggested Citation:"Appendix E Role of Test and Evaluation in Department of Defense Acquisition." National Research Council. 2008. Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/12058.
×
Page 25
Suggested Citation:"Appendix E Role of Test and Evaluation in Department of Defense Acquisition." National Research Council. 2008. Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version. Washington, DC: The National Academies Press. doi: 10.17226/12058.
×
Page 26

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix E Role of Test and Evaluation in Department of Defense Acquisition The Department of Defense (DOD) major system acquisition process is described briefly here to explain the role of test and evaluation (T&E). Further information is widely available. A recent Congressional Research Service report provides a good overview of defense acquisition (Chadwick 2007). The acquisition of a new capability starts with the generation of a mission needs statement, which identifies and supports the need for a new or an improved capability. The acquisition process is initiated on approval of the mission needs statement by the secretary of defense (referred to as Milestone A). On approval of the secretary of defense, a DOD component runs the program. The Joint Program Executive Office Chemical and Biological Defense (JPEO CBD), which coordinates biological defense efforts of the four services, is the DOD component responsible for the T&E of biological standoff detection systems. The next phase in the acquisition process is a competitive exploration of alternative system concepts. Demonstration and validation follow approval of the alternative system concepts. Pending the outcome of the demonstration and validation, a preferred system is recommended. On approval of the secretary of defense (Milestone 1), full-scale engineering development of the preferred system starts. Procurement of long-lead production items and limited production for operational test and evaluation (OT&E) are approved at this time (Milestone B). On successful completion of the full-scale engineering development and on the basis of the outcome of the OT&E, JPEO CBD may recommend production of a detection system. If approved by the secretary of defense (Milestone C), production of the system begins, and the services are authorized to deploy the system. Two types of measurements are required to verify and validate the performance of lidar systems. First, metrics must be provided to establish the quality of the modeling and simulation products used to relate the performance of the lidar during both T&E and later operation. The metrics will identify the underlying assumptions that produce a positive detection event from a certain type of data and the confidence level and variability in that decision-making process. Second, metrics must be provided to yield functional data that go into a model to produce a positive call. These measurements should identify the variation in each element of the data and explain how uncertainties are compounded. Often, they are measurements of performance and effectiveness and of the merit of the system under test (for more information, see http://vva.dmso.mil/Special_topics/Measures/default.htm). Measures of merit (MOMs) relate the effects of a concept or system to the mission that the concept or system supports. MOMs measure concept or system capabilities in terms of the effects of the capabilities on the overall mission of which the concept or system is a part. They cover mission attributes that define the overall objectives of the simulation. For example, an 25

attribute of a standoff detector is usability. Measures of detector usability might include weight, power use, mean time between failures, and difficulty in reading data. Measures of effectiveness (MOEs) assess a system’s effectiveness in the accomplishment of a task. MOEs measure capabilities in terms of task accomplishment or system attributes. Tested capabilities should be related directly to operational capabilities in terms of engagement or battle outcome. MOE evaluation criteria (acceptability criteria) should be quantitative if possible. For example, measures of standoff detection include accuracy, false-alarm rates, response time, reliability, range of detection, and discrimination between target and threat. Measures of performance (MOPs) gauge system or component capabilities or characteristics. MOPs are quantitative or qualitative measures of simulation capabilities and characteristics. They are based on capabilities and characteristics that are defined by the requirements of the intended application or that meet user-defined system performance requirements. Quantitative MOPs are used when it is difficult to assess an MOE directly or when quantitative criteria need to be established. Qualitative MOPs are categorical measures of performance that refer to the presence or absence of specified characteristics. Quantitative MOPs can frequently be related to a numerical scale. A MOP for a standoff detector would be how many simultaneous plumes can be detected, tracked, and analyzed. A qualitative MOP would be how much more rapidly a standoff detector allows a battlefield commander to decide to change a protective posture. Subjective measurement techniques are generally used to address qualitative MOPs. Associated with each measure is a criterion that shows how well the measure needs to be addressed by the simulation if it is to be acceptable for the intended use. Those criteria are typically called acceptability criteria because they define a minimal level of performance, degree of effectiveness, level of success, or the like that the simulation needs to achieve to be acceptable to the user. 26

Test and Evaluation of Biological Standoff Detection Systems: Abbreviated Version Get This Book
×
Buy Paperback | $21.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A biological warfare agent (BWA) is a microorganism, or a toxin derived from a living organism, that causes disease in humans, plants, or animals or that causes the deterioration of material. The effectiveness of a BWA is greatly reduced if the attack is detected in time for the target population to take appropriate defensive measures. Therefore, the ability to detect a BWA, in particular to detect it before the target population is exposed, will be a valuable asset to defense against biological attacks. The ideal detection system will have quick response and be able to detect a threat plume at a distance from the target population. The development of reliable biological standoff detection systems, therefore, is a key goal.

However, testing biological standoff detection systems is difficult because open-air field tests with BWAs are not permitted under international conventions and because the wide variety of environments in which detectors might be used may affect their performance. This book explores the question of how to determine whether or not a biological standoff detection system fulfills its mission reliably if we cannot conduct open-air field tests with live BWAs.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!