National Academies Press: OpenBook
« Previous: Appendix A: Workshop Agenda and Participants
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

Appendix B
Relevant Readings

INTELLIGENCE AND INTELLIGENCE ANALYSIS-GENERAL

Bennett, M., and Waltz, E. (2007). Counterdeception: Principles and applications for national security. Boston: Artech House.

Bruce, J.B. (2008). Making analysis more reliable: Why epistemology matters to intelligence In R.Z. George and J.B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 171-190). Washington, DC: Georgetown University Press.

Bruce, J.B., and Bennett, M. (2008). Foreign denial and deception: Analytical imperatives. In R.Z. George and J.B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 122-137). Washington, DC: Georgetown University Press.

Clark, R.M. (2007). Intelligence analysis: A target-centric approach. Washington, DC: CQ Press.

Davis, J. (1999). Introduction: Improving intelligence analysis at CIA: Dick Heuer’s contribution to intelligence analysis. In R.J. Heuer, Jr. (Ed.), Psychology of intelligence analysis (pp. xiii-xxv). Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.

Folker, R. (2000). Intelligence analysis in theater joint intelligence centers: An experiment in applying structured methods. Occasional Paper 7. Washington, DC: Joint Military Intelligence College. Available: http://www.au.af.mil/au/awc/awcgate/dia/analysis_structured. pdf [accessed February 2010].

George, R.Z., and Bruce, J.B. (2008). Analyzing intelligence: Origins, obstacles, and innovations. Washington, DC: Georgetown University Press.

Goodson, R., and Wirtz, J. (2008). Strategic denial and deception: The twenty-first century challenge. In R. Goodson and J.J. Wirtz (Eds.), Strategic denial and deception: The twenty-first century challenge (pp. 1-14). London: Transaction.

Heuer, R.J., Jr. (Ed.). (1979). Quantitative approaches to political intelligence: The CIA experience. Boulder, CO: Westview Press.

Heuer, R.J., Jr. (1999). Psychology of intelligence analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

Johnston, R. (2005). Analytic culture in the U.S. intelligence community. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency. Available: https://www. cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/analytic-culture-in-the-u-s-intelligence-community/analytic_culture_ report.pdf [accessed February 2010].

Kennedy, R. (2008). Of knowledge and power: The complexities of national intelligence. Wesport, CT: Praeger Security International.

Kent, S. (1949). Strategic intelligence for American world policy. Princeton, NJ: Princeton University Press.

Kent, S. (1955). The need for an intelligence literature. Reprinted in D.P. Steury (Ed.), Sherman Kent and the Board of National Estimates. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency. Available: https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kent-and-the-board-of-national-estimates-collected-essays/2need.html [accessed February 2010].

Kent, S. (1964). Words of estimative probability. Reprinted in D.P. Steury (Ed.), Sherman Kent and the Board of National Estimates. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency. Available: https://www.cia.gov/library/centerfor-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kentand-the-board-of-national-estimates-collected-essays/6words.html [accessed February 2010].

Lehner, P. (2009). The objective analysis of analysis. Paper presented at the Community of Interest for the Practice and Organization of Intelligence Ottawa Roundtable “What Can the Cognitive and Behavioral Sciences Contribute to Intelligence Analysis?: Towards a Collaborative Agenda for the Future,” February, Meech Lake, Quebec.

Mandel, D.R. (2009). Setting the stage: The role of science in applied communities. Paper presented at the Community of Interest for the Practice and Organization of Intelligence Ottawa Roundtable “What Can the Cognitive and Behavioral Sciences Contribute to Intelligence Analysis?: Towards a Collaborative Agenda for the Future,” February, Meech Lake, Quebec.

Marrin, S. (2009). Training and educating U.S. intelligence analysts. International Journal of Intelligence and Counter-Intelligence, 22, 131-146.

Milgram, S. (1974). Obedience to authority: An experimental view. New York: Harper & Row.

Moore, D.T. (2006). Critical thinking and intelligence analysis. Occasional Paper Number 14. Joint Military Intelligence College, May.

Rieber, S. (2004). Intelligence analysis and judgmental calibration. International Journal of Intelligence and Counter-Intelligence, 17, 97-112.

Rieber, S., and Thomason, N. (2006). Toward improving intelligence analysis: Creation of a National Institute of Analytic Methods. Studies in Intelligence, 49(4), 71.

Sims, J.E., and Gerber, B. (2005). Transforming U.S. intelligence. Washington, DC: Georgetown University Press.

Swenson, R.G. (2002). Meeting the community’s continuing need for an intelligence literature. Defense Intelligence Journal, 11(2), 87-98.

Warner, M. (2009). Sources and methods for the study of intelligence. In L.K. Johnson (Ed.), Handbook of intelligence studies (pp. 17-27). New York: Routledge.

Weinstein, N. (1980). Unrealistic optimism about future life events. Journal of Personality and Social Psychology, 39, 806-820.

Westerfield, H.B. (Ed.). (1995). Inside CIA’s private world: Declassified articles from the agency’s internal journal, 1955-1992. New Haven: Yale University Press.

Wirtz, J.J. (2009). The American approach to intelligence studies. In L.K. Johnson (Ed.), Handbook of intelligence studies (pp. 28-38). New York: Routledge.

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

Yarger, H.R. (2008). Strategy and the national security professional: Strategic thinking and strategic formulation in the 21st century. Westport, CT: Praeger.

ALTERNATIVE COMPETING HYPOTHESES

Billman, D., Convertino, G., Shrager, J., Massar, J.P., and Pirolli, P. (2006). Collaborative intelligence analysis with CACHE and its effects on information gathering and cognitive bias. Technical Report. Palo Alto, CA: Palo Alto Research Center.

Cheikes, B.A., Brown, M.J., Lehner, P.E., and Adelman, L. (2004). Confirmation bias in complex analyses. Technical Report. Bedford, MA: MITRE Center for Integrated Intelligence Systems.

Cluxton, D., and Eick, S.G. (2005). DECIDETM: Hypothesis visualization tool. Presented at the International Conference on Intelligence Analysis Methods and Tools, May 2-6, McLean, VA.

Convertino, G., Billman, D., Pirolli, P., Massar, J.P., and Shrager, J. (2006). Collaborative intelligence analysis with CACHE: Bias reduction and information coverage. Technical Report. Palo Alto, CA: Palo Alto Research Center.

Convertino, G., Billman, D.O., Pirolli, P.L., Massar, J.P., and Shrager, J. (2008). The CACHE study: Group effects in computer-supported collaborative analysis. Computer Supported Cooperative Work, 17, 353-393.

Good, L., Shrager, J., Stefik, M., Pirolli, P., and Card, S. (2004). ACH0: A tool for analyzing competing hypotheses. Technical description for Version 1.0. Palo Alto, CA: Palo Alto Research Center.

Heuer, R.J., Jr. (2008). Computer-aided analysis of competing hypotheses. In R.Z. George and J.B. Bruce (Eds.), Analyzing intelligence: Origins, obstacles, and innovations (pp. 251-265). Washington, DC: Georgetown University Press.

Heuer, R.J., Jr. (2008). Improving intelligence analysis with ACH. Available: http://www. pherson.org/PDFFiles/Heuer-ImprovingIntelligenceAnalysiswithACH.pdf [accessed February 2010].

Lehner, P.E., Adelman, L., Cheikes, B.A., and Brown, M.J. (2008). Confirmation bias in complex analyses. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 38(3), 584-592.

Pherson, R. (2008). Collaborative analysis of competing hypotheses (C-ACH). Available: http://www.pherson.org/pdffiles/Collaborative-ACH.pdf [accessed February 2010].

Pirolli, P. (2006). Assisting people to become independent learners in the analysis of intelligence. Technical Report. Palo Alto, CA: Palo Alto Research Center.

Pope, S., and Jøsang, A. (2005). Analysis of competing hypotheses using subjective logic. Tenth International Command and Control Research and Technology Symposium, June, McLean, VA.

Stech, F.J., and Elsaesser, C. (2005). Deception detection by analysis of competing hypotheses. Technical Report. McLean, VA: MITRE Corporation.

Valtorta, M., Dang, J., Goradia, H., Huang, J., and Huhns, M. (2005). Extending Heuer’s analysis of competing hypotheses method to support complex decision analysis.Presented at the International Conference on Intelligence Analysis Methods and Tools, May 2-6, McLean, VA.

Wheaton, K.J., and Chido, D.E. (2006). Structured analysis of competing hypotheses: Improving a tested intelligence methodology. Competitive Intelligence Magazine, 9(6), 12-15.

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

APPLIED BAYESIAN ANALYSIS

Dawes, R.M., and Corrigan. B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.

Dawes, R., Faust, R., and Meehl, P. (1989). Clinical versus actuarial judgment. Science, 243(4899), 1668-1674.

Digman, J.M. (1990). Personality structure: An emergence of the five-factor model. Annual Review of Psychology, 41, 417-440.

Fisk, C. (1972). The Sino-Soviet border dispute: A comparison of the conventional and Bayesian methods for intelligence warning. Studies in Intelligence, 16(2), 53-62.

Grove, W.M., Zald, D.H., Lebow, B.S., Snitz, B.E., and Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19-30. Available: http://www.psych.umn.edu/faculty/grove/096clinicalversusmechanicalprediction. pdf [accessed February 2010].

Kahneman, D., Slovic, P., and Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.

Kent, S. (1964). Words of estimative probability. Reprinted in D.P. Steury (Ed.), Sherman Kent and the Board of National Estimates. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency. Available: https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/sherman-kentand-the-board-of-national-estimates-collected-essays/6words.html [accessed February 2010].

Neapolitan, R.E. (1990). Probabilistic reasoning in expert systems. New York: Wiley.

Nisbett, R.E., and Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice-Hall.

Pearl, J. (1988). Probabilistic reasoning in intelligent systems. San Mateo, CA: Morgan Kaufmann.

Schum, D.A. (1994). Evidential foundations of probabilistic reasoning. New York: Wiley.

Sticha, P., Buede, D., and Rees, R.L. (2005). APOLLO: An analytical tool for predicting a subject’s decision making. Presented at the International Conference on Intelligence Analysis Methods and Tools, May 2-6, McLean, VA.

Wallsten, T.S., Budescu, D.V., and Zwick, R. (1993). Comparing the calibration and coherence of numerical and verbal probability judgments. Management Science, 39, 176-190.

PRELIMINARY CREDIBILITY ASSESSMENT SCREENING SYSTEM (PCASS)

Battelle Memorial Institute. (2007). Efficacy of prototype credibility assessment technology: PCASS final report. Technical Report. Columbus, OH: Battelle Memorial Institute (Defense Academy for Credibility Assessment).

Ben-Shakhar, G., and Eitan, E. (2003). The validity of psychophysiological detection of information with the guilty knowledge test: A meta-analytic review. Journal of Applied Psychology, 88(1), 131-151.

Bhatt, S., and Brandon, S.E. (2008). Review of the preliminary credibility assessment screening system (PCASS). Unpublished manuscript, Washington, DC.

Clapper, J.R., Jr. (2007). Operational approval of the Preliminary Credibility Assessment Screening System (PCASS). Washington, DC: U.S. Department of Defense.

Dedman, B. (2008a). New U.S. weapon: Hand-held lie detector. Available: http://www.msnbc. msn.com/id/23926278/ [accessed February 2010].

Dedman, B. (2008b). What is the PCASS and how does it work? Available: http://www.msnbc. msn.com/id/24015982/ [accessed February 2010].

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

Harris, J.C., and McQuarrie, A.D. (2006). The Preliminary Credibility Assessment System embedded algorithm description and validation results. Technical Report. Laurel, MD: Applied Physics Laboratory, Johns Hopkins University (Counterintelligence Field Activity).

National Research Council. (2003). The polygraph and lie detection. Committee to Review the Scientific Evidence on the Polygraph. Board on Behavioral, Cognitive, and Sensory Sciences and Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

Senter, S., Waller, J., and Krapohl, D. (2006). Validation studies for the Preliminary Credibility Assessment Screening System (PCASS). Technical Report. Fort Jackson, SC: U.S. Department of Defense Polygraph Institute.

Thompson, D. (2007). Evaluation of the Preliminary Credibility Assessment Screening System (PCASS). Camp Cropper, Iraq, Department of the Army. Available: http://antipolygraph.org/documents/PCASS_Camp_Cropper_memo.pdf [accessed February 2010].

VOICE STRESS ANALYSIS

Baker Group. (2007). What is DVSA. Available: http://bakerdvsa.com/whatisdvsa.htm [accessed February 2010].

Bhatt, S., and Brandon, S.E. (2009). Review of voice stress–based technologies for the detection of deception. Unpublished manuscript, Washington, DC.

Brown, T.E., Senter, S.M., and Ryan, A.H., Jr. (2003). Ability of the VericatorTMto detect smugglers at a mock security checkpoint. Report No. DoDPI00-P-0024. Fort Jackson, SC: U.S. Department of Defense Polygraph Institute.

Cestaro, V.L. (1995). A comparison between decision accuracy rates obtained using the polygraph instrument and the computer voice stress analyzer (CVSA) in the absence of jeopardy. Report No. DoDPI95-R-0002. Fort McClellan, AL: U.S. Department of Defense Polygraph Institute.

Cestaro, V.L. (1996). A comparison of accuracy rates between detection of deception examinations using the polygraph and the computer voice stress analyzer in a mock crime scenario. Report No. DoDPI-R-0004. Fort McClellan, AL: U.S. Department of Defense Polygraph Institute.

Diogenes Company. (2008). Diogenes Digital Voice Stress AnalysisTMDDVSATMfeatures. Kissimmee, FL: Author.

Haddad, D., Walter, S., Ratley, R., and Smith, M. (2001). Investigation and evaluation of voice stress analysis technology. In-house technical memorandum AFRL-IF-RS-TM-2001-7. Rome, NY: Air Force Research Laboratory Information Directorate.

Harnsberger, J. D., Hollien, H., Martin, C.A., and Hollien, K.A. (2009). Stress and deception in speech: Evaluating layered voice analysis. Journal of Forensic Science, 54(3), 642-650.

Hollien, H., and Harnsberger, J.D. (2006). Voice stress analyzer instrumentation evaluation. Final Report CIFA Contract FA 4814-04-0011. Gainesville, FL: Institute for Advanced Study of the Communication Processes, University of Florida.

Hollien, H., Harnsberger, J.D., Martin, C.A., and Hollien, K.A. (2008). Evaluation of the NITV CVSA. Journal of Forensic Science, 53(1), 183-193.

Holman, D. (2005). Nothing but the truth. The American Spectator. Available: http://spectator. org/archives/2005/12/15/nothing-but-the-truth [accessed February 2010].

Hopkins, C.S., Ratley, R.J., Benincasa, D.S., and Greico, J.J. (2005). Evaluation of voice stress analysis technology. 38th Hawaii International Conference on System Sciences, IEEE.

Janniro, M.J., and Cestaro, V.L. (1996). Effectiveness of detection of deception examinations using the computer voice stress analyzer. Report No. DoDPI-R-0005. Fort McClellan, AL: U.S. Department of Defense Polygraph Institute.

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×

Krapohl, D., Ryan, A.H., and Shull, K.W. (2002). Voice stress devices and the detection of lies. Policy Review. Available: http://hnspolygraph.com/media/voice_stress_devices_ and_the_detection_of_lies.pdf [accessed February 2010].

Meyerhoff, J.L., Saviolakis, G.A., Koenig, M.L., and Yourick, D.L. (2000). Physiological and biological measures of stress compared to voice stress analysis using the computer voice stress analyzer (CVSA). Report No. DoDPI98-R-0004. Fort Jackson, SC: U.S. Department of Defense Polygraph Institute.

Nemesysco. (2005). Old versions—VericatorTM. Available: http://www.nemesysco.com/Prod_VERICATOR.html [accessed February 2010].

Nemesysco. (2008). LVA—Layered voice analysis—Nemesysco’s technologies. Available: http://www.nemesysco.com/technology-lvavoiceanalysis.html [accessed February 2010].

NITV. (2007). Structured interview assessment of the field use of the voice stress analyzer technology. Available: http://www.cvsa1.com/VSAAssessment.pdf [accessed February 2010].

NITV. (2008a). Agencies utilizing the CVSA. Available: http://www.cvsa1.com/Agencies using.htm [accessed February 2010].

NITV. (2008b). U.S. special operations command independent evaluation validates the CVSA. Available: http://www.cvsa1.com/USSpecialOper.htm [accessed February 2010].

Palmatier, J.J. (1999). The computerized voice stress analyzer: Modern technological innovation or “the emperor’s new clothes”? ABA General Practice, Solo & Small Firm Division Magazine. Available: http://www.abanet.org/genpractice/magazine/1999/jun/palmatr.html [accessed February 2010].

Palmatier, J.J. (2005). Assessing credibility: ADVA technology, voice and voice stress analysis. In R.J. Montgomery and W.J. Majeski (Eds.), Corporate Investigations (pp. 37-60). Tuscon, AZ: Lawyers & Judges.

Sommers, M.S. (2006). Evaluating voice-based measures for detecting deception. Journal of Credibility Assessment and Witness Psychology, 7(2), 99-107.

STIG. (2008). VSA-2000 lie detection lab. Available: http://www.secintel.com/p-509-vsa-2000-lie-detection-lab.aspx [accessed February 2010].

Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 99
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 100
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 101
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 102
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 103
Suggested Citation:"Appendix B: Relevant Readings." National Research Council. 2010. Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/12854.
×
Page 104
Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary Get This Book
×
Buy Paperback | $41.00 Buy Ebook | $32.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On September 22-23, 2009, the National Research Council held a workshop on the field evaluation of behavioral and cognitive sciences—based methods and tools for use in the areas of intelligence and counterintelligence. Broadly speaking, the purpose of the workshop was to discuss the best ways to take methods and tools from behavioral science and apply them to work in intelligence operations. More specifically, the workshop focused on the issue of field evaluation—the testing of these methods and tools in the context in which they will be used in order to determine if they are effective in real-world settings.

This book is a summary and synthesis of the two days of presentations and discussions that took place during the workshop. The workshop participants included invited speakers and experts from a number of areas related to the behavioral sciences and the intelligence community. The discussions covered such ground as the obstacles to field evaluation of behavioral science tools and methods, the importance of field evaluation, and various lessons learned from experience with field evaluation in other areas.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!