|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
|
| ||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 3
Chapter 1
Program, Curriculum, and Syllabi Review
BACKGROUND
The call to arms for increased educational scientific opportunities for all Americans has been
sounded by many parties, including the members of the National Research Council's (NRC’s)
Committee on Prospering in the Global Economy of the 21st Century, characterized by Thomas
Friedman as a "nonpartisan group of America's most distinguished engineers, scientists, educators
and industrialists" (Friedman, 2010). These opportunities need to be extended not only to students
in the normal educational pipelines but also to veterans and those in active military service. There
is a clear need for increased training and education in the sciences, technology, engineering, and
mathematics (STEM) at all levels in the United States. This need, so clearly elucidated by
President Barack Obama in his "Educate to Innovate" campaign for excellence in STEM
education in November 2009 (White House, 2009), is also faced by the U.S. intelligence
community. As the world becomes more technologically advanced, the need for intelligence
officers and analysts skilled in science and technology intelligence (S&TI) increases.
The committee agrees that increased S&TI capability is critical given the current and future
threat environment. It offers several reasons why the United States needs analysts and intelligence
officers with S&TI skills:
The increased speed of science and technology breakthroughs;
The globalization of science and technology (S&T);
The convergence of various S&T disciplines (computer science, biology, physics,
neuroscience, nanotechnology, chemistry);
The impact of commercial technology and its speed of dissemination; and
The increased capabilities of potential adversaries, including both non-state and state
actors and the willingness of these parties to share with or sell to one another.
Returning veterans of the wars in Afghanistan and Iraq require additional education and
training so that their experiences and skills are not lost to the nation. In light of these needs, the
National Defense Intelligence College (NDIC) has created a new program in S&TI education.
This effort includes the opening of a new School of Science and Technology Intelligence and the
development of new educational programs (Studds et al., 2011; NDIC, 2011a).
The NRC was asked to convene a Committee for the Review of the Master’s Degree
Program for Science and Technology Intelligence Professionals to consider a proposed master of
science degree in S&TI. The statement of task associated with this request reads as follows:
To provide a national level perspective to the National Defense Intelligence
College (NDIC) as it plans for a new master’s degree program scheduled to start
3
OCR for page 4
4 Review of the NDIC’s Master’s Degree in S&TI
in the fall of 2011, an NRC committee will review a draft curriculum and
associated materials developed by the NDIC. The committee will produce a letter
report offering advice and recommendations regarding the suitability of the draft
curriculum as a basis for producing desired learning outcomes for intelligence
professionals who participate in the proposed Master of Science and Technology
Intelligence degree program in the School of Science and Technology
Intelligence.
The NDIC is an institution of higher education accredited by the Middle States Commission
on Higher Education. It is authorized by the U.S. Congress to award both bachelor’s and master’s
degrees, to conduct and disseminate research related to national security and intelligence
activities, and to engage in outreach activities. The NDIC currently offers a bachelor of science in
intelligence (BSI) and a master of science in strategic intelligence (MSSI). It also offers
certificate programs in various regional and topical study areas. One such specialization is a
certificate in foreign denial and deception (Studds et al., 2011). The NDIC serves both military
and civilian members of the intelligence community. It is currently housed at the Defense
Intelligence Agency (DIA) headquarters facility in Washington, D.C.
Created in 1962 as the Defense Intelligence School, the NDIC has evolved in response to the
emerging needs of its constituency and as directed by its board of visitors (NDIC, 2010; NDIC,
2011a). Most recently, in 2010, it was directed to create a school of science and technology
intelligence and to develop a master of science and technology intelligence degree program
(NDIC, 2011a). The curriculum under review by the committee is the material developed in
response to that directive.
When a new degree program is developed by the NDIC, the program must be shown to meet
four distinct criteria. First, it must be shown to be necessary. In other words, the degree program
must meet an unmet need that is essential for the execution of the applying agency’s mission.
Second, it must be shown to be unique. It cannot duplicate existing programs that could be
satisfactorily used to meet the identified need. Third, it must meet the standards for graduate
degree programs met by non-federal institutions. And finally, it must allow for academic freedom
for the faculty and the students so that research and classroom activities are free from undue
influence and are unbiased by either job or mission (Studds et al., 2011, p. 8).
From May to August 2011, the committee met and considered the material presented by the
NDIC. A committee meeting was held in May, when materials were presented and the committee
was able to question the faculty and administrators of the NDIC. After the meeting with the
faculty and administrators, additional material was collected to provide a comparative sampling
of other curricula. Because this is a unique offering in a unique setting, there is no exact replica to
examine. However, curricula similar in scope and nature were reviewed, representatives of the
client population were interviewed, and the program goals were examined.
Detailed review comments regarding the curriculum in general and the syllabi provided to
the committee are included in the next sections of this chapter. The committee’s discussion first
reflects an overview of what was provided to the Committee and then the findings and
recommendations follow.
DISCUSSION OF THE MS&TI DEGREE
Curriculum
A review of the curriculum reveals that it is both unique and necessary. Comments from
current members of the S&TI community revealed a true dearth of structured educational
programs to create leaders in S&TI for the intelligence community. There is a distinct difference
in the educational needs of S&TI leaders and S&TI analysts. Whereas analysts are required to be
OCR for page 5
Program, Curriculum, and Syllabi Review 5
deeply knowledgeable about one or more subject areas of science or technology, leaders are
required to have a broad appreciation for S&TI as a whole, to understand how S&TI contributes
to national security, and to know how to be effective managers within the structure of the U.S.
government. It was noted that the craft of S&TI is very different from simply being a scientist,
engineer, or technologist. The practitioner of S&TI at all levels needs to understand not only the
underlying science but also how potential adversaries use science, both systematically and
culturally. This fundamental distinction leads to a substantive difference between simply learning
a science and learning S&TI. As such, this discussion of the proposed degree begins with the
curriculum.
The mission of the master of science and technology intelligence (MS&TI) degree program
falls within the greater mission of the NDIC, which is to provide educational programs to
members of the U.S. intelligence community. As a program funded by the national intelligence
community, NDIC’s MS&TI program has both a professional and an academic duty to enrich the
skill sets of intelligence professionals, with a specific focus on improving the nation’s ability to
collect, analyze, and provide intelligence product in the areas of foreign science and technology
development. Through its core courses and concentrations, the MS&TI program sets out to
address some of the challenges identified in the following statement:
. . . our ability to identify emerging technologies and warn about the disruptive nature of
foreign developments through Scientific and Technical Intelligence (S&TI) must have a broader,
worldwide focus that builds upon the traditionally peer and near-peer competitors but extends
beyond to the world of non-traditional sources of technology and asymmetric adversaries.
(NDIC, 2011a)
Specifically, the mission of the S&TI community is to “address threats to national security
arising from globalization of science and technology; identify disruptive consequences of
adversarial technology adaptations; and provide a framework for effective collection and warning
(Studds et al., 2011).” It follows, therefore, that a master’s degree in this area should prepare
students to both contribute to the greater S&TI mission and adopt leadership roles in the
community of practitioners. And in fact, the mission for this program was clearly identified as
developing leaders for the field of S&TI rather than analysts.
The MS&TI program is offered for full-time in-residence students and part-time students as
well. Approximately half of the students admitted to existing educational programs at the NDIC
study full time. Students come mainly from the population of military officers in the O-4 and
higher ranks and civilians at GS-13 or above. Given this mature body of matriculating students,
the program is focused on equipping managers with a broad understanding of intelligence and
giving them a S&TI-specific background rather than on just developing a cadre of nationally
trained analysts (Studds et al., 2011).
In order to offer the MS&TI degree, the university population needs to have faculty and
students with a strong technical background. In particular, “the student body would have
characteristic prerequisites for graduate study that are significantly different from those needed
for the advanced policy, political and military strategic intelligence of the University‘s current
elective program.” The students are expected to have the “technical depth, skills, and tools to
understand the potential of S&T” (NDIC, 2011a).
When examined in detail, the requirements for admission to the MS&TI program are
unclear. In the briefing received by the committee, it was stated that students require
“undergraduate level STEM knowledge” (NDIC, 2011a) but in discussion with the faculty and
administrators there was no consensus on whether the requirement for STEM undergraduate
degrees or prior STEM training would be enforced. The committee feels strongly that the STEM
prerequisites must be clarified, preferably in favor of requiring knowledge of STEM prior to
admission.
OCR for page 6
6 Review of the NDIC’s Master’s Degree in S&TI
The current educational programs at the NDIC, including the master of strategic studies in
intelligence (MSSI), require that students have an active Top Secret - Sensitive Compartmented
Information (TS/SCI) clearance, which enables them to access classified resources and attend
classes taught at the SCI level. The requirements for clearances, the closed facility with cleared
faculty, and the breadth of knowledge brought to the classroom by other students with field
experience are unique aspects of the NDIC that provide a strong underpinning to the MS&TI
program.
Comprising a faculty to teach a new program of study is always a challenge. Upon
examination, the NDIC does plan for a faculty worthy of the program. It has been building its
capabilities over several years in specialist areas, which it already provides as focus areas for
students in existing degree programs. Hence, a cadre of capable and tested faculty exists.
Secondly, there are plans for recruiting and expanding faculty in critical areas that will provide
the capacity for competently serving the incoming student body.
The faculty appears to bring an appropriate set of knowledge capabilities to the task of
offering the MS&TI program. According to the NDIC catalog (NDIC, 2011a):
The faculty brings a wealth of knowledge and experience gained through earning advanced
degrees in intelligence-related fields, and through leadership positions in the national
intelligence community. They possess a wide range of expertise in Intelligence Community (IC)
topics and come from varied academic and intelligence careers. Many have served on national
boards and commissions to include the National Security Council, Intelligence Science Board,
the Weapons of Mass Destruction Commission, and selected presidential commissions.
The Defense Intelligence Agency (DIA), Department of State (DoS), Central Intelligence
Agency (CIA), National Security Agency (NSA), Federal Bureau of Investigation (FBI), Foreign
Denial and Deception Committee (FDDC), and National Geospatial-Intelligence Agency (NGA)
are all represented on the faculty. The Air Force, Army, Coast Guard, Marine Corps, and Navy
provide faculty members as service advisors for their cohorts of students enrolled in the
University. CIA, DIA, NGA, NSA, and the Department of State also provide distinguished
Chairs and greatly contribute to a more integrated Intelligence faculty. (NDIC, 2011a)
The requirements set forth for the faculty are that they have “specific technical backgrounds
and experience [including] an in-depth understanding not only of the science, but conditions and
environment of government, military, and commercial technology research, development, and
deployment.” Beyond this, they are also required to hold degrees suitable to the level at which
they teach and advise (NDIC, 2011a). The faculty is actively engaged in curriculum development.
The NDIC curriculum working group reviews all proposed course additions or changes. The
process for approving new courses was described as including several distinct steps, starting with
a course proposal, then conditional approval for offering the course, followed by several trial runs
in special topics status, and finally, if all conditions are met, status as a permanent course
offering. The NDIC requires that at least two faculty members be able and willing to teach any
new course (O’Neill, 2011).
The MS&TI curriculum was developed from the intelligence community’s review of
competencies and needed skill sets. The Office of the Director of National Intelligence (ODNI)
published a report on skills, competencies, and intelligence capabilities that served to both
identify the need for the MS&TI program and develop the structure of the program (ODNI,
2008).
The program covers the global impact of S&T, the challenges of globalization, and S&T-
enabled symmetry and S&T-enabled asymmetry. The School of Science and Technology
Intelligence “complements the capabilities of the existing College of Strategic Intelligence by
providing select science and technology focused students better opportunities for effective
OCR for page 7
Program, Curriculum, and Syllabi Review 7
science-based research, focused education and interagency outreach within a science-based
educational paradigm” (NDIC, 2011a, p 38).
The core curriculum provides students with a comprehensive and laudable foundation in
intelligence that includes the following, as partially extracted from the NDIC catalog (NDIC,
2011a):
A Global Perspective – begin with an overview of globalization and strategic intelligence issues
as the context for intelligence and U.S. national security.
Intelligence in a Dynamic Global Environment – focus on developing and applying forecasting
and analytical skills to systematically translate world events into products that identify and fill
intelligence gaps.
National Security – prepare students to be conversant with strategic intelligence demands, and to
understand how the executive branch coordinates intelligence policy in the context of national
security planning, Congressional oversight of intelligence policy, and budgeting, and how
military and diplomatic consumers contribute to and defend intelligence policies and programs
in both the executive and legislative processes.
Integrated Skills, Competencies, and Intelligence-Related Capabilities – Critical thinking,
analytic skills, communications, and intelligence related capabilities are woven throughout the
content of the curricula.
Engagement and Integration – Develop Interpersonal Skills, teaches students how to Engage
and Collaborate with others and how to Influence and persuade others.
Communication – Oral and Written Communication, Multi-Media Communication, and
Conviction.
Critical Thinking – Development of Situational Awareness, use of Creative Thinking, Synthesis
and Explore Alternatives.
Professionalism – Apply the principles of Ethics and Integrity, Initiative, and Adaptability.
Mission Awareness – Understands the Enterprise Perspective, Customer Operations and
Requirements, and Policy and Directives.
Analysis – Use Analytic Rigor, how to Investigate, use Collection Systems and Operations,
understand the intelligence Process and Exploitation capabilities, knowledge of Analytic Tools
and Methods. (NDIC, 2011a)
Before 2009, a non-thesis option was available for MSSI, but all students are now required
to develop and present a master’s-level thesis (Studds et al., 2011). The planned MS&TI thesis
will conform to academic standards as a formal, written presentation of original research
examining an S&TI-related topic in the selected specialization. Students are encouraged to pursue
research that contributes to the mission of the intelligence community. Theses may be classified
or unclassified (Studds et al., 2011; NDIC, 2011a)
The MS&TI program, therefore, requires the student to complete forty-three (43) total credit
hours, thirty-six (36) of which are in course work and seven (7) are in thesis research. The course
work is split between three categories: fifteen (15) common core courses, six (6) degree program
required courses, and fifteen (15) elective/concentration courses (Studds et al., 2011; NDIC,
2011a). The five concentration areas currently include Weapons of Mass Destruction,
Information Operations and Cyber Intelligence, Emerging and Disruptive Technologies,
Geostrategic Resources and the Environment, and Foreign Denial and Deception.
The five courses that make up the common core are:
MCR 601. Globalization and Intelligence Issues
MCR 603. Social Analysis and the Spectrum of Conflict
MCR 607. Intelligence Reasoning and Analysis
MCR 609. The Compound Eye: Intelligence Collection
MCR 611. Intelligence and National Security Policy
OCR for page 8
8 Review of the NDIC’s Master’s Degree in S&TI
These courses are all developed and available to be taught. There are faculty to teach them,
established learning objectives, and structured course material.
The courses that make up the five concentration areas are listed here by concentration. In the
following lists, each course is annotated according to the maturity of the course offering. The
annotation A refers to a course that is available, has been taught before, and is mature in program
content, although it may be modified for the MS&TI degree offering. The annotation D refers to a
course that is developed but has not yet been taught and is therefore considered to still be a work
in progress according to the NDIC course development processes. The annotation F refers to a
course that has not yet been developed but which is intended for development at a future time.
Weapons of Mass Destruction
—MST 663. WMD Counter-proliferation (A)
—MST 665. The Biological Threat (A)
—MST 666. Space and Missile Threats (D)
—MST 667. The Nuclear Threat (A)
—MST 669. The Chemical and Explosive Threat (A)
—MST 655. Advanced Conventional and Non-conventional Weapons (A)
—MST 656. The Economics of Technology (F)
Of these courses, five, marked A, are considered to be developed, available to be taught, and
mature in terms of content, learning objectives, and structured course material. One, MST 666,
which is marked D, is considered to be under development, and one, MST 656 marked F, is
considered to be a future development need (NDIC, 2011b).
Information Operations and Cyber Intelligence
—MST 680. Information Power and National Security (A)
—MST 681. Propaganda, Persuasion, and Influence (A)
—MST 682. Cyber Operations (A)
—MST 683. Foreign Information and Cyber Strategies (D)
—MST 684. Cyber Defense (A)
—MST 687. Advanced Information Power Seminar (D)
—MST 698H. Cyber Attack (D)
—MST 698J. Cyber Exploitation (D)
Of these courses, four are listed as being developed, available to be taught, and mature in
terms of content, learning objectives, and structured course material. The four that are listed as
under development (D) are MST 683, MST 687, MST 698H, and MST 698J (NDIC, 2011b).
Emerging and Disruptive Technologies
—MST 653. Advanced Science and Technology Intelligence (Process) (F)
—MST 655. Advanced Conventional and Non-conventional Weapons (A)
—MST 656. The Economics of Technology (F)
—MST 657. Case Studies in Technology Transfer (D)
—MST 658. Infrastructure Vulnerability Assessment (A)
Of these courses, two are listed as being developed, available to be taught, and mature in
terms of content, learning objectives, and structured course material. One of the remaining three
is listed as “under development”: MST 657. The remaining two, MST 653 and MST 656, are
listed as future development projects (NDIC, 2011b).
OCR for page 9
Program, Curriculum, and Syllabi Review 9
Geostrategic Resources and the Environment (Energy Resources and Power)
—MST 672. Intelligence and the Changing Global Resource Environment (F)
—MST 673. Geology and Intelligence (F)
—MST 674. Nuclear and Other Alternative Energy Sources (F)
—MST 675. Electrical Power Systems and Distribution (F)
—MST 656. The Economics of Technology (F)
—MST 658. Infrastructure Vulnerability Assessment (A)
Of these courses, only one, MST 658, is listed as having being developed, available to be
taught, and mature in terms of content, learning objectives, and structured course material. The
remaining five courses are listed as future development efforts (NDIC, 2011b).
Foreign Denial and Deception
—MST 660. Introduction to Denial and Deception: History, Concepts, Issues,
and Implications (A)
—MST 662. Denial and Deception: Psychological/Cultural Aspects, and
National Security Decision Making (A)
—MST 664. Denial and Deception: Adversaries, Organizations, Activities, and
Countermeasures (A)
—MST 668. Denial and Deception: Tradecraft, Tools and Methodology (A)
Of these courses, all are listed as being developed, available to be taught, and mature in
terms of content, learning objectives, and structured course material (NDIC, 2011b).
Overall, as it reviewed the proposed curriculum, the committee found that the common core
courses are all available and that three of the proposed specialization areas appear to be populated
enough to be offered. Two of the specialization areas, Emerging & Disruptive Technologies and
Geostrategic Resources & the Environment, require significant course development efforts before
they will be ready to be offered to students. Findings and recommendations are found later in the
chapter.
Syllabi
A detailed description of the course work of the MS&TI program was provided to the
committee. The courses annotated as “on hand” or “under development” all included a course
description, a description of the course’s contribution to the mission, expected learning outcomes,
a list of intelligence community competencies, course requirements, grading criteria, a reading
list, a session outline with subjects and issues being covered, and defined learning objectives. The
course content reflected in this material seems complete and thoughtfully compiled, with the
exception of the structure used for the learning objectives. The committee is concerned that the
learning objectives are too broadly stated and are poorly linked to the program learning outcomes.
Additionally, a close analysis of the ODNI Competency Subdirectory for S&TI, which was
used as a reference material for designing the MS&TI program, reveals that much is missing from
the current course content (ODNI, 2008; Studds et al., 2011, p 14). Furthermore, of the material
included, the committee had significant concerns about the learning objectives. These issues are
discussed here and the committee’s findings and recommendations are offered in the appropriate
section.
Learning Objectives
Learning objectives form the core of any educational program. Correctly and precisely stated
expectations of what exactly a student should learn in a course allow not only the precise
OCR for page 10
10 Review of the NDIC’s Master’s Degree in S&TI
engineering of course content to achieve those goals, but they also define the prerequisite
knowledge of the student, the competencies of the instructor, and the end product of a program.
Each of these is worth considering separately.
Learning objectives should be stated using action words and precise outcomes, which must
be empirically measurable. For example, a learning objective that is poorly formulated is this:
“The student should understand how to write a paper.” A much better crafting of that learning
objective would be “The student will write a research paper that adheres to appropriate academic
standards; the paper will include a thesis statement, background material, data presentation and
analysis, and a synthesis of research findings.” In the first statement of objective, there is no way
to measure how well a student understands how to write a paper. In the second, not only is the
expectation clearly stated but the measure of success is also clearly defined.
The problem of poorly defined learning objectives creates confusion for the program
objectives and the student learning prerequisites. Consider as an exemplar, learning objective for
MST 684, Cyber Defense: “Gain an understanding of the vulnerabilities of the DoD and IC cyber
systems.” Were this learning objective to be recrafted in active language, it could be expressed in
many ways, two of which follow: (1) “Use software modeling techniques, such as functional
decomposition diagrams (FDD) and data flow diagrams (DFD), to identify critical points of
vulnerability in an exemplar cyber system” and (2) “Reverse engineer an identified piece of
malicious software using provided tools in a laboratory environment to identify attack paths and
probable target structures.” Either and both of these alternatively stated learning objectives can be
measurable on an objective basis. In other words, one can measure both whether a student
achieved the learning objective and how well the student achieved it. Furthermore, pre- and post-
education assessments are possible to measure how much was achieved. This becomes a critical
quality control function: If students are uniformly not achieving the expected level of
competency, it might be an indicator that prerequisite knowledge has been incorrectly set or that
the teaching process is not optimized to the student population.
However, the two restatements of learning objectives differ in three important ways: first, in
the prerequisite knowledge demanded of the student; second, in the capabilities they demand of
the instructor; and, third, in overall learning outcomes called for by the program. These
differences are critical to the administration of the MS&TI program. Both the first and second
restatements are legitimate children of the original learning objective but speak to different
student populations, different faculty, and different program outcomes.
The obligatory prerequisite level of competency, for learning objective 1 would need to
include a basic knowledge of information system structures. For learning objective 2, however,
the requisite level of competency would need to be much higher; it would include knowledge of
software structures, algorithmic development methods, and attack pattern analysis. This
distinction then tracks back to the admissions process: requiring specific types of prior education
or training, or having an identified set of prerequisite knowledge to be gained prior to admission
to the program.
OCR for page 11
Program, Curriculum, and Syllabi Review 11
FIGURE 1-1 The importance of learning objectives.
The learning objectives influence faculty capabilities. For learning objective 1, faculty
having a general knowledge of software modeling could teach the course. For learning objective
2, however, someone with significant experience in software development, reverse engineering,
and cyber threat analysis would be needed to successfully teach the course. It is far easier to find
an instructor to meet the needs of learning objective 1 than those of learning objective 2, which
argues for a careful examination of the program learning outcomes. If these outcomes are truly to
produce graduates who can reverse engineer malware, then having faculty who can teach to
learning objective 1 but not to learning objective 2 will simply not do.
And it is, in fact, the overall program learning outcomes that are at stake. If the individual
course learning objectives are poorly and imprecisely worded and defined, then the outcomes will
range widely, depending on individual student ambitions and efforts. A more predictable graduate
results from precisely defined learning objectives.
Program ODNI S&TI Competencies
A detailed breakdown of competencies and the courses in which they are covered is
provided here. Two important caveats apply to the following discussion. First, it is recognized
that MS&TI faculty are still developing courses, and the committee hopes that some of the
comments included below will be addressed. Second, is that the committee understands that some
of the syllabi have classified portions that were not shared with it. For this reason, the committee
readily acknowledges that some of the following points may well be moot.
The following core competencies seem to be well addressed in the syllabi: Engagement and
Collaboration, Critical Thinking, Accountability for Results, and Communication. Engagement
OCR for page 12
12 Review of the NDIC’s Master’s Degree in S&TI
and Collaboration includes information sharing and interpersonal skills, which are natural
components of an educational environment. Critical Thinking is a priority area for the curriculum
in general and as such the elements of it are addressed in each syllabus extensively. The elements
of Accountability for Results that are addressed include continual learning, policy and directives,
results orientation, and rigor. The elements of Communication, including information transfer and
communicating with impact, are also natural elements of an academic program.
The core competencies that are not well addressed in the syllabi include Leadership and
Integrity and Management Proficiency. The elements of Leadership and Integrity identified in the
ODNI Competency Subdirectory, such as Developing Others and Leveraging Diversity, are
difficult to teach in a classroom environment. Indeed, the syllabi emphasize integrity from an
academic perspective. The elements of Management Proficiency can be taught in a classroom
environment, but they compete for time with more technical subjects. When hard choices have to
be made regarding use of available time in the classroom, it would be appropriate to prioritize the
material at the core of a program rather than more general topics. In this case, technical
knowledge, skills, and abilities (KSAs) may be rightfully prioritized over management
proficiencies, depending on the students’ backgrounds. That is not to downplay the importance of
such proficiencies; on the contrary, they are vitally important to the desired end product of the
MS&TI degree: a leader who is competent to manage an S&TI function. As discussed in findings,
this shortcoming of the MS&TI program might be addressable using clever partnerships with
other programs.
Sixteen (16) technical expertise competencies (TECs) are listed in the ODNI Competency
Subdirectory. Those TECs that are well-covered in the MS&TI syllabi include the following:
Collection Systems Capabilities, Cryptography, Customer Operations and Requirements,
Intelligence Disciplines, Research and Exploration, and Researching. The TECs that seem to be
adequately covered in the syllabi include the following: Processing and Exploitation Capabilities,
Project Management, Technology Integration/Insertion, Tools and Methods, and Transportation.
The TECs that are not well-covered, either due to incompleteness of the topic or the absence of it
altogether, include the following: Contracting/Procurement, Counter-proliferation, Intelligence
Topics, Systems Engineering, and Targets: General.
Given the time constraints of the program, it is probably logical to not cover everything and
in this case, Contracting/Procurement is an obvious topic to leave out. However, it is appropriate
to review the Counter-Proliferation TEC, described as “skill at supporting counter-proliferation
operations with S&T operations” (NDIC, 2011b), for completeness. Reviewing all syllabi
provided does not readily reveal any coverage of this competency. It is entirely possible that this
competency is covered in MCR 601 and/or MCR 611 but, if it is, this is not explicitly stated in
the syllabi and so that lack of coverage can only be conjectured. If, in fact, this competency is not
covered, it may be worthwhile reviewing the syllabi to consider whether there should be more
emphasis on this competency. One of the TECs that is not completely covered is Intelligence
Topics. While 55 KSAs are listed under Intelligence Topics, the syllabi that were provided, which
do not include sensitive KSAs, show 27 of them are not addressed. These are predominantly the
KSAs that deal with biological weapons development, deployment, and testing. Because this is a
critical element of the S&TI challenge today and an extremely complicated and difficult one, the
NDIC should consider a more comprehensive approach to educating the MS&TI student
population in biodefense capabilities. “The breadth of biological threats is much broader than
commonly appreciated and will continue to expand for the foreseeable future” (Choffnes et al.,
2006, p 28; NRC, 2006).
This very important area of ST&I, biological threats, appears to be under-represented in the
current curriculum. This deficiency is exemplified by MST665 “The Biological Threat,” which is
described as “on hand” and includes two lectures on bacteria, two on viruses, and two on toxins,
but the content of these lectures is not detailed. They do not appear to include basic principles of
epidemiology, synthetic biology, the engineering of organisms for resistance and increased
OCR for page 13
Program, Curriculum, and Syllabi Review 13
virulence, vaccines and protective immunity, or sources of new disease agents (emerging
infections).
Additionally, the TEC for Systems Engineering is viewed by the committee as a critical
omission. The interdependencies that characterize the global innovation landscape require for
ability to understand and apply systems logic when analyzing the various bits and fragments of
information collected on foreign capabilities. In particular, the integration of vastly different
technologies and sciences is resulting in capabilities that can be understood only when viewed
systemically. This is particularly true in the biosciences: “Other fields not traditionally viewed as
biotechnologies–such as materials science, information technology, and nanotechnology–are
converging with biosciences in unforeseen ways and enabling the development of previously
unimaginable technological applications” (Choffnes et al., 2006, p 26; NRC, 2006; NRC, 2005).
Developing these new technologies with fully open information and sharing is difficult enough;
the challenge of divining what is occurring in a closed environment where clues are being
gleaned from a variety of sources is vastly more difficult.
Systems engineering expertise is also useful for the analytical function from a team
communications perspective.
Useful analysis requires effective communication among diverse individuals. Members of
an analytical community must communicate with each other and with their clients. Both kinds of
communication can bring into contact individuals with very different missions, backgrounds,
and perspectives. Within an analytical community, a single analysis might require
communication among individuals with expertise in economics, anthropology, psychology,
engineering, and logistics. Each contributing discipline might have subfields and competing
theories, each needing to be heard. Setting the terms of an analysis and reporting its results might
require communication with clients who differ from the analysts in their objectives, careers, and
education. These professional differences overlay the cultural, socioeconomic, and other
differences that can complicate any communication in a diverse society. (NRC, 2011a)
Expertise in systems engineering establishes an ability to “integrate and verify system
elements” and analyze “processes and procedures [used] to implement operational schedules,”
according CIA’s description for the job of systems analyst, available online at
www.cia.gov/careers/opportunities/science-technology/systems-engineer.html, last accessed July
15, 2011. Furthermore, systems engineering expertise is increasingly being identified as a critical
core competency that is presenting short supply in today’s intelligence community (NRC, 2011a;
NRC, 2011b). In sum, the Systems Engineering TEC appears to be an increasingly valuable part
of the intelligence analysis equation, particularly in S&TI.
This discussion serves as the overarching framework for the findings and recommendations
of the committee. It is worth restating that both the discussion and the findings and
recommendations are based solely on the material presented and are made with the express
knowledge that some material may not have been shared with the committee for reasons of
sensitivity or classification, perhaps rendering some of the discussions or the findings and
recommendations moot.
FINDINGS AND RECOMMENDATIONS
General
In general, the master’s program is well thought out and gives students a solid foundation on
a wide range of S&T intelligence issues. For the most part, the courses cover a lot of ground, and
students will be exposed to a great deal of S&TI-related material. The course development
process is inclusive and requires that at least two faculty members be available to teach a
OCR for page 20
20 Review of the NDIC’s Master’s Degree in S&TI
MST 613 Science, Technologies, and Intelligence, would appear to be the fulcrum upon
which the MS&TI pivots. As such, it is a critically important course. This is the course that is
specifically focused on S&TI in the program, and its description in the syllabi clearly states that a
primary purpose of the course is to teach how to identify disruptive technologies. In support of
this goal, it would be worthwhile to have a deeper discussion on technology forecasting and
tracking strategies in the course. Seeing into the future is complex and uncertain, yet there are
techniques that have been developed to lend structure to such endeavors, particularly in the
technological arena. A survey of forecasting methodologies would be worthwhile. Special
emphasis should be placed on approaches to forecasting disruptive technology. Students should
have an understanding of the appropriate use of forecasts and how to frame and present
technology forecasts in analyses. The committee notes that a good teaching material would be the
NRC’s Persistent Forecasting of Disruptive Technologies (NRC, 2009).
Finding 17: The material included in MST 613 on forecasting is incomplete and could benefit
from increased rigor, particularly with respect to disruptive technology identification.
Recommendation 17: The National Defense Intelligence College faculty should add a session on
disruptive technology forecasting and tracking to MST 613. Further, the NRC report Persistent
Forecasting of Disruptive Technologies (NRC, 2010) should be added to the materials for
teaching this course.
Areas of Specialization/Concentration
There are currently five of concentration areas for the MS&TI: Weapons of Mass
Destruction, Information Operations and Cyber Intelligence, Emerging and Disruptive
Technologies, Geostrategic Resources and the Environment, and Foreign Denial and Deception.
The courses associated with these concentration areas of are at a mix of maturity levels. Tthe
courses that were presented with the syllabi and found to warrant suggestions are discussed next.
Those courses that were not mature or that did warrant suggestions are not covered. The
concentration Geostrategic Resources and the Environment currently has no developed courses,
so no concrete commentary is possible.. However, the committee supports the inclusion of this as
a concentration area in the future MS&TI program because the issues involved are so important
and have the potential to impact both global and regional stability, which are important elements
of national security.
Weapons of Mass Destruction
The inexorable advances in S&T increase the importance of intelligence about weapons of
mass destruction. S&T discoveries give groups and individuals who scheme to advance their
causes more options to harm the United States. As an example, the knowledge to produce and
deploy nuclear and biological weapons that may not be optimal, but it can still cause horrendous
destruction. It is mostly held in check not by insufficient S&T knowledge but by the lack of
enriched uranium or a means for delivering biological agent over a wide area. Nevertheless, as
demonstrated by the effects of modest-scale agent delivery by mail, a few letters containing
anthrax sent by one (probable) individual can cause multiple deaths, panic, and economic
calamity (FBI, 2010). Similar disruptive attacks on food supplies, electrical grids, key modes of
transportation, and economic mechanisms such as stock markets are possible, if not likely, targets
that S&T may enable.
MST 655, Advanced Conventional and Non-Conventional Weapons. This course should
include a discussion of the importance of software in both advanced conventional and non-
conventional weapons. Software subsystems are an important underlying technology of weapons
OCR for page 21
Program, Curriculum, and Syllabi Review 21
systems. Furthermore, the precision of weapons is increasingly a function of software, hardware,
and firmware elements. It is increasingly important for weapons analysts to understand the
underlying logic that guides and controls weapons, both from the platform and within the
warhead. Additionally, new weapons are increasingly developed and tested virtually, before any
physical evidence is available for either collection or observation. As such, software simulation
environments become an important element of new weapons discovery.
Finding 18: The MST 655 course related to the importance of software in weapon systems has a
shortcoming.
Recommendation 18: The National Defense Intelligence College should include software design
and simulation software in MST 655, with an emphasis on the use of these tools for the design
and testing of new systems.
Another area worth covering in the course would be autonomous and semi- autonomous
weapons systems such as UAVs. These systems are changing the way we fight on the battlefield
and how we think about tactical warfare. While relatively new, these systems are being rapidly
developed by many countries.
Finding 19: The syllabus for MST 665, The Biological Threat, is a thin outline compared with
the substance in other sections in the concentration area such as MST 667, The Nuclear Threat.
Recommendation 19: The National Defense Intelligence College should greatly expand the
content of MST 665 and label it as “under development” until it is improved.
Emerging and Disruptive Technologies
The NRC report Persistent Forecasting of Disruptive Technologies discussed in detail
current forecasting methodologies, the nature of disruptive technologies, and the characteristics of
disruptive technologies (NRC, 2010). Many researchers now see the importance of tracking
technologies not only for how they are originally used, but how they are used by different groups
of people. That NRC report stated as follows:
The value of technology forecasting lies not in its ability to accurately predict the future but
rather in its potential to minimize surprises. It does this by various means:
Defining and looking for key enablers and inhibitors of new disruptive technologies,
Assessing the impact of potential disruption,
Postulating potential alternative futures, and
Supporting decision making by increasing the lead time for awareness. (NRC, 2010)
S&TI analysts will need to understand how various technologies fit together and how out of
the box thinking may affect or change whole industries very quickly. This is not a new thought.
As Arthur Schopenhauer stated over 100 years ago:
The task is not so much to see what no one yet has seen, but to think what nobody yet has
thought about that which everybody sees.
The cultural lenses that overlay expectations color the ability to “see” potential applications
and uses. It is more important to study an approach to correctly defining these potential uses than
it is to review as many technology areas as possible.
OCR for page 22
22 Review of the NDIC’s Master’s Degree in S&TI
Finding 20: The Emerging and Disruptive Technologies concentration area includes a very
ambitious list of topics. The scope of problems is huge and rapidly changing. Employing
competent instructors for these evolving topics will be challenging.
Recommendation 20: The National Defense Intelligence College should reduce the number of
topics in the Emerging and Disruptive Technologies concentration to a manageable set that
provides core competencies across a broad range of topics without being overly ambitious in
terms of content.
Finding 21: The Emerging and Disruptive Technologies concentration area lacks a session on the
innovative use of technologies. Obvious examples of this are flying airplanes into buildings and
using the mail to distribute bioweapons-grade materials.
Recommendation 21: The National Defense Intelligence College should add a session to an
existing course or create a new course on the innovative use of technologies to create threat
capability.
Information Operations and Cyber
Cyber is a difficult area to cover: technology changes very quickly and new innovations are
adopted globally at a dizzying pace. Information Operations and Cyber Intelligence is a very
complicated topic area that spans not only raw IO & Cyber but also Cyber as a means to conduct
other forms of ST&I. In fact, the basic vocabulary of IO does not appear to be addressed in the
concentration which would allow leaders to understand at least the categories, significance, and
components of modern cyber warfare and information operations. Furthermore, the areas of
information operations and of cyber intelligence are in fact very different areas, with different
skill sets, different objectives, and different candidate students.
Finding 22: The Information Operations and Cyber Concentration requires a precise definition of
prerequisite knowledge for the students and for the instructors in order to create a slate of courses
that are meaningful. Understanding what the graduate should be able to do is critical to defining
both the learning objectives and the prerequisite knowledge requirements.
Recommendation 22-1: The National Defense Intelligence College should emphasize the
development of a student’s ability to learn, adapt, and deal with revolutions in cyberspace rather
than emphasizing the learning of specific approaches or techniques that might work today but be
obsolete within a year.
Recommendation 22-2: National Defense Intelligence College students focusing in the
Information Operations and Cyber concentration should be required to have an appropriate
undergraduate level of education, or life skills associated with information operations and cyber,
or training appropriate to the learning objectives.
MST 680, Information Power and National Security. As the Internet increasingly becomes a
conduit for both power projection and competition, all of the students must have prior knowledge
so that this course can be conducted at the graduate rather than the introductory level. Students
need an overview of how the Internet works and of how typical computer and IP networks are
designed and must know about the growing dependency of everyday devices, appliances,
systems, and infrastructure on IP.
OCR for page 23
Program, Curriculum, and Syllabi Review 23
Finding 23: The class should pay special attention to cloud-based architectures as well as mobile
devices. Students should understand the basic strategies used to attack networks and have an
understanding of defensive measures. There should be discussion of how the Internet can
multiply the power of individual ideas and how this can be seen as threatening to other nation
states.
While week 6 is devoted to Legal considerations in Information Operations, it is extremely
important to discuss the various intelligence general counsel interpretations of existing laws,
policies, and restrictions. In many cases, the agencies do not have the same interpretations,
especially on issues such as U.S. persons, identity, data handling, covert action, and the active use
of social media. Additionally, it is important to cover the role of standards, agreements, treaties in
cyberspace and the strategies other nation states such as China, Russia, and the European Union
are trying to reduce the United States influence and dominance in cyberspace. There are various
materials available from the U.S. government, such as through the Joint Staff publications, that
can be used to illustrate these issues.
Recommendation 23: The National Defense Intelligence College should add to MST 680
materials such as the Joint Information Operations Planning handbook and doctrine manuals for
electronic warfare.
MST 681, Propaganda, Persuasion, and Influence. New social media technologies such as
Facebook, Twitter, and Google are having a global impact. Recent examples of events that show
the power and influence of social media include the 2009 Iranian elections1, the Arab Spring of
20112, and the Great Firewall of China3. Analysts need to understand how social media is being
used, particularly in coordinating and spreading political and social agendas. Further, they need to
understand how social media are used to position and reinforce ideas and to generate and
mobilize large portions of the populace around those ideas, particularly youth. A significant
challenge lies in how social media are used in intelligence: At times social media blur the line
between analysis and collection, and the potential for crossing that line needs to be deeply
appreciated by members of the intelligence community. For example, active participation in
social media could trigger specific rules and regulations that restrict how an analyst may operate,
even with the most benign intentions, because some actions might be considered to fall into the
realm of covert action. There is a distinct difference between reading a blog and friending or
tweeting: One set of actions is passive while the other is active. There are challenges in how
analysts and leaders manage and constrain their interactions with social media while still taking
advantage of access to useful information.
Additionally, S&TI analysts and managers need to understand meta-social media issues,
including the mechanisms of how usage is measured, how activities are tracked, the measurement
of impact, assessing how social media is actually being used, and the identification of supernodes.
The emerging area of social media may prove to be too large to fit into one course; it might
indeed need to be distributed across several courses. Not only are the preceding aspects of great
importance, but there are also subtleties that need to be covered in the curriculum—including
how different cultures use social media differently, how social media can be overwhelming
traditional media at the same time they are feeding it, and how social media postings go “viral”
(including characteristics of viral postings).
1
See http://mashable.com/2009/06/21/iran-election-timeline/
2
See http://blogs.reuters.com/great-debate/2011/02/16/digital-media-and-the-arab-spring/
3
See http://www.datelineshanghai.com/scaling-the-great-internet-wall/
OCR for page 24
24 Review of the NDIC’s Master’s Degree in S&TI
Finding 24: The area of social media appears to be neglected both in MST 681 and in the
curriculum in general.
Recommendation 24-1: The MST 681 course should cover the use of social media and the
importance of the social graph to show how they are used to facilitate nation-state protest,
revolution, and social consciousness. The curriculum should cover the power of crowd sourcing,
the effects of the digital exhaust, and social network tools such as Facebook, Google, YouTube,
and Twitter. The class should also discuss Internet use to influence and shape culture and society.
Recommendation 24-2: The National Defense Intelligence College faculty should consider
segmenting the challenges and learning objectives associated with covering social media and
parceling out the learning needs across several different courses.
Finding 25: The backgrounds of the instructors for the Information Operations topics are critical
to ensure that the material taught is deeper than a technical seminar from a vendor.
Recommendation 25: The National Defense Intelligence College should incorporate relevant
open-source information on cyber-attacks for educational purposes and for stoking the
imagination of students. Candidate material should include numerous technical analyses readily
available online that cover recent cyber-incidents such as the Farewell Dossier, the Stuxnet worm,
the Phishing Scheme against RSA, and the anonymous attack on HBGary, as well as future
incidents as they occur.
Finding 26: The Information Operations syllabi reads more like political science focused syllabi
than science or technology focused syllabi. For example, learning objectives are very high level
and include phrases such as “gain an appreciation for” and “gain an understanding of.” The basic
vocabulary of this concentration area does not appear to be addressed so that it would allow
leaders to understand at least the categories, significance, and components of modern cyber
warfare and information operations.
Recommendation 26: The National Defense Intelligence College should repurpose the course
content to be much more focused on the underlying science and technology and introduce people
to the concept of cyber by exposing them to widely available hacking training for hands-on
knowledge of systems compromise and manipulation.
Denial and Deception
The Denial and Deception concentration area is a well-developed program of study that has
existed for quite some time. The courses for the MS&TI program have been repurposed from the
existing curriculum for MSSI. Unfortunately, the resulting program of study that appears to be
light on science and technology while strong in policy analysis.
Finding 27: There is little discussion of the implications of communications technology. The use
of black, gray, and white propaganda and psychological operations is conspicuously absent. (This
may be because it is in sensitive elements of the program.) Social media, which can be easily
used for denial and deception, are not included in the provided material.
Recommendation 27: The National Defense Intelligence College should add the following
content to the syllabi for MST 664:
OCR for page 25
Program, Curriculum, and Syllabi Review 25
Communications are now nearly instantaneous, ubiquitous, costless, anonymous, secure, and
unlimited band-width. This enables worldwide and rapid flows of information, images, and
money for good and ill. “In the global community, this leveraging power of information
technology provides previously ineffective marginalized groups with additional means for both
direct challenges to authority as well as indirect media for intra-group communications, planning,
and management. Recent cases include the B-92 radio station in Bosnia4, the use of the Internet
by the Chiapas revolutionaries to marshal global attention to their cause5, the use of tape-recorded
messages by the Ayatollah Khomeini while in exile in Paris, the use of Twitter in the Iranian
election protests, and the use of Facebook during the Egyptian protests6. These examples
illustrate the breakdown of boundaries, both physical and customary” (Ryan, 2011).
Finding 28: The current Foreign Denial and Deception concentration does not distinguish
between the MSSI and MS&TI programs. Only one course, MST 604, is different, and MST 613
would be optional in MS&TI. Additionally, there is a dearth of technology and science in the
syllabi.
Recommendation 28: The National Defense Intelligence College should incorporate a stronger
background in science and technology in the Denial and Deception concentration area by
requiring one or more additional courses that focus on S&T-based based deception.
CONCLUSION
The need for the NDIC’s new degree program was found to be justified. The user population
reflected strong support for this effort. An examination of the program also revealed that it meets
the four criteria referenced: it is necessary, it is unique, it meets general academic practice, and
the administrative processes support academic freedom.
REFERENCES
Published
Choffnes, Eileen R., Stanley M. Lemon, and David A Relman. 2006. “A brave new world in the
life sciences: The breadth of biological threats is much broader than commonly thought and
will continue to expand.” Bulletin of the Atomic Scientists September/October: 26-33.
Crawley, Edward, Johan Malmgvist, Soren Ostlund, and Doris Brodeur. 2007. Rethinking
Engineering Education: The CDIO Approach. Springer Science+Business Media, pp. 40 and
113.
Engineering Trends. 2006. “Historical trends and near-term predictions of statistics on degrees,
enrollments and research expenditures for engineering education in the United States.”
Engineering Trends Quarterly Newsletter. Summer 2006.
4
See for example “Preserving the Free Flow of Information on the Internet: Serbs Thwart Milosevic
Censorship” at http://www.usip.org/events/preserving-free-flow-information-internet-serbs-thwart-milosevic-
censorship-round-two.
5
See for example “Rebellion in Chiapas: insurrection by Internet and public relations” by Jerry W. Knudson
from Temple University, available online at http://www.bsos.umd.edu/aasp/chateauvert/rebellio.pdf.
6
See for example “The dark side of Internet for Egyptian and Tunisian protesters”, which discusses many of
these events. By Evgeny Morozov for the Globe and Mail, published Jan. 28, 2011. Available online at
http://www.theglobeandmail.com/news/world/africa-mideast/the-dark-side-of-internet-for-egyptian-and-tunisian-
protesters/article1887170/print/.
OCR for page 26
26 Review of the NDIC’s Master’s Degree in S&TI
Federal Bureau of Investigation (FBI). 2010. Amerithrax or Anthrax Investigation Summary.
February 19. Available online at http://www.fbi.gov/about-us/history/famous-cases/anthrax-
amerithrax/amerithrax-investigation.
Friedman, Thomas L. 2010. “Can’t keep a bad idea down.” The New York Times, October 26.
Available at http://www.nytimes.com/2010/10/27/opinion/27friedman.html. Last accessed on
July 20, 2011.
Intelligence Science Board (ISB). 2006. Intelligence Science Board Task Force Report on the
Intelligence Community and Science and Technology: The Challenge of the New S&T
Landscape. November. Washington D.C.: Office of the Director of National Intelligence.
Available at http://www.fas.org/irp/dni/isb/landscape.pdf. Last accessed on September 1,
2011.
National Academy of Sciences, National Academy of Engineering, Institute of Medicine (NAS,
NAE, IOM). 2010. Rising Above the Gathering Storm, Revisited: Rapidly Approaching
Category 5. Washington, D.C.: The National Academies Press. Available at
http://www.nap.edu/catalog.php?record_id=12999. Last accessed on July 12, 2011.
National Defense Intelligence College (NDIC). 2011a. NDIC University Catalog 2011-2012.
Washington, D.C.: Center for Strategic Intelligence Research.
NDIC. 2010. National Defense Intelligence-2010 Annual Report. Available at
http://www.dia.mil/college/. Last accessed on July 12, 2011.
National Research Council (NRC). 2011a. Intelligence Analysis: Behavioral and Social Scientific
Foundations. Washington, D.C.: The National Academies Press. Available at
http://www.nap.edu/catalog.php?record_id=13062. Last accessed on July 15, 2011.
NRC. 2011b. Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social
Sciences. Washington, D.C.: The National Academies Press. Available at
http://www.nap.edu/catalog.php?record_id=13040. Last accessed on July 15, 2011.
NRC. 2010. Persistent Forecasting of Disruptive Technologies. Washington, D.C.: The National
Academies Press.
NRC. 2006. Globalization, Biosecurity, and the Future of the Life Sciences. Washington, D.C.:
The National Academies Press. Available at
http://www.nap.edu/catalog.php?record_id=11567. Last accessed on July 15, 2011.
NRC. 2005. An International Perspective on Advancing Technologies and Strategies for
Managing Dual-Use Risks: Report of a Workshop. Washington, D.C.: The National
Academies Press. Available at http://www.nap.edu/catalog.php?record_id=11301. Last
accessed on July 15, 2011.
Office of the Director of National Intelligence (ODNI). 2008. Intelligence Community Directive
Number 652, Occupational Structure for the Intelligence Community Civilian Workforce.
ICD 652, April 28. Available at http://www.fas.org/irp/dni/icd/icd-610.pdf. Last accessed on
September 1, 2011.
Ryan, Julie (ed.). 2011. Leading Issues in Information Warfare and Security Research, Volume 1.
Reading, England: Academic Publishing International. Pages xiv and xv.
The White House, Office of the Press Secretary. 2009. “President Obama launches "Educate to
Innovate" campaign for excellence in science, technology, engineering & math (STEM)
education. Press Release, November 23. Available at http://www.whitehouse.gov/the-press-
office/president-obama-launches-educate-innovate-campaign-excellence-science-technology-
en. Last accessed on July 12, 2011.
OCR for page 27
Program, Curriculum, and Syllabi Review 27
Unpublished
NDIC. 2011b. National Academy of Sciences–Curriculum Review of the Master of Science and
Technology Intelligence, May 2011. Washington, D.C.: Center for Strategic Intelligence
Research, NDIC.
O’Neil, Joseph. “The faculty’s role in curriculum development.” Presentation to the committee on
May 13, 2011.
Studds, Susan, Brian Smith, Joe O’Neil, Gerry Sherrill, Duncan McGill. “National Defense
Intelligence College – School of S&T Intelligence.” Presentation to the committee on May
13, 2011.
OCR for page 28
OCR for page 29
Appendix
OCR for page 30
