National Academies Press: OpenBook

A Review of the FBI's Trilogy Information Technology Modernization Program (2004)

Chapter: 2 IT-Related Issues for the FBI Requiring Immediate Action

« Previous: 1 Background
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

2
IT-Related Issues for the FBI Requiring Immediate Action

Although limited in scope and time, the committee’s review of the approach and methodology currently being used by the FBI to drive the introduction of its new IT systems has raised a number of significant issues. To address these issues requires concerted FBI action in four major clusters. The issues in each of these clusters are serious in and of themselves. Taken in aggregate, the detrimental impact of inattention to these issues on the FBI’s IT modernization efforts is enormous despite the progress that has been made. These four clusters are:

  • Enterprise architecture. An enterprise architecture maps the linkage between the FBI’s strategy and operational needs and its IT program.

  • System design. System design is the engineering of detailed IT solutions driven by the enterprise architecture.

  • Program and contract management. Large-scale program and contract management processes are critical to the success of IT modernization endeavors as massive as Trilogy.

  • Skills, resources, and external factors. The fourth and final cluster cuts across the first three clusters and generally relates to human resources and skills, and to some of the external constraints faced by the FBI in its IT modernization efforts.

The next four sections deal with these issues cluster by cluster.

2.1 ENTERPRISE ARCHITECTURE ISSUES

2.1.1 Creating an Enterprise Architecture That Serves FBI Objectives

What Is an Enterprise Architecture?

An enterprise architecture characterizes the enterprise’s missions, tasks, and operational processes, and relates these tasks, processes, and operational objectives to IT strategy, invest-

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

ment, and design. It provides substantial detail on the structure and standards used to implement the IT system. The enterprise architecture is the framework that describes the way in which an organization such as the FBI conducts its mission(s), how it organizes and uses technology to accomplish its goals and execute key operational processes, and how the IT system is structured and designed in detail to achieve these objectives. In general, it should also include documentation that explains the rationale behind important decisions and why certain alternatives were chosen and others rejected.

An enterprise architecture thus contains much more than information about technology. The enterprise architecture becomes the template on which the IT investment is rationalized, and the enhancements to the FBI’s mission achievement, the return on the investment, are defined and quantified using appropriate metrics. When new needs emerge, as with the counterterrorism mission, an enterprise architecture provides a point of departure—a framework within which additional capabilities can be coherently designed.

Absent an enterprise architecture, it is essentially impossible for any large organization, including the FBI, to make coherent or consistent operational or technical decisions about IT investments. Among these decisions are the definitions of appropriate data structures and linkages to other systems and data sources, policies and methods of information sharing, issues of security and the tradeoffs with information access, innovation, and the exploitation of evolving technologies, and metrics of effectiveness for the IT system and its use.

The close link between good enterprise architecture planning and sound systems engineering practice on the one hand and success in large-scale IT deployments on the other has been demonstrated in numerous examples in the private sector and in the federal government.1 Further, the existence of the enterprise architecture can be a major contributor to the confidence and trust that management, users, and implementers have in a project, as well as facilitating cooperation among them. A well-documented and communicated enterprise architecture is a prerequisite for driving cultural and operational change and innovation at a pace that effectively capitalizes on the pace of technology improvement. Good models depict the as-is (today’s) environment as well as the to-be (future desired) environment, showing the transition.

The Structure of an Enterprise Architecture

While technology capability is an important input to operational strategy, it is important that IT investment not be driven primarily from the technology end, but rather that operational strategy drive technology investment and system design. To this end, the committee believes that a good starting point for achieving the required linkage is to think about architectures in three conceptually distinct but interrelated forms:2

1  

For an example in a medical setting, see Jonathan M. Teich et al., “The Brigham Integrated Computing System (BICS): Advanced Clinical Systems in an Academic Hospital Environment,” International Journal of Medical Informatics 54:197–208 (1999). For a case study in which failure was closely associated with insufficient attention to architectural issues, see National Research Council, Continued Review of the Tax Systems Modernization of the Internal Revenue Service, Computer Science and Telecommunications Board, National Academy Press, Washington, D.C., 1996. For a case study in the private sector that demonstrates similar lessons, see Christopher Koch, “AT&T Wireless Self-Destructs,” CIO Magazine, April 15, 2004, available at http://www.cio.com/archive/041504/wireless.html.

2  

The architectural triad used here originates with the Department of Defense, which has used this framework to develop its command and control systems since 1997. (A good reference on this subject is Architecture Working Group, Department of

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • An operational architecture, which describes in graphical and narrative terms the key operational objectives; the operational elements, tasks, and processes used to achieve them; a high-level description of the types of information used and created; how and with what constraints information of various types must be exchanged; and the information flows in these processes. An operational architecture relates to specific mission scenarios and functions and forms the basis for realistic process and information flow representation and prioritization.3 (An analogy for operational architecture in the construction domain is the concept of operations for the various purposes a building will serve, and how various components of the building will serve those needs.)

  • A systems architecture, which describes at a high level the data repositories, systems, applications, connectivity, and communications infrastructure that supports the operational architecture for mission needs. The systems architecture defines the logical and physical connection, location, and identification of the key nodes, circuits, networks, and platforms that are associated with information exchange and specifies some critical system performance parameters. The systems architecture is constructed to satisfy operational architecture requirements using the standards defined in the technical architecture. (An analogy for systems architecture in the construction domain is the blueprints for a building that illustrate how components fit together.)

  • A technical architecture, which captures the key technical standards, protocols, and specifications required to implement the systems architecture. A technical architecture is intended to be a compact set of rules governing the arrangement, interaction, and interdependence of the parts or elements whose purpose is to ensure that a conforming system satisfies a specific set of requirements. It identifies system services, interfaces, standards, and their relationships. It provides the framework for the derivation of engineering specifications that guide the implementation of systems. (An analogy for technical architecture in the construction domain is the set of building codes for connecting subsystems and ensuring the operational effectiveness of the building with the community electrical, water, sewage, and roadway systems.)

To indicate what some of the content of these three architectures might be, the committee attempts to describe in Figure 1 some primary and supporting activities in the FBI as they might contribute to one part of the architectural triad described above—an operational architecture.4 The purpose of Figure 1 is to focus attention on key structural elements of the FBI. Figure 2 shows a subset of FBI activities and the potential scope of planned supporting IT

   

Defense, C4ISR Architectural Framework, Version 2.0, December 18, 1997, available at http://www.defenselink.mil/nii/org/cio/i3/AWG_Digital_Library/pdfdocs/fw.pdf. The descriptions of operational, systems, and technical architectures contained in this report are adapted from this DOD document.) Note, however, that the engineering methodology of defining a “functional architecture” based on processes and information flows and then building a “system architecture” with ever increasing technical detail dates back to the mid-1970s. And the use of structured analysis to form the basis for a technical system design has been in widespread use throughout commercial and government organizations since the late 1970s.

3  

As one possible scenario, the FBI told the committee that it has some 6 million documents found in Afghanistan. How will the information in these documents be extracted and managed? Who will have access to what set of those documents? What circumstances will govern access rules? How will the information contained in these documents be used with other information to which the FBI has access? Thinking through this scenario will provide a great deal of insight into information management requirements. The key point is that however the FBI decides to handle the information from these documents, it should be able to articulate the rationale for doing so in terms that make sense from an operational point of view.

4  

See, for example, Michael E. Porter, Competitive Advantage: Creating and Sustaining Superior Performance, Free Press, New York, 1998.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

FIGURE 1 FBI value chain—an example of a framework for “the view from 40,000 feet” of a serious enterprise architecture. This adaptation of Michael Porter’s famous business value chain indicates the core processes of the FBI’s two major missions—(1) criminal investigation and (2) counterterrorism. In addition, the primary activities include process management to monitor the performance of these two missions and quality management to provide feedback on the effectiveness of both the primary outputs—information—and how users actually use that information. At the top of the diagram are the traditional resources management areas: administration, procurement, human resources, finance, and IT. These supporting activities ultimately must be integrated with the primary activities shown in the bottom two-thirds of the diagram for the FBI to work most effectively.

activities—the ACS, the VCF, SCOPE (the Secure Counterterrorism Operational Prototype Environment, discussed below in Section 2.2.3), and the IDW.

Figures 1 and 2 represent the committee’s (certainly imperfect) perspective on FBI operations and the roles of IT systems in supporting FBI missions. These figures are highly simplified (for example, they do not include the operational impact of providing law enforcement assistance to other agencies), and they lack any kind of supporting detail. However, the committee inferred the content of Figures 1 and 2, rather than receiving them from the FBI. One of the committee’s major concerns is that it was not shown anything produced by the FBI or its contractors that approaches the content or substance of even these oversimplified figures. In

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

FIGURE 2 The scope of key applications on the FBI value chain. This diagram overlays on the middle portion of Figure 1 the FBI’s currently existing IT systems (ACS and the SCOPE prototype) and the ultimate next-generation applications (VCF and IDW 1.0) and their purpose in relation to the bureau’s core processes. Clearly, the FBI has hundreds of systems to be arrayed on a chart of this kind for it to be of most value. This task is not complex, and the results would greatly facilitate understanding both the “as is” and the “to be” contemplated by the Trilogy program and other FBI IT planning.

other words, the fact that the FBI was unable to present its own version of these figures was telling to the committee about a lack of clear architectural thinking. Without such a top-level description and understanding, all of the subsidiary steps in the design and implementation of an enterprise IT system are at significant risk of failure.

It is important for the FBI to create and use its own versions of Figures 1 and 2 to help explain its strategies, and to do the supporting work required to fill out their content. Once this is done, it does not matter if the FBI’s own versions agree with the committee’s figures, and a detailed acceptance or rejection of the committee’s figures is not relevant.

The committee recognizes that the framework described above is only one way to conceptualize enterprise architectures. For example, the Federal Enterprise Architecture (FEA) frame-

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

work,5 to which the FBI has already committed itself, is used in many federal agencies. Multi-agency use of the FEA framework enables architectural efforts presented using its five-component framework to be directly compared across agencies. Nevertheless, the committee believes that the architectural triad described above is conceptually clearer and more straightforward than the FEA framework and that the FBI will make more rapid progress using the triad even if some translation between the triad and the FEA framework will likely be necessary at the end of the process.

Recognizing the Role of Top Management in Creating an Enterprise Architecture

An enterprise architecture has far more operational content than technical content. Accordingly, the FBI’s enterprise architecture must be based on mission needs and must be formulated, propagated, owned, and evolved by the operational leadership of the FBI at the highest levels (the FBI director and the directors of the FBI’s mission-oriented divisions).6

An effective process requires that the enterprise architecture be created by a combined effort involving both senior operational management and key technologists. The CIO plays a key role as the facilitator of the process; however, the creation of the enterprise architecture cannot be handed off to the CIO, and certainly cannot be outsourced. The enterprise architecture is central to an organization’s strategy, as well as important for process analysis, change, and planning. Outside expertise can play a role in assisting and advising the enterprise architecture process, but the FBI’s top management team must manage, buy into, and execute the process.

Although this task is critical, it need not be an enormously time-consuming one. With adequate time and focus on the part of senior operational management working with key IT people, major progress could be made in a week of intense work, and the creation of a reasonably complete operational architecture together with a top-level schematic systems architecture—the most critical part of the enterprise architecture—should be possible in a 6-month time frame.7 Further, committee experience indicates that progress is best made with the top-level management team supported by an architecture team composed of a small number of full-time professionals dedicated to the task. These individuals must understand operations and have some appreciation for technology, rather than being technologists with only a loose connection to the operational side.

Given the fact that the leadership of any large organization, including the FBI, has many ongoing demands on it, the temptation is large to simply hire a CIO with a great deal of technology experience, delegate the task, and then forget about the problem. Why must an organization’s operational leadership be deeply engaged in the development of an enterprise architecture for IT? Why can’t this function be outsourced?

5  

The FEA is a business- and performance-based framework to support cross-agency collaboration, transformation, and government-wide improvement. See http://feapmo.gov/.

6  

In principle, it is also possible to begin with a vision of how technology might enable missions to be accomplished—that is, to seek a technology-driven enterprise architecture. But such an approach demands an extraordinarily high level of sophistication about the capabilities and limitations of technology, and for reasons that this report documents, is not appropriate for the FBI.

7  

The executive assistant director for intelligence testified to the committee that a small team working full time for about 10 weeks was able to develop a concept of operations for the Office of Intelligence. Thus, 4 to 6 months of work does not seem unreasonable for developing the operational dimensions of a larger enterprise architecture.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

The committee’s experience is that without the drive of operational missions, technology deployments inevitably serve to automate existing functions at a relatively low level, and thus the processes associated with these functions remain static. Even worse, automation of existing functions can degrade the ability of an organization to carry out its missions. The reason is that the functions that an organization must serve are sometimes in tension—consider an organization that must keep sensitive information secure (a driver for restricted access), and yet make it available to everyone with legitimate need (a driver for broad access).

When people deal with these competing demands, they make judgments and ad hoc decisions on the basis of their experience and expertise. Despite the existence of functional tensions in a non-automated system, human flexibility and adaptability generally ensure that operations can function at an adequate level of performance. Indeed, in most organizations, what people actually do on the job varies from what is specified by the organization’s formal processes. These variances do not arise because people are poorly motivated or lazy, but rather because people are highly motivated, and the variances are usually necessary for real work to be done.

The fact that humans resolve these conflicts so smoothly, however, usually means that the tensions inherent in differing functions are hidden from view. Technology deployments bring out these tensions, and force someone to decide, in advance, how the tradeoff should be resolved. When automation is implemented in an organization without the benefit of operational knowledge (i.e., without taking into account what the organization is trying to do), the tradeoffs involved in deciding what to deploy are made in a vacuum—and often wrongly. Only the senior operational management is in a position to articulate explicitly how these tradeoffs should be resolved.

Beyond the need to develop an enterprise architecture to manage the IT investment process, it is important to recognize that the activity of developing an enterprise architecture often results in a (highly desirable) reexamination of operational processes. That is, by focusing on what technology can and should do for an organization, attention is naturally drawn to the processes it is intended to support—and often the processes themselves are seen to be out-moded, unnecessary, or ripe for streamlining, often because they were based on the limitations of older technology or on requirements that are no longer current. Such discoveries are valuable in themselves in that they help an organization function more efficiently even apart from its investments in technology. Such opportunities can be expected to arise when examining the processes of investigation and intelligence at the FBI.

Finally, as missions evolve, so must an enterprise architecture—and in that sense, an enterprise architecture is never final. Since it is only the senior operational leadership that can make basic decisions about the organization’s missions, they will need to have a continuing involvement in the evolution of the enterprise architecture and a process to periodically revisit it. This point again reinforces the idea that an articulation of the operational dimensions of an enterprise architecture cannot be outsourced.

2.1.2 FBI Activities with Respect to Enterprise Architecture

Based on FBI briefings and presentations to the committee, the committee believes that the FBI’s efforts and results in the area of enterprise architecture are late, limited, and fall far short of what is required. Indeed, in spite of the fact that the Trilogy projects are far along and substantial resources have been expended, the FBI has only very recently begun serious efforts

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

to create its enterprise architecture, and no comprehensive enterprise architecture is in place today.8

For example, the FBI reported to the General Accounting Office that it has completed and approved an enterprise architecture “foundation document.”9 However, the FBI made no reference to this document or its content in its October 2003 and December 2003 presentations to the committee, despite its publication on August 23, 2003, and despite the fact that the committee raised questions about enterprise architecture many times during those sessions. Ultimately, the committee learned of this document by reviewing a DOJ inspector general report, and it obtained a copy of the document in early 2004.

Committee members have reviewed this document and believe that the document does begin to discuss at a high level some of the issues discussed above about linking the IT system to missions and operational processes. However, little follow-on work appears to have taken place as of the time of this writing (late March 2004).

In addition, the FBI has undertaken other efforts that could reasonably represent seeds of architectural work that needs to be done more broadly. The Office of Intelligence has embarked on a laudable effort to develop the beginnings of its own architectural sub-framework, and spokespeople from that office were able to relate IT requirements to operational process at a high level in a convincing manner. Those responsible for the VCF project have also developed an architectural sub-framework that appears adequate to support the initial rollout of the application in 2004. The VCF may well be a component that can be made, with some effort, to fit into the enterprise architecture when it is created. Nevertheless, these architectural efforts—important though they are—do not substitute for an overall architecture that addresses the missions of the FBI collectively.

The committee’s concerns about inadequate enterprise architecture work are reinforced by the fact that only in the FY2005 budget enhancement request will the bureau include very substantial funding for enterprise architecture related activities, and that this request was under consideration at DOJ and the Office of Management and Budget (OMB) as of September 2003. (The FBI is further seeking to obtain an interim system engineering, integration, and test contractor to blend the Trilogy VCF, the Secure Counterterrorism Operational Prototype Environment (SCOPE), and the Integrated Data Warehouse (IDW), and several smaller efforts, into a unified and functioning whole.10)

Under the best possible circumstances, this FY2005 budget enhancement request will take effect October 1, 2004, suggesting that FBI effort in this area will not be substantial at least until then. Moreover, the fact that this request has taken the form of a budget “enhancement” for FY2005 implies that the FBI has given this task a lower priority than might be implied, for example, by a request for authority to reprogram existing funds from FY2004 for this effort. In the absence of a serious enterprise architecture effort, even the interim plan to blend the VCF, SCOPE, and the IDW will be very difficult to achieve.

8  

In this regard, the committee’s conclusion echoes the primary finding of the General Accounting Office report The FBI Needs an Enterprise Architecture to Guide Its Modernization Activities (General Accounting Office, GAO-03-959, September 2003, available at http://www.gao.gov/new.items/d03959.pdf).

9  

See Response of the FBI to the GAO report The FBI Needs an Enterprise Architecture to Guide Its Modernization Activities, dated September 22, 2003. Available at http://www.gao.gov/new.items/d04190r.pdf.

10  

See Response of the FBI to the GAO report The FBI Needs an Enterprise Architecture to Guide Its Modernization Activities, dated September 22, 2003. Available at http://www.gao.gov/new.items/d04190r.pdf.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

Beyond the lack of anything resembling a complete enterprise architecture, of most concern to the committee is the fact that most of the senior operational management of the FBI does not seem to have been deeply engaged in either the nascent architecture efforts or in shaping the Trilogy effort. For example, the Enterprise Architecture Foundation Document was produced by an entity called the Architecture Governance Board, whose members are apparently drawn from the FBI IT community—and did not include senior individuals from the major operational units of the bureau. That is, it does not appear to reflect substantial involvement by the senior operational leadership of the FBI and hence cannot be sufficiently complete to guide the FBI’s IT development and deployment efforts.

This must change. For the IT modernization to succeed, frequent and engaged participation by the FBI’s senior operational management in the creation and review of an enterprise architecture is necessary, and must be followed up with ongoing and systematic monitoring of the FBI’s and contractor plans and progress through implementation. A properly designed process for developing an enterprise architecture can allow senior management to play their essential role without placing excessive demands on their time, and being actively engaged in the creation of an enterprise architecture does not necessarily mean that the senior operational management is responsible for every step. Rather, as the enterprise architecture is being created, the senior operational management must be involved in sessions at which there is serious discussion of the enterprise architecture’s contents, and where important decisions are made. Many of these issues and decisions involve policy questions, including a determination of whether a relevant policy exists and thus should be reflected in the enterprise architecture, or needs to be created.

The committee believes that the absence of an enterprise architecture has been a major contributor to the problems faced by the implementers of the Trilogy program. That is, the lack of an architecture to guide the planning of an information and communication infrastructure has resulted in improvisation that has virtually no chance of resulting in a well-ordered infrastructure for the enterprise to build upon. In fact, merely providing parts (e.g., computers and accessories, piece-part applications, and so on) is like buying brick, mortar, and lumber and expecting a builder to produce a functional building without benefit of building codes, blueprints, or an understanding of how people will use the building.

2.1.3 Data Management Issues Arising from the Absence of an Enterprise Architecture

The FBI’s lack of an enterprise architecture has manifested itself in many ways. For example, as noted above, the counterterrorism mission requires access to and the exchange of comprehensive and timely information. Thus, it is easy to imagine that the FBI will, under some set of circumstances, need to receive and/or send information from or to state and local law enforcement agencies (e.g., local police departments), its counterparts in foreign nations, or other members of the U.S. intelligence community.11 The identification of precisely which

11  

A proof-of-principle project (the Gateway Information Sharing Project) was established in St. Louis in April 2002 to integrate investigative data from federal, state, and local law enforcement agencies into one database that will ultimately be accessible to all participating agencies via a secure Internet connection. The project merges investigative files and records from all levels of law enforcement into a single, searchable data warehouse, providing investigators and analysts the ability to search the actual text of investigative records for names, addresses, phone numbers, scars, marks, tattoos, weapons, vehicles, and phrases. The project is the result of cooperation between the FBI and a variety of state and local police departments in the St. Louis area and Southern Illinois. See DOJ press release of October 9, 2002, “Attorney General John Ashcroft Unveils Gateway Information Sharing Pilot Project in St. Louis, Missouri.” Available at http://www.usdoj.gov/opa/pr/2002/October/02_ag_590.htm.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

parties the FBI will share information with and the conditions under which information will be shared has profound implications for its enterprise architecture, and is an issue that the senior operational leadership must address.

A second issue is a potential confusion between mission and process. In discussions with the committee, senior FBI personnel repeatedly stated that “investigation and intelligence are the same thing.” While it is true that the mission of criminal investigation makes use of intelligence processes, this fact does not mean that an IT infrastructure to support investigation should be the same as an IT infrastructure that can support intelligence. For example, the measure of success of an infrastructure that supports investigation is whether it helps agents to do their investigation more effectively, more efficiently, or with greater quality of output and national impact. Agents are the primary actors in criminal investigations, and they task analysts; thus, the developers of an IT infrastructure for investigation should regard agents as their primary customers.

By contrast, the mission of counterterrorism is intelligence-heavy. The measure of success of an IT infrastructure for intelligence is whether it helps the analyst perform analytical functions more effectively/efficiently, e.g., helps analysts identify patterns, clues, or other information generally indicative of events that have not yet occurred. In the intelligence process, analysts are the primary actors, in that they are the ones primarily responsible for analyzing a wide variety of information, from both open and classified sources, in order to identify impending terrorist actions that must be disrupted or pre-empted. Thus, agents will act on the information given them by analysts, and those actions are generally mapped into a criminal investigation. Presumably, agents also serve as collectors in the counterterrorism mission, and therefore can be tasked by analysts.

The following are some important differences that must be kept visible in designing IT support for the investigation and intelligence processes:

  • Analysts tend to place a higher priority on access to a broad array of information resources, while investigators tend to place a higher priority on highly targeted information.

  • Analysts may find “bad” or “untrustworthy” information contextually useful, while investigators may regard it as a liability and would want such information purged as soon as possible.

  • Analysts must understand context and how a situation changes over time; investigators often place a higher value on information that can serve as evidence and fact.

  • Analysts must handle and manage classified data, while investigators must generally, though not always, manage “law enforcement sensitive” or “sensitive but unclassified” data, which entails an entirely different set of requirements.12

In general, criminal investigation and intelligence are sufficiently different processes that the FBI would be wise to view the development of an overall enterprise architecture, particularly at the systems level, as calling for two subarchitectures with well-defined interfaces between them. An interface between the two subarchitectures is defined by the data that they exchange and share. These interfaces will require enforcement and oversight so that only

12  

There are exceptions to these generalizations, of course (e.g., investigations for counterintelligence purposes often generate classified information). But in any case, these exceptions represent only a small fraction of the total information collected.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

necessary data are passed between them, and it will take senior operational management attention to define what data are “necessary” and what the policies are for the interface engines. Further, these interfaces have profound implications for the systems and technical architectures in areas such as standards that facilitate data exchange.

2.2 DESIGNING IT SYSTEMS TO SUPPORT FBI STRATEGY AND OPERATIONAL NEEDS

This section comments specifically on a few important applications design issues, including overall system structure; the design of data repositories; and workflow, security, and access, as well as a collection of specific deficiencies in the VCF and other applications. In most cases, these issues are a consequence of the lack of an enterprise architecture, and thus of the disconnect between enterprise architecture and technology planning and design.

2.2.1 The Virtual Case File Application of Trilogy

The Virtual Case File, the user application component of Trilogy, is a custom-designed software application that is intended to facilitate case file management by integrating data from older, separate investigative systems, including the Automated Case Support (ACS) system, and eventually replacing them. The VCF is intended to create efficiencies in entering case-related information by reducing the number of steps in filing documents and to facilitate the storage and retrieval of data for wider access, tracking, and analysis of case-related data.13

The VCF is a new IT application designed to be the primary operations system to support the investigative mandate of the FBI. The design process was under way prior to the addition of the intelligence mission, and the requirements for the intelligence mission were not included in the VCF design.

The VCF is a significant step forward from today’s ACS system, and considerable progress on the VCF seems to have been made since September 2002. (The committee underscores the term “seems,” because it was shown only a mock-up of the application, and not the application itself. That is, the committee viewed a canned presentation, because no working application was available at FBI headquarters. The committee notes that there is often a significant difference between the impression conveyed by a mock-up and what an actual application can do in practice.)

Perhaps the most important—and commendable—development in the VCF effort is the appointment of a very experienced and computer-savvy FBI special agent as program manager who has played a strong role in driving the design from user requirements. To the committee, this individual appeared eminently capable of articulating user needs based on operational experience rather than speculation, and the committee believes that it is this manager’s operational insights that served as an implicit enterprise architecture (more precisely, a subarchitecture) for the VCF and that have consequently been a primary driver of significant progress. As such, the VCF has the potential to serve agents well in their criminal investigation mission.

13  

For further information about the virtual case file, see U.S. Department of Justice, Office of the Inspector General Audit Division, The Federal Bureau of Investigation’s Management of Information Technology Investments, Report No. 03-09, December 2002, pp. 91, 94-99, available at http://www.usdoj.gov/oig/audit/FBI/0309/final.pdf.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

Nevertheless, the committee is deeply concerned about certain aspects of the VCF effort. In the wake of the VCF prime contractor’s failure to meet a critical delivery date for the Trilogy VCF component rollout,14 the FBI has had to delay achievement of full system capability. In a memo delivered to the committee on February 11, 2004, the FBI noted that the revised schedule for Trilogy calls for achieving full system network capability (i.e., full functionality to 28,000 FBI employees at 612 sites) by April 30, 2004. Based on the VCF developer’s original estimate that it would require 6 weeks of full system capability for testing the VCF, and assuming full system capability by April 30, 2004, it was estimated that the earliest activation of the first release of the VCF would be approximately mid-June 2004. Subsequently, the FBI advised that such an assumption would be incorrect given the level of immaturity of the VCF application delivered to the government in December 2003 and through continued evaluation of the application. The VCF vendor is scheduled to present to the government an estimated time to complete on April 8, 2004.

The details of the new VCF plan have not been made available to the committee. But in briefings held in October 2003, the FBI described to the committee a plan that would implement the VCF in what it describes as a “flash cutover.” That is, the VCF would be rolled out for employee use all over the bureau simultaneously (or nearly so). The date for this cutover has been postponed, but the committee is concerned that, because of schedule slippage, the amount of testing will be reduced even further below what is already an inadequate level. With limited testing, and no experience gained from a limited initial rollout, the FBI would be implementing what amounts to a prototype throughout the bureau. This approach is nearly guaranteed to cause mission-critical failures and further delays, as discussed below in Section 2.3.1, with significant implications for training, performance, coherence, internal morale, public image, and cost to recovery.

The committee is also concerned that the VCF’s current design and technical specifications lack the flexibility needed to incrementally improve the application to support additional functions important for the field agents and FBI management. This inflexibility will make further rounds of fixes and enhancements difficult. (Note that the comments below are based on a VCF mock-up in a canned presentation to the committee, rather than the committee’s own exploration of a working VCF prototype.)

Based on the committee’s understanding of and perspective on the VCF’s current design, some of the potentially important issues related to the VCF include the following:

  • As described to the committee by the FBI and its contractors, the current implementation of the VCF appears to have embedded the workflows describing how information is to be entered, reviewed, and used. Embedding the workflow in the application (that is, hard-wiring it) will make any such changes in the future much more difficult (more expensive and slower) to implement. Such changes are likely to be driven by changes in the operational processes that the VCF supports.15

14  

See U.S. General Services Administration press release, “Contractor Misses IT Delivery Date,” November 3, 2003. Available at http://www.gsa.gov/Portal/gsa/ep/contentView.do?pageTypeId=8199&channelId=-13259&P=%7C40E6C831B9572449852568AF00594486&contentId=14244&contentType=GSA_BASIC.

15  

To illustrate how workflow changes might become necessary, note that FBI practices seem to be built around the assumption that only supervisor-approved documents have any status as information worthy of access by parties other than the original creator. However, in today’s world of greater information sharing, an unapproved draft may well have information that might

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • The VCF does not have an “offline use” mode. Without an offline mode, an agent working a case cannot take copies or extracts of case materials into the field for reference, nor create content offline that will then be synchronized into the VCF server when coming online. Today’s administrative procedures may prohibit such practices, but if mobile computing and remote access for agents become desirable and necessary (as discussed in Section 2.2.4), the inability to access the VCF in an offline mode will become critical. Remote operation, whether offline or via a communications link, has proved to be a major enhancer of productivity in a wide variety of situations.

  • In the VCF, documents are indexed with terms that are manually specified by supervisory agents without any auxiliary capability for the automatic generation of index terms. Manual indexing reflects today’s practice in the FBI, but modern indexing software can do a very good job of finding index terms in a corpus of text automatically. Manual indexing will fail to supply terms that are outside the context of the human indexer, as many counterterrorism activities are likely to be. Today, full-text index systems that index every word in a document are the foundation of powerful search engines, such as Google, and are being exploited to a higher level in extracting meaning automatically in some experimental systems. It is true that basic automatic indexing systems are unable to provide index terms that are not explicitly represented in the text; for this reason, automatic indexing is not a replacement for manual indexing as much as it is a very fast and powerful supplement for it.

  • The VCF appears to lack broad automatic notification of changes that are made. Case owners are made aware when documents are added to a case file, but no other parties are notified. In an environment in which many people (investigators and analysts) may use a case file, such lack of notification places a large burden on those users to check the contents of the case file periodically. They will waste time in checking a file that has not been altered and lose currency between the point of update and their access to the file. Lack of notification is thus an inhibitor of collaboration.

  • The VCF appears to lack useful capabilities such as bookmarking, favorites, or history features for an individual user. Without such features (commonly available on most Internet browsers), users are poorly equipped to remember where they are or where they saw interesting information. This fact is particularly significant when the volumes of information with which one is working are large. Such capabilities would be useful now, but will be expected by any user in the future.

  • The VCF appears to be weak in its ability to sort data. The ability to sort columns of names or dates or other such information is quite valuable when one has only a vague memory of a name or a date associated with it. For example, a user may remember that a document was from the June 2001 time frame. If the system does not allow a sort by date, the user is forced to search explicitly for all documents dated June 2001. This does not appear to be a drawback, until one realizes that his memory may be incorrect, and in fact the document appeared on May 31. By arranging the entries according to date, the user can rapidly scan both sides of the putative date index, and the negative impact on information retrieval of errors in recollection can be reduced.

   

prove useful to another agent or to another case, and procedures may need to be changed to accommodate a different sharing arrangement. Other examples might be the need to cope with shifting organizational structure and roles, or with new missions and legislation or regulation.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • The VCF interface presented to the committee was inconvenient in some respects. For example, the committee saw no evidence that hot links to organizational and user information were provided, even though such information could be supplied easily. To increase the VCF’s convenience, a case file should have listings of everyone who accesses the file and should provide online contact information about those users to facilitate contact.

One might draw an analogy between the VCF and “blogs” (i.e., Web logs). Today, members of the Internet community often use blogs to observe and comment on some theme, or to discuss various topics. The VCF captures observations about a case, and may include a commentary describing the investigator’s reasoning. Some blogs support cross-referencing and commentary by contributors other than the primary author. The VCF, particularly when seen through an analytic lens, might benefit from richer discussion and content, dynamic cross-referencing of cases, and external commentary on the reasoning behind a case or cross-references. The culture of blogs and the technology to support them may provide a rich source of external ideas for the future functionality of the VCF, as well as commercial technology for future implementations. An example is the Really Simple Syndication protocol used with blogs to notify interested parties of changes, thereby eliminating the need to constantly revisit a blog to understand what has changed.

All of the above issues can be addressed in subsequent releases of the VCF. A more staged rollout of the VCF would likely have resulted in more focus being put on these and other still-to-be-identified shortcomings. The VCF will certainly require ongoing multiple releases, and it is essential that the FBI substantially enhance its capabilities in design and implementation for these ongoing investments.

2.2.2 Data Management and the Integrated Data Warehouse (IDW)

The FBI has a tremendous quantity of data, most of which comes from its investigative processes. This data must be organized and managed in a way optimized to promote the effectiveness of the bureau’s agents and intelligence analysts. Access capabilities required for intelligence analysis to determine possible events in the future often differ greatly from the access capabilities required for reliable case records management, which has the mission of organizing and retaining information to meet rules-of-evidence requirements. Accordingly, because it need not follow rules of evidence, the intelligence process may have different trust models governing browsing of information.

Thus, the information requirements of the investigative and the intelligence missions of the bureau are very likely to require very different work processes and information access. It is very likely (nearly certain) that different data models will be required in order to provide efficient support for these separate missions. Furthermore, each mission of the bureau has unique constraints for access to the information and will require different security models and different tradeoffs between security and accessibility. Implementation of these models is likely to show that distinct systems, with overlapping contents and interface engines to manage data sharing and inter-organization handoffs, will be needed to assure manageability in the presence of conflicting constraints.

The Integrated Data Warehouse (IDW) will serve as a repository to store external data from a variety of sources that come into the agency at different frequencies, such as criminal

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

information from other government agencies and visa data.16 The IDW will remain separate from the VCF but will replicate much of the data in the VCF. With proper security and sharing rules placed on the data, the IDW is intended to allow teams of analysts and agents to run queries horizontally across the data, and to share subsets of the data with other government agencies.17

According to briefings to the committee, the IDW is intended to be a resource for intelligence efforts (among other roles), and to be used by FBI analysts and shared with other intelligence agencies. For such analyses data from non-FBI-sources will have to be included in the IDW.

These briefings also suggested that data modeling efforts for the criminal investigation mission were just getting started. Further, the data models of the IDW presented to the committee (and the VCF for that matter) were far too abstract to be very useful, though the presentation from the intelligence side recognized that there were differences in need between the investigative and intelligence analysis functions.18

Three examples will suffice to illustrate a disconnect that needs to be resolved between the data models as described to the committee and operational needs:

  • Presentations to the committee raised the issue of data currency. That is, intelligence analysts seemed to expect to have access to live databases containing the most current information, while the design of the FBI’s data warehouse incorporated copies of production databases. If the warehouse is intended to contain copies, the issue is raised concerning the frequency at which the warehouse copies are refreshed from production databases.

  • The IDW appeared to have been designed to overwrite old copies of databases with newer copies, so data can be there one day and not the next (that is, the IDW is not equipped to handle time-series of versioned data). While having only the most recent copy of data may be appropriate for the purposes of an investigation (presuming the most recent copy is the most accurate and reliable), this assumption is almost certainly not valid for intelligence purposes.

16  

The committee notes that the IDW acronym itself is subject to some confusion. Director Mueller has made reference in congressional testimony to an “Integrated [italics from the committee] Data Warehouse [that] will link 31 FBI databases for single-portal searches and data mining.” The DOJ inspector general report states that “the FBI expects to use the network to transport the Investigative [italics from the committee] Data Warehouse, which will link 31 FBI databases for single-portal searches and data mining,” and the FBI briefed the committee on IDW referring to the “Investigative Data Warehouse.” Whether this inconsistent use of the acronym is inconsequential or reflects a deeper confusion within the FBI about the purview of data to be contained in the IDW remains to be seen. For Mueller’s testimony, see http://www.fbi.gov/congress/congress03/mueller032703.htm; for the DOJ IG report, see http://www.usdoj.gov/oig/audit/FBI/0336/exec.htm.

17  

Robert S. Mueller III, “Congressional Statement on Federal Bureau of Investigation’s Fiscal Year 2004 Budget,” House Appropriations Committee, Subcommittee on the Departments of Commerce, Justice, and State, the Judiciary and Related Agencies, March 27, 2003, available at http://www.fbi.gov/congress/congress03/mueller032703.htm.

18  

Data models are generally used by system architects and designers in an iterative refinement process that starts with a conceptual data model, which deals with the key business concepts of the database, and then proceeds to a logical data model, which allows designers to reason about the primary relationships among the key data tables and elements, and then finally ends up with a physical data model, which documents the database implementation details. Modern computer-aided software engineering tools often facilitate the refinement process, reducing the labor required to construct and manage the three levels of model. Rather than include any specific recommendations about data modeling, the committee notes that the FBI’s enterprise architecture effort should identify a data modeling framework to be used by developers and implementers. An essential element of data modeling is a data dictionary, which defines the basic structure and organization for the database with which it is associated. The construction of data dictionaries for the various databases is an essential prerequisite for sharing data between databases.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • Some of what was presented when the committee asked about data models were not data models at all, but rather hierarchical XML descriptions of various criminal justice documents (e.g., arrest forms, booking reports, sentence orders, and so on).

To illustrate what a high-level enterprise perspective on data management might entail, consider that IT systems for the FBI must serve multiple operational functions. From the committee’s perspective, the major classes of data are data to be kept in:

  • The active case records used by the field agents—these are to be served by the VCF system under development.

  • A broad-based data warehouse to serve intelligence tasks now assigned to the FBI—these were served by the SCOPE prototype demonstration and are to be served by the IDW follow-up projects.

  • A reliable repository of record that will provide a formal backup source for prosecution of criminal cases, intelligence investigations, and internal audits. It is restricted to holding data for which the FBI is formally responsible. As such, it will be smaller, less visible, and more stable than other FBI data systems, will be highly reliable and capable of becoming the FBI’s replacement for paper documentation, and will spare other FBI IT systems some of the more stringent requirements that can inhibit their operation.

  • A wide variety of management and administrative databases for personnel, capital and registered item inventory, evidence tracking and inventory, and so on, to allow FBI headquarters to carry out its responsibilities.

Data in each of these classes are not entirely independent of each other, and thus applications to manage these types of data must have linkages among them. (For example, information related to a given legal discovery motion or Freedom of Information Act request may be contained across these different types.) In addition, there may well be linkages to data contained in databases operated by other agencies (e.g., data supporting FBI intelligence tasks may well originate in databases operated by the CIA). However, the fact that the applications must have linkages among them does not mean that the databases must be co-located or should be managed by the same staff under the same access rules—a point especially likely to apply to data-linking connections to intelligence agencies.

To understand fully the high-level requirements for these applications, their objectives must be made explicit. Based on what it learned from the FBI about its data handling needs, the committee infers the following list of objectives for systems that handle each major class of data described above.

The VCF should:

1a.

Serve the field agents and their supporting analysts, wherever they are, in tasks of information collection and analyzing the collected information on their cases.

1b.

Be available 24/7.

1c.

Allow convenient upgrading of services as needed by the agents and enabled by technology.

1d.

Provide linkages to data in the IDW and other FBI databases, and allow insertion of records from other sources that have been determined to be pertinent to a case by the responsible agent and his/her supervisor.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

1e.

Purge closed cases, but keep track of them for analysis purposes.

1f.

Keep all the information secure vis-à-vis outsiders (i.e., outside the FBI).

1g.

Keep some elements secure vis-à-vis most FBI personnel.

1h.

Be flexible enough to accommodate changes in operational processes that may be made in the future.

1i.

Resist tampering with records.

1j.

Support an appropriate degree of auditing when information is entered or changed.

1k.

Support audits for system use (a security measure). (System use refers to read access, write access, and search results.)

The IDW should:

2a.

Serve broad intelligence functions of agents and analysts.

2b.

Acquire, integrate, and store data from other sources of potential relevance for broader analyses, such as sources from the Department of State, Immigrations and Customs Enforcement, and foreign sources.

2c.

Allow mining of information that is not (yet) related to a specific case within the time that might be needed to mobilize a team to respond to an incident (say, within 20 minutes). (This time appears to be an achievable goal with current technology for obtaining and integrating data from known remote sources.)

2d.

Have the capability to store tentative conclusions, and to purge such information if it is determined to be misleading.

2e.

Be flexible to support an increasing variety of data mining and analysis tools to be installed as they are found to be useful.

2f.

Protect its content so that it is secure vis-à-vis outsiders and unauthorized insiders or turncoat insiders.

2g.

Not contain information that would be forbidden to any FBI personnel.

2h.

Support an appropriate degree of auditing when information is entered or changed.

2i.

Support audits for system use (a security measure). (System use refers to read access, write access, and search results.)

The repository of record should:

3a.

Serve broad intelligence functions of agents and specialized, authorized personnel, including internal audits.

3b.

Serve legally required documentary archival storage requirements, with complete audit trails, primarily to provide backup for cases under prosecution or review, including sensitive information that would be unwise to have available within systems that allow broader access.

3c.

Maintain extremely high reliability and integrity, even at the expense of 24/7 availability and flexibility.

3d.

Be fed primarily from the VCF, and exclude material that is not the responsibility of the FBI.

3e.

Support tools used solely for records management and not for investigation or analysis. (That is, only tools for records management should be run against the data in the

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

 

repository of record, though the contents of the repository of record may be an input to a larger pool of data that can be analyzed with arbitrary tools. Thus, the data models of the repository of record should be developed in a way that anticipates this use/interface.)

3f.

Be reorganized only rarely.

3g.

Not be used as a working resource by FBI personnel.

3h.

Be maintained at a very high level of security.

3i.

Support appending of contents rather than record purges for updates.

3j.

Resist tampering with records.

3k.

Support an appropriate degree of auditing when information is entered.

3l.

Support audits for system use (a security measure). (System use refers to read and write access.)

Administrative databases should:

4a.

Serve a variety of FBI management and administrative functions.

4b.

Be maintained and updated primarily by FBI personnel at headquarters.

4c.

Be accessible to field personnel for determining the location of their colleagues, resources, and equipment.

4d.

Manage the operational security issues associated with making such data available (in 4c).

4e.

Be available at close to a 24/7 schedule, although updating of information may be constrained to business hours.

4f.

Be secure vis-à-vis outsiders.

4g.

Not contain information that would be forbidden to any FBI personnel.

4h.

Evolve to align with Department of Justice standards for administrative support systems (e.g., finance, e-mail).

4i.

Support an appropriate degree of auditing when information is entered.

The FBI may disagree with this listing of essential objectives, which the committee has created based on limited knowledge and in the absence of an FBI enterprise architecture. Agreement in detail is unimportant, but the FBI must develop its own list of essential objectives, including priorities, tradeoffs, and explicit analyses, based on its own understanding of its essential operational processes and how it expects to use these databases. In any event, the diversity of requirements makes it clear that these storage applications must be implemented in separate systems, and that different policies will be needed to ensure the proper balance of accessibility, security, availability, and reliability.

It is likely that there will be a fair amount of duplication of contents in these databases. However, the cost of replicated storage is less than the cost of the manpower needed to maintain a single complex system with multiple and potentially contradictory objectives. Replication also mitigates some security problems, such as denial-of-service attacks (see Section 2.2.5). With multiple copies, there is also a significant probability of inconsistent or out-of-synch information. This is especially true with information that can change over time. Cleaning and validating information will become a separate process unto itself.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

The IDW has a broad role, and correspondingly broad objectives and access needs.19 The committee understands the intent behind the broad access needs of the IDW and other systems for managing operationally relevant data, but from a pragmatic standpoint, it will be difficult to design a single system to meet these access needs while still retaining necessary security measures.

Consider, for example, the statement that “the ownership of FBI information is corporate; no individual division or employee ‘owns’ FBI information.” Such a principle is attractive, for entirely understandable reasons. But the agents who collect the data will undoubtedly be concerned about maintaining control over data distribution and protection of sources and methods. Today, these issues are managed through informal information-sharing networks of agents and analysts that operate not only on the basis of formal access control provisions such as level of security classification but also on personal knowledge and trust of the individuals involved.

Information management systems that do not provide these agents with some control over the release of certain kinds of information may increase the likelihood that important information may not be entered at all. Agents or other information collectors may be inclined to keep two sets of records—one for official use and one for more sensitive information, allowing them to maintain control over the disposition of sensitive information.

This record-keeping challenge, whether based on realistic concerns or perception, is unlikely to be overcome by fiat. That is, a directive that agents and other collectors record officially all information, regardless of the sensitivity of its source, is likely to be quietly disregarded in many instances. Such disregard would not necessarily be gratuitous, but rather an entirely understandable reaction of collectors in the field who might be reluctant to trust the security of their sources to another party.

2.2.3 SCOPE

The FBI demonstrated the Secure Counterterrorism Operational Prototype Environment (SCOPE) to the committee. This demonstration illustrated the analytic tool suite with synthetic data. The analytic tools were based on commercially available products that included support for data visualization, relationship analysis, and automated language translation. Because this demonstration was not done using operational data or the Trilogy network, the committee cannot comment on the performance or scalability of this approach.

19  

For example, a July 2003 draft Integrated Information Sharing Plan lists the following eight guiding principles that the FBI expects to apply to its information-sharing strategy.

  • All FBI data is to be shared within the FBI, with very few exceptions.

  • The ownership of FBI information is corporate; no individual division or employee “owns” FBI information.

  • The FBI will have a single, integrated information space, in which the default will be to share with agencies with due consideration for the protection of sources and methods, and the security and prosecutive objectives of investigations.

  • The FBI will not filter information internally but instead will create an overarching FBI-wide policy that balances the need for cross-correlation with the risks of misuse.

  • The view of what the FBI collects and what the FBI creates will look very similar.

  • All data collected must also be recorded, searchable, retrievable, and easily cross-correlated.

  • FBI employees will have the ability to conduct federated queries across multiple systems to identify relationships.

  • Technology will be in service of people instead of people in service of technology. The FBI must have interconnectivity with Intelligence Community systems. Additionally, the FBI must leverage existing technologies instead of rebuilding them.

The committee did not review this draft plan. The text above is contained in a DOJ Office of the Inspector General report, The Federal Bureau of Investigation’s Efforts to Improve the Sharing of Intelligence and Other Information, Report No. 04-10, December 2003, available at http://www.usdoj.gov/oig/audit/FBI/0410/findings2.htm.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

The committee believes that the FBI has an important opportunity with SCOPE. Here, the FBI is not burdened by a legacy of analytic tools and data models. It has a unique opportunity to assemble and integrate the most promising analytic ideas and tools. A well-conceptualized enterprise architecture and data models are critical to successfully exploiting this opportunity and creating an environment that can incorporate the best analytic ideas and products over time. (This point reinforces the immediate need to develop and understand the data models and the workflow.)

The FBI’s use of commercial products is laudable. This willingness to look beyond conventional sources for the best available ideas can provide the FBI with an analytic advantage. The commercial market for such tools is being driven by much broader trends than just counterterrorism. These trends include the private sector’s desire to analyze and support market data, and the increasing globalization of markets. By harnessing the commercial market for analytic tools, the FBI can gain access to far more creativity than it could ever capture with internally developed tools.

Commercial products can also be expected to incorporate the best practices in human-computer interfaces. Even if the FBI does not use commercial tools, it should study their human-computer interfaces and incorporate the best ideas into the tools it develops internally. An approach rooted in commercial products does not rule out one-of-a-kind tools that are internally developed.

2.2.4 Mobile Computing

The FBI’s interest in wireless data communications appears to be driven primarily by operational continuity considerations. For the most part, the FBI’s experience with wireless data communications has been through personal digital assistants (PDAs) used by a small number of senior leaders. Anecdotal reports suggest these individuals have found that wireless data communications enhance the conduct and accelerate the pace of day-to-day business.

To make further progress in this area, the FBI is planning a trial of PDAs at a field office. Such a trial can be the basis for establishing a baseline characterization of agent and other activities in that office to understand if the deployment of PDAs has any impact. Thinking about how to create a baseline characterization of office activities, which in the future might involve mobile computing platforms such as PDAs or laptop/tablet computers, can also contribute to quantifying an office’s implementation of the FBI’s operational processes that are the critical input to the FBI’s enterprise architecture. The planning and results of this trial should be coordinated with the FBI’s enterprise architecture efforts.

2.2.5 Security

Loosely speaking, the security of an information system or network refers to its ability to continue to support authorized parties when it is under attack and its ability to resist the efforts of unauthorized parties to compromise its operation or data. For this report, it is helpful to conceptualize security issues along two dimensions—those that arise from the disconnect between the FBI’s operational mission needs and its enterprise architecture, and those that arise from inadequate thought about the implementation of whatever security policy is adopted.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

The disconnect between the FBI’s operational mission needs and enterprise architecture is reflected in the FBI’s expression to the committee of a philosophy of zero tolerance for security risk, seeking not to manage risk but to avoid it. Given that the FBI’s information systems are prime targets for criminals, espionage organizations, saboteurs, and terrorists (some possessing significant technical and human resources), the sentiments underlying this view are quite understandable. Nevertheless, the only approach that truly provides zero risk is one that destroys all information as soon as it is collected or that otherwise makes information entirely inaccessible to anyone.

A zero tolerance for security risk is doubly problematic in an environment in which information must be shared. There is a deep and unavoidable tension between information sharing and information security, and while there are mechanisms that allow design architects to mitigate this tension (a matter of implementation discussed below), some tensions are unavoidable, and at some point the senior leadership must decide what degree of security risk is acceptable and under what circumstances in return for the advantages of broad-based information sharing.

For example, it is the committee’s understanding that agents and intelligence analysts are not routinely provided with Internet access for security reasons. It is certainly true that a lack of Internet access will prevent Internet-based security compromises, but it also impedes the use of the Internet to search for potentially valuable publicly available information. (For example, FBI agents assigned to field divisions without Internet access must go to public libraries to search the Internet, and must use various commercial e-mail accounts.) This is a reasonable policy only under the implicit assumption that publicly available information is not useful in either intelligence or investigation—and this assumption is simply not tenable for the counterterrorism mission.

Furthermore, isolating FBI staff from the Internet precludes their being educated by casual exposure to the best new ideas and tools emerging in the global Internet community. While there are individuals with deep expertise, often gained by dedicated people spending their own time and money, and there are specific provisions for controlled access to the Internet, there is no substitute for the institutional and cultural impact of broad availability of Internet access to all FBI staff.

Another important issue with profound consequences for security is raised by the attempt to make a single physical IT infrastructure serve the needs of both intelligence analysts and law enforcement investigators. Much intelligence information is classified, with rigid security requirements imposed by legal and national policy. However, much of the data related to investigations is unclassified (sensitive but unclassified (SBU) or law-enforcement sensitive). Devising technologies and policies to manage both kinds of data under one rubric is thus potentially contradictory, unless one is willing to accept the severe operational penalties of treating unclassified information (SBU or law-enforcement sensitive) as classified.

Intelligence analysts also have a need for relatively unfettered “browsing” through information. However, sensitive information (e.g., identities of informants) can be derived or deduced from aggregation and inference from large amounts of apparently unrelated data.

There are ways to manage these tensions, such as allowing intelligence analysts access to “pointers” to protected internal information rather than the information itself, so that the analyst can make a direct request for access to the party responsible for protecting that information. In the area of unclassified versus classified data, it would help to attempt to balance the threat of misuse or inappropriate access against the benefits of allowing more use, e.g., by

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

remote agent access in the field; this process is known as a risk management assessment. Nevertheless, the tension cannot be eliminated, and only the FBI’s senior operational management is in a position to judge how this tension should be resolved.

A third security tradeoff will manifest itself when mobile computing becomes an issue. The value of mobile computing is likely to be high, but mobile computing is inherently more vulnerable than office-based computing, and the senior operational leadership will have to decide if the increased security risks are worth the added operational flexibility. Empirical data and a formal evaluation process for mobile computing would be useful inputs to the leadership. The FBI’s PDA experiment will help the leadership to think through how to differentiate and measure communications that involve classified information, SBU information, and unclassified information, each of which has different security requirements.

In the absence of any specific information on the subject but based on previous experience, the committee suspects that a significant fraction of, if not most, FBI data communications traffic need not be carried at the classified level. If the measurements associated with a PDA trial support this conjecture, the FBI may be in position to take advantage of wireless data communications in designing an infrastructure, systems, and processes to implement its enterprise architecture. Such systems and policies could differentiate between SBU and unclassified communication, handle them differently, and significantly improve FBI metrics for efficiency and effectiveness. If this conjecture turns out to be true, the FBI would also have to establish a clear policy governing SBU communications.

The committee is also concerned about a number of security issues related to implementation. These include:

  • The use of passwords for authentication. The weaknesses of passwords as an authentication mechanism are well known (e.g., they are easily compromised without the owner’s knowledge),20 and yet no thought seemed to have been given to alternatives.

  • The lack of consistent security models between the IDW and the VCF. In particular, this inconsistency suggested that searches of data contained in the VCF from the data warehouse side would not be made visible to case owners. Moreover, as long as security mechanisms only provide access control, and do not log the information that is accessed, then misuse of systems by authorized users will not be visible until the information turns up in the wrong places. (And, of course, logs must be reviewed regularly by automated log analysis tools supporting human analysts.)

  • The operating system monoculture. The Trilogy IT modernization put into place a single operating system environment, and the security vulnerabilities of an operating system monoculture are well known.21 Such an environment carries with it the risk that a single exploit, such as a worm or virus, can result in global failure of the Trilogy network.22 The technology

20  

See, for example, Department of Defense, Password Management Guidelines, April 12, 1985, available at http://www.radium.ncsc.mil/tpep/library/rainbow/CSC-STD-002-85.html.

21  

National Research Council, Cybersecurity Today and Tomorrow: Pay Now or Pay Later, Computer Science and Telecommunications Board, National Academy Press, Washington, D.C., 2002.

22  

Note also that security risks may be especially great during a time of transition between old and new systems. The reason is that anomalous system behavior can have multiple causes, and may reflect operator inexperience, inherent system bugs, or adversary-planted exploitation. Uncertainty about the actual cause may lead to inaction for a longer period of time than would be the case during normal operation.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

deployment plan relies on isolation from the Internet and diligent network monitoring and response to protect against such problems. But neither of these approaches, taken singly or in combination, can eliminate the risk of catastrophic failure, and recent worm/virus experience has shown that self-propagating, malicious code can enter isolated networks through an unwitting laptop user connecting to the isolated network after having been connected to, and infected through, the Internet. Furthermore, the propagation speed of modern worm/virus code is sufficiently rapid to overwhelm any monitoring and response center staffed by humans.

  • Seemingly inadequate contingency plans for operating under attack. A basic principle of managing a critical operational network is that plans for maintaining operation in the face of a compromised element must be made in advance.

    • How are nodes in the network protected against other nodes that may have been compromised? Rather than a simple perimeter defense, a principle of mutual suspicion should apply that does not allow an attacker who has breached the perimeter to have free access to the entire network.23

    • How will the FBI’s public-key infrastructure be managed in the face of an insider threat?

    • How will key revocation be handled? Emergency re-keying? Access overrides (i.e., how will authorized parties obtain access if access tokens and/or information are lost?)

    • How will worms and viruses be purged from the system if they are introduced?

    • How will the network respond to a denial-of-service (DOS) attack? There are many different flavors of DOS attacks, ranging from a deliberate attack on the inside of the network to an attack occurring on the “outside” of the network that merely consumes bandwidth and degrades the performance of the public links that carry the virtual private networks of the FBI.

  • Missing or incomplete validation of a security architecture, which the committee inferred on the basis of presentations to it. Specifically, the FBI did not provide to the committee evidence that it had validated:

    • The putative design principle for security, which seemed to be characterized by an approach of “build to DISA (Defense Information Systems Agency) gold standard and back off until tools work.”

    • The model of the threat that faces the FBI’s IT systems. (Note that a threat model must include threats emanating from both inside and outside the organization.)

    • The use of the electronic key management system developed by the National Security Agency (NSA) in the context of the specific needs of the FBI.

In each of these cases (the design principle, the threat model, the use of the NSA electronic key management system), the approach may be plausible and ultimately desirable. But a process of systematic validation is necessary before those conclusions can be drawn. More generally, a security architecture is based on an understanding of the data flows in the organization (derived from the enterprise architecture). Based on these data flows, the databases that need to be isolated are identified. When necessary, controlled interfaces between isolated and non-isolated databases are also identified. These interfaces are then mapped onto the physical network configuration to decide what links/nodes must be encrypted, physically protected, or both. Lastly, the security architecture requires both a plan for managing encryption keys and a vulnerability assessment (i.e., a paper attack on the paper architecture).

23  

To illustrate, this principle might call for the use of firewalls on every node of the network and anti-virus protection running continuously on all computers connected to the network to mitigate denial-of-service or intrusion risks.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • Potentially inadequate proactive counterintelligence against inappropriately trusted insiders. As the Robert Hanssen case illustrates, security must be maintained against deliberate exfiltration of sensitive or classified information. Counterintelligence tools for this problem include logging of access, analysis of logs, and possibly filtering of outgoing information. Note also that notifying the owner of a sensitive information item that some other party has accessed that information is a potentially powerful way of raising red flags.

  • Potentially inadequate security of hardware and software implementation. When hardware is deployed and software is written by outside contractors, there are greater risks that hostile parties working for the contractors may seek to insert trapdoors and other ways of bypassing security in the systems being deployed. As a general rule, these risks can be managed, but the weakness of the FBI’s contract management abilities may mean that the FBI’s efforts in this area are not up to par (e.g., as with Defense Department special access projects). The same considerations apply to commercially acquired infrastructure software. Though it might be difficult for an attacker to compromise a component of such software, the possibility cannot be ruled out. Moreover, even security patches and upgrades are not always guaranteed to work properly, which means that installation of such upgrades cannot be undertaken blindly or without explicit thought from IT-savvy managers.

2.2.6 Privacy

As part of the FBI’s briefings to the committee, the FBI’s privacy office addressed the committee, from which the committee concluded that the FBI did have some sensitivity to some of the relevant privacy issues. However, the committee has not undertaken a comprehensive study of protection of individual and corporate privacy in the context of the FBI’s IT modernization program, and thus the comments below can only suggest some of the issues that may be involved.

  • The FBI must proceed with great sensitivity to privacy issues as the counterterrorism mission assumes greater importance, and both substance and perception are important in this domain.

  • There are many substantive issues associated with privacy. For example,

    • The FBI privacy office seemed to the committee to be geared primarily to protecting personally identifiable information from being improperly shared outside the FBI (e.g., with contractors), and in the short presentation to the committee, its approaches to this problem seemed reasonable.

    • Another privacy issue not addressed in briefings but of great concern to the privacy-sensitive segment of the public is the protection of the public from the abuse or improper use of such data by rogue FBI employees acting on their own or through some official though perhaps not publicly acknowledged FBI program. The committee has no comment on this issue, other than noting that the simple exhortation “trust us” is not likely to be reassuring to this segment even if in fact the FBI is doing everything possible to prevent such abuses from happening.24

24  

Note also that Section 223 of the USA PATRIOT Act allows individuals to recover monetary damages and litigation costs from the United States in the event that information or records are willfully and improperly disclosed, a fact that increases the importance of keeping good records of who is accessing what data and under what circumstances. In addition, federal employees found to have improperly disclosed information are also subject to administrative action under the PATRIOT Act.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
  • Perception is at least as important as substance. That is, a technically well-conceived privacy protection program solves the substantive problem. But even with such a program in place, the FBI must also pay attention to external perceptions, because those are a major influence on the public trust that the FBI must maintain in order to be effective.

2.3 ENSURING EFFECTIVE MANAGEMENT OF IT DEVELOPMENT AND IMPLEMENTATION

This section addresses a variety of serious management issues in the FBI’s IT modernization program. Issues and problems identified here relate to the management approaches and processes used by the FBI across the implementation life cycle, including requirements determination, development methodology, contracting and project management, system rollout, training, evaluation, and metrics of effectiveness. While the committee’s comments regarding implementation are derived from its consideration of the Trilogy infrastructure and VCF projects only, the committee believes that dealing systematically with these issues is essential to success in follow-on efforts such as the IDW and SCOPE.

2.3.1 Overall Development Methodology

In the committee’s judgment, the FBI’s management approach to the Trilogy modernization violates a number of basic principles that should govern the development of large systems.

Principle 1: Any organization that depends on the continued operation of its IT systems and is modernizing those systems should plan on the simultaneous operation of both old and new systems for some period of time, so that failures in the new system do not cripple the organization.

Large-scale deployments involving new technology almost always come with problems (e.g., system crashes, unacceptably slow performance), and there is some risk that the problems will be so severe as to prevent the effective use of the entire system for some period of time. Accordingly, there must be backup and contingency plans in place that anticipate a wide range of failure conditions. However, the committee saw little evidence that such plans had been formalized, and indeed has been informed that the current transition plan does not envision the availability of the ACS after the cutover to the VCF, even though the hardware supporting the ACS is neither being redeployed to support the VCF nor being immediately decommissioned.

Effective transitions are usually managed gradually. Even where the transition is abrupt, maintaining the function of the old system for some period as a fallback option is good practice. Any ACS-to-VCF transition without a backup plan is far too risky for the FBI and for the nation, especially in light of the FBI’s myriad and critical operational responsibilities to the nation that will continue during any such transition.

The costs of maintaining a fallback plan and a supporting infrastructure are small compared to the operational costs associated with large-scale VCF problems. Moreover, because the hardware supporting the ACS will not be converted to support the VCF, there is a residual and pragmatic disaster-recovery capability to revert to the ACS. But contingency plans that anticipate a continuing if temporary need for the ACS must be made before the transition to exploit this residual capability, or else the use of the ACS in this contingency mode will suffer.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

Expenditures to maintain an infrastructure that can support concurrent use of the ACS and the VCF for a limited transition period would be a worthwhile insurance policy.

Principle 2: Any organization undertaking a large-scale IT system development and deployment, but especially an organization without a strong track record in IT development and use, should develop, test, evaluate, and iterate a small-scale prototype before committing itself to an organization-wide program.

The development methodology for Trilogy seems to be based on a one-way non-iterative process where rigid specifications are generated in advance. Subsequent changes can then be argued to be “specification changes” that open the door for schedule delay and increased expense. In truth, it is essentially impossible even for the most operationally experienced applications developers to be able to anticipate in detail all of the requirements and specifications in advance. Therefore, development contracts should not make such an assumption, but rather should call for an approach to specification of user requirements that is based on a process of extensive prototyping and usability testing with real users.25

To the best of the committee’s knowledge, apart from SCOPE, no prototype has been developed for any of the major components of Trilogy (the Trilogy network or the VCF) or for the IDW. As the FBI recognized in developing SCOPE, the advantage of a small-scale prototype is that it can be iteratively developed with strong user feedback and involvement, thus increasing the chances that what is ultimately delivered to the end users meets their needs. This point is relevant to many dimensions of system development, including the functionality desired in a new application, the convenience and intuitiveness of a user interface, and the nature and mix of the operating load that the system must support under real operational conditions.

In some instances, the intimate involvement of project managers with extensive operational experience can play an important role in some of these dimensions (e.g., defining the required functionality); indeed, the committee believes that it is the involvement of such an individual in the development of the VCF that has been responsible for what success it is likely to have when it is deployed. Nevertheless, in the end there is no substitute for realism in the evolution of a prototype.

Principle 3: In large-scale system development and deployment, testing should account for as much as 35 to 50 percent of the project schedule if a successful deployment is to be achieved.

Testing is necessary to shake out bugs and flaws in a new system, and flaws will often be invisible until after actual users deal with actual data. The 35 to 50 percent figure above is an experience-based rule of thumb that is widely accepted among those involved with large-scale system design. Moreover, an organization without a strong track record in IT could reasonably be expected to require even more time for system testing and integration. Testing is a

25  

These comments do not mean that prototyping is a panacea that guarantees success. While prototyping usually has high value in developing requirements for user functionality, it produces results that are only as valid as the user group selected for involvement. Furthermore, obtaining realistic results from prototyping exercises requires using real-world data. Finally, prototyping is most valuable when understanding user requirements is the task at hand; it is less useful for determining reliability and response times for applications that will be deployed on a large scale.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

critical dimension of system development and deployment requiring a well-developed testing strategy and a progression from unit testing to full-blown integration testing, and usually calls for dress rehearsals for system use.

The project schedule for completion of Trilogy as it was represented to the committee in October 2003 appears to leave inadequate time for testing. The committee was shown briefing charts indicating that the FBI allocated less than 10 percent of its schedule for testing, and under schedule pressure the contractor was trimming that amount even further. It was also surprising for the committee not to see, at a very late stage in VCF development, any kind of risk-based prioritization for the items remaining to be done. Instead, these items were lumped together in the category “unfixed bugs.”

To provide a simple example of the need for testing, it is important to realize that the term “works” (as in the system “works”) has a wide spectrum of possible meaning. Consider a simple system, consisting of a network transport layer that moves packets and is responsible for encryption and decryption and an application running on top of it. Beginning with the least demanding definition, the expression “the system works” can plausibly mean that:

  1. The transport layer “lights up” but does not move application packets. Nodes can find each other, basic protocol functions such as “ping” work, and bits flow between a subset of the network and on the whole network.

  2. Definition (1), and in addition data packets move between a subset of the network and on the whole network.

  3. Definition (2), and in addition encryption and decryption are successful between nodes on a subset of the network and on the whole network.

  4. Definition (3), and in addition the transport layer can move application packets under simulated “realistic” load between nodes on a subset of the network and on the whole network.

  5. Definition (4), and in addition users in different locations can use the application to pass the appropriate information among each other.

  6. Definition (5), and in addition the system functions with actual users using actual data.

  7. Definition (6) with realistic numbers of users and the full complement of files.

  8. Definition (7) with some reasonable assurance of the overall security of the applications, as opposed to the encrypted transport of bits alone.

In the absence of a clearly specified test plan that is acceptable to all users, it is possible to make a claim that the system “works” according to any of these definitions, even though the point of failure in the development of many systems is encountered when they are required to continue to function under load. It is unclear for Trilogy which definition of “working” is being used by those claiming that a system is working. In the case of the Trilogy infrastructure network, the VCF application is not yet available, and so it is clear that the higher levels of functionality (definition (5) and above) have not yet been attained.

2.3.2 Contracting and Contract Management

Both key contracts for Trilogy (Trilogy infrastructure and the VCF) were awarded on a cost plus/cost reimbursable basis. Only optional “tools” were awarded on a fixed-price basis. While the task orders (T0001AJM026 and T001AJM028) for both phases of the Trilogy program

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

detail the firm fixed price to eight or nine significant figures, the task order schedules were almost totally lacking in specifications and a commitment to checkpoints. For example, the documents list a large number of plans to be created, with dates “TBD.” (The committee believes that one reason underlying the lack of detail in these critical documents is that the underlying architectural work has not been done. In the absence of such architectural work, system specifications are hard to develop with any coherence.) Under these circumstances, effective program and contract management becomes essentially impossible. This weakness, aggravated by turnover among key FBI staff (e.g., the FBI chief information officer), makes it unsurprising that Trilogy is significantly behind schedule and over budget.

To illustrate some of the problems with contract management, the FBI told the committee that the Trilogy network had been made operational on March 28, 2003, only because FBI personnel in the program office were pressed into overtime service to compensate when contractors failed to meet commitments. While such efforts point to the FBI’s admirable dedication to duty, the need for the program office to stand in for a contractor is a sign of contractor failure. If these “above and beyond” efforts of FBI staff could be converted into penalties for (or dismissals of) the contractors, it would send a very positive signal that the FBI program office is becoming serious about contract management.

For a contract of this magnitude and importance to the FBI and to the United States, it is imperative that senior management of the FBI monitor contractor progress closely and step in when necessary to forestall difficulties seen down the road, although day-to-day involvement is not necessary. Senior-level contract managers, experienced and empowered, should be assigned for the duration of the project, and should provide periodic actionable status information to FBI senior management. In briefings, it appeared to the committee that contractors may be viewing this project as governmental “business as usual,” without due regard to the critical importance and congressional visibility of Trilogy’s success or failure. Clear and detailed schedules with intermediate milestones, earned-value metrics, and severe penalties for missed delivery dates and missing functionality are desperately needed.

2.3.3 Program Management

In the committee’s judgment, the FBI’s program management of its IT projects is weak, and has been weak for over a year (based on the individual observations of many of the committee members who participated in the September 2002 meeting). Weak program management leads to a reactive mode in dealing with issues and a lack of overall project control. Continued cost overruns and delays often result from a lack of effective program management. By contrast, effective program management with effective management of costs and proactively managed IT projects significantly increases the likelihood of success.

For the committee, one of the most serious issues is that many essential tasks have been outsourced to vendors (e.g., development of data models and architectures). In essence, the FBI has left the task of defining and identifying its essential operational processes and its IT concept of operations to outsiders. This is not to say that outside contractors should have no role at all in these tasks, but rather that it should be the FBI, not the contractors, that defines and drives the process.

A small but telling example of the FBI’s dependence on contractors is that the FBI reported no contract provisions calling for the escrow of source code for the applications. Escrow calls for the deposit of source code files and appropriate documentation with a mutually trusted

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

third party during and after the completion of the contract. In the event that the vendor becomes unable or unwilling to continue to provide service to the FBI (e.g., the developer must be replaced because of poor performance, or the developer goes bankrupt), the source code is released to the FBI so that it can seek another vendor to assume the first vendor’s responsibilities. Source code escrow is a common, even routine, commercial practice.

Program management is a well-understood discipline and set of processes. A strong program management function must be established and supported by FBI management at all levels to provide appropriate oversight and management of FBI IT projects. An effective program management function will provide the FBI with a focal point for monitoring and collecting project data and allow for the reporting of the progress of active IT projects based on well-defined metrics. (Note that program management entails a set of skills and background different from those associated with operational experience in doing investigation or intelligence analysis. That is, a single individual may have strong program management skills and extensive operational experience, and arguably a “program manager” ought to have both qualities, but there is no a priori reason to expect that someone with the former will necessarily have the latter. In any event, a program manager should at least have access to both sets of expertise.)

For program managers to be effective, they must participate deeply in the negotiation of contracts with outside vendors. The contracts must include performance measures for key deliverables, milestones, and service levels with penalties and escalation procedures. Contracts should ensure that the vendor suffers severe penalties if it is not meeting the performance measures, and major vendor failure should result in a penalty that allows the FBI to transition to a new vendor with little or no financial impact to the FBI.

Program managers are responsible for validating, monitoring, supporting, and assisting in the area of project and life-cycle costs. They may also supply information that can alter project priorities, based on resource availability or interdependent project conflicts. Most important, program managers are responsible for implementing an appropriate methodology for project management, including at least the following:

  • Establishing a framework for all project communications, reporting, procedural, and contractual activity;

  • Developing task-specific project plans with timelines, critical paths, and specific deliverables;

  • Conducting project team meetings, setting schedules and agendas, and managing the meetings;

  • Monitoring milestones and deliverables and ensuring tracking to timelines;

  • Providing project status updates to key stakeholders;

  • Identifying deviations from plan and addressing issues on a timely basis for resolution;

  • Managing project budgets; and

  • Creating an environment where the project team can succeed.

An important point is that effective program management generally requires close and frequent interactions between managers and the project team and among team members. Besides well-trained staff, the team also needs an environment that facilitates working effectively, such as proximity of offices, meeting spaces, and areas in which information can be passed informally over lunch or in a chance hallway encounter rather than relying only on

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

formal channels of communication such as e-mail, phone calls, voice mail, teleconferencing, and meetings of high-level managers.26

A particularly useful management tool is a project charter. The charter includes detailed project requirements, resources, and the related roles of the project team, along with identified milestones and deliverables. Risk, issue and scope management, and communication plans are included.

The project charter is designed to do several important things. The charter should:

  • Facilitate communication among stakeholders.

  • Document approved purpose, scope, objectives, cost, and schedule baselines.

  • Document the agreement between groups.

  • Define roles and responsibilities inclusive of the overall project governance.

  • Document planning assumptions and decisions.

  • Provide the baseline for scope and expectation management.

In effect, the charter establishes a baseline of understanding among all stakeholders and especially between the project manager and the project sponsor(s), business owner(s), and vendor(s). The success of the project may be measured against this charter. Just as importantly, any changes must reference the charter.

2.4 ENSURING THE GROWTH OF FBI IT EXPERTISE AND DEALING WITH EXTERNAL FACTORS

The FBI recognizes, and the committee agrees, that the FBI is severely lacking in many of the technical and management skills to successfully plan, contract for, and implement a project of the breadth and scope of Trilogy. This situation must be remedied both to complete the initial phases of Trilogy, and more important, to deal with the issues described in the preceding sections. In addition to filling the top jobs such as the CIO with people who are willing to and capable of effectively managing the larger issues, a number of other gaps and constraints, some not of the FBI’s making, must be addressed.

2.4.1 Human Resources

With a few exceptions, the presentations to the committee persuaded it that the FBI lacks a human resource base adequate to deal with its IT modernization program. The FBI lacks experienced IT program managers and contract managers, which has made it unable to deal aggressively or effectively with its contractors. Inexperienced managers generally lack the ability to assume proactive management roles and are often held hostage to the perspectives of the contractor.

Given the importance of IT personnel and analysts to the FBI’s broadening missions, such individuals must have career track opportunities that have status and respect that are compa-

26  

Although many large organizations are able to effectively manage large-scale efforts with geographically dispersed staff, success in this regard requires a considerable degree of IT sophistication and adherence to well-defined and well-understood IT policies and practices.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

rable to those of traditional personnel tracks (such as agents) within the bureau.27 Such tracks would also enable stability in IT jobs that require extended oversight and long institutional memory.

On the positive side, the committee wishes to single out as especially noteworthy the presentations it heard from the head of security for the IT modernization program, the executive assistant director for intelligence, and the VCF project manager. These presentations reflected serious thought about their respective responsibilities from the user perspective. For example, the executive assistant director for intelligence presented a coherent and well-articulated concept of operations, as well as a clear vision of how to draw talent from the rest of the FBI. The presentations of the head of security, though not long enough to provide very much detail, were well-considered. And, the VCF project manager’s operational experience has been invaluable in keeping the VCF intellectually on track.

Although the committee met only a few of them, it believes that the FBI has an important IT resource in its younger agents. The FBI agents now being hired are in their late 20’s, and, as with others in their peer group, have been heavily exposed to modern forms of information technology for most of their lives. Given this fluency with information technology, it is likely that these newest agents will be the most enthusiastic about embracing new technologies to enhance their effectiveness. (Of course, the flip side is that if the organization in which they serve is unable to provide technology tools that they believe they ought to have and perceive to be adequate, they are likely to be disappointed in the organization, with all of the consequences regarding morale, work attitude, and retention.)

Finally, the committee notes that the FBI has the authority to obtain IT personnel from the private sector.28 Under the Intergovernmental Personnel Act (IPA), the FBI can borrow personnel from other government agencies (federal, state, and local), from institutions of higher education, and from federally funded research and development centers. Generally, these arrangements call for the “home” agency or institution to pay the salary of the employee while on detail status, though the FBI could agree to reimburse the home agency as part of the terms of the cooperative agreement. IPA details cannot exceed 4 years in duration.

Under the Information Technology Exchange Program (ITEP), the FBI can enter into a cooperative agreement with a private sector organization for the exchange of personnel who work in the field of IT management, are considered exceptional performers, and are expected to assume increased IT management responsibilities in the future. While on detail to the FBI, the private sector organization employee may continue to receive pay and benefits from his/ her employer, and the assignment may be made with or without reimbursement by the FBI for the pay, or a part thereof, during the assignment and for any contribution to the private sector organization’s employee benefit systems. ITEP details cannot exceed 2 years in duration.

The FBI also has the authority to hire IT employees outright. Senior IT positions in the FBI fall under the FBI’s Senior Executive Service (SES) pay scale, the range of which is $103,700 to $157,000. In exceptional cases, the FBI can pay up to $174,500, though this step requires Office of Personnel Management and Office of Management and Budget approval. The FBI is also authorized to pay a lump-sum recruitment bonus of up to 25 percent of the annual rate of basic

27  

Note that the problem of enhancing the status of workers in a critical but supporting field is common in many organizations, both inside and outside the federal government.

28  

Information on FBI hiring authorities is derived from a memo from the FBI to the committee transmitted on February 11, 2004.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×

pay to an employee, including individuals covered by the SES, who are newly appointed to a difficult-to-fill position.

The FBI also has available positions classified as Scientific or Professional positions that are above the GS-15 level but do not meet the SES functional criteria. These positions encompass the performance of high-level research and development in the physical, biological, medical, or engineering sciences, or a closely related field. The pay scale for such positions in the Washington, D.C., area ranges from $117,627 to $144,600.

In the judgment of the committee, these rates of pay should be sufficient to attract highly qualified IT talent. While it is true that highly qualified senior IT personnel in the private sector can command significantly higher salaries, stock options/bonuses, and the like, the FBI also offers opportunities for public service that compensate, in part, for the lower salary scales. These opportunities include the chance to have a substantial impact in service to the nation in an area of extraordinary import. The recent shift in the opportunity environment for IT people has no doubt increased the relative attractiveness of government service.

2.4.2 External Constraints

The FBI also operates under a variety of constraints that originate externally to the bureau. Two of the most significant are the following:

  • Lack of management flexibility. Agile organizations are managerially flexible so that they can take prompt action. It is the committee’s understanding that the FBI is unable to take managerial actions such as reprogramming amounts in excess of $500,000 without explicit congressional approval. This constraint and others with a similar micromanagement flavor are inconsistent with the expectation that the FBI will move quickly and forcefully to reshape itself to deal effectively with the terrorism challenge.

  • Audit pressure. The FBI reported that it had undergone more than 100 investigations and audits in the IT area. While it is true that the FBI’s performance in this area is seriously deficient (and this makes the FBI an attractive target), responding to such investigations uses the time of senior management, both in preparation and presentation of material to those investigators. And because personnel in the IT field are scarce within the FBI, the pace of work is slowed by such investigations. A better use of an equivalent amount of talent and energy would be to assist the FBI in dealing with its problems. Nevertheless, it remains the case that some audits, such as the one prepared by the General Accounting Office as The FBI Needs an Enterprise Architecture to Guide Its Modernization Activities of September 25, 2003 (released to the FBI August 22, 2003), contain valuable guidance.

A third area of concern is the presence of inflexible and stringent rules for personnel qualification, though the committee is not certain whether these rules originate within the FBI or outside it. Stringent rules reduce the pool from which potential employees are drawn, and inflexible rules—even undertaken in the name of upholding standards—may inappropriately disqualify otherwise qualified individuals. Of course, job applicants whose history indicates a substantial variance from the standards for employment are unacceptable. But a rule of reason ought to apply, and an otherwise-qualified applicant whose history is only minimally at variance with those standards should not be automatically excluded.

Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 16
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 17
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 18
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 19
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 20
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 21
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 22
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 23
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 24
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 25
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 26
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 27
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 28
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 29
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 30
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 31
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 32
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 33
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 34
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 35
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 36
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 37
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 38
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 39
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 40
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 41
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 42
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 43
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 44
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 45
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 46
Suggested Citation:"2 IT-Related Issues for the FBI Requiring Immediate Action." National Research Council. 2004. A Review of the FBI's Trilogy Information Technology Modernization Program. Washington, DC: The National Academies Press. doi: 10.17226/10991.
×
Page 47
Next: 3 Recommendations »
A Review of the FBI's Trilogy Information Technology Modernization Program Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Federal Bureau of Investigation (FBI) is in the process of developing a modern information technology (IT) system—the Trilogy program— that is designed to provide a high-speed network, modern workstations and software, and an application—the Virtual Case File (VCF)—to enhance the ability of agents to organize, access, and analyze information. Implementation of this system has encountered substantial difficulties, however, and has been the subject of much investigation and congressional concern. To help address these problems, the FBI asked the National Research Council (NRC) to undertake a quick review of the program and the progress that has been made to date. This report presents that review. The current status of four major aspects of the program—the enterprise architecture, system design, program management, and human resources—are discussed, and recommendations are presented to address the problems.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!