Executive Summary

THE MAIN MESSAGE

Although the Federal Bureau of Investigation (FBI) has made significant progress in its information technology (IT) modernization program in the last year or so, the committee believes that the FBI’s IT modernization program is not currently on a path to success. To get on that path, the committee recommends several key changes.

First, foremost, and most critical in light of the impending rollout of the Virtual Case File (VCF) application, the FBI should not proceed with deployment of the VCF until it has a validated contingency plan for reverting completely or partially to the Automated Case Support (ACS) system, if necessary, together with clear and measurable criteria to determine when the ACS can safely be turned off. In the absence of a validated contingency plan, the FBI runs a very high risk that its planned “flash cutover” from the old ACS system to the VCF will cause mission-disruptive failures and further delays. This issue is a consequence of the fact that the VCF has been developed without the benefit of prototyping, with the result that the VCF application will not have been tested in an operational context.

Second, the success of the FBI’s information technology efforts will require the development of a close linkage between IT and a coherent view of the bureau’s mission and operational needs. The development of this strategic linkage—the enterprise architecture—cannot be delegated inside the bureau to the chief information officer (CIO) or outside to contractors. Only the senior leadership of the FBI can establish the policies, define the operational frameworks and priorities, and make the tradeoffs that are necessary to formulate this strategic view. To do so, they must be deeply and directly involved in its creation.

Third, because testing is such a critical dimension of system development and deployment, the FBI must allow adequate time for testing before any IT application (including the VCF) is deployed, even if dates of initial operational capability are delayed. Testing must include a full systems integration test and adequate scale, volume, and stress tests.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program Executive Summary THE MAIN MESSAGE Although the Federal Bureau of Investigation (FBI) has made significant progress in its information technology (IT) modernization program in the last year or so, the committee believes that the FBI’s IT modernization program is not currently on a path to success. To get on that path, the committee recommends several key changes. First, foremost, and most critical in light of the impending rollout of the Virtual Case File (VCF) application, the FBI should not proceed with deployment of the VCF until it has a validated contingency plan for reverting completely or partially to the Automated Case Support (ACS) system, if necessary, together with clear and measurable criteria to determine when the ACS can safely be turned off. In the absence of a validated contingency plan, the FBI runs a very high risk that its planned “flash cutover” from the old ACS system to the VCF will cause mission-disruptive failures and further delays. This issue is a consequence of the fact that the VCF has been developed without the benefit of prototyping, with the result that the VCF application will not have been tested in an operational context. Second, the success of the FBI’s information technology efforts will require the development of a close linkage between IT and a coherent view of the bureau’s mission and operational needs. The development of this strategic linkage—the enterprise architecture—cannot be delegated inside the bureau to the chief information officer (CIO) or outside to contractors. Only the senior leadership of the FBI can establish the policies, define the operational frameworks and priorities, and make the tradeoffs that are necessary to formulate this strategic view. To do so, they must be deeply and directly involved in its creation. Third, because testing is such a critical dimension of system development and deployment, the FBI must allow adequate time for testing before any IT application (including the VCF) is deployed, even if dates of initial operational capability are delayed. Testing must include a full systems integration test and adequate scale, volume, and stress tests.

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program Fourth, the FBI’s contract management process is inadequate, and contract schedules lack the specificity necessary to determine whether a project is making adequate progress within schedule and budget constraints. This weakness can be remedied through the aggressive use of standard contract and project management tools. Finally, while the FBI’s IT team includes a number of very capable individuals, the overall human resource base for IT is not nearly adequate to meet the challenges it faces. For Trilogy and subsequent IT projects to have access to the human talent they need to succeed, the FBI must dramatically grow its own internal expertise in IT and IT contract management as quickly as possible. BACKGROUND The FBI’s Trilogy IT modernization program is intended to upgrade the IT infrastructure of the FBI by providing a high-speed network linking the offices of the FBI, modern workstations and software within each office for every FBI employee, and a user application known as the Virtual Case File to enhance the ability of agents to organize, access, and analyze information. However, the Trilogy program’s development and implementation have been troubled. The Trilogy program has been the subject of a number of General Accounting Office (GAO) and Department of Justice (DOJ) inspector general investigations, as well as a source of considerable concern to the U.S. Congress. In July 2003, the FBI requested the assistance of the National Research Council (NRC) to review the Trilogy program and the progress that had been made, and further to consider other nascent IT efforts to support the bureau’s new priorities in counterterrorism. In response to this request, the NRC convened the Committee on the FBI’s Trilogy Information Technology Modernization Program, consisting of experts with considerable experience in large-scale IT deployments. The committee met twice in 2-day sessions to receive briefings from the FBI about Trilogy and other related matters, and except as explicitly noted otherwise, those briefings constitute the factual base for this effort. THE SITUATION TODAY In the wake of the events of September 11, 2001, the FBI is undergoing a significant expansion of its mission responsibilities and a reordering of its priorities to emphasize its counter-terrorist mission, though it still retains its very important criminal investigation mission. The FBI recognizes very well that it will become ever more dependent on information technology in the future to manage the large quantities of information associated with these missions. It is challenging for any organization engaged in a complex set of activities to introduce new technologies and to reengineer its key processes to exploit them effectively. It is doubly challenging, as it is for the FBI, to do so when under intense operational pressures—the FBI’s traditional work must continue while new technology is introduced and while a culture more adapted to the use of IT is evolved. And it is triply so for the FBI in the face of the added strain of its new focus—preventive counterterrorism—in which mission success demands a different mind-set, different operational skills, and the exploitation of an expanded set of information sources. The FBI has made significant progress in certain areas of its IT modernization program in the last year or so. For example, it has achieved the modernization of the computing hardware

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program and baseline software on the desktops of agents and other personnel, and has taken major strides forward in the deployment of its networking infrastructure. Nevertheless, as this report documents, the committee believes that the FBI’s IT modernization program is not currently on a path to success. The committee’s review of the approach and methodology being used by the FBI has identified significant issues in four major areas: enterprise architecture, system design, program and contract management, and human resources. ISSUES Enterprise Architecture As in any organization, private sector or government, the operational needs of the FBI must be the driver of its information technology investments. If it is to be successful in its efforts to exploit IT, the FBI must first and as a matter of its highest priority in its IT efforts formulate an enterprise architecture. Such an architecture is necessary to provide a strategic view of its mission and operational needs, and would begin with a detailed characterization of the bureau’s goals, tasks, strategies, and key operational processes. This view links operational objectives and processes to IT strategy and will allow the FBI to specify how investment is tied to the achievement of operational objectives. Based on presentations to the committee by the FBI (as well as a review of certain documents produced by the FBI, GAO, and DOJ), the committee has concluded that the FBI’s efforts and results in the area of enterprise architecture are late and limited, and fall far short of what is required. The committee was encouraged by early efforts driven by the recently appointed executive assistant director of intelligence to develop a concept of operations of the intelligence process from which appropriate IT systems support can be architected and designed. However and overall, the FBI’s senior leadership is insufficiently engaged in the development of the enterprise architecture, with the result that this development task is delegated in large part to outside contractors and to a CIO. Though these parties are essential players, only the senior leadership of the FBI can establish the key policies, set the operational priorities, and make the significant tradeoffs that must be reflected in the complete enterprise architecture and IT system design. Among the most important decisions to be made are the risk tradeoffs involved in ensuring sufficiently broad controlled access to sensitive information. Such decisions must be made at the level of the senior leadership. System Design Although the committee recognizes that the bureau has made significant progress on the Virtual Case File in the last year or so, it has concerns about the VCF and the Integrated Data Warehouse. The VCF has many positive attributes. Based on a canned demonstration of a VCF mock-up, the committee believes that the VCF should significantly enhance the information management capabilities of FBI agents in their investigative role. However, the bureau-wide rollout of this application is months delayed from its originally scheduled deployment in December 2003. Going forward, the committee has a number of concerns about the VCF. First, the FBI described to the committee a plan for a “flash cutover” from the old Automated Case Support system to the VCF, rather than a limited initial rollout that would shake

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program out problems in an operational context. The committee’s concerns in this area are heightened by the fact that in the interests of rapid deployment, the current VCF schedule appears to give little consideration to testing and presumes success at every stage—a highly risky approach. The current choice facing the FBI on this matter of scheduling is between (a) delaying VCF deployment so that adequate testing can be completed, and (b) forcing operational users to do the testing themselves after implementation, with all of the potential negative consequences that such an approach can produce. The current plan is likely to result in (b). With limited testing, and no experience gained from a limited initial rollout, the FBI would be implementing what amounts to a prototype throughout the bureau. This approach is nearly guaranteed to cause mission-critical failures and further delays, with implications for training, performance, coherence, internal morale, public image, and cost to recovery. Second, the VCF was designed to support the investigative mandate of the FBI. The design process was well under way prior to the expansion of the intelligence mission, and the requirements for the processes supporting the intelligence mission were not included in the VCF design. For this reason, and because of the significant differences in IT requirements between systems supporting investigation and those supporting intelligence, the committee strongly recommends that the FBI refrain from using the VCF as the foundation on which to build its analytical and data management capabilities for the intelligence processes supporting the counterterrorism mission. Rather, the FBI should conceptualize an architecture for the counterterrorism mission from scratch, and then design explicit interfaces to the VCF when information must flow between them. Another application, still in the design stage, the Integrated Data Warehouse (IDW), also seems to suffer from a lack of deep consideration of how and what sources of data are used by different operational elements and in different processes of the FBI. For example, presentations to the committee suggested a mismatch between the expectation that intelligence analysts would have access to live databases containing the most current information and the reality of what the IDW as designed would actually provide. That is, the IDW would provide only the latest copies of production databases, replacing old copies of data with newer copies. Thus, data could be there one day and not the next, since the IDW apparently was not designed to retain older or historical data. While having only the most recent copy of data may be appropriate for the purposes of an investigation (presuming the most recent copy is the most accurate and reliable), this process may not serve intelligence purposes very well. Program and Contract Management The committee has serious concerns about the approaches and processes used by the FBI to develop and field both IT infrastructure and applications. In the committee’s view, a major weakness is that the FBI does not appear to employ user-vetted prototypes in its applications development process. In practice, it is essentially impossible for even the most operationally experienced IT applications developers to be able to anticipate in detail and in advance all of the requirements and specifications. Therefore, internal development plans, and the development contracts with supporting organizations, should call for an approach that is based on a process of extensive prototyping and usability testing with real users. Doing so allows iterative development with strong user feedback and involvement, thus increasing the chances that what is ultimately delivered to the end users meets their needs. This point is relevant to many dimensions of system development, includ-

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program ing the functionality desired in a new application, the convenience and intuitiveness of a user interface, and the nature, scale, and mix of the data entry, management, and retrieval load that the networks and systems must support under real operational conditions. The committee believes that both contract management and program management need substantial improvement. For example, while task orders viewed by the committee detailed pricing to eight or nine significant figures, the corresponding contract schedules were almost totally lacking in specifications, deliverables, and commitment to checkpoints. Under these circumstances, effective program and contract management is essentially impossible. Current contracting and management problems, aggravated by frequent turnover among key FBI staff, make it unsurprising that Trilogy is significantly behind schedule and over budget. Furthermore, the FBI appears overly dependent on outside contractors to undertake essential tasks, such as identifying key operational processes, defining the FBI’s IT concept of operations, and making decisions about the major tradeoffs that are inevitably required. While outside contractors play important roles, it is the senior FBI management who must lead in assuming responsibility for these tasks. Human Resources and External Constraints Although the committee did not undertake a comprehensive assessment in the human resources area, presentations to the committee persuaded it that with a few exceptions, the FBI lacks a human resource and skill base adequate to deal with the bureau’s IT modernization program. Specifically, the FBI is extremely short on experienced program managers and contract managers and senior IT management team members with good communications skills. At the same time, the FBI appears to have the authority to hire highly qualified IT personnel without requiring them to make excessive financial sacrifices, and to borrow personnel from other agencies and even from the private sector. The committee is encouraged to learn that an acting chief information officer was put into place at the beginning of 2004. Of lesser concern, but in the committee’s view still worth noting, is the fact that the FBI also operates under a number of external constraints that diminish its management flexibility. For example, it is the committee’s understanding that the FBI is unable to take actions such as reprogramming amounts in excess of $500,000 without explicit congressional approval. This constraint is inconsistent with the expectation that the FBI will move quickly and forcefully to reshape itself to deal effectively with new challenges. RECOMMENDATIONS The first and most urgent recommendation, indeed critical in light of the impending VCF system rollout, is that the FBI not proceed with deployment of the VCF until it has a validated contingency plan for reverting completely or partially to the ACS, if necessary, and clear and measurable criteria to determine when the ACS can safely be turned off. Beyond this critical recommendation, the committee makes a number of recommendations, grouped into four areas, that will significantly increase the likelihood of success in and drive an accelerated pace for the FBI’s IT modernization efforts. The most important of these recommendations are described below, and they are, in the committee’s judgment, imperatives for the success of the FBI’s IT modernization program.

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program In the area of enterprise architecture, the development of a complete enterprise architecture is central to the FBI’s IT efforts. The most important recommendations in this area (others are presented in the main text) are the following: If the FBI’s IT modernization program is to succeed, the FBI’s top leadership, including the director, must make the creation and communication of a complete enterprise architecture a top priority. This means that they must be personally involved and invested in the key decisions that the process will require be made, such as the tradeoffs between the security of and access to information in the various data sources that are used in criminal investigation and counterterrorism efforts. While a contractor might well assist the FBI in developing the enterprise architecture, no contractor will fully understand the operational issues that must be reflected in the enterprise architecture, nor be empowered to make decisions about how to make the tradeoffs with competing concerns. A small team, consisting primarily of senior operational managers from the Criminal Investigation Division, the Office of Intelligence, and the Counterterrorism Division, and a senior IT executive to translate what these managers say into architectural terms, should be able to develop the broad outlines of the operational aspects of the enterprise architecture as well as a top-level schematic view of the systems design in a matter of 4 to 6 months of full-time work. To decide on the many operational and policy tradeoffs that will inevitably arise, this team must have direct access to and the frequent involvement of the most senior management of the FBI, including the director and the deputy director. The FBI should seek independent and regular review of its enterprise architecture as it develops by an external panel of experts with experience in both operations and technology/ architecture. When the first draft of the enterprise architecture has been prepared, it should be reviewed by an external panel of independent experts charged with helping the FBI to improve how it uses IT in the long term. Given that the counterterrorism mission requires extensive information sharing, the FBI should seek input on and comment from other intelligence agencies regarding its enterprise architecture effort. The reason is that the FBI’s information systems must have interfaces to those agencies to ensure that the information resources of those agencies are appropriately linked to FBI systems so that those agencies are able to work collaboratively. In the area of system design, the most important recommendations (others are presented in the main text) are the following: The FBI should refrain from initiating, developing, or deploying any IT application other than the VCF until a complete enterprise architecture is in place. The FBI should develop a process map for information sharing that clearly defines the current state of and a desired end state for the information-sharing process so that the numerous information-sharing initiatives can be coordinated and properly monitored and managed. The FBI should immediately develop plans that address recovery of data and functionality in the event that essential technology services come under denial-of-service attacks (e.g., from viruses and pervasively replicated software bugs). In the area of program and contract management, the most important recommendations (others are presented in the main text) are the following:

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program Because testing is such a critical dimension of system development and deployment, the FBI must allow adequate time for testing before any IT application (including the VCF) is deployed, even if dates of initial operational capability are delayed. Testing must include a full systems integration test and adequate scale, volume, and stress tests. In future IT applications development, particularly of large-scale end-user-oriented applications, procurement contracts should be conditioned on the development of small-scale prototypes that can be built rapidly and tested with user feedback before committing to large-scale development. For IT applications beyond the VCF, the FBI should exploit proven methodologies of contracting and contract management, including the use of detailed functional specifications, specific milestones, frequent contract reviews, and earned-value metrics. In the area of human resources, the most important recommendations (others are presented in the main text) are the following: For Trilogy and subsequent IT projects to have access to the human talent they need to succeed, the FBI must dramatically grow its own internal expertise in IT and IT contract management as quickly as possible. In the short term, this effort will almost certainly involve borrowing experienced and capable contract managers from other agencies. In the long run, establishing its own internal IT expertise will involve the creation of long-term high-status career tracks with the FBI for IT personnel. Because of their importance to the short- and long-term success of the bureau’s IT modernization efforts, the FBI must permanently fill the positions of chief information officer and chief enterprise architect, and the committee concurs with the director’s judgment that filling these positions with appropriately qualified individuals ought to have the highest priority. The FBI should develop an improved system for internally reviewing the state of progress in key IT programs and for communicating relevant findings to key stakeholders, thus preempting the perceived need for and distraction of constant external investigations. The committee believes that the FBI has made significant progress in some areas of its IT modernization efforts, such as the modernization of the computing hardware and baseline software and the deployment of its networking infrastructure. However, because the FBI IT infrastructure was so inadequate in the past, there is still an enormous gap between the FBI’s current IT capabilities and the capabilities that are urgently needed. Some useful and valuable returns from the investment in the Trilogy program appear to be within reach. Nevertheless, the committee believes that a major effort is needed to bring the FBI to the state where it can be characterized as an effective exploiter of information technology. The committee has made recommendations that, if adopted, will significantly increase the likelihood that the FBI’s Trilogy IT modernization program will enhance the FBI’s effectiveness in carrying out its crime-fighting and counterterrorism missions. But it emphasizes the difference between a pro forma adoption of these recommendations and an adoption of these recommendations that is both fully embraced throughout the agency and aggressively executed. The former may be the metric that auditing and oversight agencies and offices often use in assessing agency performance, but it is the attitude and willingness of senior staff to act that really count. The senior management of the FBI has a substantive and direct role to play

OCR for page 1
A Review of the FBI’s Trilogy Information Technology Modernization Program in the FBI’s IT modernization efforts. This role either has not been understood or it has been given a lower priority based on the perception of more immediate operational priorities. Given the importance of IT to the FBI’s future success in carrying out its missions, the FBI’s senior management must concern itself as much with developing a coherent vision for using IT as with budgets, training programs, equipment, and organization. As the complexities of the FBI’s evolving role are understood, the committee believes that investment by the FBI’s senior management team in the IT process will yield major enhancements to mission achievement as well as substantial operational efficiencies.