Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 32
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions 3 Lessons Learned and Best Practices As discussed in prior chapters, interagency or multiagency collaboration may occur under a variety of arrangements and govern a wide range of engineering, technology, and acquisition elements in mission development and subsequent operations. Recall that, for the purposes of this report, the term “interagency” refers to multiple agencies of the U.S. federal government and the term “collaboration” is an overarching term that refers to more than one agency working together. REASONS FOR INTERAGENCY COOPERATION The most interdependent and highest-risk form of collaboration, interagency cooperation, occurs when multiple agencies are on the critical path to mission success. Such cooperation can result when multiple agencies are involved in funding or providing mission-critical hardware. Whether interdependence is stated explicitly or not, and whether it is intended or not, interagency cooperation exists in any space mission when the loss of the contribution from any agency would cause the mission to fail or would require drastic changes in mission scope or structure. The failure can be technical, cost-, schedule-, or risk-related and might occur during any phase of the project, from initial development through mission operations. Although interagency cooperation is the riskiest form of collaboration, there are reasons that it might be considered for a particular mission. For example, cooperation might be considered when scientific opportunities organically emerge, when technical capabilities exist in one agency that address a mission need in another, or when a new mission need emerges that cuts across existing agency responsibilities. However, the committee found that cooperation is ill-advised if it is motivated by factors not directly related to mission performance. These include, for example, an imposed merger of technical requirements for political reasons, external hopes for cost savings, the addition of unfunded mandates, or directed interagency collaboration for the sake of collaboration. Interagency cooperation on a particular space mission can be encouraged or even mandated to address budgetary, political, or industrial base objectives that are in tension with the ostensible technical objectives of the mission. Advocates of cooperation can underestimate the difficulties and associated transaction costs while assuming no increase in risk to performance. If external political pressure for more interagency cooperation continues without appropriate attention to mitigating the inevitable associated risks with such agency interdependence, it should be expected that costs to the nation would rise as well as would the chances for mission failure. Although cooperation should be encouraged when good reasons support it, interagency cooperation should be treated as an exception rather than a norm.
OCR for page 33
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions Agencies considering collaboration should engage in a formal decision process to assess whether the coordination of multiple agency activities is preferable to formal cooperation and at what level collaboration should occur (e.g., mission, spacecraft, subsystem, component, data standard), if at all. Such strategic decisions should be made by the agencies prior to consideration of the more tactical aspects of a proposed collaboration. IMPEDIMENTS TO INTERAGENCY COLLABORATION Impediments to interagency collaboration can result from sources both internal and external to the agencies themselves. Internal sources can include agency goals and ambitions, culture, stakeholders, and agency-unique technical standards and processes. External sources can include the different budget cycles for each agency, different authorization and appropriation subcommittees, budget instability, and changes in policy direction from the administration and Congress. These impediments manifest themselves as impacts to mission success and as changes in cost, schedule, performance, and associated risks. The most serious impediments are external to the agencies. They are typically symptoms of conflicting policies that are often not made explicit at the beginning of proposed cooperative efforts. Such impediments manifest themselves as different budget priorities by agencies, the Office of Management and Budget (OMB), and the Congress toward the same collaborative activity. While there may be acknowledgment of the value of collaboration at a national level, decision makers at the implementation level can be unwilling to prioritize collaboration above other agency mission assignments and constraints. Many of the impediments to interagency collaboration appear as impediments to good systems engineering and project management.1 A general tenet of systems engineering is that risks tend to occur at the “seams” between major system elements.2 Thus the more organizations that are involved, the more interfaces there are to manage, and the greater the risk of something being missed or of miscommunication. The increased number of stakeholders also serves to complicate the requirements development process, as a number of technical, programmatic, and political requirements are typically implied by the decision to collaborate that may not be explicitly stated at the outset.3 Yet, collaborative requirements development and prioritization constitute one of the most important elements of successful collaboration. The challenge then becomes to understand the similarities and differences between the agencies’ requirements prior to a commitment to work together on a collaborative mission. Of course, a prerequisite is that the agencies need to fully understand their own requirements (and the traceability thereof) to manage partner and stakeholder expectations for meeting those requirements. MITIGATING THE RISKS OF INTERAGENCY COLLABORATION Given a decision to pursue a collaborative space mission with interdependence greater than “the use of resources” level, there are several dimensions and distinct modalities in which risks need to be mitigated. The most important ones include: 1 By “systems engineering” the committee means the process by which the performance requirements, interfaces, and interactions of multiple elements of a complex system such as a spacecraft are analyzed, designed, integrated, and operated so as to meet the overall requirements of the total system within the physical constraints on and resources available to the system. Thus, for example, managing the effective integration and operation of a spacecraft requires that the physical, electrical, and thermal interfaces between different spacecraft subsystems, as well as their performance requirements and characteristics, be clearly defined and that responsibility for each side of an interface be well understood and as simple as possible. When organizational responsibilities for such systems interfaces are ambiguous or unnecessarily complex, the situation often leads to a less effective systems engineering process. By “project management” the committee means the overall management of the budget, schedule, performance requirements, and assignments of team member roles and responsibilities for the development of a complex system such as a scientific spacecraft. 2 See, for example, E. Bardach, Turf barriers to interagency collaboration, pp. 168-192 in The State of Public Management (D.F. Kettl and H.B. Milward, eds.), Johns Hopkins University Press, Baltimore, Md., 1996. 3 Agencies have both internal and external stakeholders for their space missions, and so they can be simultaneously both developers and consumers. One agency can also, at times, be a stakeholder of another (e.g., NASA is a stakeholder for long-term NOAA observations; NOAA has an interest in NASA’s research observations). These interdependencies, which evolve over time, can make relationships more complex than one might infer from examination of the publicly stated agency missions.
OCR for page 34
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions Policy. There should be a single, There should be a single, clear memorandum of understanding (MOU) or similar agreement between participating agencies. The MOU should define the project’s chain of command and dispute resolution mechanism. There should be an explicit treatment of how requirements are to be decided and budget disputes are to be resolved. Since OMB acceptance is crucial to resolving budgetary issues and taking a unified executive branch approach to Congress, it is highly preferable that the MOU be endorsed at a policy level that covers all participating agencies, e.g., OMB program associate director(s) or higher. Management. One of the collaborating agencies should be designated as the lead agency. Ultimate responsibility and accountability for executing the mission—within the agreed set of roles and responsibilities, command structure, and dispute resolution process defined by the MOU—should rest with the lead agency. For missions for which there is expected to be a transition of major responsibility at a point in time—for example, the transition between acquiring a new sensor, putting it on orbit, and receiving its data stream for research to ensuring follow-on sensors and long time-series of data—the single lead agency might also change. If so, then the mission budget could shift as well from the first lead agency to the later lead agency. Systems engineering. There should be a single, well-defined, established systems engineering process with a single chief systems engineer. There should be no duplicate milestone reviews or redundant appeals processes. Acquisition. A single acquisition authority should be used by the lead agency. Different authorities might be used for clearly separable components that are not on the critical path, but they should be used only sparingly and should be entirely separable. For industry, the transaction costs involved for interagency collaboration should be tracked and treated as explicitly allowable costs under contract. Operations. Interagency collaboration does not end with a successful launch but extends into the operational phase. As such there should be a common, agreed-upon concept of operations. This can evolve with time and experience, but it is preferable to have an explicit agreement on an operational approach at the outset. QUESTIONS TO ADDRESS BEFORE COLLABORATING Interagency collaborations do not start with a blank sheet of paper. Agencies and their overseers need to ensure that they are organized, trained, and equipped to implement collaborative efforts at all levels (policy, acquisition, systems engineering, and operations), whether they have existing internal capabilities, or whether precursor, confidence-building engagements are needed. After the decision to involve Russia in the space station, for example, there was an extensive Shuttle-Mir effort to build a foundation for what would later become routine operational cooperation with Russia on the International Space Station.4 Similarly, agencies engaging in collaboration need to realistically judge their readiness for collaboration and adjust their expectations and resources accordingly. Taking into consideration the mixed results of interagency collaboration, the committee compiled a series of questions, organized into topical categories that should be carefully considered before committing to a collaborative Earth or space mission. These questions do not have simple yes or no answers but, rather, address the multiple layers of collaboration that must succeed for the overall mission to succeed. In some cases, variants of the same question appear in multiple categories to ensure consideration from multiple points of view. As might be expected, the most important questions, with the greatest potential impact for mission success, are those asked before agencies commit to collaboration. Evaluation-Deciding Whether to Collaborate Why is the collaboration being contemplated? What are the arguments for and against separate missions? How real are potential synergies? What assumptions, if changed, would cause significant increases in cost and complexity? What kind of collaboration is being contemplated? (See specific types defined Chapter 1.) 4 The history of Shuttle-Mir operations is available in NASA, Phase 1 Program Joint Report (G.C. Nield and P.M. Vorobiev, eds.), NASA SP-1999-6108, January 1999, available at http://spaceflight.nasa.gov/history/shuttle-mir/welcome/goals.htm.
OCR for page 35
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions Who is advocating collaboration? Options may include agency leaders, OMB, the Office of Science and Technology Policy (OSTP), Congress, agency workforces, scientific users, industry contractors, and others. Are all or just some of the stakeholders supportive? What does each agency bring to the table? Examples include expertise in acquisition, insight/oversight capability, and technical skills. What happens if one partner leaves the collaboration? What can be done to minimize the impact of one agency’s default? What is the level of support from the agency’s workforce or from the scientific community for the collaboration? To what extent will the proposed collaboration encourage or preclude involvement of third parties in mission implementation (e.g., universities, agencies, international partners)? How will agreement (e.g., on the scope and funding of a proposed collaboration) be secured among administration, legislative, and agency stakeholders? Who will be tasked with building and maintaining consensus? Policy—Setting Priorities and Resolving Disputes How high does the cooperative project rank on each agency’s priority list? What level of leadership support is available for the project at each agency? How will project decisions be made? Are there clear lines of authority, responsibility, and accountability? Is there an agreed-upon decision-making process that includes an effective dispute resolution approach? Are the respective organizations adequately defined and structured in accordance with agreed-upon roles and responsibilities? How will funding be provided to and from each agency? How will cost overruns be paid for? What is the process to resolve disputes at the project, program, and agency levels? What are the criteria for terminating the project? Systems Engineering—Achieving Mission Success Is there an agreement on a single process for systems engineering? If so, what is the process? Is there an agreement on a single process for requirements definition, and what is the process? How will project decisions be made, and who is empowered to make them? How will the interfaces and work breakdown between agencies be determined? To what extent are the mission systems defined at all levels so that each participating agency understands how its roles and responsibilities translate into work and products? To what extent does each participant understand what it needs to provide to the other team members and when? Is there a written plan (e.g., project plan) including this level of detail? To what extent do good open communications between all parties occur, and what provisions can be made to ensure good future communications? To what extent do the participants trust and respect each other, and can appropriate commitment be demonstrated? Acquisition-Achieving Technical and Programmatic Success Which agency’s acquisition process will be used? Are there independent cost estimates at each major milestone, and is there a process for reconciling differences between the project office’s estimates and independent estimates? Which agency’s quality assurance process will be used?
OCR for page 36
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions TABLE 3.1 Leadership Responsibility for Interagency Cooperative Missions Evaluation—Deciding Whether to Collaborate Policy—Setting Priorities and Resolving Disputes Systems Engineering—Achieving Mission Success Acquisition—Technical and Programmatic Success Operations—Successful Mission Execution Executive Branch Leaders * * Agency Leaders * * * * Project Leaders * * * Which agency’s spaceflight project and/or flight instrument selection process will be used? What evidence suggests that good open communication between all parties happens? What evidence suggests that the participants trust and respect each other and are committed? Operations-Successful Mission Execution Is there an agreement on a single operational concept, and if so, what is it? To what extent do good, open communications exist between all parties? To what extent do the parties trust and respect each other, and are they committed? Different levels of government will have different roles and responsibilities for answering these questions. As a collaborative endeavor progresses from policy decision and strategic planning phases into tactical planning and execution phases, the level at which officials need to be responsible for decision making and program direction also change. Senior agency leadership and the executive branch (notably OMB) need to take ownership of the strategic acquisition and policy aspects of interagency collaboration. If interagency cooperation, requiring a significant level of interdependency between participants, is intended, primary responsibility for systems engineering and acquisition will need to be clearly assigned, preferably to one agency or organization. The collaborating agencies may increasingly share responsibilities as the mission enters an operational phase, but this, too, requires a clear mechanism for setting priorities and quickly elevating any disputes for resolution. Table 3.1 shows how responsibility for addressing the categories of questions listed above might be allocated. CHARACTERISTICS OF SUCCESSFUL INTERAGENCY COLLABORATIONS Despite the numerous impediments and potential pitfalls of interagency collaboration in Earth and space science missions, it is nonetheless possible to have successful outcomes. As discussed in previous chapters, and as drawn from the committee’s examination of case studies, agency briefings, existing reports, and members’ own personal knowledge and direct experience, successful interagency collaborations share many common characteristics.5 Those characteristics are, in turn, the result of realistic assessments of agency self-interest and capabilities and a disciplined attention to systems engineering and management “best practices.” The committee finds that successful interagency space mission collaborations are characterized by: A small and achievable list of priorities. Projects address a sharply focused set of priorities and have clear goals. Agreement is based on specific projects rather than general programs. A clear process to make decisions and settle disputes. Project decision making is driven by an intense focus on mission success. This is facilitated by formal agreement at the outset on explicitly defined agency roles 5 Those same characteristics, and others related to them and mentioned earlier in this report, were also identified in a document prepared by the senior representatives of the five International Space Station international partner agencies. See “International Space Station Lessons Learned as Applied to Exploration,” International Space Station Multilateral Coordination Board, Kennedy Space Center, Fla., July 22, 2009.
OCR for page 37
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions and responsibilities and should involve agreed-upon processes for making management decisions, single points of accountability (i.e., not committees), and defined escalation paths to resolve disputes. Long-term planning, including the identification of exit strategies, is undertaken at the outset of the project and includes consideration of events that might trigger a descope or cancellation review and associated fallback options if there are unexpected technical difficulties or large cost overruns that make the collaboration untenable. Clear lines of authority and responsibility for the project. Technical and organizational interfaces are simple and aligned with the roles, responsibilities, and relative priorities of each cooperating entity. Project roles and responsibilities are consistent with agency strengths and capabilities. Expert and stable project management has both the time and the resources available to manage the collaboration. Specific points of contact for each agency are identified. Agency and project leadership provides firm resistance to changes in scope. When possible, one of the collaborating agencies should be designated as the lead agency with ultimate responsibility and accountability for executing the mission within the agreed set of roles and responsibilities, command structure, and dispute resolution process defined by the MOU. In some cases the lead agency might change as a function of time, as for missions in which the lead agency differs between the implementation and operations phases. Well-understood participation incentives for each agency and its primary stakeholders. There is a shared common commitment to mission success, and there is confidence in and reliance upon the relevant capabilities of each partner agency. Each agency understands how it benefits from the collaboration, and recognizes that collaborative agreements may need to be revisited at regular intervals in response to budgetary and political changes. There is buy-in from political leadership (e.g., executive branch, Congress, and agency-level administrators), which can help projects move past rough spots that will inevitably occur in funding and support. There is a general spirit of intellectual and technical commitment from the agency workforce and contractors to help projects mitigate the disruptive effects of technical and programmatic problems that will also inevitably occur. Early and frequent stakeholder involvement throughout the mission keeps all stakeholders informed, manages expectations, and provides appropriate external input. Single acquisition, funding, cost control, and review processes. There is a single agency with acquisition authority, and each participating entity accepts financial responsibility for its own contributions to joint projects. Reliance on multiple appropriation committees for funding is avoided or reduced to the greatest possible extent. Cost control is ideally the responsibility of a single stakeholder or institution, because without a single point of cost accountability, shared costs tend to grow until the project is in crisis. Single, independent technical and management reviews occur at major milestones, including independent cost reviews at several stages in the project life cycle. Adequate funding and stakeholder support to complete the task. Funding adequacy is based on technically credible cost estimates with explicitly stated confidence levels. The committee recommends that all of the above characteristics be incorporated in every interagency Earth and space science collaboration agreement. Beyond the formal interagency MOU creating an interagency collaborative space mission, there should be a joint (signed) implementation plan. The committee found that while such documents are commonplace in international collaborations, they are equally crucial for interagency collaborations. The implementation plan establishes management authority, organizational responsibilities, integrated review plans, budgets, schedule, and priorities at the outset and explicitly spells out how conflicts over scarce resources are to be handled. This implementation plan should be responsive to each agency’s needs for involvement and oversight. Once agreed to, the implementation plan supersedes individual agency policies when there is conflict and remains invariant throughout the project life cycle regardless of any participating agencies’ internal changes in policies and procedures. This prevents the phenomenon of “moving the goalposts” during implementation, which can create dramatic disruptions to project budgets and schedules. A prerequisite for the implementation plan is the existence of a well-formed requirements document that allows explicit trade-offs between priorities that may become necessary and provides a foundation for resisting changes in mission scale and scope once underway. Finally, the committee notes the critical importance of open, honest, effective, and complete communications. This encompasses all types of communication—written, oral, formal, and informal—from program and project
OCR for page 38
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions plans, schedules, requirements, and contracts, to technical interchange meetings, interface control documents, MOUs, and configuration control boards, including telephone calls, e-mail, and on-site visits. Communication is important in any space mission, but even more important when organizations with different cultures, procedures, vocabularies, and roles come together to achieve a common goal. Differences of culture, language, and procedures are expected in international space cooperation but are often underestimated in interagency collaborations until problems become quite obvious. As part of having good documentation and open communication, a collaborative project should strenuously avoid having separate agency project plans covering the same work content. Aside from the duplication of effort, separate plans tend to perpetuate areas of project team miscommunication that should be resolved early and quickly. SUMMARY OF LESSONS LEARNED As evidenced by the case studies discussed in Chapter 2, interagency collaboration is both difficult and expensive. It should never be pursued solely for political expediency or in hopes of reducing total costs. As with other types of collaborations, including international ones, the costs of collaboration (i.e., the additional administrative burdens and management complexity, and the real financial costs to deal with those) can be significant. Although parties may still find collaboration attractive if their share of a mission is more affordable than it would be if they were going it alone, they may nonetheless want to consider alternatives to interdependent reliance on another government agency. For example, agencies might consider buying services and/or coordinating observations preferable to fully interdependent cooperation. Because of increased costs and complexity when space projects are conducted on a cooperative6 multiagency basis, as opposed to under the auspices of a single agency, the committee recommends that agencies should conduct Earth and space science projects independently unless: It is judged that cooperation will result in significant added scientific (and possibly follow-on operational) value to the project over what could be achieved by a single agency alone; or Unique capabilities reside within one agency that are necessary for the mission success of a project managed by another agency; or The project is intended to transfer from research to operations necessitating a change in responsibility from one agency to another during the project; or There are other compelling reasons to pursue collaboration, for example, a desire to build capacity at one of the cooperating agencies. Good systems engineering and project management techniques are important in any space mission, but especially where multiple organizations are involved. As has already been noted, the seams that characterize the interface between collaborating organizations are a source of technical and programmatic risk that requires proactive management and attention. In the event of a decision to proceed with collaboration, conscious steps to mitigate risk are required at every stage of development—from identification of the potential partner agencies and assignment of their respective roles; through project definition, management, and acquisition; to working with the administration and Congress to ensure mission success. Interagency collaboration is not the norm for federal agencies pursuing Earth or space science missions. However, when agencies do collaborate, grassroots collaboration is preferred because it is based on technical necessity and a desire to work together. The committee understands that there may be national policy reasons to require collaboration in certain situations, but top-down collaboration will be burdened from the beginning with a lack of working-level buy-in. Teams that want to work together far outperform those that are forced together, and they also facilitate the application of the tools and techniques associated with good program and project management. Successful collaboration is more likely when each agency considers the partnership one of its highest priorities; 6 This means a relationship that has multiple agencies on the critical path to mission success. Necessary funding and/or mission-critical hardware may come from multiple agencies. If one agency leaves the program, the program falls apart unless that agency’s elements are replaced.
OCR for page 39
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions such an understanding should be codified in signed agreements that also document the terms of the collaboration’s management and operations. Interagency collaborations for Earth and space science missions will occur over the development period of instruments and spacecraft, a time that will typically be measured in years and extend over multiple congressional budget cycles. During this period, agency priorities or external events may result in one of the partners wishing to terminate the collaboration. It is important to recognize the potential for breakdowns in collaboration to impact the likelihood of future collaborations. Not all collaborations will be of critical priority for an agency or group of agencies. If, for example, the collaboration consists of procuring services or using another agency’s resources, then a formal high-level agreement may not be necessary. The level of criticality depends on the impacts of one party or another pulling out. As with international agreements, interagency agreements should not be entered into lightly, and even then, entered into only when alternatives are considered and discarded. Numerous impediments challenge the successful implementation of collaborative missions, requiring numerous parties to mitigate these challenges. At the policy level, the projects may have to deal with multiple appropriations committees, and at the engineering level there may be overlapping review processes and conflicting acquisition rules. The commitment necessitated by interagency collaborations means there is a need for coordinated oversight by the executive and legislative branches of these special projects. The success that the U.S. Global Change Research Program enjoyed in its early years illustrates the value of an OMB-led budget cross-cut process as well as the active leadership and effective partnership of OSTP and OMB. However, the current oversight roles of OMB and OSTP are not suited to the kind of day-to-day operational oversight needed to facilitate interagency cooperative efforts, and so some other governance mechanism may be needed to facilitate decisions across multiple agencies and to provide accountability and support for those decisions by the administration and Congress.7 The committee recommends that if OSTP, OMB, or the Congress wishes to encourage a particular interagency research collaboration, then specific incentives and support for the interagency project should be provided. Such incentives and support could include cross-cutting budget submissions; protection of funding for interagency projects; freedom to move needed funds across appropriation accounts after approval of a cross-cutting budget; multiyear authorizations; lump-sum appropriations for validated independent cost estimates; minimization of external reviews that are not part of the project’s approved implementation plans; and unified reporting to Congress and OMB, as opposed to separate agency submissions. 7 National Research Council, Earth Science and Applications from Space—National Imperatives for the Next Decade and Beyond, The National Academies Press, Washington, D.C., 2007, pp. 13-14.
OCR for page 40
Assessment of Impediments to Interagency Collaboration on Space and Earth Science Missions This page intentionally left blank.