EVALUATION OF THE PLANNING PROCESS
The draft plan was developed largely by the Climate Change Science Program (CCSP) Office and the participating agencies without involvement of the external community. As a consequence, and as pointed out in this committee’s first report, the draft plan was of mixed scientific quality, with the result that those chapters drawing upon preexisting expert working groups and science initiatives (e.g., atmospheric composition and the carbon cycle) were better developed and more consistent with the community consensus about priorities than other chapters in the plan (NRC, 2003b).
Once developed, however, a number of steps were taken to solicit input on the draft strategic plan. The CCSP organized a Planning Workshop in December 2002, which was open to all interested parties. The effort it took to organize such a large workshop for the discussion of the draft report was notable and widely appreciated. Comments on the draft plan were solicited from numerous scientists and stakeholders, at the workshop, by e-mail, and by other means. These approaches succeeded in communicating the thoughts and ideas of hundreds of people; well over 1,000 people attended the workshop and some 900 pages of written comments were received. In addition, the CCSP requested and received a detailed report from this committee. Overall, the mechanisms for gathering and organizing input relevant to the draft plan were commendable. In the view of the committee, the approaches taken by CCSP to receive and respond to comments from a very large and very broad group of scientists and stakeholders sets a high standard for all government research programs related to the development and use of science and engineering information.
The workshop was structured to elicit a wide variety of ideas and suggestions for improvement. The agenda included keynote addresses by many top Administration and international officials, breakout sessions focused on individual chapters or crosscutting issues, and plenary session summaries of the breakout sessions. In each breakout session, an overview presentation was made by an agency employee, two to four invitees then presented a critique of the designated section of the plan, and finally the session was opened to comments from the audience. The workshop attendees were able to engage openly in discussions, to express ideas, and to offer suggestions for improvement. A message of transparency and openness was constantly communicated to all participants. The format of plenary sessions, breakout groups, and breakout group summaries was generally effective in facilitating the exchange of ideas at such a large gathering.
There are opportunities for improvements in future workshops of this type. First, in several sessions, the balance between presentation and discussion should have been modified to permit more of the latter. The constrained schedule for the conference meant that the printed document dominated the agenda, leaving insufficient time to discuss questions about the underlying assumptions and gaps in the program’s intellectual underpinning. Second, particular efforts should be made to attract stakeholders and scientists from programs now targeted for enhancement, such as decision support. Participation in the workshop was dominated by agency employees and scientists supported by federal funding, with significantly smaller attendance by scientists from previously underemphasized program elements, the private sector, state and local natural resource and land-use decision makers, and the environmental community.
It was clear that the comments elicited were welcomed and would receive consideration. The process used to make decisions regarding the comments was not well communicated. The committee recognizes the difficulty associated with specifying exactly how comments would be evaluated, as such activities inevitably involve extensive discussions among the plan drafters and managers. At the same time, more transparency would have been desirable regarding how comments would be weighed, how conflicting comments would be resolved and how the program would respond to suggestions not to be implemented. For the most part, the CCSP’s revisions to the strategic plan are quite responsive to comments expressed at the workshop, in written input, and by this committee. One notable exception is the fact that the revised plan does
not acknowledge the substantive and procedural contributions of the U.S. National Assessment of the Potential Consequences of Climate Variability and Change (NAST, 2001), a major focus of the Global Change Research Program (GCRP) in the late 1990s. Many participants at the December workshop criticized how the draft strategic plan treated the National Assessment, as did this committee in its first report (NRC, 2003b). The revised plan does not reflect an attempt to address these concerns, and no rationale for this decision has been provided.
As the program moves forward from planning to implementation, regular opportunities should be provided for interested parties to comment on the specific details of the program. The overall plan, and its individual components, will benefit from review boards, steering committees, and other structures that can provide external expert advice to the program’s managers. In fact, at the committee’s August 2003 meeting, several chapter authors indicated that they are planning workshops with research and stakeholder communities to further revise their portions of the strategic plan and to develop implementation plans. The committee commends the program managers for seeking input from expert communities in this manner. These smaller expert workshops would have been of even more value if they had taken place before the strategic plan was prepared and before the large planning workshop. Increasing the involvement of the decision support community and various stakeholders is an important way to improve future planning. This involvement should be given a high priority in the near term, starting with areas where there is already a receptive decision-making group, such as water resource managers.
THE NEXT GENERATION OF STRATEGIC PLANNING
The current strategic planning effort of the CCSP has been impressive. It has identified goals and objectives for the program, proposed an ambitious series of products that will shed light on issues perceived to be important for national decision makers, and stimulated a great amount of cooperation among the many participating agencies. But, as the CCSP itself has pointed out, planning and implementing such an ambitious program is itself something of an experiment. It is an experiment not only in managing activities among a diverse group of agencies but also in trying to produce near-term results and analyses helpful to decision makers while simultaneously assuring that the long-term nature of the climate change issue continues to receive sufficient attention. Even with the substantial history of the GCRP behind it, continued planning and management of the CCSP remains a work in progress.
While many of the activities that are envisioned in the current strategic plan will succeed, some will fail, and others will achieve their goals more slowly than anticipated. Some agencies will perceive their involvement in the CCSP to have advanced their missions; others will not. The science will proceed quickly in some areas and frustratingly slowly in others. It is critical that the program management and the agencies use these experiences in an adaptive way to adjust their own management practices as they identify the next series of tasks in a dynamic scientific, budgetary, and political environment. Embracing adaptive management for the program as a whole will require ongoing and rigorous evaluation and redirection. As discussed in Chapter 3 of this report, to identify which program elements are succeeding and which are lagging, the CCSP will need to conduct rigorous independent program reviews.
The committee believes that one way to ensure that adaptive learning occurs will be for the CCSP to conduct future strategic planning exercises, perhaps in collaboration with relevant international programs. The CCSP should update the strategic plan every three to five years. The updated strategic plan need not be as extensive as the current plan; it could instead focus largely on those areas of the science and the program for which adjustments are needed, and should spell out what those adjustments are intended to be. It will be critical that the updated plan be developed in cooperation with scientific and stakeholder communities, and that the updated plan identify the management responsibility and accountability for all the elements of the program, including its crosscutting functional components, such as communications and data management.
Recommendation: The CCSP should plan for the generation of an updated strategic plan every three to five years.
The process of producing the updated plan should reflect the learning that has accompanied the current CCSP strategic plan. Any strategic plan is a balance between the top-down goals of the organization and its bottom-up capabilities to deliver information and products. The current plan reflects this tension in the often poor linkage between the products and milestones identified in the individual science chapters and the five goals for the overall CCSP. The updated plan should resolve persisting linkage problems. This can be done effectively only by engaging the scientific community responsible for generating measurements and knowledge in each of the program’s areas. This engagement should happen early and often, to provide timely feedback to the CCSP.
Involving the potential users of climate science (broadly defined) early in the updated strategic planning
effort will be equally important. Many of the activities proposed in the current plan could be used to structure such engagement, and their success will be critical to the overall success of the CCSP. Engaging users in an open and transparent way will strengthen the credibility of the plan. The CCSP should hold open workshops to review users’ needs as a precursor to the development of an updated strategic plan.
Another improvement to the planning process should be a greater interaction with the global audience than has been achieved to date. This interaction should be rooted in both the science and decision support activities of the CCSP. By engaging the scientific and user communities in critical countries, the CCSP could be more effective in addressing its scientific objectives and in investing resources.
The CCSP should document and publish its process for strategic planning and implementation. The CCSP intends to become a learning organization, and one of the characteristics of such organizations is their documentation of what they have learned. Because documentation typically leads to institutionalization, the CCSP will be able to learn effectively from the current process of planning and implementation, and will be able to demonstrate the progress that the nation can reasonably expect in the future.