CHAPTER TEN

Computational Infrastructure—Challenges and Opportunities

It is in the nature of infrastructures to be ignored when they are functioning. When they crumble, it can cause major disruptions or collapse of the enterprises and communities that depend on them. Computational infrastructure underpins the entire climate modeling enterprise. This chapter reviews the risks posed to the current climate modeling infrastructure by changes in technology and proposes an expansion of the infrastructure that could dramatically alter the landscape of climate modeling. (Definitions of selected terms used throughout this chapter are provided in Box 10.1.)

Future generations of climate simulation models will place an ever-increasing demand on computational infrastructure. There are compelling scientific needs for higher levels of spatial resolution (e.g., cloud-resolving atmospheric component, eddy-resolving ocean component, landscape-resolving land-surface component), more extensive vertical domains (e.g., whole atmosphere), increased model complexity (e.g., treatments of parameterized processes as well as the addition of other simulation component models, such as marine and terrestrial ecosystem components), and larger simulation ensembles (e.g., 50-100 members). All these developments are seen as essential for more accurate, reliable, and useful climate projections and predictions. The current generation of supercomputer systems deployed partly or mostly to support climate modeling, such as the National Oceanic and Atmospheric Administration’s (NOAA’s) Gaea supercomputer and the National Center for Atmospheric Research’s (NCAR’s) Yellowstone system, provide an important capability that enables progress toward these goals, but they are only a first step in deploying rapidly evolving computational capabilities. Scientific advances and applications will motivate the national climate modeling enterprise to exploit these new computational capabilities; the complexity and diversity of climate model codes coupled with the expected nature of hardware advances will make this an increasingly challenging task over the next two decades.

As a rough guide, the ability to simulate 5-10 years per wall clock day of computing time continues to be regarded as necessary to make the climate modeling problem tractable.1 With the current generation of high-performance computing systems,

___________________

1 Climate model runs often simulate time spans of hundreds of years, meaning that a single run can take days or weeks to complete.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 175
CHAPTER TEN Computational Infrastructure— Challenges and Opportunities I t is in the nature of infrastructures to be ignored when they are functioning. When they crumble, it can cause major disruptions or collapse of the enterprises and communities that depend on them. Computational infrastructure underpins the entire climate modeling enterprise. This chapter reviews the risks posed to the current climate modeling infrastructure by changes in technology and proposes an expansion of the infrastructure that could dramatically alter the landscape of climate modeling. (Definitions of selected terms used throughout this chapter are provided in Box 10.1.) Future generations of climate simulation models will place an ever-increasing demand on computational infrastructure. There are compelling scientific needs for higher lev- els of spatial resolution (e.g., cloud-resolving atmospheric component, eddy-resolving ocean component, landscape-resolving land-surface component), more extensive ver- tical domains (e.g., whole atmosphere), increased model complexity (e.g., treatments of parameterized processes as well as the addition of other simulation component models, such as marine and terrestrial ecosystem components), and larger simulation ensembles (e.g., 50-100 members). All these developments are seen as essential for more accurate, reliable, and useful climate projections and predictions. The current generation of supercomputer systems deployed partly or mostly to support climate modeling, such as the National Oceanic and Atmospheric Administration’s (NOAA’s) Gaea supercomputer and the National Center for Atmospheric Research’s (NCAR’s) Yellowstone system, provide an important capability that enables progress toward these goals, but they are only a first step in deploying rapidly evolving computational capabilities. Scientific advances and applications will motivate the national climate modeling enterprise to exploit these new computational capabilities; the complexity and diversity of climate model codes coupled with the expected nature of hardware advances will make this an increasingly challenging task over the next two decades. As a rough guide, the ability to simulate 5-10 years per wall clock day of computing time continues to be regarded as necessary to make the climate modeling problem tractable.1 With the current generation of high-performance computing systems, 1  Climatemodel runs often simulate time spans of hundreds of years, meaning that a single run can take days or weeks to complete. 175

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G BOX 10.1  GLOSSARY OF SELECTED TERMS USED IN THIS CHAPTER Computational infrastructure: the software basis for building, configuring, running, and analyzing climate models on a global network of computers and data archives. See Edwards (2010) for an in-depth discussion of “infrastructure” and software as infrastructure. Refactoring: rewriting a piece of software to change its internal structure without in any way altering its external behavior. This is often undertaken to increase the efficiency or ease of use and maintenance of legacy software. Core: an element of computational hardware that can process computational instructions. Some current computers bundle several such “cores” onto a single chip, leading to “multicore” (typically 8-16 cores per chip) and “many-core” systems (tens of cores per chip). Node: an object on a network. In the context of high-performance computing (HPC) architecture it is a unit within a distributed-memory supercomputer that communicates with other nodes using network messaging protocols, i.e., “message passing.” It is the smallest entity within the cluster that can work as an independent computational resource, with its own operating system and device drivers. Within a node there may be more than one, indeed many, integrated but distinct computational units (“cores”) that can communicate using more advanced fine-grained communication protocols (“threading”). Concurrency: simultaneous execution of a number of possibly interacting instruction streams on the same computer. Flops: floating-point operations per second, a unit of computational hardware performance. Prefixed by the usual metric modifiers for orders of magnitude; a petaflop is 1015 flops and an exaflop is 1018 flops. Exascale: computers operating in the exaflop range, coupled to storage in the exabyte range. Threads: a stream of instructions executing on a processor, usually concurrently with other threads in a parallel context. this allows global resolutions of 50 km. Each additional 10-fold increase in resolution leads to a more than 1,000-fold increase in operation count, before considering ad- ditional complexity. As recent history has demonstrated, the testing, debugging, and evaluation (e.g., participation in formal model evaluation processes, like the Coupled Model Intercomparison Project, Phase 5 [CMIP5]) required for any given model version continues to require ever-greater amounts of computing time as the model becomes more complex. Overall, the climate modeling enterprise relies on sustained improve- ments in supercomputing capabilities and must strategically position itself to fully exploit them. Finding 10.1: As climate models advance (higher resolutions, increased complex- ity, etc.), they will require increased computing capacities. 176

OCR for page 175
Computational Infrastructure—Challenges and Opportunities PREVIOUS HARDWARE TRANSITIONS As discussed in Chapters 3 and 4, higher spatial resolution is an important element of improving the fidelity and usefulness of climate models, but also dramatically increases their computational requirements. Thus, climate models have historically been among the principal drivers of HPC. The first coupled ocean-atmosphere model (Manabe and Bryan, 1969) was recognized as a landmark in scientific computing in a Nature survey (Ruttimann, 2006). Coding models to take advantage of advanced and novel computational architectures has always been a significant activity within the climate and weather research and operations communities. During the 1980s and early 1990s, the climate simulation en- terprise benefited greatly from computer architectures focused on high-performance memory systems, mainly in the form of vector computing, a technique of fine-grained array concurrency pioneered by Seymour Cray. Several models from many institutions were able to deliver sustained computing performance operating near the theo- retical limits of these computing architectures. In the late 1990s, proprietary vector supercomputing architectures began to be supplanted by commodity off-the-shelf architectures. This was partly a result of market forces driving U.S. manufacturers of proprietary architectures out of this field. At least one company (NEC) soldiered on, with the SX series of machines successfully operating in subsequent generations until the present in this field in Japan, Australia, and Europe. A disruptive transition in the late 1990s pushed climate modeling toward parallel and distributed computing implementations. Although this was viewed with trepidation at the time (NRC, 1998, 2001b; USGCRP, 2001), it is possible to see from the hindsight of 2011 that the community in fact weathered the transition with distinction. While actual sustained performance of climate codes relative to the theoretical peak hard- ware performance fell by an order of magnitude, time to solution continued to be reduced through exploitation of the aggregate performance of massively parallel machines. Adapting to parallel computing architectures did require pervasive refac- toring of highly mature codes. This recoding process necessarily started prior to the establishment of a programming environment standard (e.g., shared memory parallel directives that evolved to SHMEM (Barriuso and Knies, 1994) and distributed-memory approaches (like Parallel Virtual Machine [Geist et al., 1994] that eventually evolved into the Message Passing Interface standard [Gropp et al., 1999]). Being out in front of the development of a programming model has been essential to navigating previ- ous architectural transitions. But rather than insert these new methods throughout model codes, climate and weather modelers began to see these methods as part of infrastructure. Most institutions developing high-end models resorted to high-level 177

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G libraries where the standard and highly reusable computational methods were encap- sulated. Scientists and algorithm developers were able to use abstractions that were scientifically intuitive, and the gory details of parallel programming were effectively hidden from those developing the scientific aspects of climate models. When the underlying hardware changed, the infrastructure changed with it, and the scientific codes remained largely intact. Finding 10.2: The climate modeling community adapted well to the previous hardware transition by moving toward shared software infrastructure. A similarly disruptive moment is now upon us. As described in the next section, all indications are that increases in computing performance through the next decade will arrive not in the form of faster chips, but more of them, with considerably more complex embodiments of concurrency. Deep and abstruse memory hierarchies, and processing element counts that push the limits of current parallel programming stan- dards, both make for a challenging environment for application programmers. In this chapter, the committee argues that a renewed and aggressive commitment to shared software infrastructure across the climate and weather communities will be needed to successfully navigate this transition, which may prove to be even more disruptive than the vector-to-parallel transition. Indeed, conventional wisdom in the HPC com- munity (see Zwieflhofer [2008] and Takahara and Parks [2008] for examples) is that the next generation conversion will be significantly more complex and unpredictable than previous changes, given the absence of a clear technology path, programming model, performance analysis tools, etc. The ratio of sustained performance to theoreti- cal hardware peak may once again fall precipitously, as it once did during the vector to distributed-memory-parallel transition. A second element of infrastructure that now pervades the field is the global data infrastructure for models. While this “vast machine” networking the globe is historically associated with the global observing networks, it now encompasses models as well (Edwards, 2010). The committee argues below that this infrastructure too is invisible and taken for granted, but that without proper provisioning for the future, it may be overwhelmed by projected growth and demand for access to data. NEW ARCHITECTURES AND PROGRAMMING MODELS This section summarizes trends in HPC hardware, the programming model for using such platforms, and the system software expected to be in place over the next two decades. 178

OCR for page 175
Computational Infrastructure—Challenges and Opportunities Hardware Assessment: Architectural Prospects for the Next 10-20 Years Conventional high-end multicore microprocessor technology is approaching multiple limits, including power consumption, processor speed, per-core performance, reliabil- ity, and parallelism. Hardware assessments such as Kogge (2008) have shown that ex- trapolating current technologies such as those used in Oak Ridge National Laborato- ry’s (ORNL’s) Jaguar machine or NOAA’s Gaea leads to exascale machine configurations that can be ruled out on the basis of power consumption alone, which would reach the 100-megawatt range. Alternate technology paths in the next generation of ma- chines include the Blue-Gene system-on-chip design. Using flops/watt as a key metric, this technology path scores better on the power efficiency front. However, the newest and largest machine of this class, the 20-petaflop Sequoia platform at Lawrence Liver- more National Laboratory,2 will still require 6 megawatts of power to operate. A second technology track aims at exploiting the fine-grained parallelism employed in contemporary graphics chips. Graphics processing units (GPUs) have been used to achieve very high concurrency for specialized graphics operations (such as rendering three-dimensional objects on a two-dimensional screen). More recently GPUs have be- come a viable alternative to conventional high-end microprocessors (CPUs), in part be- cause of their high performance, programmability, and efficiency at exploiting parallel- ism for use in scientific and other nongraphics applications. This accelerator approach has been extended to allow for general-purpose computing on graphics processing units (GPGPUs) using a technique that combines programmable stages and high- precision arithmetic with the fine-grained parallelism for which GPUs were designed. This approach allows mapping of standard computational concepts onto the special- purpose features of GPUs (Harris, 2005). The Intel Knights Corner chip, released at the 2011 Supercomputing Conference, is the latest extension of GPGPU technology called Many Integrated Core (MIC) architecture, utilizing many parallel lower-power cores. Such computer architectures as GPUs and MICs are expected to be an important step toward the realization of exascale-level performance in the next one to two decades. Other novel technologies under consideration such as field-programmable gate arrays have not proved very suitable to complex multiphysics applications such as Earth sys- tem models: they are more intended for applications where a single set of operations is repeated on a data stream. Even more experimental approaches, based on biology, nanotechnology, and quantum computing, are not expected to be suitable for con- 2  http://nnsa.energy.gov/blog/sequoia-racks-arriving-llnl (accessed October 11, 2012). 179

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G ventional codes and will almost certainly require complete and radical rethinking of computational Earth system modeling if they are to be exploited. Although there are architecturally different solutions to enhancing system perfor- mance, they share the characteristic that the increase in performance will come from exploiting additional concurrency at the node level. Extrapolating to an exaflop (1018 flops) with single thread clock speed of 1010 Hz leads to a required concurrency factor of 108-109. (Gaea, the largest machine today entirely devoted to climate modeling, is rated at ~1 petaflop to 1015 flops.) There is no current comprehensive climate model (as opposed to process study model) operating anywhere remotely close to that level of concurrency. The best examples of parallel concurrency exhibit about 105 based on maximum processor counts reported for models in the CMIP5 archive (model descriptions are online3). The level of concurrency in comprehensive climate models lags the hardware concurrency of the leadership-class machines by at least a factor of 100. Based upon the commu- nity’s experience with the previous disruptive technology transition, where the ratio of sustained to theoretical peak performance dropped from ~50 percent to ~10 percent for typical climate codes, it would not be surprising if this ratio dropped again during the coming transition. This cannot be made up without a very substantial investment in research into basic numerical approximations and algorithms, which must then be adopted quickly by the simulation community. Finding 10.3: Climate models cannot take full advantage of the current parallel hardware, and the gap between performance and maximum possible perfor- mance is likely to increase as the hardware advances. Programming Models for the Next 10-20 Years At this time the many emerging architectures do not adhere to a common program- ming model. While new ways to express parallelism may well hold the key to progress in this decade, from the point of view of the developer of scientific applications, a transition path is far from evident. Assessments undertaken by the Defense Advanced Research Projects Agency (DARPA) and the Department of Energy (DOE) (e.g., DOE, 2008; Kogge, 2008) indicate profound uncertainty about how one might program a future system that may encompass many-core chips, coprocessors and accelerators, and unprecedented core counts re- quiring the management of tens of millions of concurrent threads on next-generation 3  http://q.cmip5.ceda.ac.uk/ (accessed October 11, 2012). 180

OCR for page 175
Computational Infrastructure—Challenges and Opportunities hardware. The President’s Council of Advisors on Science and Technology (PCAST) has called for the nation to “undertake a substantial and sustained program of fundamen- tal research on hardware, architectures, algorithms and software with the potential for enabling game-changing advances in high-performance computing” (PCAST, 2010). This challenge will grow to a billion threads by the end of this decade. The standard du jour programming model for parallel systems is based on MPI (Lusk and Yelick, 2007), shared-memory directives (e.g., OpenMP [Chandra et al., 2001]), or a hybrid of both. These are likely to fail at the scales projected for next-generation systems. Several new approaches are being proposed: for instance, Partitioned Global Address Space lan- guages (Yelick et al., 2007), which grew out of the DARPA High Productivity Comput- ing Systems (HPCS) program to develop “high-productivity” programming approaches (Lusk and Yelick, 2007). In some instances they are entirely new languages such as X10 (Charles et al., 2005) and Chapel (Chamberlain et al., 2007), and in other instances extensions to existing languages, like Co-Array Fortran (Numrich and Reid, 1998) and Unified Parallel C (Carlson et al., 1999). On longer time scales the exploitation of fine-grain parallism that minimizes data movement will be necessary. For certain hardware directions such as GPUs, a fine- grained parallelism approach is immediately needed. A new programming standard suitable for GPUs and many-core chips is being proposed (OpenCL: see, e.g., Munshi, 2008). Experiments with hardware-specific programming models such as CUDA for Nvidia graphics chips indicate a potential for significant speedups at very fine grain (see, e.g., Michalakes and Vachharajani, 2008) but very extensive intervention is needed to translate these speedups to the entire application. There are efforts under way to introduce a directive-based approach that could potentially be unified with OpenMP. The most promising avenue at the moment appears to be OpenACC,4 but it is still early in its development. The climate/weather modeling community has never retreated from experimenting with leading-edge systems and programming approaches to achieve required levels of performance. The current architectural landscape, however, is particularly challeng- ing, because it is not clear what direction future hardware may follow. The Interna- tional Exascale Software Project (Dongarra et al., 2011) promises a “roadmap” by 2013. The challenge is starkly presented as one that cannot be met without a concerted effort at exposing concurrency in algorithms and computational infrastructure. Finding 10.4: Increases in computing performance through the next decade will arrive not in the form of faster chips, but more of them, with considerably more 4  http://www.openacc-standard.org/ (accessed October 11, 2012). 181

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G complex embodiments of concurrency; the transition to this new hardware and software will likely be highly disruptive. System Software System software, such as operating systems, file systems, shells, and so on, will also undergo radical change to cope with the changes in hardware. Input/output (I/O) in particular will be a profound challenge for climate modeling, which is generating data volumes on an exponential growth curve (Overpeck et al., 2011). Furthermore, and potentially more significantly, the highly concurrent machines of this decade have the potential for decreased reliability as the component count increases. Fault-resilient software to account for decreased reliability could reduce the effective computation rate, as many such methods involve redundant computations (Schroeder and Gibson, 2007). With regards to fault resilience, architectures with extreme levels of concurrency and complexity may not guarantee that a program executes the same way every time. The possibility of irreproducible computation presents a challenge to testing, verification, and validation of model results. Chaotic systems with sensitive dependence on initial conditions will wander arbitrarily far outside any tolerance bound, given enough time. The key question in the climate context is to see whether trajectories subject to small changes at the hardware bit level stay within the same basin of attraction, or do these small errors actually push the system into a “different climate state.” Currently there is no other way to prove that an architectural or software infrastructure change has not pushed the system into a different climate other than computing the climatology of long (usually 100-yr, to take into account slow climate processes) control runs. This requirement is hugely expensive and a significant barrier to testing. Can the climate modeling community adapt to a world where in silico experimenta- tion is more like in vitro biological experimentation, where reruns of experiments are only statistically the same? Such adaptation would entail profound changes in meth- odology and be an important research challenge for this decade. Substantial progress must be made in fault-resilient software. Fault tolerance gener- ally implies redundancy layers, which can also imply a further decrease in achievable execution rate. Currently, the development efforts in high-end computing emphasize “peak flops,” and the climate modeling community would benefit from redressing this with increased efforts in fault-tolerant system software and workflows, echoing from previous National Research Council (NRC) reports that endorsed balancing support across the software life cycle. In conclusion, sharp decreases in reliability should be 182

OCR for page 175
Computational Infrastructure—Challenges and Opportunities anticipated as the hardware concurrency in machines grows. Investments that em- phasize system software and workflow reliability and investments in achieving faster execution rates both need to be balanced. Adoption of these approaches will involve extraordinary levels of effort by the climate modeling community, who will probably have a minimal influence on the hardware or the software roadmap. Various studies on exascale computing have noted that the computational profile of climate science is unique because of the tightly coupled mul- tiphysics nature of the codes. SOFTWARE INFRASTRUCTURE FOR MANAGING MODEL HIERARCHIES The community is best served by adopting a domain-specific technical infrastructure under which its developments can proceed. At the scale of one lab, this has been widely adopted and extremely successful. For example, at most modeling centers, sci- entists do not directly apply MPI or learn the intricacies of tuning parallel file-system performance, but use a lab-wide common modeling infrastructure for dealing with parallelism, I/O, diagnostics, model coupling, and so on, and for modeling workflow (the process of configuring, running, and analysis of model results). Many things that are done routinely—adding a new model component, adapting to new hardware— could not be done without a growing reliance on shared infrastructure. It was recognized over a decade ago that such software infrastructure could usefully be developed and shared across the climate modeling community. The most ambi- tious such project was the Earth System Modeling Framework (ESMF; Hill et al., 2004). It introduced the notion of superstructure, a scaffolding allowing model components to be coupled together following certain rules and conventions to permit easy inter- change of components between models (Dickinson et al., 2002). See Box 10.2 for a further description of ESMF and how it has unfolded after a decade of development. ESMF and other examples of common infrastructures, including the Flexible Modeling System (FMS)5 developed by the Geophysical Fluid Dynamics Laboratory (GFDL), the Model Coupling Toolkit, extensively used in the CESM, and OASIS, a European model coupling project, are the subject of a comparative survey by Valcke et al. (2012). This committee, which includes several members closely involved in the development of ESMF and other single-institution frameworks, observes that the idea of frameworks and component-based design is no longer novel or controversial. While switching 5  http://www.gfdl.noaa.gov/fms (accessed October 11, 2012). 183

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G BOX 10.2  THE EARTH SYSTEM MODELING FRAMEWORK (ESMF): CASE STUDY Following reports on U.S. climate modeling in 2001, federal agencies made substantial in- vestments in software infrastructure and information systems for both modeling and analysis. ESMF is a high-profile activity that was funded in 2001 by the National Aeronautics and Space Administration (NASA), and continues to be developed and maintained under funding by the Department of Defense, NASA, the National Science Foundation (NSF), and NOAA. The first cycle of ESMF was funded as a computational technology activity focused on model coupling in both the weather and climate community. In the second cycle of ESMF, the focus was extended from technology to the formation of a multiagency organization. As discussed in Chapter 2, this focus on process and governance included development of ways to manage sponsor and user expectations, requirements, and delivery. Building upon earlier work at GFDL and Goddard Space Flight Center (GSFC), ESMF intro- duced the notion of a superstructure (Collins et al., 2008; Hill et al., 2004) providing a common vocabulary for describing model components (e.g., ocean, atmosphere, or data assimilation pack- age), with their own gridding and time-stepping algorithms, and the fields they exchange. The common vocabulary permitted components to be coupled with relatively little knowledge of the internal working of other components, other than those exposed by the interface. This approach has proved very powerful and attractive for communities whose scientific activities depend upon component-level interoperability. In particular within the numerical weather prediction (NWP) community, many participants in the National Multi-Model Ensemble are building the National Unified Operational Prediction Capability,a which utilizes ESMF as part of its foundation. ESMF is also used within some single-institution frameworks, such as the GEOS system from NASA GSFC, and the Coupled Ocean/Atmosphere Prediction System. NCAR and GFDL have not adopted ESMF as their central framework. GFDL already had internally developed the FMS, from which ESMF borrowed ideas. NCAR was also actively test- ing another framework alternative, the Model Coupling Toolkit (MCT), while ESMF was under development, and for pragmatic reasons adopted MCT for high-level coupling while using some of the lower-level functionality of ESMF. However, their own frameworks remain architecturally compatible with adoption of ESMF; and in fact ESMF is often the lingua franca, or the common language, when their components are widely used in other communities (such as GFDL’s Modular Ocean Model). a http://www.nws.noaa.gov/nuopc/ (accessed October 11, 2012). between functionally equivalent frameworks has costs, few technical barriers to doing so exist should a compelling need arise. With ESMF and other infrastructure activities, the climate modeling community is seeing the natural evolution of infrastructure adoption. Individuals, communities, and institutions are seeing advantage. The committee believes that the community is now at the point where the benefits of moving to a common software infrastructure out- 184

OCR for page 175
Computational Infrastructure—Challenges and Opportunities weigh the costs of moving to it. With the experience, successes, and lessons learned of the past decade, the climate modeling community is positioned to accelerate infrastructure adoption. Cross-laboratory intercomparisons are now routinely con- ducted and, more importantly, are the way forward. End users require climate model information to be robust and reliable. Common infrastructure improves the ability to enforce scientific methodology (e.g., controlled experimentation, reproducibility, and model verification) across institutions and is one of the primary building blocks of that robustness and reliability. So far, no one software framework has become a universal standard, because model- ing centers that initially invested in one framework have had insufficient incentive to switch to another. Nevertheless, we believe that two critical strategic needs—that the U. S. climate community needs to more effectively collaborate, and that it needs to nimbly adapt to a wave of disruptive new computing technology—position the community for a further unifying step. The vector to parallel disruption led to wide- spread adoption of framework technologies at the scale of individual institutions. The climate modeling community can now conceive of a framework that could be sub- scribed to by all major U.S. climate modeling groups, supports a hierarchy of models with component-wise interchangeability, and also supports development of a high- performance implementation that enables climate models of unprecedented resolu- tion and complexity to be efficiently adapted to new architectural platforms. This idea is explored below. Finding 10.5: Shared software infrastructures present an appealing option for how to face the large uncertainty about the evolution of hardware and program- ming models over the next two decades. A NATIONAL SOFTWARE INFRASTRUCTURE FOR CLIMATE MODELING Very complex models have emergent behavior whose understanding requires being able to reproduce phenomena in simpler models. Chapter 3 makes a strong case for hierarchies of models adapted for different climate problems. From the computational perspective, some model types can be classified by a rough pace of execution needed (i.e., model simulated time per computer clock time) to make efficient scientific progress: • process study models and weather models (single component or few compo- nents; dominated by “fast” physics; 1 year/day), • comprehensive physical climate models (ocean-atmosphere, land and sea ice, 185

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G includes “slow” climate processes important on decadal to centennial climate scales, 10 years/day), and • Earth system models for carbon-cycle studies; paleoclimate models (most complex physics; dominated by slow processes and millennial-scale variability, 100 years/day). A single national modeling framework could allow the climate modeling community to configure all of these models from a palette of available components of varying complexity and resolution, as well as supporting high-end modeling. This idea has been proposed in the past: the history of previous efforts is recounted in Chapter 2. The committee believes that current trends in our methodology, both its strengths and weaknesses, point in the direction of a concerted effort to make this a reality, for reasons that are outlined below. A related methodological advance is the multimodel ensemble and the model inter- comparison project, which has become ubiquitous as a method for advancing climate science, including short-term climate forecasting. The community as a whole, under the aegis of the World Meteorological Organization’s World Climate Research Pro- gramme—through two working groups, the Working Group on Climate Modeling and the Working Group on Numerical Experimentation—comes to consensus on a suite of experiments, which they agree would help advance scientific understanding (more information in Chapter 8). All the major modeling groups agree to a suite of numeri- cal experiments defined for the current-generation Coupled Model Intercomparison Project (CMIP5) as a sound basis for advancing the science of secular climate change, assessing decadal predictability, etc., for participation in defining the experiments and protocols. The research community addressing climate variations on intraseasonal, seasonal, and interannual (ISI) time scales agrees on similar multimodel approaches for seasonal forecasting. A globally coordinated suite of experiments is then run, and results shared for a comparative study of model results. The model intercomparison projects (MIPs) are sometimes described as “ensembles of opportunity” that do not necessarily sample uncertainty adequately. A second major concern is the scientific reproducibility of numerical simulations. Even though differ- ent models are ostensibly running the same experiment, there are often systematic differences between them that cannot be traced to any single cause. Masson and Knutti (2011) have shown that the intermodel spread is much larger than differences between individual ensemble members of a single model, even when that ensemble is extremely large, such as in the massive ensembles of QUMP (Collins et al., 2011) and CPDN (Stainforth et al., 2005). To take but one example of why this is so trouble- some in the public sphere, consider different studies of Sahel drought made from the 186

OCR for page 175
Computational Infrastructure—Challenges and Opportunities CMIP3 experiments. Two studies, based on the GFDL (Held et al., 2005) and Community Climate System Models (CCSM; Hurrell et al., 2004), have roughly similar skill in repro- ducing late-20th-century Sahel drought, and propose roughly equivalent explanations of the same, based on Atlantic meridional temperature gradients. Yet the results in the future scenario runs are quite different, producing opposite sign projections of Sahel drought in the 21st century. Hoerling et al. (2006), in their comparative study of the CMIP3 models’ Sahel simulations, acknowledged these differences but could not easily point a finger at any feature of the model that could account for the differences: The differences are not easily attributable to any single difference in physics or process between the models, nor can the community easily tell which, if any, of the projections is the more credible. Tebaldi and Knutti (2007) also address this fact, that intermodel spread cannot be explained or even analyzed beyond a point. This weakness in methodology requires the climate modeling community to address the issue of scientific reproducibility. That one should independently be able to repli- cate a scientific result is a cornerstone of the scientific method, yet climate modelers do not now have a reliable method for reproducing a result from one model using another. The computational science community has begun to take a serious look at this issue, with a considerable literature on the subject, including a special issue of Computing in Science and Engineering (CISE, 2009) devoted to the subject. Peng (2011) summarized the issue as follows: Computational science has led to exciting new developments, but the nature of the work has exposed limitations in our ability to evaluate published findings. Reproduc- ibility has the potential to serve as a minimum standard for judging scientific claims when full independent replication of a study is not possible. Having all of the nation’s models buy into a common framework would allow this research to be systematized. Maintaining the ability to run experiments across a hier- archy of models under systematic component-by-component changes could hasten scientific progress significantly. Research in the science of coupling is needed to make the vision a reality. Existing framework software does not specify how models are coupled. Software standards are one part of the story, but there will remain work to be done to define choices for coupling algorithms, fields to be exchanged, and so on. An effective common modeling infrastructure would include • common software standards and interfaces for technical infrastructure (e.g., I/O and parallelism); 187

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G • common coupling interfaces across a suite of model components of varying complexity; • common methods of expressing workflow and provenance of model results; • common test and validation methods; • common diagnostics framework; and • coupled data assimilation and model initialization framework. ESMF and other frameworks meet many, but not all, of these requirements.Within the infrastructure described above, there is considerable scope for innovation: • different dynamical cores and discretization methods for global and regional models; • different vertical coordinates; • new physics kernels where there is still uncertainty about the physical formu- lation itself (“structural”); and • different methods contributed by different groups based on their interests and specializations (e.g., data assimilation). Finding 10.6: Progress in understanding climate model results requires main- taining a hierarchy of models. There are barriers to understanding differences between model results using different models but the same experimental proto- cols. Software frameworks could offer an efficient way of systematically conduct- ing experiments across the hierarchy that could enable a better characterization and quantification of uncertainty. Data-Sharing Issues The rapidly expanding archives of standardized model outputs from the leading international climate models have heavily contributed to the IPCC assessments. They have also made climate model simulations accessible to a wider community of us- ers as well as researchers. This effort has been led by the DOE-sponsored Program on Climate Model Diagnostics and Intercomparison (PCMDI) at LLNL. While a centralized data archive was initially developed, the volume of model output has grown so much that this internationally-coordinated model data set is now distributed through the Earth System Grid,6 a linked set of data storage locations. Both the PMDI and Intercom- parison and the Earth System Grid are fragile institutions, maintained by a mixture of volunteer effort and a succession of competitive grant proposals, but are a vital backbone for efficiently providing current climate model output to diverse user com- 6  http://www.earthsystemgrid.org/ (accessed October 11, 2012). 188

OCR for page 175
Computational Infrastructure—Challenges and Opportunities munities. The data infrastructure benefits from a “network effect” (where value grows exponentially as more nodes are added; see, e.g., Church and Gandal, 1992; Katz and Shapiro, 1985). It involves developing operational infrastructure for petabyte-scale (and soon exabyte-scale; see Overpeck et al., 2011) distributed archives. This infrastruc- ture development has occurred through an international grassroots effort by groups such as the Global Organization of Earth System Science Portals and the Earth Systems Grid Federation. More generally, needs for data storage, analysis, and distribution have grown signifi- cantly and will continue to grow rapidly as models move to finer resolution and more complexity and the needs of an increasing diversity of users become more sophisti- cated. Furthermore, the scientific research, observational devices, and computational resources are becoming increasingly nonlocal. NCAR has recently built the NCAR-Wy- oming Supercomputer Center to house its supercomputer and data repository. NOAA has placed its Gaea supercomputer at ORNL serving research labs across the country. It is increasingly common for scientists at many institutions to modify and enhance models at remote institutions and to analyze results from other models. The need to use large volumes of remotely stored data places extreme strains on systems and requires systematic planning and investment as part of a national climate modeling strategy. This need is similar to that described in Chapter 5 related to the handling of observational data. This combination of rapidly increasing climate simulation data objects with a more distributed set of supercomputers and data archives requires the climate modeling community to begin to make use of a separate backbone data-intensive cyberinfra- structure, based on dedicated optical lightpaths on fiber optics separate from the Internet, connecting these facilities with each other and with data-intensive end users. These “data freeways” have been developed nationally (ESnet, Internet2, National LambdaRail; see Figure 10.1) and internationally (Global Lambda Integrated Facility) over the past decade outside of the climate community, providing 10-100 gigabit per second (Gbps) clear channels for data movement. Dedicated 10-Gbps optical path- ways enable moving a terabyte of data between two sites in 15 minutes, compared to 1-10 days to move a terabyte on the 10-100 Mbps shared Internet. An example of the use of such supernetworks in climate is the recent NOAA 10-Gbps optical network for moving large amounts of data between key computational facilities (including ORNL and GFDL) and end users. The new Advanced Networking Initiative7 is already devel- oping a prototype scientific network that can operate at 100-Gbps speeds (Balman et al., 2012). 7  http://www.es.net/RandD/advanced-networking-initiative/ (accessed October 11, 2012). 189

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G FIGURE 10.1  N-Wave, the NOAA Research Network. A high-performance network dedicated to envi- ronmental research and connecting NOAA and the academic and state research network communities. SOURCE: N-Wave website, http://noc.nwave.noaa.gov/. Finding 10.7: The data-intensive cyberinfrastructure that will be needed to en- able the distributed climate community to access the enormous data sets that will be generated from both simulation and observations will require a dedi- cated data-sharing infrastructure. THE WAY FORWARD Climate simulation is difficult because it involves many physical processes interacting over a large range of space and time scales. Past experience shows that increasing the range of scales resolved by the model grid ultimately leads to more accurate mod- els and informs the development of lower-resolution models. Therefore, to advance climate modeling, U.S. climate science will need to make effective use of the best possible computing platform and models. To facilitate the grand challenges of climate modeling in support of U.S. national interests (Chapter 4), increased model resolu- 190

OCR for page 175
Computational Infrastructure—Challenges and Opportunities tion and complexity will be required, which in turn will result in the need to exploit enhanced computing power, (i.e., new hardware). Therefore, the committee recom- mends an expansion of the capabilities within the climate modeling community that includes three main elements: (1) a common software infrastructure shared across all U.S. climate modeling groups; (2) a strategic investment in continuing the ongoing de- ployment of advanced climate-dedicated supercomputing resources and in research about how to design climate models to exploit new computational capabilities, based on the common infrastructure; and (3) a global data-sharing infrastructure that is op- erationalized. The data-sharing infrastructure that exists is already vital to the climate modeling enterprise, but it is at risk because of tenuous recognition of its importance in the form of resourcing. Evolving to a Common Software Infrastructure The committee believes the time is ripe for a systematic cross-agency investment in a common U.S. software infrastructure designed in close collaboration with the major modeling centers (e.g., CESM, the Goddard Institute for Space Studies, the Global Modeling and Assimilation Office, GFDL, and the National Centers for Environmental Prediction). There is no reason it could not be globally shared, but the committee limits its discussions here to U.S. modeling centers in line with its statement of task (Appendix A). The infrastructure needs to support interoperability of climate system model components and common data-handling standards that facilitate comparisons between component models developed by different modeling groups, and across a hierarchy of component models of different levels of complexity, including regional models. Within this framework, there is still scope to address structural uncertainty (Chapter 6) by allowing competing representations of processes to be systematically compared. The common software infrastructure will follow coding conventions and standards enabling the construction of hierarchies of models of smaller scope to be run on a broad class of computing platforms (see Box 10.3 for further discussion). While this investment could be justified purely on its potential for allowing more ef- ficient use, comparison, testing, and improvement of U.S. climate models, it is more urgent in light of the upcoming transition to high-end computers that are based on much higher levels of concurrency (Figure 10.2). In particular, it would position the U.S. climate community to make the additional investment to redesign climate models to effectively use new high-end supercomputers, because of the possibility of configur- ing multiple scientifically credible versions of individual model components to run on such systems without enormous additional effort. 191

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G BOX 10.3  SOFTWARE INFRASTRUCTURE ANALOGY TO OPERATING SYSTEM ON A SMARTPHONE The software infrastructure described in this chapter can be thought of as similar to the operating system on a smartphone. The software infrastructure is designed to run on a specific hardware platform (i.e., the phone), and climate modelers develop model components (i.e., apps) to run in the software development to simulate parts of the climate system like the atmosphere or ocean. Right now, different modeling centers in the United States have different software infra- structures (operating systems) that run on different pieces of hardware; this would be like the iPhone compared to the Android, for example. This means that climate model components (apps) written for one software infrastructure will not work with another (i.e., iPhone apps will not work directly on an Android). Ultimately, the vision is that the U.S. modeling community could evolve to use the same common software infrastructure (operating system), so that model components (apps) could be interchanged and tested versus one another directly. This would also mean that when the hardware (phone) advances, the software infrastructure (operating system) can be updated to continue to work with the new hardware without having to completely rewrite the climate model components (apps). The committee recommends that the best pathway for achieving a common software infrastructure involves a community-based decision process. As discussed in Chapter 2 and Box 10.2 (ESMF), efforts to dictate transformations to specific infrastructures have met with less success than those that have come from the bottom up. This is described further in Chapter 13 in the discussion of the formation of an annual national climate modeling forum. The evolution to a common modeling infrastructure will require on- going work, and a working group at such a forum could provide a venue for that work. The evolution to a common software infrastructure will not be without risks and costs. A single infrastructure could inhibit the exploration of alternative approaches to soft- ware design and development. Frontier-scale computing is an evolving problem that could be challenging to adapt to a preexisting community framework. Community- based decision processes require nurturing, have inefficiencies, and require com- promise. Individual modeling centers may not easily be convinced to migrate from their current infrastructure. However, given a decade of experience, combined with a bottom-up, community design process, the committee believes that the climate mod- eling is ready to develop a capable and ambitious common software infrastructure whose overall benefits far outweigh the costs. This evolution can only succeed with adequate sustained resources. It will benefit from the coordinated involvement of the funding agencies to help organize and sup- 192

OCR for page 175
Computational Infrastructure—Challenges and Opportunities FIGURE 10.2  The development of a common software infrastructure will facilitate the migration of mod- els to the next generation of computing platforms and allow interoperability of model components. port the development of the new infrastructure and its implementation in the lead- ing U.S. global and regional climate models, as well as its use in model comparisons and national climate model data archival. The U.S. Global Change Research Program (USGCRP) is expected to play an important role in this effort, having stated in its recent strategic plan that “Promoting the development and widespread use of such frame- works is a central task for USGCRP so as to maximize collaboration, co-development of models, and, ultimately, coordinate integrated research efforts” (USGCRP, 2012). Addressing Climate Model Computing Hardware Requirements As described earlier in this chapter, the future needs of U.S. climate modeling to pro- vide climate data for decision makers and other users will outstrip its current com- puting capabilities. Chapter 13 discusses options for how to address these needs in 193

OCR for page 175
A N AT I O N A L S T R AT E G Y F O R A D VA N C I N G C L I M AT E M O D E L I N G more detail in the context of how they might interact with existing climate modeling centers and regional modeling activities. Operationalizing the Global Data-Sharing Infrastructure Two observations can be made about the distributed nature of climate science today. First, the growth rate of climate model data archives is exponential. Without substan- tial research effort into new methods of storage, data decimation, data semantics, and visualization, all aimed at bringing analysis and computation to the data, rather than trying to download the data and perform analysis locally, it is likely that the data might become frustratingly inaccessible to users. Research efforts under NSF’s Earth- Cube and smaller projects funded by NOAA and DOE/SciDAC attempt to address this, but this is one realm that would benefit from the network effect and consolidation of effort across agencies, and in fact internationally. The challenges posed by a federated global archive demand an equally federated response. Second, globally coordinated modeling experiments have become central to climate science. Climate science and policy-relevant science and decision making are increas- ingly dependent on the results of these experiments. The enterprise is critically depen- dent on a global data infrastructure for disseminating these results. It cannot continue to be developed and run by a dedicated but small cadre of technologists funded by separate, uncoordinated, unstable pools of grant-based resources. European organiza- tions are attempting to do this through enabling an operational European Network for Earth Sciences. (See Chapter 9 for a further discussion of what constitutes “operational” infrastructure.) The committee believes that climate science will continue to make advances based on globally coordinated modeling experiments based on a small but richly diverse set of models from different research institutions. The results of these experiments will con- tinue to be a treasure trove for climate science, climate impacts science, and services for policy and decision support. The data requirements for modeling data described in this chapter and the observational data described in Chapter 5 both support the need for an enhanced information technology infrastructure for climate data. The com- mittee believes that the existing data infrastructure efforts are too important to be allowed to rely on volunteer efforts and less than stable funding. Recommendation 10.1: To promote collaboration and adapt to a rapidly evolv- ing computational environment, the U.S. climate modeling community should work together to establish a common software infrastructure designed to facili- 194

OCR for page 175
Computational Infrastructure—Challenges and Opportunities tate componentwise interoperability and data exchange across the full hierarchy of global and regional models and model types in the United States. Recommendation 10.2: In order to address the climate data needs of decision makers and other users, the United States should invest in more research aimed at improving the performance of climate models on the highly concurrent computer architectures expected in the next 10-20 years, and should sustain the availability of state-of-the-art computing systems for climate modeling. Recommendation 10.3: The United States should support transformational research to bring analysis to data rather than the other way around in order to make the projected data volumes useful. Recommendation 10.4: The data-sharing infrastructure for supporting inter- national and national model intercomparisons and other simulations of broad interest—including archiving and distributing model outputs to the research and user communities—is essential for the U.S. climate modeling enterprise and should be supported as an operational backbone for climate research and serv- ing the user community. 195

OCR for page 175