The DOE 2000 Project
MARY ANNE SCOTT
U.S. DEPARTMENT OF ENERGY, OFFICE OF ADVANCED SCIENTIFIC COMPUTING
Collaboratories are limited only by our imaginations and the time technology takes to advance enough to fulfill that vision. Electronic collaboration is not only an important research tool; it can also leverage resources in today's constrained R&D environment. The U.S. Department of Energy's DOE 2000 Project has therefore been fostering specific collaboratories as well as funding research on networking and communications technologies essential to support collaborations. We are still building the foundations.
Just over three years ago, DOE decided that some collaboration tools were mature enough to warrant a focused program: it was time to put them in the hands of scientists to determine what works and what doesn't. The measure of success would be a positive impact on the way science is done and what it accomplishes. Our hope was that collaboratories would enable scientists to perform research that was impossible before.
The DOE 2000 program was funded in fiscal year 1997. Because collaboratories are aimed at linking people, computers, data, and facilities, the program was designed to reveal how all of these elements could contribute to a wide range of R&D and applications.
The program has three parts. First, there are the R&D projects on advanced computing software tools. These projects use research in applied mathematics and computer science to develop an integrated set of high-performance tools that can simulate complex systems in various disciplines. The intent is that these tools will remain in use over many generations of computer hardware. These tools will represent complex geometries, solve diverse equations, simplify the execution of codes assembled of modules written in different languages, evaluate and enhance the performance of applications codes, and dynamically steer calculations—for example, changing the convergence threshold of a module.
The second part of the program funds a mix of short and long-term R&D on tools for making collaboratories themselves possible. Again, we take the results of fundamental research, this time in computer science and networking, and develop an integrated set of tools to enable scientists to remotely access and control facilities and share data and information in real time. Specific projects include:
- Defining and demonstrating security architecture based on Public Key Infrastructure (PKI) that can protect proprietary data and hence make such data remotely accessible. The same architecture will be designed to provide remote access to those authorized to operate experimental devices.
- Development of differentiated services within the research network that will provide reserved bandwidth to support applications requiring sustained bandwidth.
- Developing a prototype modular electronic notebook that can be used in a number of desktop computer environments. This notebook will enable scientists to design experimental procedures jointly and share their data from scientific instruments.
- Developing such tools as video conferences to manage distributed collaborations. These tools range from those that allow "whoever is speaking" to have "the floor" to those that allow a meeting leader to control the floor.
- Developing advanced techniques for managing the electronic record of the collaboration—that is, the creation of persistent representations of sessions and people that include audio/video, electronic notebooks, and electronic whiteboards.
- Exploring such techniques as virtual reality that enable large groups to work together effectively at a distance.
- A collaborative framework for pursuing these projects will allow all of the tools to interoperate.
The third and most important part of the DOE 2000 program is pilot collaboratories, in which we test, validate, and apply the tools we have developed in partnership with other programs in the Division of Energy Research and other DOE offices. DOE 2000 now maintains two pilot projects:
- The Materials Microcharacterization Collaboratory (MMC) is a partnership between DOE's Office of Basic Energy Sciences and Office of Energy Efficiency and Renewable Energy to provide remote access to facilities that perform electron-beam microcharacterization of materials. Its goal is to furnish a common interface to remote users, both novice and advanced. The facilities, which are unique but complementary, are located at Oak Ridge National Laboratory (ORNL), Lawrence Berkeley National Laboratory (LBNL), Argonne National Laboratory (ANL), the National Institute of Standards and Technology, and the University of Illinois.
- The Diesel Combustion Collaboratory (DCC) is a partnership to develop the next generation of clean diesel engines. This collaboration brings together the same three divisions of DOE plus researchers at three U.S. manufacturers of diesel engines and Sandia National Laboratory, Lawrence Livermore National Laboratory, LBNL, and the University of Wisconsin. These pilot
- projects use tools developed under the DOE 2000 program as well as those available commercially.
Brian Toner, who runs a small research laboratory for surface studies in the Physics Department at the University of Wisconsin in Milwaukee, is a participant. He is developing bioremediation methods for cleaning up transition metal actinide pollutants and has done microscopy work at several locations, including LBNL. About four years ago a group from LBNL asked whether he was interested in participating in Distributed Collaborative Experiment Environments, a program that would enable him to use the Advanced Light Source (ALS) to run his experiment remotely. The ALS lends itself especially well to a remote collaborative environment or virtual laboratory, having been designed to support remote collaborations and operations as technology advances to make this possible.
Dr. Toner was initially skeptical but decided to become part of the team for two years. It didn't take long for him to become an advocate. Today the SpectroMicroscopy facility at the Advanced Light Source (ALS) Beamline 7 enables a large and geographically distributed collaboration, part of a rapidly expanding user community, to make analytical use of synchrotron radiation. Dr. Toner now does his research without leaving Milwaukee, although his graduate students travel to the facility to set up new experiments. Funding for the original project has expired, but Dr. Toner maintains his participation because the ALS tools have changed the way he does his research.
ALS Beamline 7 was designed to provide spatially resolved chemical information at lengths from below 1 micron to the atomic scale, in the case of photoelectron diffraction structural imaging. The facility's capabilities are beyond those of any other in the world, with the possible exception of one or two sites with similar soft x-ray undulator beamlines. Because of the unique capabilities of these instruments, the SpectroMicroscopy Project was conceived from the outset as a rather large collaboration. These machines represent a substantial investment in training, staffing, time, and travel costs, so plainly the success of the collaboratory would have far-reaching implications for users of the SpectroMicroscopy facility, ALS, and synchrotron radiation sources in general.
In the other pilot project, Chaitanya Narula, a research scientist at Ford Research Laboratory in Dearborn, Michigan, is trying to find new catalysts to reduce NOx emissions from diesel engine exhausts. The platinum (Pt) clusters (the catalyst) are supported on titania (TiO2) particles. To be effective, the Pt clusters should be small and uniformly distributed on the TiO2 (titania) particles, which should also be as small as possible. But were they? Ford sent a sample to ORNL's High-Temperature Materials Laboratory to be examined on its electron microscope.
Dr. Narula didn't have to leave his office to get the answer because he can use his PC and the Internet to control the microscope at ORNL remotely and also to obtain advice from ORNL microscopy experts. When he examined the results,
he found that the platinum clusters were 2 to 5 nanometers in size and not uniformly distributed, while the TiO2 particles averaged 20 to 50 nm in diameter. So it was back to the drawing board. Dr. Narula tried a different processing technique and sent a new sample to ORNL within a week. Ford and ORNL researchers examined the new samples together across the Internet. This time the platinum clusters were uniformly distributed and only 0.5 to 1 nm in diameter on TiO 2 particles that were 5 to 10 nm in diameter.
Dr. Narula still needed to find out if the clusters were really platinum. The sample was forwarded to ANL, another member of the DOE 2000 project. Using the Advanced Analytical Microscope at ANL and a three-way telepresence session, participants confirmed that the particles were indeed platinum. All of this was accomplished in a few short weeks rather than months as would normally be required.
The Diesel Combustion Collaboratory (DCC) provides another example of a successful electronic collaboration. For a dozen or so years the major U.S. diesel manufacturers have teamed with laboratory researchers in precompetitive research to understand how to design better diesel engines. The agreements they are working under call for quarterly meetings in which experimentalists and modelers discuss progress and analyze results with representatives from the manufacturers. The problem is that the emission standards of the U.S. Environmental Protection Agency are very stringent, and manufacturers are hard pressed to meet them. The goal for the DCC is to use collaborative technologies to speed up the process whereby the experimentalists and modelers agree on how to improve engine design. Toward that end, a group is working on such tools as Web-accessible data archives, with the appropriate security, and remote execution of computational models.
Early on in the collaboratory, at one of the quarterly review meetings, a representative from Cummins Engine Company showed a slide comparing his analysis with the latest data from a Sandia National Laboratory experiment. The Sandia experimentalist, John Dec, immediately asked, "Where did you get that? That looks just like my slide." The Cummins engineer had taken advantage of the collaboratory's newly installed shared work space to download an electronic version of the slide without having to bother Dr. Dec. This is a simple example, but it illustrates how such techniques save time and effort.
Dr. Dec may have been surprised the first time he saw his data being reused, but he has many reasons to be pleased with similar capabilities. He has generated more than 35 gigabytes of experimental data over the past few years from his combustion rig, but proprietary software, obsolete hardware, and other factors have made simply collecting that information in a form that could be analyzed time consuming. The collaboratory has provided a secure data archive to replace this outmoded approach. Dr. Dec can now access his data from any location with a Web browser instead of making a trip to the laboratory.
Of course, protecting proprietary information in this environment is very important, and new security mechanisms being developed on this project are ad-
dressing this issue. For example, the image library used by researchers in the DCC is protected by a software implementation of a policy-based access system—the Akenti Policy Engine—that uses public-key-based authentication coupled with secure communications.
Instead of sending massive amounts of data to individual routers, the Internet's Multicast Backbone, Mbone for short, routes real-time communications over the Net by distributing and replicating the data stream only as needed, thus efficiently distributing data packets without congesting any single route. Mbone was created by Van Jacobson of LBNL; Steve Deering, then of Xerox Corporation's Palo Alto Research Center; and Steve Casner of the University of Southern California. Mbone technology was used to establish the first multicast video and audio link to the South Pole—between the LBNL and scientists at the U.S. Amundsen-Scott Station—in early April 1998. Mbone video conferencing tools, developed by Van Jacobson and Steve McCanne at LBNL, exchange live sound and pictures between remote locations far less expensively than any other method. This link is permanent, although it works only when a satellite is in the right position. The connection allows scientists in the Antarctic and the United States to jointly manage and work on the Antarctic Muon and Neutrino Detector Array, which uses instrument probes thousands of meters deep in boreholes in the polar ice.
When this link was initiated, school children also got into the act. Real-time interaction via Mbone between students in the United States and researchers at the South Pole was featured in "Live from the Poles," an hour-long television special produced by the National Aeronautics and Space Administration's Passport to Knowledge project. This project, distributed by almost 300 public television stations across the nation and by NASA-TV, is just one example of what this technology has to offer education. Schools everywhere can interact with astronomers at mountain-top observatories, biologists in the rain forest, geologists on the slopes of live volcanoes, oceanographers under the sea, and astronauts aboard the Space Station.
In 1997 one of the members of the MMC, Edgar Voelkl, visited his hometown of Regensburg, Germany, to attend a conference. The conference organizer became quite excited when Edgar suggested operating his U.S.-based electron microscope remotely during the program. Edgar also encountered a lot of skepticism, but he didn't let it influence his plans. The local newspaper announced the remote operation as one of two highlights of the upcoming meeting: "World premier at the university: A highly sophisticated instrument in the American Oak Ridge (Tennessee) will be operated live through the Internet."
On the night of the session the lecture hall was almost filled. It was obvious that many came to scoff, but it was all in vain. Toward the end of Edgar's talk, the connection to ORNL was established and the microscope was used to remotely obtain high-resolution images of gold particles. Astigmatism and focus were corrected live, and the final image was downloaded to a laptop in Regensburg. The connection was great—throughout of greater than one image per second. The
outcome of the session exceeded expectations and surely converted many skeptics that night.
Several factors helped things go so well. The location had good network connectivity: the University of Regensburg is part of a 225 megabytes/second ring that includes the Universities of Nürnburg, Berlin, and Frankfurt. Frankfurt maintains a direct connection into the Energy Sciences Network in the United States, the research network serving DOE scientists to which ORNL is connected via a T-3 link. However, good connections do not necessarily ensure that applications run as expected when these networks become congested. That's why at DOE we are continuing to work on providing differentiated services—or quality of service, to use a more commonly used term—to provide scientists the means to access sustained bandwidth when needed.
As you can see from these examples, the DOE 2000 program has made progress toward the goal of enabling scientists and engineers to interact as if they were physically collocated—sharing data, high-performance computing systems, and instrumentation independent of location. Tools are becoming available, but issues remain, such as hardening these tools, providing interoperability, and assuring availability across a wide variety of platforms.