Developing a Digital National Library for Undergraduate Science, Mathematics, Engineering and Technology Education: Report of a Workshop (1998)

Using the themes described in the project loci section of this report, Jack Wilson, Chair of the Steering Committee, opened the workshop on August 7, 1997, by asking participants to be able to answer the following major cross-cutting questions by the end of the workshop:

Is an NL a good idea for improving undergraduate SME&T education?

Is the proposed NL a better idea than other initiatives that might compete for the same funds?

If the NSF does commit to supporting an NL, what kinds of information and issues will it need to consider so the project can be undertaken efficiently and cost effectively?

Hal Richtol, Director of the Laboratory and Technology Development Section of NSF's Division of Undergraduate Education and Program Officer for an NL initiative, spoke next. He noted that there have been many important innovations in SME&T education during the past 30 years. These innovations have led to a broad spectrum of educational materials and methodologies for use by both faculty and students and an expanded research base to support local and national efforts to improve teaching and learning. Richtol noted that while much of this progress has been catalyzed by sponsoring agencies (both public and private), a great deal of progress also has been the result of dedicated individual faculty members working without formal support to improve their courses and programs, and some of their work goes unrecorded. Given that many innovations are never disseminated, Richtol said that the NSF felt that there would be great value in developing an electronic system that could serve both as a central repository for existing materials and as a reliable gateway to other collections of materials and ideas. This system also might serve as a forum for both faculty and students for the submission of project materials or information for formal review and evaluation—a venue for the systematic generation and preservation of informed discussion and review of educational material and a quality control agent for what is posted to the system. Faculty might then sample from these resources to determine suitability for their programs and institutions, communicate their findings to broader audiences, and participate in far-reaching discussions, current thinking, and debate about undergraduate SME&T education. This system also could prove very useful both to the K-12 educational community and to the concerns of business and industry as they work with the undergraduate sector to address such issues as school-to-work transitions, partnerships between these communities, and communicating the value of undergraduate SME&T education to the public at large.

Richtol reiterated Wilson's message that the NSF needs advice from workshop participants about whether or not to proceed with the concept of an NL as an educational infrastructure for faculty, students, and the public. If the NSF does proceed with this project, how should it be developed and by whom? What kind of entity would administer and manage an NL so that it might eventually become a self-sustaining enterprise? Richtol emphasized that consideration of these complex issues will go well beyond this workshop and any developmental efforts that might occur over the next year, five years, or during the next decade.

Following introductory remarks, the remainder of the morning of Day I was devoted to formal remarks by eight workshop participants. The speakers had been selected by the Steering Committee and NRC staff to provide a variety of perspectives. Some speakers had written commissioned papers; others had not.

These formal remarks were offered in two plenary sessions of four speakers each. Each speaker answered questions following her/his presentation. General discussion ensued after each plenary session. ⁴

Session I: Presentations

The first session presentations were given by:

William Arms, Corporation for National Research Initiatives

Miriam Masullo, T.J. Watson Research Center, IBM Research Center

Michael Raugh, Interconnect Technologies Corporation

Lee Zia, Department of Mathematics, University of New Hampshire

William Arms spoke on three themes:

First, although he believes there is a need for a resource such as a digital National Library, Arms concluded that the commissioned papers collectively had not yet made a convincing ease for how any NL initiative could benefit undergraduate education and urged participants to focus on this issue during the workshop. He pointed out that some of the commissioned papers suggested that libraries traditionally have not been central to undergraduate science education in many institutions of higher education. In trying to decide how to make the case for an NL, Arms provided two examples of how technology has added value to undergraduate education, sometimes in ways that were unexpected and unanticipated. In both examples (a distance learning program at the British Open University and the introduction of technology to all facets of undergraduate education at Carnegie-Mellon University), success was realized, Arms said, because the faculty at both institutions (the users) took control of the projects and shaped them to fit user needs. In both eases, faculty emphasized how the technology could best be employed to abet teaching and learning. Subsequent studies have indicated that these tools also have enabled teaching and research to be more closely conjoined than in the past because faculty have found they can use the same tools for teaching as for research. Arms emphasized that the undergraduate SME&T community must articulate a vision of how computing can be best employed in undergraduate education and that such a vision could be best articulated by teaching faculty. He noted that teaching faculty were underrepresented at the workshop.

Second, Arms addressed the issue of a "library without collections." He suggested that, unlike contemporary, libraries and electronic databases where information is housed in a building or stored and distributed from a central computer, an NL project should look seriously at a structure that would guide users to collections of materials that are located and maintained elsewhere. Pointers might refer to commercially available materials, online collections, curricula (e.g., the online collection of curricula provided by the Mathematics Department at Dartmouth College⁵), and course notes and modules. The function of an NL would be to identify, evaluate, review, and index these materials.

Third, Arms emphasized that developing and maintaining an NL will be technically difficult. If an NL is to provide a high-quality service to users, it will be absolutely necessary to find people who are dedicated to the service aspects of putting the technology together. Users must define how the technology is employed; technology must not dictate how a library can and cannot be utilized.

Miriam Masullo continued the discussion of the role of an NL with respect to teaching and learning. She agreed with several authors of commissioned papers that the term "library," as applied to an NL project, places limitations on what this entity might become and how it might evolve. In contrast, if a digital library is defined as "a class of tools that includes capturing, authoring, storing, managing, searching, organizing, retrieving, indexing, sharing, and collaborating, we are probably talking about computer science and several other disciplines as well."

Masullo next described some of her experiences with digital libraries in K-12 education in the United States and similar projects around the world. Her experience convinces her that such entities will be butt in unexpected places and in the near future because the enabling technology is now available. She thinks that it is justified to associate a sense of urgency with the proposed NL project.

However, Masullo emphasized that the focus should be on how to organize information in this new medium. If traditional transport and dissemination mechanisms are changed, new problems could emerge that could frustrate users and discourage use of an NL. In short, much more attention should be paid to how people acquire and use information from computers and networks.

Masullo also raised the issue of equity of access to information in an NL and how information in an NL might be distributed to or shared with other countries. After working with UNESCO on such issues, Masullo said that while the Internet is probably the delivery vehicle of choice for the United States and more developed nations, it may not be the avenue for distributing information to other parts of the world. However, if information from an NL is to be globally accessible, then transport parameters must be changed. If these parameters are changed, many other features of the system also may have to be altered, resulting in a multitude of technological issues to be resolved before the System can be used. Different delivery mechanisms also might have a severe impact on the quality of service, again raising the important issue of equity of access. Masullo disagreed with the notion that the challenge of the "have-nots" will be resolved with more sustained connectivity because she has not seen this happen to date with today's network technology.

Masullo also urged the group to think about what the term "national" actually connotes. Is an NL to be constructed primarily for use by people in the United States or will it be our contribution to the international dissemination of information electronically? How inclusive would an NL be? Many other countries are investing heavily in similar kinds of information infrastructures. If the United States means to be first in this new means of disseminating information, it also might eventually be left behind.

Masullo also emphasized that other countries (e.g., Singapore, Brazil, Chile, Malaysia, and Egypt) are investing hundreds of millions to billions of dollars each in their projects and choosing to focus on dedicated education infrastructures rather than "information superhighways." She advocated that the United States consider this emphasis as it develops its own facilities.

Michael Raugh described himself as an academic mathematician "who happened to wander on to the Internet one day" and who then formed a company to create digital libraries. His company currently is building a digital library for research and development in aviation safety information for the Federal Aviation Administration. Raugh reiterated Arms' point that it is difficult to build a digital library. He added that librarians already know this. Whether traditional or digital, libraries are complex structures that involve curating content, indexing, cataloging, abstracting, and providing for preservation. Raugh said that the issues of preservation of materials and other similar long-term issues have not yet been addressed seriously by the cooperative federal agency initiative that is involved with constructing the other aforementioned digital libraries.

Lee Zia addressed Arms' challenge to articulate a need or use for an NL. He pointed out that some of the commissioned papers suggested that faculty would be the primary users of this resource. Other papers stated that students should be the targeted clients. Zia felt that targeting an NL either to faculty or to students establishes a dichotomy that should be avoided since both faculty and students are two groups of learners. By focusing on the needs of learners, the dichotomy between serving the needs of faculty vs. students disappears.

There are examples of materials that achieve what Zia envisions, but he also agreed with Richtol that much of this work is being carried out by individuals. Technology has allowed people to rethink not only what they are teaching but also how they are teaching it and, more importantly, how their students are learning and interacting with the content. He also agreed with commissioned paper authors and previous speakers that the use of the term "library" evokes an image that is hard to alter, and he advocated finding an alternative term.

Zia emphasized that the proposed project, if properly designed to support learning and focus on the needs of diverse learners, could provide

unprecedented opportunities for all students, not just the top students in a class, to engage more in the inquiry process and in research.

Zia then provided an example of how he might use information technology in his classroom to enhance student understanding of complex concepts such as differential equations—courseware that would allow students to venture well beyond reading about particular concepts and descriptions of the equations involved. With the courseware he envisioned, students could manipulate the initial conditions and parameters of the equation and instantaneously see the consequences of their manipulations. They could then apply their work to other, related systems in biology or chemistry to better understand how differential equations are employed in those other disciplines.

However, such outstanding programs require an enormous commitment of creative time and effort, not to mention technological manipulation so that they will operate on a variety of hardware platforms. Because such resources still are not widely available (and are seldom evaluated systematically for their ability to improve learning), not all students may have access to or benefit from them.

Are faculty members willing to make the required investments of time and effort to produce and use such materials? Will students use and benefit from such efforts? Zia did not offer specific answers but, rather, closed by saying that he believes there is a definite need for this resource and that the NSF should continue to pursue the concept of an NL for undergraduate SME&T education.

Session II: Presentations

The second session presentations were given by:

Mary Case, Office of Scholarly Communication, Association of Research Libraries

Michael Lesk, Bellcore

Francis Miksa, Graduate School of Library and Information Science, University of Texas at Austin

Nisha Vora, Association of American Publishers

Mary Case began by stating that if the agenda for an NL is not broad or visionary enough to stir people to create and use this facility, it is unlikely that any agency will be able to overcome the challenges and face the issues required to get the job accomplished. Case sees an NL as an entity that will 1) improve SME&T education; 2) attract more students to careers in these fields; and 3) by virtue of the first two, strengthen the United States' position in the global marketplace. However, she added that it also is important to develop an educated citizenry in SME&T to understand the importance of research. Case said that any NL initiative should take into account the billions of dollars that colleges and universities already have spent in computer hardware, software, and networking and find ways to use that existing infrastructure to better educate undergraduates.

Case pointed to the general agreement that collaborative activity is one of the most effective ways for students to learn and noted that such collaborative learning could take place in a research setting. In fact, many in higher education are promoting research by undergraduates. Given this, an NL could serve in part as a learning laboratory for undergraduates. It could be an active environment that provides resources, tools, and collaborative opportunities to support teaching, learning, and the creation of new knowledge by both faculty and students. The resources that might be incorporated into such an NL include primary research resources, raw data, published literature, reference sources, courseware, interactive modules that allow users to manipulate data, computer-aided design, lab simulations, and virtual reality applications. Case advocated an NL that supports multimedia, makes research tools available electronically, provides tools for users to create papers and courseware, and allows users to videoconference. She added that the enterprise is not worth pursuing if it does not take advantage of the interactiveness of these resources.

Meanwhile, the library community continues to struggle with many of these same issues. Case said it will be critical to bring people with this experience and perspective into discussions about the nature and course of development of an NL. Construction of an NL also must be examined in light of the funding crisis in higher education. The Council for Aid to Education predicts a $38 billion shortfall in funding for higher education by the year 2015. This shortfall, coupled with spiraling costs and the volume of new materials in SME&T that libraries must purchase, will have a grave impact on how universi-

ties organize themselves, how teaching is done, and what the higher education and library communities can afford to do.

An NL also must address other legislative and technical challenges. Likely to contain both public domain and proprietary resources, an NL should nonetheless work to insure that all materials associated with it are available for all educational purposes. An NL system must recognize the need to compensate authors and publishers, to allow robust educational and research uses, and to keep all of this “affordable." Real. issues of intellectual property rights, copyright, and other legal challenges will need to be addressed before materials are made available to users through any new NL system.

Finally, Case stated that construction of an NL must accompany cultural changes in higher education so that the system of promotion and tenure values teaching as much as it does research and other types of scholarship. As with scientific research, the realm of higher education needs to find ways to evaluate the contributions of individuals in a highly collaborative environment. In many ways, these types of social and cultural issues can be more difficult than the technical ones, Case observed. Maintaining peer review and having support from national organizations like the NSF, professional and scholarly societies, and approval by colleges and universities will be very important to this effort, she added.

Michael Lesk began his presentation by displaying recent data from NSF on the number of bachelor's degrees awarded in different areas of SME&T (e.g., see Figure 1, from National Science Foundation, 1996a). With the exception of the life and agricultural sciences, the number of degrees awarded in. engineering, mathematics, and computer science has been declining in recent years. The number of degrees awarded in the clinical sciences has remained low. Furthermore, most of the advanced students in SME&T are not coming from U.S. undergraduate institutions. For example, more than half of the Ph.D.s awarded in mathematics have gone to individuals who are not U.S. citizens.

Many people already are using what amounts to a digital library. It is called the World Wide Web. Importantly, Lesk said, arguments remain about how best to use the material one can make available in this way. Computer-aided instruction traditionally has been factual in nature or has encouraged drill. Most researchers agree that students learn better with applications that enhance creativity and make locating information easy. Bellcore has performed studies showing that people who have to answer questions to problems can do so 25% faster and 25% more accurately if they search a digital book rather than a paper text (Egan et al., 1991). Other, similar studies have measured student learning using electronic text and hypermedia (e.g., Friedman et al., 1989; Marchionini, 1994), although very few definitive reports exist about the efficacy of this process. The upshot is that an NL may not be able to achieve its mission simply by providing scans of information from textbooks; rather, new types of materials will be needed, as well as more usable and searchable traditional materials.

While some have predicted that the cost of digitizing information will become prohibitive, Lesk reported that current prices for scanning books range from 8.5 cents to 40 cents per page ($25 to $120 for a typical SME&T book). More important than cost are educational content and benefit. Is it educationally beneficial to digitize existing materials? Perhaps, Lesk said, since undergraduates resort to the Internet so frequently to find information. Lesk advocated that underlying any discussion of digitizing and storing information should be the question of whether doing so will enhance education and whether we can measure that enhancement.

Francis Miksa began by saying that after having read the commissioned papers and listening to comments all morning, he had arrived at a "state of ignorance" about the proposed NL project. The more people talked about it, the less he felt he knew about it.

As a librarian and professor of library science, Miksa said he found the prospect and reality of

Figure 1.

Science and engineering bachelors degrees awarded, by major field: 1966-1994. 

Reprinted from Chart 2, National Science Foundation (1996a).⁶

delivering information electronically tremendously exciting. He characterized it as an idea whose time has come, whether people like it or not. However, there is a difference between electronic delivery of information per se and the concept of a digital library, Miksa said. The latter is informed ultimately by the idea of the library, which has a long history. In its present context, a library is a function of a series of value-adding activities that are performed with respect to media, information, and the housing of various media containing information. Traditionally, librarians have performed these activities. In short, a library is not, by any definition, an inert mass of things that have been collected and indexed; rather, it is a process of adding value to that mass.

In a digital library, the roles of librarians include value-adding activities such as selecting information sources from among all those available and acquiring them in digital form, organizing those sources of information into an intellectually cohesive structure, providing assistance to others in finding and using such resources, and preserving information as required. Perhaps the newest challenge imposed by digital libraries, Miksa said, is the potential for combining the concepts of publishing and electronic commerce. Another challenge to consider for the proposed NL project is the need for it to be constructed in a manner that supports its primary purpose of improving undergraduate SME&T education specifically and the educational process in general. All other parameters must flow from that purpose. This premise will cause such questions to arise as, What actually goes on in undergraduate education? What kinds of information-bearing entities are used in that process? How are they used?

Miksa said that what should be happening in undergraduate education is enhancement of the process of discovery by students. He noted that this notion is similar to the idea expressed in John Jungck's commissioned paper of a direct and conscious confrontation with ignorance and Harold Billing's idea of students as mavericks who contribute to the digital library in addition to using it (see Appendix A). Despite his opinion, however,

Miksa cautioned that an NL whose primary focus is on discovery might attract only a small number of users. Given the general temperament of people when they are students and the myriad conditions under which they attend school and pursue education, the discovery approach alone may be insufficient to meet the needs of this diverse group. Undergraduate education should be structured to allow students to experience the discovery of new knowledge.

Nisha Vora explained that her role at the American Association of Publishers (AAP) is to work with anti-piracy enforcement and copyrights. She expressed concern that with the explosive growth of information technology, a project like an NL for undergraduate SME&T education would need to be thoroughly informed about and respect copyright law and licensing. She advocated discussions about how to balance the interests of teachers, students, and proprietors when such a project is developed. She also advocated the need to look at existing laws and models for guidance.

Vora addressed the misconceptions that copyright holders want to “lock everything up" and do away with the doctrine of fair use or that educators only want materials to be free of charge. She returned to the idea of balance, with publishers and higher education working together to resolve such misconceptions. Vora said that opportunities exist to provide vast amounts of resources and tools and to facilitate creativity and communication among students, teachers, and other scholars around the world.

Vora noted that legislation is being proposed and negotiations are going forward regarding issues of database protection and fair use. She said the publishing community has interests on both sides of these issues because publishers that create database directories are pulling their raw data from other sources. Thus, both publishers and authors are users as well as holders of copyrights.

Vora mentioned that AAP is addressing a problem that affects most electronic databases and libraries: keeping URLs current. In collaboration with William Arms' group (CNRI), AAP is developing a digital object identifier, an electronic "license plate" that stays with a digital object and allows businesses and others to identify and track the object wherever it moves on the Internet. This device, if successful, could be immensely important in the maintenance and curation of information in an NL for undergraduate SME&T education.

In closing, Vora emphasized the need to promote the incentive for creativity and development of new materials while at the same time protecting what already has been developed.

The following questions and discussion points emerged after the plenary presentations on Day 1.

Is there a need for an NL?

William Arms was concerned that all of the discussion to this point had focused on what an NL for undergraduate SME&T education could do for other people, not the participants themselves. Were participants convinced that they could be better teachers if they had ready access to an NL? Robert Lichter addressed Arms' concern by stating that not enough potential users of an NL were present at the workshop. ⁷

Elizabeth Dupuis was uncertain about who would use this proposed NL and whether it actually would improve undergraduate education in ways that other programs and resources could not. She pointed out that some workshop participants saw this as a resource for faculty, while others felt that students should be the primary beneficiaries. What is placed into a library will, in large part, depend upon the resolution of this issue, Dupuis maintained.

Lorraine Normore emphasized the importance of knowing who the users of an NL for undergraduate SME&T education will be and how they currently obtain information. Science students today use textbooks as primary sources for finding information. In contrast, people at the graduate level and beyond do not rely on textbooks for such purposes. These people are using the Internet and other information sources to generate new ideas. There currently is no common mechanism for getting undergraduates to work as graduate students and others do. Rather than saying that an NL should contain information from or pointers to textbooks, we should work to change the process of

learning so that we foster the kinds of thinking and work habits that trained scientists and technologists ultimately will need to use.

Francis Miksa urged NSF to proceed modestly with the project, at least in the near future. Although an NL is a wonderful goal, Miksa outlined an alternative project. He suggested focusing on mathematics, engineering, and technology as central loci for this initiative. Faculty, students, librarians, and computer technologists all would be key players in establishing such an NL initiative. These groups would work together within disciplines to catalog (with editorial comment) information that is available and appropriate to the discipline, whether it is available on the Internet or not. Tools would be employed to preserve Internet materials that are deemed worthy of preserving, to catalogue and track them automatically, and to remove more ephemeral materials. These groups would develop standardized vocabulary, thesauruses, and syntax methodologies for their disciplines so that users could find information more accurately and easily. The project also would proceed as existing libraries do to sort, catalogue, and disseminate information.

Amanda Spink pointed out that ongoing experiments with digital libraries on individual university campuses could inform the present project and ultimately be integrated with it. Jack Wilson agreed, saying that this project should tie into those other projects being sponsored by the NSF and private foundations.

How large is the potential audience of users?

Roberta Lamb urged the workshop participants to think beyond faculty and current undergraduates as the primary users of an NL. She said that an NL also should be useful and accessible to continuing learners. Because many people reenter the realm of education after their initial formal education has ended, these students blur the boundaries between learners and workers and between teachers and learners. Ruth Seidman also emphasized the importance of constructing an NL to include materials that will transcend the undergraduate years and be valuable to people who wish to revisit subjects or explore new topics later in life, especially since careers and disciplines change often and rapidly.

Gordon Freedman reminded the workshop participants that by the time any NL initiative is well established, it will be serving a primary audience that is 12 years old today. Therefore, the project must account for the ways in which these students will work when they reach college age. The spectrum of students is very broad, ranging from those who will be able to create three-dimensional models on computers to those who think that they will not be able to participate in our society at all. Taking into account the current status of K-12 education and the students who are in it now will be vitally important to the future success of an NL.

Content of and access to the proposed NL

Michael Lesk suggested that many undergraduates now rely on the World Wide Web for most or all of the information that they research. Thus, we should ask not whether we will have a digital library for undergraduates but what its content will be. What would we like to see in an NL that is better than what is on the Internet now? Students will use online information, but which information will they seek out?

Gordon Freedman said that the primary issue is less about content than an access system that is sufficiently standardized and universally available so that any high school teacher or college professor could find desired information. What universality can be applied to an NL so the least amount of time will be spent in acquiring appropriate and relevant information? Similarly, Nabil Adam noted that developers of materials appropriate for the proposed NL initiative will need to have development tools that allow them to spend most of their creative energies on content rather than on issues of a technical or technological nature.

Edward Fox said that at his institution and all over the world, people are creating rich curricula and learning tools at many different levels, ranging from some that are commercially available to local innovations that could be adopted and adapted elsewhere. Students at his institution have begun to depend on these resources, and data collected there indicate that these resources are effective for enhancing student learning. A major question is how such resources can be distributed. Some authors of materials refuse to make them available without compensation or to provide them to entities that do not have mechanisms in place for peer review and evaluation.

Wayne Wolf felt that there should not be a "go/no go" decision from the workshop about whether to establish an NL or what it should look like. The proposed NL should have some uncertainty built into it (as do conventional libraries) so that students will learn that there often is no single answer or solution to a problem. Students then would need to deal with information as professional scientists and engineers do. Ronald Stevens expanded on this point, saying that another potential use for an NL would be to point out what is not known. For example, an NL could point to emerging concepts, disagreements, and conflicts in new disciplines. A compilation of such emerging information could be a great resource for researchers and others who want to examine "cutting edge" information about specific topics or scientific disciplines.

Wayne Wolf said that an NL that served to point to repositories of books available in other libraries would not be very useful. Rather, an NL should help users find information about emerging new topics that are not readily or easily available elsewhere. Students find such topics very interesting, and faculty should have a resource that permits easy access to such supporting materials. While some of this kind of information already is available on the Internet, an NL could formalize its indexing and evaluation. An NL also should make sure that the users substantially benefit by computerized access to information. Searching speed should be high and access rapid. Effective dissemination tools are important for the proposed NL, Wolf said.

Jack Wilson expanded on Wolfs comments by saying that another advantage of electronic media beyond rapid access is the ability to offer continuous annotation and review of materials. Such interactivity among users is very difficult to achieve in a traditional, text-based library setting. As Zia mentioned during his presentation, there also should be opportunity for users to interact with and change the parameters of the materials they are using.

Clifford Lynch spoke on the issue of undergraduate students both as creators and users of content. He questioned how much dynamic, meaningful content undergraduates, especially those at the beginning of their undergraduate careers, actually could produce. Lynch also pointed out that as it is structured currently, much of higher education does not stress such modes of teaching and learning for the vast majority of undergraduate students. Much current learning is textbook-based. Given that cultural change in education often is slow, Lynch asked how an NL could be constructed to enhance the kinds of educational experiences that are now offered to most undergraduates. Jack Wilson responded that, at his institution, teams of undergraduates, graduate students, and faculty author electronic materials. However, undergraduates do much of the work. Graduate students and faculty may be responsible for quality control and overall direction of the project, but undergraduates play important roles in the process. James Davis said that when he was at Cornell University, undergraduates in computer science did not create content, but they provided public, electronic annotation of existing materials and the courses in which they were enrolled. Student commentaries also included discussions of the processes of learning.

Christine Borgmann again raised the issue first articulated by William Arms about building an NL that accommodates the ways in which users would actually employ the resources. She emphasized that surprisingly little research has been done at the nexus of information seeking and problem solving. However, an NSF-sponsored conference held a year ago examined the social aspects of digital libraries, such as learner-centered designs, information life cycles, and incentives for using, preserving, and creating information (Borgmann et al., 1996). Other initiatives similar to the present one are under way and are addressing similar issues. How can this workshop carve out a specific niche for undergraduate SME&T education without duplicating other efforts? The current project, if funded, must coordinate its efforts with those other projects and learn from their research and experience. If the community waits to move forward until it resolves how students actually will use information in an electronic environment, the proposed NL for undergraduate SME&T education might never be built.

Robert Lichter reminded workshop participants that we cannot predict what information will be important to archive⁸ for use 100 years from now,

and so we must build this uncertainty into the equation of what is selected for inclusion in an NL. Hard and fast decisions should not be made that would limit what is going to be important for the next generation of educational leaders who are today's 12 year olds (or even younger), and whose focus on teaching and learning is likely to be very different from ours.

Tora Bikson countered that while people tend to view archivists as individuals who try to keep and preserve everything, archivists actually work systematically to determine what is and what is not worth keeping. For instance, when a new library is established or materials from a new area of knowledge are collected, archivists may apply a generous retention policy to all of those materials for a few years. These materials may then be evaluated to determine what is worth retaining in the long-term. Such decisions are typically made on classes of items rather than on individual items. Bikson suggested that similar procedures be applied to materials in an NL. For example, she thought that annotations by students about the efficacy of a particular learning process might be valuable for several years but then could be removed as the process to which they refer evolves and new comments are added. Decisions about selective long-term retention are particularly important for an NL because of the cost of maintaining older digital materials in formats that can be accessed with contemporary software.

Is the proposed NL really a library?

Francis Miksa suggested that a useful analogy for the Internet is a publishing realm or publishing empire rather than a library. A library makes informed decisions about its own physical collections and catalogs, as well as connections to other paper and electronic resources. A library is not the same as a publishing realm. Thus, it would be helpful to all concerned to develop different terminology to describe this proposed NL. Barbara Polansky expanded on an idea in James Keller's commissioned paper by agreeing that we should focus on encouraging creative thinking rather than simply on constructing a library. She suggested that this entity be seen more as a national electronic resource locator to materials in many realms that helps users gain access to the materials themselves or to the authors of those materials. Peter Graham agreed, saying that the library community has been looking at very similar issues involving the blurring of the distinction between how libraries use information and what is happening outside the control and influence of library structures. Graham felt that the entity at hand is less a library than what might be called "a reserve book room in electronic form." In short, an NL should be more a mechanism and structure for accessing information.

Timothy Ingoldsby countered that an NL should serve as more than a pointer to other sites and sources. It also should serve as a "repository of last resort" for projects that are especially innovative or on the cutting edge of a discipline or pedagogy but not yet developed sufficiently to attract a commercial publisher or major funding.

Robert Lichter, as president of a private foundation, said that an important additional function of an NL would be to allow educators to realize that many good ideas and materials are already available for use. This knowledge alone could save potential developers of materials from "reinventing the wheel" and significantly reduce the amount of money expended by funders on projects that replicate earlier efforts. Lichter felt that this is a non-trivial issue that has to be factored into the economic equation of an NL as well as into resolution of issues of content, evaluation, and access.

Tora Bikson emphasized the need to pay careful attention to acquisition and evaluation processes. Currently, it is very difficult to search large databases or electronic clearinghouses unless the user is armed with sufficient information about references or keywords before beginning a search.

What is the meaning of "national" in NL?

Christine Borgmann revisited Miriam Masullo's discussion of what is implied by use of the term, "national." In central and eastern Europe, individual governments view national libraries as their nation's contributions to the world. Is the workshop's view of an NL that of an entity by and for the United States or are we actually thinking about building the United States' contribution to the international information infrastructure for undergraduate SME&T education? Edward Fox replied that government agencies such as the NSF have provided millions of dollars for the exploration of educational and research components of digital libraries and that while a "national" digital library should be international, the United States must play an active leadership role in the process. Any NL initiative also must be for the national benefit of the United States. In another take on the meaning of "national" in this context, Borgmann added that the term implies that other agencies with overlapping interests in education (e.g., FIPSE, the Department of Education's Fund for the Improvement of Post-secondary Education) should team with the NSF to pursue their parallel goals and objectives.

Technology/economic issues

James Davis expressed concern that many of the programs and other materials either available or being written now will not be usable in three years because of rapidly changing technology. It is difficult to think of a library that contains content that "self-destructs" at every release of a new computer operating system. Richard Furuta emphasized that if the proposed NL is to be established, the planners must work with existing technology that can serve all potential users over the next several years while building a system that still will provide the same services on computer platforms 20 and more years from now. On the other hand, Robert Lichter urged that the technology associated with an NL be dictated by the needs of users rather than vice versa.

Keith Stubbs asked what role Internet II⁹ might play in the development of an NL. Since it is supposed to be accessible only by institutions of higher education, placing the proposed NL on this platform could immediately limit what materials go into it. Internet II also would create problems of access for users such as lifelong learners and those involved with continuing education. While undergraduates can put anything they author onto Internet I, their access to Internet II might be more restricted. Stubbs concluded that the development of Internet II needs to be monitored carefully for its potential impact on an NL for undergraduate SME&T education.¹⁰

Robert Panoff pointed out the high costs of digitizing materials that are now in other formats. Moreover, digitized materials have to be placed into a format that can be readily searched and that is reliable, robust, and accessible to all people through existing hardware and software systems. Tora Bikson expanded on this concern, saying that many organizations that are now preserving documents digitally are finding that maintenance costs are great because of the constant need for technology

upgrades (e.g., conversion) to retain these documents in an interactively usable, manipulable form. Another serious problem is maintaining dynamic links and pointers among documents. Currently available technology does not make this task easy or inexpensive.

Harvey Keynes said that although many materials are available on the Internet, not much is developed to the point that faculty can use them well or on short notice in their classrooms. Getting faculty to create useful, content-rich materials will require external funding so they can devote large blocks of time to their projects.

Clifford Lynch again raised the issue of whether an NL should commission the creation of new content or organize and point to existing materials. If the decision is for the former, then much of the financial support from the NSF and elsewhere must be devoted to producing this content, at least toward the beginning of the project. Some examples of attempts to create content already exist. The NSF-sponsored Synthesis Coalition allows engineering faculty access to the National Engineering Education Delivery, System (NEEDS) database of objects and materials that they can use in their courses.¹¹ This project has shown that it is costly to translate such a concept into usable content because faculty have had to invest a lot of time to adapt the material that was made available to them. Lynch expressed concern about the problems with sealing up to produce content for an NL that covers all areas of SME&T. Brandon Muramatsu agreed with Lynch's assessment of the problem. Muramatsu has been involved with a project to develop a database of engineering courseware (NEEDS). One question has been, How does another instructor or student take a piece of courseware and easily adapt it to her/his own use and teaching/learning objectives? Muramatsu said that a large problem has been the lack of instructor's guides and clear articulation of the learning goals of the software. Without such supplemental materials, much of the content of an NL may have only limited use and success in enhancing teaching and learning.

Tora Bikson agreed that even well planned interactive materials that are produced de novo by one person are difficult for others to use. Bikson said that RAND research on building educational resources for teaching and learning high school science suggests that it is best to create new materials on top of existing, common software packages that then are used in a variety of contexts. People who are familiar with the basic package will not have as many problems in getting the new innovation to work as those who are unfamiliar with the software platform. In addition, both written and electronic documentation are more likely to be provided with commonly used platforms, whereas educators are reluctant to spend the time and effort producing such documentation for their own software creations. Finally, this approach enables students to learn authentic skills—ones that transfer to veridical contexts of use. An educational interface developed for a standard Geographic Information Systems, for instance, allowed students to master some of the inquiry techniques employed by social demographers (McArthur et al., 1995).¹²

Intellectual property issues

Robert Panoff expressed concerns about how copy-fight law might impede the development of content for an NL. There are many examples of students authoring materials that could be used in an NL, and they should be encouraged. However, because students are transient at an institution, their contributions raise concerns about intellectual property rights. Even if students create materials, faculty ultimately must take responsibility for the use of these materials in their classrooms. If the proposed NL is to be a dynamic entity, Panoff said that ways must be found for users to take, use, and mold available materials without having to be concerned about who owns the intellectual property involved. When and how should materials developed by others be made available for fair use?

Shamkant Navathe revisited the issue of archiving features of an NL with respect to copyright and other intellectual property rights. He proposed a multitiered architecture for an NL, where one tier would archive journals, conference proceedings, working papers, and reports. This tier would be subject to all copyright and other laws governing

intellectual property. Other tiers, or layers, might contain unreferenced materials and possibly "exploratory materials" made available for use and revision but not subjected to the same level of legal scrutiny or archived as meticulously as materials in the first tier.¹³

Richard Furuta suggested that issues regarding copyright, fair use, and intellectual property transcend this particular project. They are so important and encompassing and have such widespread commercial implications that they will be addressed in other realms. This initiative should track those efforts and plan the development of an NL to take into account and incorporate the legal precedents that will inevitably emerge regarding these issues.

Edward Fox was pleased that publishers were well represented at the workshop. Many digital library conferences in the past have had no representation from the publishing community. Fox suggested that the NRC broker a highly publicized collaborative agreement or arrangement between universities and publishers to work together on resolving the issues of fair use and intellectual property rights that were raised at this workshop. If an arrangement were struck, the proposed NL could tap into a vast resource of materials that is now largely unavailable, such as the information contained in the more than 200,000 graduate and undergraduate theses produced in the United States. each year. With support from FIPSE, an initiative is currently under way to build a national digital library of theses and dissertations. ¹⁴

In counterpoint, Ronald Stevens said he has heard that approximately 80% of people who earn a Ph.D. never publish their dissertations, which calls into question the usefulness of at least some of the materials that are being produced. In addition, the most prestigious journals typically accept only between 10% and 20% of submissions. So, professional publishing is a mature field that selects the best materials submitted. If the proposed NL becomes a new professional model where many more additional contributions are included within it, the overseers of this resource must worry about maintaining the quality of information that becomes available. This issue also applies to teaching materials. If virtually everything is made available, will teaching suffer as a result? Any NL initiative must consider what kinds of standards for quality and rigor it will set before accepting materials.

An NL and the culture of higher education

Francis Miksa asked how the creation of teaching materials intersects with the tenure process at post-secondary institutions. At his institution, there is "slow movement" toward giving credit for the creation of teaching and learning materials. Is this a trend at other colleges and universities? Jack Wilson responded that much work is now going on in this area. Rather than teaching per se, Wilson emphasized that the discussion at hand is really about professional development and the recognition of this kind of work as another form of scholarship. Wilson agreed that the reward system has to change.

During the first day's plenary session, participants requested a straw poll about whether or not NSF should support the establishment of an NL for undergraduate SME&T education and what its purpose might be. The resolution on which the vote was taken read as follows:

The outcome of the vote was 25 in favor, 1 opposed, and 16 undecided. Jack Wilson interpreted the vote as indicating that participants are committed to having NSF do something in this realm but that they are not completely certain what should be done. It should be noted that a second straw poll was not taken at the end of the workshop.

Four break-out groups met during the afternoon of Day 1 of the workshop. One group each focused on "Curricular, Pedagogical, and User Issues" and "Economic and Legal Issues." Two groups considered "Logistic and Technology Issues." Each breakout group appointed one or more members to report their group's discussions and findings.

Given the issue discussed earlier in the day regarding whether this project should be considered a library in the traditional sense, this break-out group worked to find a different name for an NL.¹⁵ They also discussed issues of audience, the kinds of materials that should be placed in this facility, the kinds of controls and governance needed for what is placed into, maintained by, and supported in an NL, and how to maintain user interest in and support of this enterprise.

The group decided that an NL should definitely target the needs of SME&T faculty. Some people felt that it also should be aimed at students, although others in the group worried that few students would utilize it if the materials focused primarily on needs of faculty. However, government agencies, foundations, publishers, and other commercial interests also would have to find benefit from the development of an NL for undergraduate SME&T education for it to succeed. The proposed NL must provide a clear, compelling message to its community of users about its objectives and how the materials contained within it contribute to those objectives.

The proposed NL project should take advantage of and learn from existing quality control programs for making decisions about materials to be included. In addition, an NL should incorporate and promote materials that have been created based on existing national education standards (e.g., National Council of Teachers of Mathematics Standards in Mathematics and the National Research Council's National Science Education Standards ).¹⁶ An NL should serve other interests, such as policy-making organizations in individual states and in the federal government. Importantly, an NL should focus on providing materials that target specific goals for improving teaching and learning that could not be undertaken without it. Finally, outcome measures for success of the proposed NL should be developed at the beginning of the initiative, with input from the community of users.

Expanding on Shamkant Navathe's comments, the group also produced the following two tables of "things" that might be placed into an NL (Table 1) and tools that could be made available to users of an NL for producing their own materials (Table 2):

In Table 1, "Formal Materials" refers to resources, many of which already are online. Incorporating such materials into an NL would require licensing agreements with many different publishers and other proprietors. Materials listed under "Archival Materials" would be suitable for more permanent types of cataloging, indexing, and storage. "Other Refereed Materials" would include those that have undergone less scrutiny via formal refereeing processes. Some materials, such as software that has been peer-reviewed prior to its release, might fall within either of these two categories.

Under the "Informal Materials" heading are unrefereed, exploratory, and experimental resources. These kinds of materials might originate from both faculty and students and include items such as course syllabi, laboratory exercises, computer applications, and theses and other papers produced by students that are available for sharing, interactive commentary, and further development. How this material would be submitted and posted to an NL is an open question. The group felt that the NSF should support projects for various models of NL governance boards. These boards would experiment with criteria for including materials in an NL in the same time frame as other components of the system were developing (see "Recommendations," page 53). The group did not attempt to identify these criteria.

As Table 2 shows, the group thought that the proposed NL also must provide tools that allow users to work from and contribute to it effectively. "Creation and Authoring" tools would enable contributors to produce materials with standardized software that is easy to use. "Usage" tools would facilitate rapid searching and browsing of the pro-

posed NL's contents, while "Resource Locator Services" would provide electronic pointers (e.g., URLs) to materials located elsewhere. "Downloading and Packaging" tools would enable people to take materials that are available in an NL and elsewhere and package them for their own use in courses, laboratories, and so on. All of these resources pose their own challenges in terms of hardware and software platforms and operating systems on which to mount an NL, which languages to use, how the different components of an NL would operate seamlessly with other components, and how to serve the needs of users who will have vastly different levels of expertise and comfort with using electronic resources. The system also must have careful oversight through a clearinghouse mechanism to monitor contributions, to measure their value (specific metrics for doing so were not discussed), and to help in decisions about what should and should not go into this resource. Expectations for what would be included in an NL's collection will likely change as the project evolves.

The group suggested that, in its earliest stages, an NL might focus on enabling faculty to learn about effective practices and approaches to enhancing learning. As the system evolved, it could enable a restructuring of learning environments to support the needs of both faculty and students in various SME&T disciplines. Ultimately, the system could serve as a driver of change and the focus of communication for a community of learners in SME&T.

The group concluded that any NL initiative must "improve with age." The collection of materials and tools must continue to grow in ways that add value for its intended users. The tools and resources must keep pace with rapidly advancing technology and research into learning about SME&T. Its overseers must develop self-sustaining and self-funding models for operation that engage support from both the public and private sectors.

Of the two break-out groups that considered these issues, the members of the first decided that they would focus on what the NSF wishes to accomplish through its NL initiative. The agency's goal is not to build a digital library per se but to improve undergraduate education and instruction. Thus, this breakout group decided its question was whether building the proposed NL would enhance or contribute to the NSF's goal of improving undergraduate education. With this premise, the group focused on teachers as the primary users and community for the proposed NL's collections and content. They also explored the question of helping instructors create materials that other instructors can easily use and adapt to their own classroom and institutional situations. Thus, dearly identifying and attending to the needs of the intended users was seen as more important in the beginning of the process than making decisions about the technology and other issues associated with traditional libraries. How can an NL make more effective use of instructors' time and resources as well as leverage investments made already by the NSF and individual instructors?

Clearly defining the intended users should be of paramount concern. The social milieu in which students live, study, and work, and their motivations for using this kind of resource are not the Same as for faculty. Therefore, deployment strategies for students could limit usage by faculty. If one says that the primary stakeholders are teaching faculty, many of whom will be in tenure-track positions, then the proposed NL's structure would have to consider faculty rewards and incentives, motivation, other tasks teachers perform that constrain instructional time and effort, and sources of funding for instructional and scholarly productivity. Given the current culture in higher education, the question arose as to why faculty, and especially untenured faculty, should spend time creating and upgrading instructional materials and then sharing them when these activities typically are not recognized as criteria or accomplishments for professional advancement.

Of all teachers, faculty in higher education usually have had the least amount of formal training in pedagogy or learning theory and processes. Most have not been educated in instructional design, how to evaluate instructional models and materials created by others, or how to adapt these materials to their own classrooms. Also, because teaching is not openly discussed with professional colleagues to the same degree as research, individual instructors often are idiosyncratic in their teaching methods. Again the question arose as to why college and university faculty should be expected to seek out someone else's materials and spend time adapting them to their own purposes.

An NL could address lack-of-incentive issues by having as a primary component a registry for pedagogical materials and learning resources. These resources could be evaluated through a peer-reviewed process. The proposed NL also would need to develop a strategy for achieving "buy-in" from faculty who previously have not used these kinds of resources. The developers of such a resource will need to identify early adopters and work closely with them to determine their patterns of usage and the problems they encounter. Prominent individuals in all SME&T disciplines should be recruited to tout the benefits of this kind of resource. In addition, materials would need to be easily transferable and to operate with minimal outside assistance on computer configurations with which faculty were familiar and comfortable. Faculty should be able to pull materials from an NL, modify them to suit their own needs, and then make them available to students to enhance learning. Under this model, students would benefit from an NL, if indirectly, by using downloaded materials locally.

Continuing to build on this model, the group thought that an NL's resources should be distributed across the Internet. This resource would be expected to account for and resolve issues centrally associated with copyright and intellectual property. On the technological side, an NL would still also need to deal with issues such as metadata, description of information, searching, archiving and maintenance, preservation, and access.

The members of the second break-out group on "Logistics and Technology" also considered user and pedagogical issues. They emphasized that, unlike other digital library projects, an NL for undergraduate SME&T education should deliver services and not be a site for conducting additional research about digital libraries. An NL should be expected to keep abreast of research in other digital libraries but primarily for the purpose of improving service to users.

This group also strongly agreed that an NL would need to stay in close contact with its customers, whoever they might be. They stated that any project must incorporate extensive' market research and evaluation of customer needs from the beginning so that it can change in concert with the needs of its users. Faculty especially must become engaged in these efforts from the start of any NL project that might emerge.

This break-out group thought. that the design, technology, and logistics employed in an NL project also must become integrated with all other digital library efforts under way both domestically and internationally. They saw this NL project as ultimately moving toward one coordinated, distributed system that would be overseen by some high-level governance body. Initially, however, organizations such as scientific societies that have access to materials and can review them for their scholarly content and suitability for inclusion in an NL might serve as overseers of the process. Since these organizations were founded along disciplinary lines, what initially went into an NL for undergraduate SME&T education through their oversight also would likely be disciplinary in nature.

Members of this group thought that the NSF might establish an initiative to decide the best use and applications for an NL for undergraduate SME&T education. In Phase I of this initiative, some group members suggested that NSF issue a Request for Proposal (RFP) in the format of a cooperative agreement to create four or five models of an NL. By testing different models over time, NSF might then be able to decide the best paradigm to support. Other members of the group felt that an RFP in the format of a cooperative agreement to start a single project could be issued. Regardless of the number of contracts issued, the group emphasized that these projects should be structured in such a way that there would be collaboration and cooperation among them. Phase I might last up to two years. Phase II would coordinate best models and practices from Phase I to develop a single NL that is distributed physically but coordinated centrally.

The group proposed that any RFP should contain suggestive rather than prescriptive language to encourage the emergence of new ideas and technologies that otherwise might not be considered in a more restrictive venue. At a minimum, the members of this break-out group said that a successful proposal should demonstrate awareness of other similar projects and include a plan for integrating and coordinating with these projects. The proposed project also should identify users and their needs, deal with customer satisfaction, define a rating system for adding materials into an NL (including which kinds of people or organizations would be responsible for overseeing the system), and demonstrate how such an NL could be economically viable, sustainable, and robust. In addition, successful proposals would detail how to address technological issues such as identification and naming, authentication, classification, and descriptions of metadata.

The discussion that followed this group's report focused on some of the details of how these proposals might be divided, the topics on which each might focus, and the development of appropriate evaluation tools.

Like the groups dealing with "Logistics and Technology," this break-out group also struggled, with user issues, in this case to give some context to the economic and legal issues they were asked to consider. The role of an NL and its users could be considered on a continuum, the group thought. At one end of the continuum, an NL would provide content and instructional materials that could stand-alone and serve as replacements for textbooks. At the

other end of the continuum, an NL would be mediated by teachers who would select, adapt, and launch some of these materials into a broader pedagogical context.

The members of the group predicted that an NL ultimately would assume a format where teachers were its primary users, and these teachers would contextualize and personalize the available materials for their students. The development or acquisition of stand-alone materials for students would not be precluded by this model, but an NL might simply point to such resources elsewhere rather than commissioning their production or storing them.

The group next considered the scope of an NL for undergraduate SME&T education. They agreed that an NL primarily would be an index and, to a lesser extent, a repository for materials not available elsewhere (e.g., faculty-authored materials). The proposed NL might accept submissions from elsewhere but would not make commissions nor fund in other ways the creation of content; other organizations, such as the NSF or publishers, would be responsible for supporting such projects, the group thought. Neutral about its specific origins, an NL would index content and assist users in finding, evaluating, and sharing it. Faculty were seen as the primary users, but students also were viewed as being able to extract useful information from an NL. If materials were directed primarily toward faculty, however, the group surmised that most students would not be able to use them directly unless faculty provided additional context within a course.

Conversations continued about what might be included in the index of an NL and also what should not be included. The group thought that classic literature from scholarly publishers that is being digitized does not belong in the proposed NL since this material falls more within the realm of traditional research libraries. However, an NL for undergraduate SME&T education should work with conventional libraries to develop good pointers to these materials. Because the proposed NL also should be viewed as a "working facility for faculty" to improve teaching, long-term preservation of materials would not need to be a high priority and would be a responsibility that fell more to research libraries.

What should be included in an NL according to the members of this break-out group are models and simulations, courseware modules, curricular materials, syllabi, reading lists for courses, and annotated reviews and evaluations of these materials by other users. The group could not agree on whether an NL should point to data sets and sources of data. For example, public domain databases, such as the National Library of Medicine's Visible Human Project,¹⁷ could become a part of an NL in the sense that the NL might point readers to such databases and then also provide additional tools and value-added components (e.g., course modules where students could work with the database in a particular context).

As conceptualized by this working group, the indexing service provided by the proposed NL would not require a large physical infrastructure. However, to point effectively to what is likely to be a large and rapidly expanding volume of available materials, infrastructures for technology, networks, compatibility with local computer systems and servers, and coding of materials would need to be developed carefully.

The group also discussed the issue of "critical mass." If an NL does not contain enough material to meet the needs of users, it will fall into disuse. The group was uncertain how much and what kinds of materials collectively constitute critical mass and suggested that librarians be brought into this discussion, as the establishment of any new library raises these same issues.

Quality control also presents an interesting dilemma for a digital library. In a scholarly journal, quality control consists of a decision to publish or not to publish a submitted or commissioned paper. For the kinds of materials being suggested for this proposed NL, different rules probably would apply, especially for materials such as syllabi or course modules.

Economic and legal issues that the group considered centered on the following:

Paying for the proposed NL's services: Several possibilities emerged. If an NL provides resources that replace textbooks and more traditional materials, then perhaps students should bear some of the costs through university user fees, lab fees, or the cost of software or other materials that they use in

place of textbooks. For indexing services, contributors could be libraries, scholarly societies that are looking to increase their involvement in improving SME&T education,¹⁸ and federal and private sources. The group did not think that directly billing users for services would be a viable option.

Intellectual property rights: Many of the materials likely to be included in or indexed by the proposed NL are composite multimedia works that continually are moved from one institution to another and repeatedly adapted and modified to address users' needs. This kind of use results in problems with derivative works, works in which many institutions may claim property rights, and the rights of multiple authors from different institutions as colleges and universities become more interested in claiming part of the rights to materials that their employees produce.

Liability of authors who produce such materials: In cases of liability claims against multiple authors from different institutions, will the institutions stand behind their employees?

Privacy and confidentiality: The break-out group expressed concern that if materials from an NL were to be used to evaluate programs (e.g., to measure the effectiveness of courses or instructors based on student learning), legal problems could arise with regard to access to student records. Some issues also are ethical; for example, in the case of simulations or self-assessments, would it be appropriate for faculty to know how many hours students have worked with a program or how they are using it?

Any of these issues, if not resolved early in the development of the proposed NL initiative, could become a major barrier to its use.

Following the group's presentation to the workshop-at-large, Edward Fox asked how much such an NL project might cost. Clifford Lynch, facilitator and spokesperson for the break-out group, responded that his colleagues had not considered the actual cost. However, he contended that such an NL, as perceived by this group, would be far less expensive than a system that commissions or licenses content on behalf of its users. Lee Zia asked the librarians participating in the workshop how much of a student's tuition goes to support the library system at a university. Richard Lucier responded that, in California, university libraries are line items in the state budget. Michael Lesk indicated that 3-4% of a typical university's budget goes to its library system, although there can be considerable variation depending on the type of institution.

Michael Lesk wondered about sources for long-term financial support of an NL. He stated that unless universities paid some of the costs and then charged students additional tuition in exchange for having to purchase fewer textbooks and other materials, an NL could not be financially viable. Additional discussion ensued about the use and importance of various types of student fees, including those for laboratories and information technology being applied to support access to the proposed NL. Tryg Ager worried that financial support for such an enterprise would be in peril if each state had its own method for covering these kinds of costs or for collecting money from students.

With regard to intellectual property rights, Michael Lesk said that if universities were to become more involved with issues of intellectual and other property rights for their employees, such matters would be easier to resolve since far fewer people would be involved in negotiations. Clifford Lynch said that some indication of who holds property rights to an object should accompany that object when. it is transmitted to users. Michael Raugh said that one advantage of the registry or catalog model for an NL is that it avoids many of these intellectual property issues because the burden for resolving them is placed on users of the materials, who would most likely enter into negotiations or purchase arrangements with providers.

Jack Wilson opened the second day of deliberations about whether the NSF should support the establishment of an NL for undergraduate SME&T education by reviewing the discussion to date. He pointed out that he had not heard a strongly articulated need for an NL expressed the day before. He also noted that workshop participants had not

reached agreement on many of the issues. Wilson summarized the sense of the workshop from Day 1 as follows:

Audience—The greatest area of agreement was who the proposed NL's “customers” were. Workshop participants seemed to feel very strongly that faculty were the primary customers. Students, librarians, SME&T professionals, and lifelong learners also might be considered as customers, but agreement could not be reached about this. It was noted once again that potential users who could best articulate who would use this facility and whether it would enhance undergraduate teaching and learning were not well represented at this workshop.

Content—Wilson stated that he sensed no agreement about content. Rather, the following questions consistently recurred in the discussion: Should an NL absorb pre-existing materials and/or point to them? Also, should an NL support the generation of new materials?

Legal issues—Workshop participants appeared to agree that these problems transcend their discussions. Legal issues would be addressed by others and could not be meaningfully resolved in this forum. The directors of an NL would need to stay closely connected to ongoing discussions and legislation in this realm.

Technology issues—Most speakers and break-out groups that considered these issues agreed that the key element was to let the needs of users drive the development and implementation of the technological features of any NL initiative. How this will be achieved is a more difficult question. The rate at which technology advances far outpaces both the development of content of undergraduate SME&T education and the cultural features of higher education that catalyze and reward educational reform.

In closing, Wilson returned to the point that the workshop had not yet strongly articulated a need for an NL despite the outcome of the straw poll on the previous morning. He asked participants to address this point clearly in subsequent deliberations since the Steering Committee was charged with advising the NSF about whether or not to proceed with a project or series of projects to establish an NL for undergraduate SME&T education.

Several participants addressed the conundrum of the previous day's straw vote in favor of establishing an NL, on the one hand, and the apparent lack of clear objectives or raison d'être for it on the other. Francis Miksa saw the problem in the nature of undergraduate education itself. Libraries currently do not play a major role in the education of many undergraduates. To make a strong case for an NL, the culture of SME&T education itself would have to change. Christine Borgmann continued this discussion, saying that before an NL can be implemented, we must ascertain how potential users of the proposed NL actually do their work, what their needs are, and how they work within their organizations. Then a system can be designed that is capable of being embedded in and compatible with its potential users' working environments. This goal must be the first priority in any design for an NL. Further, it is more of an organizational issue than a technology one. Robert Lichter suggested that the proposed NL be developed as commercial products are. Businesses require extensive market research to determine users' needs and desires before deciding how to produce a product that will meet those needs. In the present case, much more input is required from users. Richard Furuta said that if the proposed NL was designed to promote professional development for faculty, it also could catalyze the reform of undergraduate teaching which, in turn, would change the ways and cultures in which students learn.

Tryg Ager said that the Library of Congress, which has a mandate to undertake K-12 outreach, is placing into its digital library a photographic record of American history, primarily for students. Ager suggested that a sheet of paper be passed around so that everyone present could add her or his top priorities for materials that could be included in an NL for undergraduate SME&T education. An assessment of user needs and preferences could go far toward articulating not only what an NL should contain but its importance, as well.

Edward Fox emphasized that, ultimately, this proposed NL has to support students, who are the end users. Faculty create materials for students to use, not just to share among faculty. Thus, rather than being considered a primary resource for faculty, an NL should be set up in ways that millions of

students can use it. The proposed NL also should be designed so that faculty can share teaching materials easily. Functionality and interoperability should be of paramount concern in its design and establishment. Francis Miksa stated that the word "student" has too narrow a meaning in this context. The proposed NL actually should be thought of as a science digital library for independent learners—a perspective that would include everyone who needs to use this kind of resource, whether in a school setting or not and regardless of the user's professional status. Let an NL be independent of any formal educational structure and put materials into it that are useful to learn, Miksa advocated.

Margaret Gjertsen felt that the group should not spend so much time on the question of whether or not the need for an NL can be articulated. The group agreed that a specific need had not yet been stated. If that need were well known, some commercial entity already would have begun to undertake this project. Gjertsen emphasized that the real issue is in how the proposed NL is developed so that many people will have access to the materials in it. Precisely because there has been no articulated need that could be met by a commercial enterprise, an NL should be built by non-commercial sources.

On Day 2, two break-out sessions focused on "User and Pedagogical Issues." One break-out session focused on "Logistic and Technology Issues." One break-out session concentrated on "Economic and Legal Issues.”

Because there was general agreement that legal issues would have to be resolved elsewhere, this break-out group concentrated on economic issues. Clifford Lynch, group facilitator for both. days, reported that the morning's discussions were almost opposite to those of the day before, which had focused on the proposed NL's mission of supporting teaching. Today's discussions focused on how to help people learn SME&T. The second-day break-out group discussed the facilitation of learning and the level of content in an N L for undergraduate SME&T education. It was thought that the proper level for content would take into account the experience, skills, and knowledge one would likely encounter in undergraduate students, but that the content need not focus exclusively on the needs of individuals formally enrolled in undergraduate SME&T courses. The group discussed how a primary means for constructing an NL would be to impanel groups of specialists and charge them with identifying and selecting materials for inclusion. Here, professional and scholarly societies and organizations could play a leading role by identifying core subjects and the kinds of materials needed to provide an effective and exciting picture of different disciplines. It was thought that organizations such as the NSF might decide to support a major initiative in some disciplinary area (similar to the chemistry and calculus initiatives). Partial funding for such projects might come from individuals or organizations interested in developing specific content and tools to promote learning in and appreciation of certain subject areas.

Economically, the group thought that any NL initiative should emphasize open access to materials, including those already in the public domain and those for which an NL had negotiated broad distribution rights for educational purposes. For example, professional societies might be encouraged to make available to this NL selected materials to further development of subject matter and educate the public about their particular disciplines. It was noted that the American Chemical Society makes materials available through its Education Division that are already widely distributed elsewhere through other arrangements. Perhaps an NL also could provide or point to other copyrighted items to which users could have open access, either through the NL itself or from other sources. Commercial interests also might be encouraged to become involved as a means of promoting other salable goods. The sense of the group was that an NL should not purchase rights to materials and then try to recover those expenses by charging users or by seeking reimbursement through government or other sources.

This group also agreed that "library" is not a good descriptor for this project. Members suggested that what is envisioned is more an electronic science center or some modification of a science encyclopedia.

Discussion—In response to a workshop participant's question about who would pay for the development of content, Lynch replied that much of the material is already available. Materials commissioned specifically for the proposed NL could be handled outside its operation. For example, some other organization such as a professional society or the NSF might decide that some specific content should be added to an NL and would provide the funds to develop or acquire and insert that information into the system.

This group agreed that any NL initiative should be driven by an overarching vision that this resource should incorporate all kinds of opportunities for improving undergraduate SME&T education into its infrastructure. However, until this grand vision is more fully developed, a set of smaller, tightly defined initiatives should be undertaken. Thus, most of the session was devoted to discussion of how to support undergraduate learning and to defining a set of functional requirements for users that could encourage the most innovative set of responses from the technology experts who would build the proposed Nits infrastructure. This group agreed with other groups that a needs assessment focused on communities of users should precede any actual effort to build an NL.

This group basically agreed that an NL should be a registry of information, although some members felt it also should be a repository. Regardless of how an NL ultimately is constructed, its registry should allow users to understand and interface easily with the system. The proposed NL would include both formal and informal materials (see Figure 2) that would start with input from professional organizations in the SME&T disciplines about what constitutes both core information and "critical mass" (as described on pages 50 and 38) for each discipline. An important key to success in promoting such activities would be to develop an incentive structure that recognizes and rewards these kinds of scholarly activities. Materials might include examples Of best practices, information about how to teach, tools to support simulation, visualization, and animation. Indeed, the NSF might require all of its grant recipients who develop curricular materials to place them and evaluations of their efficacy for enhancing student

Figure 2. Another perspective of the content for an NL.

learning in an NL's registry. The system should also support bringing into an NL existing digital materials developed with grant support, for example, from the Division of Undergraduate Education. The group discussed how authentication and accurate, interactive searching features will be important components of an NL, especially if it does not serve as a repository for materials. Extremely important will be data authentication devices, such as algorithms that digitally sign data to make tampering easier to detect. These will help reassure users that the data or other materials they have accessed outside an NL actually correspond to data. or other materials in the Nits indices. An NL also should incorporate tracking features that will allow its overseers to determine what and how often users are accessing particular materials in the collection. This information would assist their making informed decisions about future acquisitions and features as well as retention periods for current holdings. An NL also should support features to promote interaction among users such as discussion groups.

Discussion—Lee Zia offered the counter position that an NL should be much more than a registry, even if the material listed has been peer-reviewed. An NL should, at minimum, allow users to at least sample snippets of the materials to which it points. But much more material could be stored and distributed from an NL. This proposed NL project should be seen as being much more dynamic and evolutionary than would be possible if it assumes only a registry function.

This group concentrated its efforts on articulating a need for an NL. The members agreed that a sense of urgency to construct this facility cannot be found in the SME&T higher education community. However, by committing funds to its development, the NSF can help drive this community toward desired reforms while simultaneously providing tools to help accomplish change. Another reason for the NSF to support this project is that it is not yet commercially viable. Providing such a service now could provide models to the commercial sector, which is seeking guidance about what needs to be done in this realm. However, the impetus for beginning such a venture must come from a non-commercial source, such as the NSF. In addition, the NSF is the only organization at this point that can provide the guidance, breadth of knowledge, and access to resources for interdisciplinary activities, promote standards across disciplines for materials to be included in an NL, and foster the kinds of professional recognition that is needed for those individuals who undertake this kind of work.

This facility could facilitate SME&T education-related activities that otherwise would be difficult or impossible to accomplish. Professional people who want to begin new projects for improving education could be informed by the materials already listed or provided in an NL. If the proposed NL had high standards for accepting or listing materials, having such materials accepted for inclusion would be considered professionally prestigious and would promote recognition that developing such materials is an important aspect of scholarship.

The group recommended that the NSF support the creation of an index of existing quality courseware materials, including work that has been sponsored by NSF grants. In this ease, courseware was broadly defined to include all types of materials to support teaching. The NSF also might consider supporting individuals or organizations to review comments made by users of these items or to create comments about the usefulness of products in the proposed NL. Such comments would be for the benefit of users rather than serving as an "accept/do not accept" decision by a governing body for inclusion of these items in the proposed NL.

Discussion—Lee Zia questioned the assumption that there is no sense of urgency to improve undergraduate SME&T education. Tora Bikson argued that the urgency should be evident from the data on SME&T undergraduate rates in the United States that Michael Lesk had presented. Ronald Stevens responded that, while change is coming, he does not feel that the SME&T community is willing to abandon the present system yet or to design a digital library that will set the standard and direction for undergraduate SME&T education for the next 15 years. By adopting the establishment of this proposed NL as a priority, the NSF could help catalyze such change. Jack Wilson pointed out that a sense of urgency does indeed exist, even if it is not within the SME&T community itself. Many state legislatures and governors are proposing legislation and regulations that could fundamentally restructure higher education.

Figure 2 summarizes this group's discussion about the types of content that might be included in an NL for undergraduate SME&T education. This scheme arose from earlier discussions about the purpose of teaching and learning technologies: improving the quality of classroom education vs. increasing the productivity of faculty so that they can teach more students. The latter purpose reflects more of an administrative rather than a learning-oriented perspective. This group agreed that the role of relevant technology should first and foremost be to improve learning. Increasing teaching effectiveness should be a secondary goal.

Toward that end, this group agreed that the NSF should support an initiative to provide worldwide access to resources that improve student learning in SME&T, that are usable in a variety of settings, and that also increase faculty teaching productivity by 1)

enhancing existing courses and labs, 2) encouraging development of new courses and labs, and 3) engaging learners in the process of doing science. This group agreed with the definition of digital library provided by Borgmann (1996, see footnote 1). They noted that this definition implies that a digital library is not only for information retrieval. Rather, it also is a place where users can create and deposit their scholarly contributions as well as search for and use materials already there. The proposed NL should contain rich collections of materials with the kinds of choices available that allow users to tap its resources for multiple purposes. Students should be able to learn about SME&T through this resource in ways that would be difficult or impossible through other media, including access to such items and information as NASA's up-to-the-minute accounts and pictures of its missions which also engage the public at large. An NL should engage learners from all walks of life in the thrill and beauty of science via learning opportunities such as live discussions with experts and with each other. While the scheme in Figure 2 separates faculty resources from other components of an NL, it is meant to imply that faculty also are learners and can benefit from these resources in the same ways as all other learners.

Further, this group felt that 1) the proposed NL should be a national digital library in the sense that resources should be constructed by and for SME&T instruction in the United States, and that 2) the proposed library also should be a U.S. contribution to international cooperation in education. However, the United States should monitor carefully developments in other countries to take advantage of international advances in technology and/or in available content.

This break-out group also considered the logistics that would be required to implement their model. If the NL project were to go forward, important components for its overseers to consider would be its registry, classification of materials, standards for accepting and posting objects, and interoperability. Although users should have direct access to the content listed in an NL's registry, this access could have important ramifications for the cost of establishing, supporting, and maintaining the enterprise. Searching and indexing features might be tailored to different user communities so that users could access the same information through different search paths. The group noted that training of users and authors will have to be a very important component of this project since the information in an NL and the platforms on which it operates will change frequently.

Discussion—Michael Lesk asked whether people in other parts of the world should have to pay for the materials and services provided by this proposed NL. The group thought that fee structures for users outside the United States should be examined. Lesk also suggested that a way to deal with this issue is to trade with other nations. For example, the United States might provide access to digital libraries dealing with subjects which have been well developed here in exchange for access to information in digital libraries elsewhere in the world that focus on other topics. For example, nations that have developed extensive materials for language education might provide otherwise restricted access to that information in the U.S. and gain similar access to United States digital libraries, such as the one proposed for SME&T education.

Ensuing discussion focused on several points raised by the presentations from the break-out groups. Richard Furuta said that members of his break-out group concurred that the NSF should view this project in broad terms initially rather than focus on the specific components that will be needed to develop an NL. Furuta also stressed that this effort should be a long-term one, although he acknowledged that other groups thought that shorter term projects (six months to two years) should be undertaken before the NSF commits to a particular paradigm for developing an NL.

The proposed NL was compared to a zoo, where the "inhabitants" of both constantly change over the course of their "lives" and therefore require changes in nurturing, care, and sustenance. Unlike a traditional book that remains available in a traditional library for many years, an entry in a digital library may not be useful or even readable five to ten years in the future. Tora Bikson said that the NSF is key to the success of the proposed NL because the agency is now the only organization that can imbue the project with both a coherent vision and overarching organization and coordination of its various components. Without this oversight, it is unlikely that the individual pieces of the project will meld into a usable whole.

Bikson underscored the issue of time scale, saying that short-term efforts are likely to be futile because faculty users will not invest the time and effort necessary to locate and evaluate materials in an NL if they cannot be assured that these materials will remain available when they are actually ready to use them in their classrooms or teaching laboratories. She suggested that we need to start now to learn how to construct this entity because it will take a long time both to educate faculty in its use and to learn from efforts by faculty to make the proposed NL even more useful and accessible. Bikson related a sense of urgency to begin this project because it could be important to the national economy. Governors of several states and leaders in business and industry are contemplating the creation of virtual universities on an interstate scale and will be recruiting more and more people who are well versed in science and technology (e.g., Western Governors University). An NL could be one way to increase the number of U.S. undergraduates who enter these fields.

Ronald Stevens explored further the perception of urgency for establishing an NL for undergraduate SME&T education. He argued that while there may be an urgent national need to establish this NL, there is no such sense of urgency to do so among faculty or academic SME&T departments. Faculty are not currently receiving tenure or promotions based on professional activities in this realm. Many faculty do not explore new teaching methods or actively change their courses on a monthly or even yearly basis. Little research is available on which teaching practices are most effective¹⁹ and outcomes of effective teaching practices are not widely disseminated. However, Stevens emphasized that a well-constructed NL could stimulate a sense of urgency among faculty to reexamine their teaching practices. Robert Panoff agreed, saying that it is difficult for faculty to sense urgency or crisis about the state of undergraduate education when most undergraduates with degrees in SME&T are finding jobs. Further, given the perception among some faculty that problems with their students originate in the lower grades and the fact that undergraduate SME&T majors find employment or places in postgraduate education after leaving their institutions, some faculty may believe that shortcomings in undergraduate SME&T education are someone else's problem.

Roberta Lamb stated that by encouraging potential users to visit its registry, an NL might help them to understand better what already has been developed and what is usable. By providing this forum, the NSF could advertise and promote good science and science education and showcase attractive environments for learning that also might increase participation in and appreciation of science by the larger community. Thus, an NL should appeal to many potential user constituencies. The NSF is the government agency that can best coordinate the continuing discussion about these diverse and potentially conflicting goals.

Jeanne Narum said that an NL would be the newest tool in an arsenal of SME&T education projects and initiatives that the NSF has promoted during the past ten years to disseminate examples of "what works" to improve undergraduate SME&T education. She stated that the sense of urgency should be to capture the richness of the innovations that are occurring in this realm, many of which have been supported by the NSF. There is a need to articulate a vision and mission that will move the SME&T education community into the next century, utilizing the best tools available to do so.

James Davis then challenged the group by proposing that NSF should not construct or support the construction of an NL for three reasons. First, workshop participants had not articulated specific needs or demands for such a resource. If undergraduate SME&T education is indeed in crisis, an NL is not the current solution for this problem. Additional work with curricula, new institutional initiatives, or metrics may be better tools to mitigate these problems at this point. Second, Davis stated that a high percentage of the functionality of an NL as it had been discussed in this workshop would likely emerge from grassroots efforts and the commercial sector without assistance from the NSF. Third, given the likely emergence of these resources from other sectors, the funds that would have to be committed to this project could be better applied elsewhere, especially in efforts to develop better content materials.

Michael Raugh agreed with Davis on most aspects of his statement but added that the current situation for locating materials on the Internet is chaotic

because reasonable standards for cataloging and indexing materials have not been applied to this medium. Through an NL project, the NSF has the opportunity to bring some order to this chaos by catalyzing the creation of a standardized infrastructure that can serve as the model for the many different approaches to searching for and sharing information on the Internet. Raugh also felt that by placing its imprimatur on faculty and other developers of such materials who agree to participate in rigorous evaluation of their creations, the NSF would provide support for work that too often is not recognized by the academic community as scholarly and promote greater participation and professionalism in the evaluation of content on the Internet. No other agency in the United States has the ability to encourage this kind of activity. Peter Graham added that the humanities and social and behavioral sciences communities also would benefit from work done on an NL for undergraduate SME&T education in resolving issues such as authentication, incorporation of metadata, and engines for searching and retrieving information in their disciplines. Thus, the benefits of the NSF's support of an NL for undergraduate SME&T education potentially could extend well beyond the sciences.

Michael Lesk felt that evidence for the efficacy and success of existing digital library projects should be provided to the NSF so the agency can make a more informed decision as to whether additional support of them is warranted. Several participants offered examples of successful projects.²⁰ Tora Bikson said that the NSF itself has done a very good job in supporting the development of content for undergraduate SME&T education but that it has fallen short in making sure that this content is developed in such a way that it can be placed into formats and forums where it can be readily located and used by others. She also noted a need on the part of the NSF to relay to potential users of this content the kinds of standards the developers had to meet to receive project funding. If the fruits of these projects simply appear on the Net, their true value is likely to be obscured by the morass of other similar information. Bikson suggested that the NSF should make available in some well-publicized forum the results and products of the projects the agency has supported. This forum should allow additional commentary on and evaluation of these materials by users. In addition, many NSF-sponsored, large-scale research projects are usually undertaken at only a limited number of research universities with large graduate programs. Making the results of such research available to the larger scientific community via an NL could advance inquiry-based teaching and learning at schools that emphasize undergraduate education. Another participant²¹ suggested that the NSF might require those who produce educational materials with the agency's financial support to serve as mentors for some period of time or at certain events so that they could interact with and offer their expertise to other potential users of those materials. An NL for undergraduate SME&T education could facilitate a level of dynamic interaction between the creators and users of materials that would be impossible in other types of media.

Gordon Freedman next approached these issues from a public policy perspective. The federal government could have a multiplier effect on reforming education by improving the dissemination of innovative educational materials and activities, many of which have been produced and field tested with public funds. If the government does not support ways to connect these materials and make them available and useful to the widest audience possible, then citizens do not receive the best value for their tax dollars. Whether an NL is the best avenue to accomplish these goals remains open to discussion. However, Freedman contended that next steps must include mechanisms to maximize the effectiveness of programs and materials that are already being supported by the federal government.

Following this discussion, the workshop was adjourned.