EXECUTIVE SUMMARY

A central role of the National Science Foundation (NSF) is to support the improvement of science, mathematics, engineering, and technology (SME&T) education for all students in the United States at all grade levels. In its quest to catalyze and sustain educational reform at the undergraduate level, the NSF issued a report in 1996 on the status of undergraduate SME&T education (National Science Foundation, 1996b). That report called for fundamental changes in the ways in which SME&T subjects are taught and urged the agency to sponsor the development of "a national electronic library for validating and disseminating successful educational practices" (National Science Foundation, 1996b, page 72) and to "provide specific problem training sessions for faculty across institutions, in topics such as how to do inquiry and collaborative learning in large 'lecture' classes, how to assess learning outcomes, and how to document learning gains at the departmental and institutional levels" (National Science Foundation, 1996b, page 72).

Digital libraries¹ are currently under construction for a number of scientific research communities with support from the NSF, the National Aeronautics and Space Administration (NASA), and the Department of Defense's Advanced Research Projects Agency (DARPA). The Library of Congress also is developing a digital library to disseminate its vast holdings more readily. Given the potential of digital libraries to provide rapid access to large amounts of information and the research base on digital libraries that these other projects already had generated, the NSF's Division of Undergraduate Education asked the National Research Council's (NRC) Center for Science, Mathematics, and Engineering Education (CSMEE) to undertake a study that would 1) explore the feasibility of establishing a digital National Library for undergraduate SME&T education and 2) examine various challenges that would have to be overcome in order to build a library that is both educationally innovative and cost effective.

In collaboration with the NRC's Computer Science and Telecommunications Board (CSTB), CSMEE responded to NSF's request by forming a project steering committee consisting of representatives from the NRC's four postsecondary boards and committees (Mathematical Sciences Education Board, Board on Engineering Education, Committee on Undergraduate Science Education, and the Committee on Information Technology). The Steering Committee, in turn, commissioned ten "white papers" from individuals with expertise in SME&T education, technological aspects of digital libraries, library science, and economic and legal aspects of this rapidly evolving area of knowledge and research. These commissioned papers (revisions of which are reprinted in Appendix A of this report) served as the basis for plenary and break-out discussions at a workshop that was held at the National Academy of Sciences on August 7-8, 1997. Some 50 guests from academe, digital library initiatives, private laboratories, private foundations, research and teaching libraries, and the commercial publishing sector participated in this workshop.




ISSUES CONSIDERED

The issues that these papers and workshop participants considered are diverse and exceedingly complex. They include the following:

Curricular, Pedagogical, and User Issues

(e.g., Who is the potential user population? What types of materials should be included? What impact can be expected?)

Logistic and Technology Issues

(e.g., What kinds of editorial oversight are needed? What kinds of technology are currently available to build such a national library (NL)? How can a multi-year project like this adapt to new technologies that may emerge?)

Economic and Legal Issues

(e.g., How can we estimate or measure the costs and benefits of establishing an NL? What are the long-term financial implications? How could intellectual property, copyright, and "fair use" issues be resolved?)

At the workshop, Steering Committee Chair Jack Wilson also charged participants with trying to arrive at answers to the following cross-cutting questions:


Is an NL a good idea for improving undergraduate SME&T education?

Is an NL a better idea than other initiatives that might compete for the same funds?

If the NSF does commit to supporting the proposed NL, then what kinds of information and issues will it need to consider so that the project can be undertaken efficiently and cost effectively?

Accordingly, this report provides a detailed summary of the presentations at the workshop and a synthesis of the discussions that were generated there. The report also identifies those issues on which workshop participants were able to reach substantial agreement and those which remained unresolved by the conclusion of the meeting. The report presents the conclusions of members of the Steering Committee who attended the workshop and provides a number of recommendations to the NSF from the entire Steering Committee about both the value and feasibility of proceeding with this project.




OVERVIEW OF CONCLUSIONS

The conclusions of the workshop are organized here by major issues addressed.

Users and Needs

These issues pervaded the entire workshop. A broad agreement developed that faculty engaged in SME&T education would be included among the primary users targeted by an NL. Workshop participants also concurred that the central focus of an NL should be to improve and enhance learning of SME&T.

Nevertheless, there was considerable divergence of opinion about the extent to which an NL also should provide learning resources directly to undergraduate students and possibly other users (e.g., advanced high school students, adults engaged in distance learning through a university program, lifelong learners seeking information on specific topics, or those wishing to increase their understanding and appreciation of SME&T in general). There also was a divergence of opinion about how often these "student users" would access an NL.

Most workshop participants agreed that the establishment of an NL could potentially be a useful tool for improving undergraduate SME&T education. However, some workshop participants noted that a large part of the SME&T teaching community has not yet felt a sense of urgency about the need for reform. Indeed, for the most part, participants believed that the workshop discussions had not made a convincing case that an NL was an essential component of SME&T education reform. However, some workshop participants and commissioned papers pointed out that, in addition to providing high-quality materials for improving learning of SME&T, an NSF-sponsored initiative to support an NL could have an important impact on undergraduate SME&T education by underscoring and showcasing the importance of educational reform in highly tangible ways. On the other hand, funds used to support an NL would then not be available to support other educational initiatives, and there is no current analysis available that indicates the relative efficacy of an NL compared to alternatives.

Because only a few science and mathematics teaching faculty and undergraduate students were present at the workshop (a list of workshop participants and their institutional affiliations is provided in Appendix C; biographical sketches of workshop participants are in Appendix D), it is not clear to what extent the proposed NL actually would be utilized by faculty to improve their teaching of SME&T courses or by other stakeholders of a larger NL. Input and advice from potential users also will be critical for designing and developing an NL's content.

Content

There was considerable discussion about what kinds of information the proposed NL should contain. Workshop participants agreed that an NL could offer a large variety of materials, such as digitized text (e.g., from professional journals, course syllabi, student works-in-progress, reports about the outcomes and evaluation of SME&T education projects that have been funded by NSF and other grantmaking agencies), videos and still images, instructional software and simulations, and anything else of relevance that could be stored digitally. However, there was little agreement about which classes of these materials an NL should make available, either immediately or in the future. In part, these disagreements were related to the issue of who the primary users of an NL will be. Thus, defining the intended audience for any NL initiative will aid in making decisions about its content. Discussion focused on several important issues related to content:


1. Should an NL commission and store discipline-based source content or serve primarily as a cataloging resource that electronically "points" users to information stored on other computers and hard copy? A broad agreement developed that, at a minimum, an NL should contain pointers to useful materials. Pointers are much less expensive to create and maintain than stored source content, more easily allow for contributions from a wider spectrum of interests and organizations in the SME&T community, and minimize current legal challenges related to intellectual property rights, copyright law, and licensing agreements. However, an NL could face several important constraints if it were to rely exclusively on the use of pointers rather than commissioning and storing at least some materials. Until consistently reliable software is available to enable an NL's registry to update the addresses of materials stored elsewhere on the Internet, tracking the location of materials will be problematic. An NL's ability to catalyze development of or to exert quality control over materials specifically suited to this medium could be very limited. Also, an NL's holdings should reach a "critical mass" of quality materials that will attract wide usership; whether the quality and quantity of materials currently on the Internet is sufficient to reach this "critical mass" in different subject areas must be determined.

2. Should an NL simply make materials available (either directly or by pointing to other Web sites), such as traditional libraries do now, or allow users to add materials to a library? Contributed materials might include such items as new teaching tools and modules or annotations (e.g., reviews, comments by users, supplemental information) about materials already available from an NL.

3. Who should exert editorial oversight of the proposed NL's contents? What types of standards should be established for accepting materials for an NL? Minimum standards and strategies for including materials and the level of editorial oversight would likely be very different if an NL were simply to point to other resources rather than storing and disseminating them directly to users. Different standards also would have to be developed for accepting materials that have undergone peer review vs. materials that have not been subjected to such scrutiny (e.g., course syllabi, courseware applications, annotations and discussions about other materials included in the proposed NL). The workshop participants broadly agreed that some mechanism for distinguishing formally reviewed from unreviewed material would be necessary, both from the standpoint of the user and for the credibility of the proposed NL itself.

4. Who should create content for an NL? Here there was fairly broad agreement that creators could include faculty, publishers, professional societies, and students. The issue of who creates content for an NL also relates to the issue of "critical mass." If the information in an NL is not sufficient in quantity or quality, user disappointment followed by disuse are likely consequences. All of these issues have implications for what materials are placed into an NL and how long they are archived.²

5. What kinds of tools will be needed to facilitate browsing and searching of an NL by users? Experience with currently available search engines for the Internet clearly indicates that simple keyword searches, though sometimes useful, are inadequate when searching through large volumes of information. Development of interactive, "intelligent" tools that facilitate searching for materials, especially those that have been designed to exploit an NL's specific electronic capabilities, should be an important component in any design of an NL for undergraduate SME&T education.

6. Is the proposed NL a library? Most workshop participants agreed that an NL for undergraduate SME&T education certainly would embrace many of the characteristics of traditional libraries. However, this resource also could incorporate many other features not found in traditional libraries, such as the capacity for the NL's users to add materials and to work interactively with and upgrade materials already in the NL. Thus, workshop participants suggested, and the Steering Committee concurs, that a better set of descriptors be devised to reflect more accurately this resource's vision and objectives and to convey better to users how it might be utilized.

Economic and Legal Issues

In addition to focusing on the potential value or desirability of an NL for undergraduate SME&T education, the workshop also addressed a number of economic and legal issues. These topics were considered by workshop participants primarily in the context of the implementation and deployment of this resource. The following issues were raised.


1. Economic issues:  While government agencies and private foundations might provide key start-up funding for an NL, workshop participants agreed that this resource would eventually need to become financially self-sustaining. However, there was no general agreement about how best to address questions of economic viability and sustainability.

2. Legal issues:  Workshop participants identified a number of legal issues that would need to be solved before the proposed NL could become operational. These include


• Intellectual Property (IP).  IP issues in the context of an NL are similar to those that any online provider of content faces. However, inclusion of some types of material (e.g., course notes) may not be as problematic as other materials because remuneration to the authors or developers is not necessarily involved.

• Liability.  NL materials that involve some potential risk to users (e.g., instructions for performing undergraduate laboratory exercises) may involve liability for those responsible for administering the proposed NL initiative or for the authors and creators of materials to whom an NL points.

• Privacy.  To the extent that students use materials or information found in an NL (e.g., an online diagnostic test), well-meaning faculty may be interested in the extent and nature of such usage. Obtaining such information might impinge on students' expectations for privacy.


Workshop participants concluded that these issues could not be solved independently for the proposed NL. Rather, a regime of general law and practice will evolve as online publishing and dissemination of information becomes more extensive. An NL for undergraduate SME&T education will have to be flexible enough to accommodate a wide range of possible legal regimes and challenges.

3. Technology issues:  Workshop participants discussed many technology-related issues, including


• The need for an NL to be oriented to satisfying user needs rather than to being a vehicle for advancing the creation of technology or research about digital libraries. Any technologies employed by an NL also should be developed and deployed to accommodate the needs of users. Associated with this requirement, some workshop participants questioned the conventional wisdom of making the proposed NL available exclusively via the Internet. Because some institutions of higher education in the United States and other parts of the world do not now enjoy access to the Internet and others have only limited access through data lines that would require too much time for the downloading of large applications or data sets, important issues of equity and access must be considered carefully and addressed. Other formats, such as CD-ROM sets, might be considered as components of vehicles for disseminating information from any NL initiative (although interactivity could be compromised compared to access to the Internet). Workshop participants also noted that the development of Internet II could possibly restrict access to an NL only through selected colleges and universities.^3,4 Again, equity of access should be an important consideration in any discussions of delivery systems for this NL.

• An NL for undergraduate SME&T education should employ technologies that are adaptive, flexible, and responsive to unforeseen user needs and problems. New applications and modules should be designed to operate with software that is widely available for other applications (e.g., commonly used spreadsheets). This design would reduce the time required for users to learn how to work with such materials. An NL initiative also will need to deal with content prepared to run on older computers and software platforms that may be incompatible with newer hardware and software platforms.

• Technology employed in the proposed NL should be developed with advice and oversight from the professional communities who are most knowledgeable about how people both organize and use information: librarians and social and behavioral scientists. Without these informed perspectives, an NL is not likely to optimize opportunities for teaching and learning.




STEERING COMMITTEE RECOMMENDATIONS

Workshop participants generally agreed that the idea of an NL for SME&T education was sufficiently promising that the NSF should pursue it further, and the Steering Committee concurs. Although workshop participants did not agree on specific next steps, the Steering Committee makes the following recommendations based on information in the commissioned papers and presentations and discussions at the workshop to guide the NSF's planning for an NL initiative and its issuance of one or more request for proposals (RFPs). The Steering Committee recommends that these steps be acted upon sequentially. The recommendations that are summarized below parallel the discussion in the "Synthesis and Conclusions" sections of the report, and readers should consult that section for additional details. The following text is cross-referenced to relevant text in that section.


1.  Clarify the potential customers of an NL for undergraduate SME&T education (page 47)

1.1  Because workshop participants were unable to delineate the stakeholders or to specify the content for this proposed NL, the NSF should do so. The level of funding that the agency can devote to this project may dictate the breadth of the proposed NL's users, and that, in turn, may help with content decisions. However, the Steering Committee recommends that, prior to making final decisions about this issue, the NSF should make a concerted effort to bring together in a series of focus groups representatives from all communities that might be an NL's likely users and service providers. Focus groups should be small and should be structured to encourage participants to discuss freely 1) their requirements for resources and tools that would help them improve teaching and learning of undergraduate SME&T, and 2) the ways in which the digital National Library could address those requirements. At a minimum, participants in these focus groups should include


• College and university SME&T faculty from all types of postsecondary institutions, including two-year colleges, undergraduate liberal arts colleges, predominantly undergraduate comprehensive universities, and research universities.

• College and university SME&T faculty at different stages of their academic careers.

• College and university faculty involved with research and practice in science and mathematics education, including the preparation of future K-12 teachers.

• SME&T faculty from middle- and high-schools across the United States.

• Undergraduate students from different types of colleges and universities. This group should include both "traditional" and "non-traditional" students.

• Graduate and postdoctoral students who are likely to enter careers in academe also should be consulted since they will define future needs of faculty.

• Librarians.

• Social and behavioral scientists with expertise in organizational constructs and in the ways in which people learn new information.

• Computer and information system specialists with specific experience with digital libraries.

• Directors of college and university information technology services.

• Representatives from the commercial publishing sector.

• Representatives from professional SME&T societies.

• Representatives from the private non-profit sector, such as foundations.

1.2  The Steering Committee suggests that two different types of focus group meetings be held. Some focus groups should concentrate on receiving input from single communities, especially SME&T faculty and students. Others should involve people from many or all of the aforementioned sectors in crosscutting sessions, with the primary objectives of having convenors listen and respond to the ideas and expressed needs of potential users.

1.3  The Steering Committee recommends that NSF also might employ the services of one or more professional organizations to organize these focus groups, to facilitate discussions within the groups and to prepare an independent assessment of user needs and desires based on the group discussions.


2.  Articulate priorities for content, technological considerations, and economic and legal models before committing to the establishment of an NL (page 48)

The Steering Committee can offer no specific recommendations about whether the proposed NL should commission the creation and storage of materials vs. developing a sophisticated system of pointers to materials that reside and are maintained elsewhere. Differences in cost between the two systems, evolving legal precedents with respect to copyright and fair use of materials, and the emergence of new technologies that may overcome some of the limitations of pointing to information stored elsewhere all must be factored into the final structure of an NL. Moreover, these parameters are likely to change during the development phase of the project. Ongoing advice from appropriate experts in all of these fields is warranted if the project proceeds.


2.1  The Steering Committee recommends that the proposed NL be viewed primarily as a resource for improving and inspiring learning of undergraduate SME&T rather than merely as a means to promote more effective teaching of these subjects. If an NL is to be a central component of current efforts to reform and improve undergraduate SME&T education, it must offer more than teaching tools alone. The NSF should appoint a Board of Overseers consisting of acknowledged experts in SME&T education, library sciences, and digital libraries that is charged to work with a broad spectrum of intended users and the other stakeholders before decisions are made about what kinds of materials should be placed into the proposed NL. If an NL initiative cannot afford to support all areas of SME&T, then the Board should decide on the initial areas of focus and look to expand coverage as the project develops.

2.2  Steering Committee members also agree with many workshop participants and recommend that an NL should strive to focus on collecting or pointing to materials that either are inaccessible through other media formats or are so innovative that they are unlikely to be commercially available or viable in the short-term. Because a "critical mass" of materials is vitally important to the success of an NL, the acquisition of such innovative new materials will likely need to be balanced with more traditional materials, at least initially.

2.3  The Steering Committee recommends that the NSF emphasize involvement by professional SME&T societies in developing content that could be appropriate for an NL. Many of these organizations already have produced materials that might be incorporated into an NL at little or no cost. By promoting the development of these kinds of teaching and learning tools and by officially recognizing their members who do so, professional societies could become key catalysts in changing the culture of higher education to embrace as legitimate scholarly activities the promotion and evaluation of teaching and the promotion of effective learning by students.

2.4  The Steering Committee recommends that an NL should provide information about and access to projects in undergraduate SME&T education that the NSF and other agencies have supported financially.

2.5  The Steering Committee recommends that the NSF also seek a new, more encompassing descriptor for this project. Workshop participants recognized, and the Steering Committee concurs, that "Digital National Library" or "National Library"—the terms that have been most commonly used to describe this entity—may be more confusing than enlightening to anyone who envisions the potential stakeholders in this project and the services it may provide. Any NL initiative is likely to transcend the functions of many conventional libraries. A more appropriate descriptor might help to focus the higher education community on the need for such a resource and its importance.


3.  Develop and issue one or more RFPs to establish an NL for undergraduate SME&T education

As the NSF receives additional input from stakeholders about the goals of and need for an NL (via Recommendations 1 and 2), the scope and potential cost of the project should become clearer. During the workshop, Steering Committee Chair Jack Wilson charged participants with trying to arrive at answers to the following major crosscutting questions: 1) Is an NL a good idea for improving undergraduate SME&T education and 2) Is an NL a better idea than other initiatives that might compete for the same funds? If the NSF is convinced on the basis of its explorations that it can answer these questions in the affirmative, then the question of how to implement this project should become the central focus. Options for proceeding at that point would include


Option 1:  Undertaking a single, large initiative that would result in an operational NL within several years.


Option 2:  Undertaking several smaller initiatives for shorter periods of time (12-24 months). These initiatives might be competitive and operate independently of each other or they might be components of some larger cooperative agreement. These various models for establishing an NL could then be evaluated against each other, with a final coordination of best practices that might lead to a single, integrated project.


3.1  Given the tremendous complexity of this project and the number of communities that must be directly involved if it is to have any chance for success, the Steering Committee recommends that NSF consider adopting Option 2. Steering Committee members envision that the smaller initiatives suggested in Option 2 might be incorporated into a program similar to those that the NSF's Division of Undergraduate Education has sponsored in recent years to change the ways in which chemistry and calculus are taught. Optimally, this new initiative would incorporate many similar components, including those delineated in Recommendations 3.2 and 3.3 below.

3.2  The Steering Committee recommends that the NSF, in following through with Recommendation 3.1, should develop an RFP articulating the need for and issues involving the establishment of an NL as outlined in this report. The RFP would encourage diverse groups of stakeholders to focus on some subset of the issues. Collaboration among stakeholders and interdisciplinary approaches to address the questions posed here would be encouraged. Preproposals could be sought, with funds then awarded to successful groups to encourage them to develop full proposals. Depending on the funds available, the NSF might then award larger contracts to one or more groups to tackle specific issues or sets of issues. Each of these final awardees would be expected to inform each other of their progress and problems through routine communications, reports, and through meetings of teams convened on a regular basis (at least annually).

3.3  Because the central concern of workshop participants was to define the users of and the need for an NL for undergraduate SME&T education, the Steering Committee recommends that RFPs for preproposals not be formulated until the NSF sponsors the focus groups described above. Feedback and evaluation of information from these groups of users and providers could then serve as the basis for constructing RFPs that would help eventual awardees to address specifically the established needs and requirements of potential NL users.



---

NOTES

¹Dr. Christine Borgman, UCLA, offered the following definition as determined by participants at the UCLA/NSF Workshop, "Social Aspects of Digital Libraries": "[Workshop participants] determined that digital libraries encompass two complementary ideas: 1. Digital libraries are a set of electronic resources and associated technical capabilities for creating, searching, and using information. In this sense they are an extension and enhancement of information storage and retrieval systems that manipulate digital data in any medium (text, images, sounds, static or dynamic images) and exist in distributed networks. The content of digital libraries includes data, metadata that describe various aspects of data (e.g., representation, creator, owner, reproduction rights), and metadata that consists of links or relationships to other data or metadata, whether internal or external to the digital library; and 2. Digital libraries are constructed—collected and organized—by a community of users, and their functional capabilities support the information needs and uses of that community. They are a component of communities in which individuals and groups interact with each other, using data, information, and knowledge resources and systems. In this sense they are an extension, enhancement, and integration of a variety of information institutions as physical places where resources are selected, collected, organized, preserved, and accessed in support of a user community. These information institutions include, among others, libraries, museums, archives, and schools, but digital libraries also extend and serve other community settings, including classrooms, offices, laboratories, homes, and public spaces." (Borgman et al., 1996)
This report is available on line at http://www.gslis.ucla.edu/DL/UCLA_DL_Report.html.

²In this report, "to archive" and/or "to serve as an archival function" mean to preserve in readable form over the long term any material determined to have enduring value. "To store" and "to preserve" are used in this report in a technological sense, as in to save copies offline of material no longer in active use but possibly desirable at some future date.


³President William Clinton's announced goals for the "Next Generation" Internet initiative are as follows: 1. Connect universities and national labs with high-speed networks that are 100 to 1,000 times faster than today's Internet. These networks will eventually be able to transmit the contents of the entire Encyclopedia Britannica in under a second; 2. Promote experimentation with the next generation of networking technologies. For example, technologies are emerging that could dramatically increase the capabilities of the Internet to handle real-time services, such as high-quality videoconferencing; and 3. Demonstrate new applications that meet important national goals and missions. Higher speed, more advanced networks will enable a new generation of applications that support scientific research, national security, distance education, environmental monitoring, and health care. (Smith and Weingarten, 1997)

⁴A reviewer of this report, who must remain anonymous under the Report Review Guidelines of the National Research Council, wrote to disagree with the workshop discussion regarding lack of wide accessibility to Internet II: This person indicated that his institution has had access to the Very Broadband Network Service (VBNS), the precursor of Internet II, for some time. The institution has a switch that routes outgoing messages to the VBNS or the Commodity Internet (Internet I), depending on the destination. No one on the Commodity Internet has had problems reaching this reviewer or others at this university. The reviewer acknowledged that there may be issues of performance between Internet I and II, particularly if streaming audio or video applications are developed, but this reviewer does not believe that access will be an issue.