SYNTHESIS AND CONCLUSIONS

As the narrative from the workshop proceedings shows, key themes emerged at the workshop, and some level of agreement was achieved on several issues. Most workshop participants agreed that the establishment of some kind of NL could be a useful—possibly critical—addition to the existing arsenal of tools and other resources that have been created by grants from the NSF's Division of Undergraduate Education and other sources to improve undergraduate SME&T education. However, workshop participants expressed a sufficient variety of different views and perspectives about the clients an NL might serve, the materials it should (and should not) contain, and how it might be constructed, organized, maintained, and supported that the Steering Committee urges the NSF to proceed cautiously and judiciously before making a final decision about establishing an NL for undergraduate SME&T education.

Highlighted below are key observations and critical issues that the NSF may wish to consider before deciding whether to issue one or more Requests for Proposals (RFPs) to sponsor the establishment of a digital NL for undergraduate SME&T education. The Steering Committee offers these ideas with the suggestion that the NSF address them either internally or by seeking additional advice from outside experts. Where appropriate, specific recommendations are provided to assist the NSF in deciding how to proceed with this project (see Steering Committee Recommendations beginning on page 53). The Steering Committee notes that both the workshop and the papers provide a breadth of possible strategies for the NSF to pursue if an NL initiative goes forward.




KEY OBSERVATIONS

In synthesizing the workshop proceedings and stating its conclusions and recommendations, Steering Committee members would ask the NSF to give serious consideration to the following key observations:


• In terms of feasibility, the pedagogical, technical, economic, and legal issues surrounding the establishment of an NL for undergraduate SME&T education are complex, interrelated, and will quickly assume new dimensions as technology improves and litigation proceeds through the courts. However, the challenges posed by these issues are most likely surmountable.

• In terms of desirability, if the NSF is to commit its financial support to a digital NL project, workshop participants and the Steering Committee agree that the agency also must be satisfied that it can answer the following questions affirmatively before proceeding:

1. Would the establishment of an NL improve undergraduate SME&T education?

2. Would establishing an NL be a more effective alternative for improving undergraduate SME&T education than other initiatives that might compete for the same funds?




CRITICAL ISSUES

Audience: To whom should an NL be directed?

This issue pervaded the entire workshop. Broad agreement developed that faculty engaged in SME&T education would be among those targeted as primary users of an NL. Specifically, an NL might enable faculty rapidly to locate, download, use, and modify and adapt materials that would assist and improve their teaching in undergraduate classrooms and laboratories.

Workshop participants also concurred that the central focus of an NL should be to improve and enhance learning of SME&T. By providing useful pedagogical tools, an NL would partially realize this mission. However, there was considerable divergence of opinion about the extent to which an NL should provide learning resources for undergraduate students and possibly other users (e.g., advanced high school students, adults engaged in distance learning through a university program, lifelong learners seeking information on some specific topic, or those wishing to increase their understanding and appreciation of SME&T in general). Some workshop participants argued that the proposed NL should provide such materials for use and exploration by students and other users that would enhance and improve self-developed learning goals. Because they too are lifelong learners, faculty also would benefit from the availability of such resources, both by increasing their own disciplinary and interdisciplinary expertise and by having tools readily available to develop and enhance their undergraduate courses.

Again, as also noted in the NSF's vision statement above (page 18), an NL makes the most sense in the context of a broader culture of higher education that provides both strong incentives and the necessary tools to promote continuous improvement in education. Only when the undergraduate SME&T community broadly shares the values of such a culture will an NL achieve its full potential for disseminating and sharing information.

A number of workshop participants made arguments about why such a cultural change was inevitable. For example, they pointed to various reports decrying the state of undergraduate SME&T education (e.g., National Research Council, 1996a; National Science Foundation, 1996b) and to increasing pressures on institutions of higher education to improve undergraduate teaching in general and SME&T education specifically. However, some workshop participants noted that a large part of the SME&T teaching community has not felt a sense of urgency about the need for reform. Indeed, for the most part, workshop participants believed that the case had not been made that an NL would be an essential component of SME&T education reform.

Specifically, many workshop participants thought that further research is needed to address the following:


• The need for an NL in SME&T education. At present, the extent to which an NL would actually be used by faculty intending to improve their teaching of SME&T courses is unknown.

• The leverage for educational reform that would be provided by an NL. Even if an NL is used by SME&T faculty interested in improving their teaching, whether the numbers of useful innovations and concerned faculty would be sufficiently large to warrant such an investment is unknown.


It is possible that an initiative to support an NL would be an important statement by the NSF that would facilitate cultural changes by underscoring the importance of educational reform in a highly tangible way. On the other hand, funds used to support an NL would not then be available to support other educational initiatives, and there is simply no analysis one way or another that indicates the relative efficacy of an NL compared to alternatives.

Thus, workshop participants emphasized that making specific recommendations about the breadth of users of an NL was very difficult because the community for whom this resource might be created (faculty from SME&T disciplines who routinely teach undergraduate students and undergraduate and graduate students) did not attend the workshop in sufficient numbers to articulate clearly their needs for content and how they might or might not utilize an NL. Workshop participants who submitted additional comments were unanimous in their concern that potential users become much more intimately involved in all aspects of this project.

Despite the Steering Committee's efforts to include teaching faculty from different kinds of undergraduate colleges and universities, insufficient numbers attended to provide a clear vision about how much and under what circumstances the proposed NL would be utilized. Lack of attendance by a broad spectrum of teaching faculty also highlighted the importance of articulating a clear, unambiguous mandate for establishing an NL.

Content: What should an NL for undergraduate SME&T education contain?

Workshop participants expressed many ideas about what types of information an NL might offer. They agreed that, from a technical standpoint, an NL for undergraduate SME&T education could offer materials such as digitized text (e.g., from professional journals, course syllabi, student works-in-progress), videos and still images, instructional software and simulations, and anything else of relevance that could be stored digitally. However, there was little consensus about which classes of these materials an NL should make available. Part of the disagreement was related to the issue of who an NL's primary users will be. If an NL is designed primarily for the benefit of teaching faculty, then it should focus on making high-quality and easily adapted teaching materials available. If an NL is to satisfy the needs of a broader spectrum of learners, then its content should include many other types of materials and media.

Workshop discussions also focused on whether an NL should store as well as commission content (e.g., an interactive, multimedia-based calculus curriculum), serve primarily as a resource that electronically "points" users to information stored on other computers (e.g., the Web sites on which such calculus curricula are maintained), or do both. A broad consensus developed to support, at a minimum, an NL that points to useful materials. Pointers are much less expensive to create and maintain than what is required to acquire, store, and provide access to stored materials. Pointers also allow input about content from a wider variety of interests and organizations in the SME&T community and enable owners of the materials to update and revise content as disciplines and knowledge advance. Importantly, pointing to materials rather than storing and disseminating them allows an NL to minimize current legal challenges related to intellectual property rights, copyright law, and licensing agreements for commercially produced content or content produced by individuals seeking compensation for their efforts (see additional discussion of this issue beginning on page 51).

However, an NL could face several important constraints if it were to rely exclusively on the use of pointers rather than the storage of existing materials or the commissioning of at least some materials. First, the project would have to await the availability of consistently reliable software and standardized protocols that would enable its registry regularly to update the addresses of materials stored elsewhere on the Internet (these tools are currently under development; see page 27). Second, the proposed NL's ability to catalyze development of or exert control over the quality of materials specifically suited to this electronic medium could be very limited. Finally, if the intellectual content or quality of materials in some disciplines was not sufficient to provide the proposed NL with "critical mass," its development and evolution would have to be accompanied by other, potentially costly programs that create such content and make it widely available to users.

Also discussed was whether an NL should simply make materials available (either directly or by pointing to other Web sites), such as traditional libraries do now, or also allow users to contribute materials. Contributed materials might include items such as new teaching tools and modules or annotations (e.g., reviews, comments by users, supplemental information) of materials already available.

With regard to the selection of materials, workshop participants also asked, on what basis should materials, pointers to material, or annotations be reviewed and evaluated for inclusion in the proposed NL? Strategies for editorial oversight of this resource would differ depending on whether it merely points to other materials and resources or whether it stores and disseminates them. It is more difficult to establish standards for materials outside a library than for materials already owned by that library. The workshop participants agreed that some mechanism for distinguishing formally reviewed from unreviewed material would be necessary, both from the standpoint of users and for the credibility of the resource itself.

At the same time, many participants pointed to the value of an NL's making available "works-in-progress" and informal papers, syllabi, notes, and other similar unrefereed materials that could help inform the community of users about new and interesting ideas. Some workshop participants noted, however, that nonrefereed material now constitutes the bulk of materials and information on the World Wide Web. Including significant amounts of such unvetted material in the NL's holdings or registry could lead to rapid deterioration of the overall quality of materials made available or pointed to by the proposed NL.

Precisely how to review material was the subject of lively discussion. Some of the possibilities discussed included


• Using approaches currently employed by refereed journals. There was concern, however, that given the variety of materials that might be included in the proposed NL, current review processes may result in the exclusion of experimental or other innovative materials that embrace new media, formats, or methods.

• Delegating at least some oversight to professional societies. There was concern, however, that this approach would compromise recent trends toward interdisciplinarity in undergraduate SME&T education because most professional societies promote knowledge and activities that are primarily within the purview of their specific disciplines.

• Soliciting and encouraging commentary by users of an NL's materials. There was concern, however, that this approach might not allow for a representative or informed sampling of opinions from users.


A fourth issue for the workshop was who the creators of content should be. Here there was fairly broad agreement that creators could include faculty, publishers, professional societies, and students.

The issue of "critical mass" arose again in the context of the creation of content. Experience with the management of information resources suggests that if a library does not contain sufficient amounts of information that have been evaluated for quality and value it will fall into disuse due to user disappointment. Such disappointment is subjective, but if the information in an NL is either not sufficient in quantity or quality, it is to be expected and could be devastating. Issues of critical mass have implications for what is entered into an NL and how long those materials are archived.

A final content issue was the need for tools to facilitate browsing and searching by users. For example, experience with the Internet clearly indicates that simple keyword searches, though sometimes useful, are inadequate and too indiscriminate when searching large volumes of information. Interactive, "intelligent" tools that facilitate searching for materials, especially those that have been designed specifically to exploit an NL's electronic capabilities, would be very useful.

Is the proposed NL a library?

Most workshop participants agreed that, as envisioned during this meeting, an NL for undergraduate SME&T education certainly would have many of the characteristics of traditional libraries, such as tools for indexing and searching. This resource also might have other traditional library capacities, such as classifying and archiving content. However, the digital National Library for undergraduate SME&T education was envisioned as being designed with many other features that are not found in traditional libraries, such as the capacity for users to publicly comment about or add new materials and to work interactively with and upgrade materials already available in the proposed NL. Workshop participants suggested that a better set of terms or descriptors be devised to reflect more accurately the proposed NL's broader vision and objectives and to convey better to potential users how the resource might be utilized. Additional discussion of this issue can be found on page 34.




IMPLEMENTATION: ECONOMIC, LEGAL, AND TECHNOLOGY ISSUES

In addition to focusing on the potential value of an NL for undergraduate SME&T education, the workshop also addressed a number of critical issues related to the implementation and deployment of this resource.

Economic Issues

While government agencies and private foundations might provide key start-up funding, workshop participants agreed that, over the long term, the proposed NL will likely need to become self-sustaining. However, there was little consensus about how best to address questions of economic viability and sustainability, as well as equity of access for all people who wish to utilize this resource as a teaching and learning tool.

Economic viability reflects both costs and revenues. In terms of costs, any NL initiative will require a variety of editorial services (similar to those required of journals). Basic technology services also will be required to store materials, to develop and update pointers to materials on other computers, and to oversee the annotation of materials in the collection or list of pointers. Searching and authoring tools will need to be developed and continually updated as technology advances. Decisions will need to be made about how much material should be archived and for how long. The problem of archiving and the costs associated with it will be difficult to determine until the overseers of the proposed NL establish what types of non-traditional materials will be included and from which sources. Here again, there is a great need to identify potential users and patterns of use of such a resource. Workshop participants noted that an added benefit of investing in an NL is that it will lessen development time and thus lower costs for future projects in other learning communities such as the humanities and social and behavioral sciences.

Revenue is likely to be a continuing problem and must be confronted directly and early in the development process if this NL is to remain viable and able to grow and adapt to changing user needs and advancing technology. Workshop participants discussed the imposition of fees on students, educational institutions, and publishers but came to no agreement about the desirability or feasibility of charging any of these communities for an NL's services. Here again, the feasibility of any given option will depend on the clients an NL is intended to serve. If students utilize this resource extensively, then user fees collected by colleges and universities that subscribe to this NL (possibly in lieu of money that students might otherwise have spent for textbooks or laboratory fees) could be used to support it. Publishers might support an NL if it provided a plausible advertising showcase for their wares. Educational institutions (e.g., through some kind of consortia) and professional societies also might support an NL financially if it provides recognized value to faculty and members, respectively. It also is important to note that an NL is likely to emerge in an era of electronic commerce, which may provide the means for micro-payments that could sustain it in the long term.

While the costs of hardware and software are likely to decrease over time, initial costs for establishing an NL and the certainty of upgrades that will be needed to maintain this NL on the cutting edge of technology²² must be factored into total costs along with the considerable costs of the personnel that will be required to maintain its infrastructure and oversee its content. Other direct or indirect costs that could affect how extensively the proposed NL is utilized include those associated with training users. Should those costs be borne by the users themselves or their places of employment? Or, should public or private funds be made available to provide this training, at least initially? Equity of access always must be a high priority in any discussions or decisions about the costs and financing of an NL. Workshop participants discussed but could not provide definitive answers to these difficult questions.

Legal Issues

Workshop participants identified a number of legal issues that would need to be addressed before an NL could become a practical reality. These include


• Intellectual Property (IP).  IP issues in the context of an NL are very similar to those facing any on line provider of content. However, some types of material may not be as problematic (e.g., course notes) because remuneration to authors of these materials is unlikely to be involved. Some of these issues could be resolved if an NL pointed to rather than provided materials. Users would then incur responsibility for compensating providers through arrangements defined by the providers (although the governing body for the proposed NL may wish to help define some of those financial arrangements as a condition for referring users to the provider's Web site).

• Liability.  Given that online references are sometimes perceived as more "threatening" than the same information contained in books (e.g., a laboratory experiment that could potentially harm an unwary student), NL materials that involve risk to students may involve liability for the overseers of that NL or for authors and creators of materials to which it points.

• Privacy.  To the extent that students use materials or information found in an NL (e.g., an online self-diagnostic test that may be packaged with educational materials), well-meaning faculty may be interested in the extent and nature of such usage. Would obtaining such information impinge on students' rights of and expectations for privacy?


The general conclusion of the workshop was that these legal issues will not be resolved in the context of this project. Rather, a regime of law and practice will gradually evolve as online publishing and dissemination of information becomes more extensive. Any NL that is built for undergraduate SME&T education will need to be flexible enough to accommodate a wide range of possible legal regimes and challenges.

Technology Issues

Workshop participants discussed many technology-related issues. First and foremost, consensus was reached that the primary purpose of the proposed NL should be to satisfy the needs of users rather than to be a vehicle for advancing technology or research about digital libraries. Participants did recognize that, from time to time, new tools will be created that might be used in entirely unexpected ways. Nevertheless, on balance, they expressed a strong sentiment for placing users first and making sure that any technologies employed to operate this resource are developed and deployed to accommodate the needs of NL users.

Associated with their desire to serve the needs of users, some workshop participants questioned the conventional wisdom of making this proposed NL available only via the Internet. Some institutions of higher education do not now enjoy access to the Internet, and others have only limited access through data lines that do not accommodate rapid downloading of large applications or data sets. The problem is exacerbated in other countries; if the proposed NL is designed to serve the needs of users beyond the United States, the Internet may not be the appropriate vehicle for delivering such services to everyone. Rather, it was suggested that other formats, such as CD-ROM sets, might be included as components of any strategy that develops for delivering information to users from this resource. Other participants noted that the development of Internet II also could limit access to the proposed NL, since Internet II may only be available to selected colleges and universities. Here again, equity of access must become a prime issue for consideration.

Workshop participants felt that emerging technologies that are being developed by other digital library projects now under way and should be utilized by this project. But such technologies should be deployed with careful attention to how they improve services to users, life-cycle costs, and long-term usability and sustainability. Workshop participants emphasized that an NL should employ technologies that are adaptive, flexible, and responsive to unforeseen user needs and problems. It is impossible to anticipate every way in which individual users might want to take advantage of this NL's resources, but customizable tools should be available to facilitate its use by as many people as possible. New applications and modules should be designed to operate with software that is widely available for other applications (e.g., commonly used spreadsheets). This design would reduce the time required for users to learn how to work with such materials.

An NL that is sustained over time must plan to accommodate new information technologies as they emerge while dealing with content that was designed to run on older computers and operating systems that are or may become incompatible with newer hardware and software. This will apply to search engines and other tools that might be used in an NL, as well as for pedagogical materials developed by one user or institution and offered to or shared by others.

Finally, the workshop participants reached broad agreement that the technology in an NL must be developed with advice and oversight from professionals who are most knowledgeable about how people organize and use information: librarians and social and behavioral scientists. Librarians grapple continually with technology interface issues as they strive to make electronic information available and complementary to materials in other formats. They also give much consideration to how such information should be stored and archived. Similarly, social and behavioral scientists can help with considerations of how individuals organize and contextualize information. Without the informed perspectives of librarians and social and behavioral scientists, an NL is likely not to optimize opportunities for teaching and learning.



---

NOTE

²²While incorporating the latest advances in technology into an NL's arsenal is tempting, this desire must be tempered by the fact that many of the computers that will be used to access the NL may not have the capacity to use new software platforms or programs that require excessive amounts of memory. Here again, it is imperative that any project include a complete discussion of users' and institutions' needs and limitations.