Appendix D
Assessment and Evaluation Data on Case Study Projects

This Appendix includes assessment and evaluation data from several of the workshop’s case studies. This information is provided to show the variety of approaches used in measuring the effectiveness of a program as judged by its various participants. These data were gathered from presenters and from resources available on the Internet. When known, Web site addresses for the data have been provided.

CASE STUDY 2, BEN VAN DER PLUIJM, GLOBAL CHANGE PROGRAM, UNIVERSITY OF MICHIGAN

Below are data from the end-of-semester survey administered to Global Change I students in fall 1999. a

a  

These data were collected by an evaluation team led by Professor Eric Dey (Higher Education, University of Michigan) as part of the School of Education’s Undergraduate Curriculum Development Testbed (UCDT). See http://www.wcer.wisc.edu/nise/cl1/ilt/case/michigan/michigan.htm.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 53
Appendix D Assessment and Evaluation Data on Case Study Projects This Appendix includes assessment and evaluation data from several of the workshop’s case studies. This information is provided to show the variety of approaches used in measuring the effectiveness of a program as judged by its various participants. These data were gathered from presenters and from resources available on the Internet. When known, Web site addresses for the data have been provided. CASE STUDY 2, BEN VAN DER PLUIJM, GLOBAL CHANGE PROGRAM, UNIVERSITY OF MICHIGAN Below are data from the end-of-semester survey administered to Global Change I students in fall 1999. a a   These data were collected by an evaluation team led by Professor Eric Dey (Higher Education, University of Michigan) as part of the School of Education’s Undergraduate Curriculum Development Testbed (UCDT). See http://www.wcer.wisc.edu/nise/cl1/ilt/case/michigan/michigan.htm.

OCR for page 53
I. LAB EXPERIENCE   Strongly Agree Agree Neutral Disagree Strongly Disagree 1. The lab assignments seem carefully chosen. 12.3% 67.7% 10.8% 4.6% 4.6% 2. The lab assignments are intellectually challenging. 7.7% 72.3% 12.3% 3.1% 4.6% 3. Laboratory assignments make an important contribution to my understanding of the topics discussed in lecture. 7.7% 44.6% 32.3% 7.7% 7.7% 4. ArcView has helped me understand Global Change concepts and principles. 13.8% 56.9% 20.0% 4.6% 4.6% 5. I feel confident in my ability to use ArcView to construct models. 24.6% 67.7% 4.6% 3.1% 0.0% 6. ArcView helps me understand the relationships among different variables. 18.5% 63.1% 10.8% 6.2% 1.5% II. LECTURE EXPERIENCE     Strongly Agree Agree Neutral Disagree Strongly Disagree 1. Having several instructors give the lecture contributes to my understanding of the concepts and principles related to Global Change II. 38.5% 46.2% 10.8% 3.1% 1.5% 2. The transition from one instructor to the next interferes with my ability to learn. 6.2% 7.7% 12.3% 66.2% 7.7% 3. I have learned a good deal of factual material in this course. 36.9% 56.9% 6.2% 0.0% 0.0%

OCR for page 53
    Strongly Agree Agree Neutral Disagree Strongly Disagree 4. The knowledge I have gained through this course has improved my ability to participate in debates about global change. 30.8% 63.1% 6.2% 0.0% 0.0% 5. This course has encouraged me to think critically about global change. 46.2% 50.8% 3.1% 0.0% 0.0% 6. It is difficult for me to understand how topics covered in the lecture fit together. 3.1% 16.9% 7.7% 60.0% 12.3% III. WEB EXPERIENCE   Strongly Agree Agree Neutral Disagree Strongly Disagree 1. Using the web has made a significant contribution to my learning. 32.3% 50.8% 12.3% 4.6% 0.0% 2. The links from the Global Change website to other internet websites have provided me with helpful information. 15.4% 24.6% 50.8% 6.2% 3.1% 3. I feel confident in my ability to use the web to gather information about global change. 52.3% 41.5% 4.6% 1.5% 0.0% 4. I have used the web skills I have acquired in this course to complete academic work for other classes. 16.9% 43.1% 26.2% 12.3% 1.5% 5. I have utilized the web skills I have developed in this course to investigate areas that interest me. 21.5% 38.5% 29.2% 9.2% 1.5%

OCR for page 53
IV. PERSONAL GROWTH     Strongly Agree Agree Neutral Disagree Strongly Disagree 1. I have deepened my interest in the subject matter of this course. 39.4% 51.5% 6.1% 1.5% 1.5% 2. I am enthusiastic about the course material. 31.8% 50.0% 15.2% 1.5% 1.5% 3. I feel like I make an important contribution to the learning of others in the course. 12.1% 36.4% 40.9% 9.1% 1.5% 4. I have had opportunities to help other students in the course learn about Global Change concepts and principles. 10.6% 42.4% 34.8% 7.6% 4.5% 5. I feel empowered to act on what I have learned. 22.7% 57.6% 16.7% 3.0% 0.0% CASE STUDY 4 MONICA ELSER, CENTRAL ARIZONA-PHOENIX LONG TERM ECOLOGICAL RESEARCH PROJECT (CAP-LTER), ARIZONA STATE UNIVERSITY http://caplter.asu.edu/progress.htm From CAP LTER 2000-2001 Annual Report (the entire report can be found at http://caplter.asu.edu/progress.htm) We reach out to the K-12 community through Ecology Explorers, a program that aims to: develop and implement a schoolyard ecology program where students collect data similar to Central Arizona-Phoenix Long Term Ecological Research Project (CAP LTER) data, enter results into our database, share data with other schools, and develop hypotheses and experiments to explain their findings; improve science literacy by exposing students and teachers to real research conducted by university-level scientists; enhance teachers’ capabilities to design lessons and activities that use scientific inquiry and encourage interest in science; provide access to and promote the use of CAP LTER materials and information; encourage collaboration between CAP LTER researchers and the K-12 community; provide

OCR for page 53
students an opportunity to share their research with other children, adults, and CAP LTER researchers through poster presentations at SEE ASU and the CAP LTER poster symposium, and through our new Kid’s Online Newsletter. From the initial collaboration sparked with 12 schools in 1998, Ecology Explorers has expanded to include 34 schools, 46 teachers, 14 school districts, and 3 charter schools. Popular summer workshops and internships have engaged numerous teachers in our schoolyard sampling protocols for the vegetation survey, ground arthropod investigation, bird survey, and plant/insect interaction study and biogeochemical cycles. This year we have developed threenew day-long workshops based on teacher requests. The topics covered in the workshops were: (1) mapping the schoolyard; (2) analyzing data; (3) insects in the classroom. A total of 21 teachers participated in these workshops. The teacher evaluations suggested that these workshops addressed their needs and were beneficial. This summer’s program will include 16 new teachers (2 of whom are from school districts new to our program) and more than 12 ASU personnel. We will be offering two 2-week internships that allow the teachers to participate in a research project and learn how to collect and analyze data. They will be introduced to several hands-on, inquiry-based lessons developed from previous workshops and create new lesson plans that will be added to the Ecology Explorers Web site. In January 2001, we surveyed our teachers to assess whether our programs are meeting their needs: 79% use our protocols in some way (29% follow the protocols and enter data, 26% conduct the protocols but do not enter data, and 24% pick and choose among parts of the protocols that meet the needs of their class). Eighteen percent were not currently involved but planned to be soon, while 6% had been involved in the past but not currently. Six percent were not currently involved and did not plan to be involved in the future. We also found that 92% of the teachers had worked with the CAP LTER education personnel and that this was an important component of the program. We found that teachers use the Ecology Explorers Web site more than their students. Items that the teachers would like to see included on the Web site were: lesson plans (47%), Web links (47%), extension activities (50%), graphs (53%), easier data entry and retrieval (30%). Teachers consistently reported that the reason they like this program is the integration of real research projects into their curriculum and the support they receive from CAP LTER staff. Participating teachers have applauded Ecology Explorers for the following attributes: “Authentic

OCR for page 53
learning activities for students. Life skills for students. Outstanding support from CAP LTER staff. Chance to participate in a long-range project. Good way to integrate skills and curriculum.” Based on feedback from teachers, we have developed several new Web features this year (http://caplter.asu.edu/explorers): online lesson plans developed by Ecology Explorer teachers, online slide sets, resource lists (Web-based and print),“Meet the Scientist” interviews, and some extension activities for several of the protocols. We have been working with the CAP LTER data personnel to make the data entry and retrieval features easier to use and hope to be able to produce real-time graphs within the next year. We are also working on a flash animation to simulate the ground arthropod protocol. Through informal discussions with teachers, we know that they have a better understanding of ecological research, students’ enthusiasm for projects exceeded expectations, students felt projects were important because of the ASU connection and were willing to put in extra effort to carry out the projects, more parents were involved than anticipated, and workshops/internships were valuable and enhanced their ability to teach science. Teacher’s have also reported that students’ math abilities improved as a result of participating in Ecology Explorers. Participating in poster presentations enhanced students’ communication skills. The program is aligned with the AZ State Education Standards, including science, math, writing, social science, and technology standards. CASE STUDY 5, BETTY CARVELLAS, ESSEX HIGH SCHOOL, VERMONT The information below is an excerpt from the TEA Program Evaluation Year Two Report. More information is available at http://tea.rice.edu. Research principal investigators (PIs) saw the experience as improving teachers’ scientific research skills, while at the same time helping the researchers and their teams gain a better understanding of K-12 education. One research PI said, Our teacher expressed an interest in our research well over a year before joining us in Antarctica. This allowed us to build a strong working relationship well before reaching the field. Before reaching the field and while in the field she continued her strong interest in the underlying experience making her a valuable member of the team. Finally, the improved understanding of the K-

OCR for page 53
Researcher PIs Overall Assessments (n = 21) Somewhat Strongly Disagree Disagree Somewhat Disagree Statement Agree Agree Strongly Agree N/A 1. Having a TEA on my team has been a rewarding experience for me. — — 4.8 4.8 19.0 71.4 — 9. Having a TEA teacher on my team helped me and my team gain a better understanding of K-12 education. — — — 23.8 33.3 42.9 — 10. Working with a TEA teacher has provided me a good opportunity to share my research and scientific interests with other teachers and students. — — — 19.0 33.3 47.6 — 25. My TEA teacher gained a better understanding of how scientific research is conducted as a result of being a member of my research team. — 4.8 — — 19.0 76.2 — 26. The field research experience increased my TEA teacher’s scientific knowledge and skills. — — — 9.5 33.3 57.1 — 30. I would take another TEA teacher with me into the field. — 4.8 — 4.8 28.6 61.9 — 31. I would take the same TEA teacher with me into the field again. 4.8 4.8 — 9.5 19.0 61.9 — 32. I would recommend to my colleagues that they take a TEA teacher into the field. — — 4.8 9.5 28.6 57.1 —

OCR for page 53
Teacher’s Interaction with PI Statement Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree N/A 9. My principal investigator (PI) involved me in a meaningful way in his/her research before going into the field. 7.7 7.7 — 23.1 23.1 38.5 — 10. My PI involved me in a meaning ful way in his/her research during the field research experience. 7.7 — — — 38.5 46.2 7.7 11. My PI helped me define a field experience that aligned with my interests. 7.7 7.7 — 7.7 38.5 38.5 — 12. My PI made his/her expectations of me clear before I went into the field. 7.7 — 7.7 15.4 46.2 23.1 — 13. I was able to make my expectations and responsibilities clear to my PI and his/her team before I went into the field. — — 7.7 7.7 61.5 23.1 — 14. My PI provided me the time needed during the field experience for me to complete my TEA program responsibilities. — — — 7.7 15.4 76.9 — 15. My PI treated me like a professional. — — — 15.4 15.4 69.2 — 17. I think my PI developed a better understanding of teachers by my involvement in the TEA project. 7.7 — — 30.8 — 61.5 — 30. My PI has continued to involve me in a meaningful way in his/her research since my field research experience. 15.4 23.1 15.4 23.1 15.4 7.7 —

OCR for page 53
12 environment enriched us all, as well as the understanding of what sorts of ideas and activities would translate well to her students. Another researcher described the experience in these words, Overall, the experience was an excellent opportunity for my TEA teacher to learn first hand about Antarctic science and ecology. I gained a tremendous insight, and rewarding experiences, after 2 days of classes teaching junior high students. Every scientist should have this experience. Based on this survey evidence, several factors in addition to conducting scientific field research itself, contribute to successful field experience for teachers. These include: (1) clear communications of responsibilities and expectations, by both parties, prior to the field work; (2) aligning of field experiences with teachers’ interests; (3) involving teachers in meaningful ways in the PIs research projects; and (4) treating teachers like the professionals they are, by both the research PIs and their research teams. CASE STUDY 6, CARY SNEIDER, MUSEUM OF SCIENCE, BOSTON, MASSACHUSETTS Predicting the Future: The Science and Technology of Weather Forecasting NSF Informal Science Education grant to the Museum of Science, Boston. Grant period 8/1/02 to 7/31/05 Despite the important role that weather plays in our lives, misconceptions and ignorance about weather are widespread. Few adults can read a weather map well enough to predict how conditions are likely to change in the next few hours, and most are unaware of typical weather patterns where they live. Yet there is a great deal of interest in the weather, and many people turn on the news just to see the current forecast. In fact, opportunities to learn about the weather through informal means may be one of the best opportunities to teach people about Earth systems. Additionally, meteorologists (atmospheric scientists whom people encounter daily) provide wonderful opportunities to communicate the processes of science, such as data collection and analysis, modeling, and prediction. This proposal is predicated on the conviction that everyone can make

OCR for page 53
accurate short-term predictions of the weather (called nowcasting.) Furthermore, in the process of learning to forecast the weather, learners can deepen their understanding of Earth systems as well as gain insight into the nature of science and technology. To accomplish these goals, experienced Museum of Science personnel will work with meteorologists and other atmospheric scientists to develop a project entitled Predicting the Future: The Science and Technology of Weather Forecasting, which will have the following components: An exhibition on weather will be available to more than 1.2 million people who visit the Museum of Science’s exhibit galleries every year. At the exhibit, visitors will learn how to forecast the weather over the next few hours using different levels of technology, including naked eye observations, data from weather maps, and real-time images from space satellites and ground radar stations. Programs for Students and Teachers will help this important group of visitors make effective use of the exhibition, and extend the field trip by using the interactive website described below. In-depth programs will be provided for teachers and students from more than 100 WeatherNet schools in New England, while other programs will be made available to more than 250,000 students and teachers who visit the Museum every year. An interactive website will provide guidance in using online weather resources to predict the weather several hours in advance. The website will be accessible to anyone, free of charge, through the Museum’s home page, or in the exhibit gallery. While website numbers are difficult to estimate, the audience for this website is at least in the hundreds of thousands. Television spots will address the broadest audience—mostly adults who tune in the weather broadcast, and who may be open to spending one additional minute to learn something new about the process of weather forecasting. The television spots will be aired on WBZ-TV Channel 4 in the Boston area, shared with CBS affiliates, and provided as part of a larger series of educational programs to The Weather Channel. Viewership is expected to be in the millions. This project will build on a long-term partnership with WBZ Television and more than 100 local schools in which students and teachers monitor the weather on a continuous basis, providing a dense local network of weather data available to the public via WBZ and Museum of Science websites. Special workshops will be held for WeatherNet teachers, and

OCR for page 53
WeatherNet students will be invited to serve as volunteer interpreters in the exhibit gallery. WBZ will also create the television spots and provide assistance with other aspects of the overall project. Other partners include TERC in Cambridge, with responsibility for developing the website, AER, Inc. will provide real-time analysis and display of satellite images to simplify interpretation, Blue Hill Meteorological Observatory will serve as one site for teacher education, and Selinda Research Associates, will conduct an independent evaluation of the project. Evaluation To ensure that the website, exhibit components, school programs, and TV spots are effectively engineered and coordinated to accomplish the project’s goals, a significant portion of the effort will be devoted to evaluation. These evaluation activities will include: (a) front end evaluation to check and refine previous studies concerning people’s understanding and misconceptions about weather; (b) formative evaluation to improve the educational effectiveness of the exhibit, programs, and website; and (c) summative evaluation to measure the effectiveness of the overall effort. Over the years, the Museum of Science has come to appreciate the importance of evaluation for maximizing the potential of creating successful visitor experiences. In “Predicting the Future,” evaluation will be planned and implemented by Selinda Research Associates and will be an integral and indispensable component of the overall project. In this project, Selinda Research Associates—in consultation with Museum of Science Staff—will plan and implement three evaluation studies: front-end, formative, and summative. Front end. This project will require a thorough understanding of the target audiences and their connection to weather. Selinda Research Associates will develop and implement an evaluation study that investigates visitors’ preconceptions and understandings of weather. It will investigate the extent to which and the ways in which the potential audience for this project—including users of the exhibition, programs, website, and television spots—think about and understand weather and the process of weather forecasting. It will also assess the ability of different age groups to grasp the requisite concepts and learn forecasting skills. Beginning with an extensive review of the literature from the fields of the public understanding of science and visitor studies, and building on

OCR for page 53
previous museum evaluation work done on weather-related projects, a preliminary understanding of the target audience will be developed. Building on the knowledge gained from the review of the literature, in-depth interviews will be also be conducted to develop deeper understandings of visitors and weather. Research questions will include: What topics will be most interesting to visitors? How do visitors understand and think about weather and its effect on their lives? What is their understanding of what meteorologists do? How do they interpret weather maps? What questions do they have? What may be effective hooks for them? What misconceptions do they have? The results of the front-end evaluation study will be summarized and presented to the project team to help inform the planning process. Formative evaluation will be conducted to identify revisions and improvements to the components of “Predicting the Future.” Exhibit prototypes will be tested in the Museum’s “Test Tube Gallery.” Formative evaluation will also be conducted of the website, TV spots, and the program throughout the development process. Using a rapid-prototyping process, results from the formative evaluation will immediately inform the design/ development decisions. Techniques will include participant and unobtrusive observation, and in-depth interview. Data analysis will use a modified inductive constant comparison approach (Lincoln, Y.S. and E.G. Guba. 1985. Naturalistic Inquiry. Beverly Hills, CA: Sage Publication). Questions will include: Can visitors use this component? What seems to work well? What doesn’t work as well? How can this component be improved so that more visitors will be able to be more successful? Throughout the formative evaluation process, findings will be quickly communicated to the project team so that changes can be made to the prototype units. Summative. At the end of the development of the components for this project, it will be important to assess the visitor experience and determine the extent to which the overall effort was successful. Using a combination of tracking and timing, unobtrusive and participant observation, and in-depth exit interviews, the visitor experience will be assessed. Emphasis will be placed on (a) the outcomes of visitors’ experiences (What did they learn that they didn’t know before? Were they able to successfully use what they learned to make weather predictions? Did they develop a greater appreciation for the processes of science? And for the role of mathematics and engineering in weather prediction?) (b) the ways they engaged (What did

OCR for page 53
they do? What was the quality and meaningfulness of their social interactions? In what ways were they intellectually engaged?) and (c) the experi ence itself (Did they enjoy themselves? Was their curiosity piqued? Were they appropriately challenged? Did they feel in charge of their learning? Were there opportunities for playfulness?). Data will be gathered and analyzed, and a written evaluation report submitted to the project team and the National Science Foundation. Finally, the Museum of Science has budgeted funds to remediate the exhibit based on the results of the summative evaluation.