This chapter presents a preliminary evaluation of the Institute.56 It begins with an account of the final facilitator debriefing at the end of the Institute and also includes data from the survey (see Appendix G) sent to participants three weeks later and their open-ended comments about particular aspects of the meeting. Insights from the outcomes of the implementation grants and the discussions at the reunion are included as well. Finally, the committee offers its judgments, based on the experience of designing and implementing the Institute, to inform similar current and future activities.
Final Facilitator Team57Debriefing
The facilitator team members met immediately after the Institute to share their thoughts about the event. The majority of the participants were committed to implementing the educational methods with a focus on responsible science at their home institutions. However, it was clear that many of the ideas introduced at the Institute were new to the participants. It also was clear that, in contrast to the National Academies Summer Institutes on Undergraduate Biology Education (NASI), a smaller amount of instruction about pedagogy per se (versus modeling pedagogy during discussions about responsible science) would be easier for participants to absorb and process.
Some of the Institute’s potential impact was lost because of the lack of advance preparation. Unfortunately, the committee’s expectation that participants would read the background materials prior to the Institute was not made clear. For future institutes it will be important to convey as clearly as possible everything what the participants are expected to do in advance. This also might include offering a series of questions or dilemmas to be considered during the Institute. Such questions, conveyed in cover letters or emails, rather than the background readings themselves, would engender greater interest and curiosity and alert participants to the kinds of problem solving to be undertaken during the Institute.
To be accepted to the NASI, one member per team was expected to participate in the reunion meeting during the academic year 2012-2013. The facilitator team agreed that it would be very important to (1) continue to provide all participants access to the Institute’s materials
56 As discussed in Chapters 3 and 4, in the field of education research “assessment” and “evaluation” are different concepts. Assessment refers to “tools for measuring progress toward and achievement of the learning goal” (Handelsman et al., 2007:19), while evaluation refers to “the process of analyzing the results of assessment and determining whether the goals have been achieved” (Handelsman et al., 2007:20).
57 The “facilitator team” includes members of the committee and invited individuals who worked with participants at the Institute.
and (2) be available to help participants address questions after they return to their home institutions.
The facilitators and other leaders of NASI routinely identify participants who might be invited to serve as facilitators at future NASI. Identifying and preparing facilitators from the participant pool enables them to reflect on the goals, objectives, and implementation strategies from two perspectives and enables them to become leaders for disseminating NASI’s goals and practices. The facilitator team concluded that a similar model would be appropriate for future Institutes.
As part of the evaluation process, the facilitator team developed a web-based survey for the participants. Three weeks after the Institute participants received an invitation to take the survey together with the Request for Applications for the implementation grants (see Chapter 5). Twenty-six of 28 participants responded to the survey, the results of which are described in the next several sections.
General Information about the Participants General characteristics of the participants were discussed in Chapter 4 as part of the committee’s approach to recruitment. Figure 6-1 shows that most participants identified themselves as university faculty while a few identified themselves as academic administrators.
The survey asked participants to indicate whether they teach primarily undergraduates, graduate or postdoctoral students, or others. The results are shown in Figure 6-2. Among the responses to the third choice (“Other”) were demonstrator (i.e., a master’s-level student), chief researcher, and faculty who teach both undergraduate and graduate students.
Participants were also asked to indicate up to three reasons they chose to attend the Institute. Figure 6-3 shows the percentage of each selected option.
A To reconnect with colleagues who share my interest in responsible conduct of science
B To meet colleagues from my country who share interests in responsible conduct of science
C To meet colleagues from other countries who share interests in responsible conduct of science
D To become more involved with future efforts to improve education about the responsible conduct of research internationally
E To deepen my understanding of the issues related to the responsible conduct of science
F To become more involved with future efforts to improve education about the responsible conduct of research in my country
G To discover tools, resources, and best practices for incorporating evidence-based teaching techniques into my courses
Participants’ Overall Rating of the Institute
Participants were asked to rate different aspects of the Institute; Figure 6-4 shows that more than 80 percent rated the quality of the sessions as either excellent or very good. There was a greater diversity of responses on questions about the use and balance of time spent in plenary and breakout sessions.
Figure 6-5 illustrates participants’ high levels of satisfaction with the overall goals of the Institute, the instructional materials, and the relevance of the topics to their professional careers.
Participants' Ratings of the Institute's Sessions, Delivery of Workshop Material and Group Work
A Quality of sessions about the responsible conduct of science
B Quality of sessions about the scientific basis for the use of active learning techniques
C Inclusion of information and perspectives from a diverse range of views
D Amount of time devoted to discussions during plenary sessions
E Balance of time spent in whole group and team breakout sessions
F Helpfulness of your breakout group’s facilitators
Participants' Ratings of the Institute's Goals, Instructional Materials, and Relevance to Their Careers
A Clarity of Institute’s goals and objectives
B Relevance of topics that were presented in relation to the stated goals of the Institute
C Usefulness of resources provided by the organizers and presenters (e.g., background resources in the Dropbox and briefing book)
D Value of the Institute as a learning or professional development experience
E Relevance to you and your work of the issues presented
F Time to meet and interact with other participants
When asked “If the National Academies were to organize and host additional Institutes or related activities on this topic in the future, would you be interested in participating?” 81 percent of the participants selected “definitely,” with the remaining selecting “maybe.” Of those who indicated they would like to be involved in future Institutes, 62 percent wrote that they would like to be a facilitator.
Participants were asked what they found to be particularly effective or not effective about the Institute. The majority of comments indicated that the Institute was effective for many reasons, but some reflected that the pace of the Institute was intense and the schedule crowded with too many subjects. Table 6-1 lists the participants’ responses (edited for clarity), organized by effective and ineffective aspects of the Institute.
TABLE 6-1 Effective and Ineffective Aspects of the Institute. SOURCE: Information compiled by the committee.
Effective Aspects of the Institute
• Interactive comprehensive coverage of all topics in a friendly yet responsible environment.
• Round table discussion and cases are effective
• The open discussion was very effective…the organization of the groups at the beginning and during the workshop was great. Talks were unexpectedly awesome.
• Effective points: 1- Active interaction of well qualified trainers. 2- Time management. 3- Clear follow up plan. 4- Appropriate class facilities. 5- Hospitality.
• Everything was very interesting and very exciting: 1. Active Learning Techniques 2. Trainers 3. Scientific Material 4. Work in Teams 5. Exchange of Experiences
• Highly experienced faculty with simple transfer of data to participants
• Effective: Knowing other faculties nationally and internationally. The spirit of cooperation made the institute pass like one day.
• Smooth cruising into the presentation and discussion of the contents of the Institute and also dealing firmly and friendly from the institute presenters and facilitators with the participants.
• In my opinion all training sessions were effective.
• Conducting research responsibly; the development of professionalism in science; being part of the responsible scientific community.
• The committee and facilitators were serious and friendly at the same time. The use of all materials used in a manner not boring.
• The use of new approach in teaching and the use of dual science
• Most effective was Pedagogy
• Most of the activities in the institute were particularly effective.
• I found the effective points were the group discussion and how the facilitator helped us to get correct aspects and encouraged every participant to integrate with each other. Using the clickers during the lecture was new to me. How to teach the complex and difficult scientific topics in thoughtful ways.
• This is the first time I’ve attended such an intensive educational workshop. The tools such as case studies and role playing, I found more effective for me. The iclicker was also an effective tool to use for evaluation, however, I don’t think I will use it at my institution with the large numbers of students….probably very expensive to get.
• I think the workshop was very valuable and gave me more experience and also gave me the chance to meet and deal with other international colleagues.
• The Institute was effective for many reasons: - It was an excellent training for me to be confronted to work with people from developed countries and countries who are facing the same problems as in my country. - To learn new tenets and pedagogical techniques for active learning. - Learn more about the different facets of what it means to conduct responsible science. - To share thoughts and learn on case studies about relevant topics: co-authorships, biosafety and biosecurity, international collaboration etc. - Develop a new network for future collaboration with mutual
• New teaching techniques and the assessment methods I found particularly effective for me.
• I found that using case study and other methods of interactive learning was very effective and I will apply in teaching courses in my institution. Moreover, subjects of discussion like misconduct and safe laboratory standards are very important and direct our minds to very critical issues.
• They worked as one team and shared in all discussing points and activities in the workshop
• The responsible conduct of research thru discussing issues related to mentoring, authorship and active learning
• Interactive session on scientific misconduct cases
• J’ai sincèrement admiré le sérieux des organisateurs et facilitateurs et leur engagement dans le travail pour mener bien et réussir les objectifs qu’ils s’étaient fixés. Personnellement j’ai énormément appris sur le plan professionnel bien sûr mais aussi sur le plan humain ou j’ai vu à l’oeuvre la générosité sans limite ni faille de certaines personnes, leur disponibilité à tout instant ainsi que leur penchant naturel à donner, à se rendre utiles sans pour autant espérer une contre partie. Tels furent à mes yeux les personnes qui ont pris en charge cette entreprise. Le groupe américain a était exemplaire à plus d’un titre… Que ses membres soient tous remerciés! (I sincerely admire the 'seriousness' (professionalism, effectiveness) of the organizers and facilitators and their commitment to successfully pursue the set goals. Personally, I learned an enormous amount at the professional level but also at the level of human relations observing in practice the limitless generosity and availability (of the aforementioned people). (I admired) their natural (spontaneous) offer to give (share) and to be of use without any compensation. Such have been the individuals in charge of this (whole) endeavor. The American group has been exemplary in more than one way (above and beyond the call of duty). All its members deserve (our) gratitude.)
Ineffective Aspects of the Institute
• The programme was very crowded.
• Dual use issues were delinquent.
• On the other hand, there were some issues regarding the place and the time of the workshop: (1) We took about 4 hours to travel from Amman to Aqaba and from Aqaba to Amman and this was fatiguing for me. (2) We start every day from (8 Am to 7 Pm) and this is too much time. (3) There is no entertainment and fun means during the workshop.
• The survey is ineffective.
• The intensive working hours is one drawback.
• I think the contents were very superficial as the institute tried to give us more than one subject in only one week as the pedagogy.
• Each topic should be a separate workshop.
NASI, which involves a variety of evidence-based approaches to active teaching, learning, engagement, and assessment, can be adapted to different topics, cultural contexts, and countries. In the course of reviewing the design and implementation of this Institute, the committee identified a number of insights that could help to improve future projects. They include logistical, academic, and cultural challenges and realities.
• Active engagement of committee members and Institute leaders before, during, and after the Institute is crucial.
• A detailed application and merit-based selection process can identify enthusiastic and committed participants who will, in turn, demonstrate the importance of such approaches to colleagues at their home institutions and in their disciplines.
• Teaching about and modeling pedagogy can play a significant role in the success of an Institute.
• The demanding pace of the Institute made it hard for some participants to comprehend the concepts and techniques fully and apply them during small group work. Future Institutes will benefit either by providing more time to integrate active learning with new content or by reducing the breadth or both.
• The design of resources and assessments for an Institute benefits from particular attention to linguistic and cultural differences among participants and facilitators. Working with partners from the region where the Institute will take place allows organizers to take into account local customs, traditions, and cultures in ways that remove barriers and foster stronger relationships among organizers and participants.
• NASI have demonstrated that a reunion of some participants after an Institute can provide new insights about participants’ challenges, resources, and opportunities for networking and for sustaining programs (for details, see Chapter 5). The Institute described in this report further confirmed that a reunion can be especially important for participants from developing countries. For example, by the end of the reunion in Jordan, the scientists who attended agreed that their ability to conduct their own work around responsible conduct and to reach other colleagues at their home institutions, across their individual countries, and in the MENA region as a whole could be expanded and sustained by establishing a network among them. They decided to use this network to share ideas, common challenges, and opportunities, and to develop joint proposals for future work.
• As with the development of NASI, new Institutes will require continuing experimentation with and evaluation of all aspects of their design. Feedback from the participants, combined with the results of their projects, can play an important role in future iterations.
• The introduction of both new pedagogies and new content at the same time can be a significant challenge for some participants. Reviewing background materials in advance of the Institute can lessen this impact. However, materials written in English about new concepts, such as active learning and dual use, may present obstacles for non-English speakers.
• Framing biosafety and dual use issues in the context of responsible science was
meaningful to many participants. However, based on conversations during plenary discussions with the participants who attended the reunion meeting in Amman, practical realities such as the lack of basic scientific equipment, reliable Internet connections, and access to scientific journals impede scientists in this region, and especially those from more impoverished nations, from undertaking research at a level where dual use issues raise concerns for them. People undertaking activities where research with dual use potential and/or misuse of technologies is to be a topic need to take this reality into account when planning their events or programs.
• Some concepts that are crucial to active learning, responsible science, and dual use cannot be expressed in Arabic. In most of the countries represented at this Institute, teaching about science occurs in English but instructors sometimes provide additional explanations or contexts in Arabic (or French in Algeria). Arabic-speaking scientists and students may interpret English words in ways that are different from what the organizers intend. For example, the facilitator team learned that there is only one Arabic word for the two English words “search” and “research,” which may contribute to misunderstanding the standards for plagiarism in English-language journals among Arabic-speaking scientists and students. For example, several participants told the group that when they ask their students to define “research,” the common response is to find the information in question on Google or another search engine. The students are not concerned about copying and pasting information from the Internet into their essays and research reports.
• Scientific research in the MENA region has advanced remarkably over the last generation. But participants reiterated that the lack of a formal framework and infrastructure for research in their countries (e.g., the absence of comprehensive policies and oversight structures regarding authorship, peer review, research with laboratory animals and human subjects, and biosafety) makes it difficult for scientists to follow international standards and to teach best practices in responsible science to their students.
• As the committee learned from the active learning exercise conducted on day 1 of the Institute, in which participants from each nation worked together to describe their country’s system of higher education, there are both similarities and differences in education philosophies, approaches to teaching and learning, facilities, and resources among nations. These differences need to be taken into consideration when planning future Institutes. A recent report funded by the Carnegie Corporation of New York provides useful data to help organizers of future events take these differences into consideration (Bhandari and El-Amine, 2012).
• The small grants awarded to participants were used creatively to address an array of educational needs that they identified, as noted in Table 5-1. In many cases these funds prompted subsequent institutional support to sustain participants’ instructional activities. However, as also occurs in the United States, limited funding restricted the ability of these motivated science educators to reach larger audiences who would benefit from instruction on responsible science, biosafety, and dual use issues.
At the reunion, discussions after each presentation and after all presenters had described their post-Institute activities revealed a great deal of variation in the ways participants in those activities were surveyed about their learning and the project’s efficacy. Assessment and evaluation are an issue for science faculty around the world. Providing additional guidance and models of survey instruments before such projects are undertaken could provide much more useful and usable data for future initiatives.
Taken together, these insights offer important lessons for the design and implementation of future programs in the MENA region as well as in other parts of the world.
IMPLICATIONS: NEXT STEPS AND SUSTAINABILITY
As discussed briefly in Chapter 2, NASI, which are intended to transform how undergraduate biology is taught in the United States, have recognized that fundamental change takes time. Similarly, the committee agrees that for meaningful change to be sustainable, the projects and lessons learned from the first Institute need to be followed by additional efforts. In modifying future Institutes or similar activities, these efforts would also need to take into account the insights gained through the committee’s evaluation work for the first Institute discussed above.
New Possibilities and Needs
Based on feedback from Institute participants, and others who became familiar with the Institute format as well as the committee, a series of ideas emerged about ways to reconfigure or extend the potential reach of the Institutes. The following four broad categories represent an amalgam of these suggestions:
• Implications of dual use
The committee was charged with addressing research with “dual use” potential in the context of responsible conduct of science as part of its Statement of Task (Box 1-2 in Chapter 1). However, as a result of both designing the Institute and engaging with its participants, it became clear to the committee that the term “dual use” might not be the most appropriate one to use to communicate to the next generation of scientists and the various publics the complexity of the issues. Through the case studies presented and the discussions, it became apparent that “multiple uses” might be a preferable descriptor since virtually all scientific activities are on a continuum from exemplary to malicious conduct.58 Given the differences between cultural norms, perspectives, and levels of scientific research among countries, scientists may be uncertain about boundaries of ethical/unethical behavior that “dual use” connotes because these behaviors are more complex than these two categories imply. There could, therefore, be value in emphasizing a continuum rather than a starker dichotomy of research and behavior as part of the discussions at the Institutes.
58 The term was adopted by the International Union of Pure and Applied Chemistry for the educational materials on Multiple Uses of Chemicals that it developed in 2007 in cooperation with the Organization for the Prohibition of Chemical Weapons. The material, which was being updated when this report went to press, is available at http://multiple.kcvs.ca/. The IAC-IAP project on Responsible Conduct in the Global Research Enterprise chose the term “misuse” (IAC-IAP, 2012).
• Involving policymakers and regulators in both the planning and conduct of future Institutes
This Institute focused on recruiting faculty and lecturers who would likely teach about the issues themselves. But policymakers and regulators of scientific policy, education, development, rules, and funding from the region where an Institute is conducted could contribute valuable insights, perspectives, and doses of reality for participants interested in developing RCS educational programs in their countries. In turn, policymakers and regulators could benefit from learning about the perspectives of scientists from their own and other countries in an environment that fosters respectful dialogue and challenges assumptions of individual participants.
Consulting with policymakers and regulators from the region prior to an Institute also could help organizers to better understand and tailor the subjects and issues that they hope to address in ways that will be more meaningful to participants at an Institute. The importance of these connections became apparent when a committee member and a member of the project staff spent several days in Algeria consulting with representatives from various government offices as well as educators at Algerian universities in anticipation of a workshop there in June 2013 that is also sponsored by the Biosecurity Engagement Program at the U.S. Department of State. That workshop will assist Algerians in developing a national curriculum in bioethics.
• Assessment of learning and evaluation of programs during institutes and in subsequent activities
As noted in Chapter 5 and earlier in this chapter, assessment of learning and evaluation of the efficacy of a program can be difficult because (1) people whose native language is not the one used to communicate may interpret words and phrases differently than the Institute organizers had intended, and (2) assessment of learning in higher education has traditionally been restricted to summative assessments that are given infrequently and are created by people with little expertise in psychometrics (the quantitative measurement tools and techniques developed in psychology). At the reunion meeting in Amman it became clear that grantees had used a broad spectrum of assessment and evaluation instruments. Participants were eager to know what instruments are already available that they could modify for their own purposes. Online instruments, such as the NSF-supported Student Assessment of Their Learning Gains (www.salgsite.org/), offer such templates. When developing future institutes, it would be helpful to provide a list of such resources and to spend some time helping participants understand their uses and value.
For purposes of evaluating individual programs consistently, developing evaluation instruments that could be used by all participants who undertake subsequent activities could ease their workloads and make comparable data more readily available to Institute organizers.
• Use of online technologies and resources Institutes that involve regional or international travel for a small number of participants from any given country will, by themselves, have minuscule impact in addressing a very large set of national issues. NASI has begun to address this limitation by
expanding to a series of seven regional institutes each summer based on the annual institutes that were held in Madison, Wisconsin, for many years. However, regional institutes still cannot address the magnitude of change that is needed across hundreds of institutions and tens, perhaps hundreds of thousands of students. In addition, the costs for the Institute approach may be prohibitive in many parts of the world.
Thus the use of increasingly sophisticated online technologies and the development of online resources to reach much larger numbers of scientists, educators, and policymakers should be considered and supported. Social media, massive open online courses (MOOCs), and other forms of distance learning are some possible solutions. However, given this Institute's emphasis on evidence-based active teaching and learning, it must be recognized that overreliance on online technologies might compromise this aspect of the experience. A great deal of research is now under way to explore how such technologies might both enhance and compromise deep learning. The results of this work, in combination with the ability of web-based approaches to reach great numbers of students, should be taken into consideration by those who plan future programs.
Potential Next Steps in the MENA Region
A second regional Institute where the lessons from the first Institute would be applied is a logical activity to take advantage of the insights gained through this committee’s evaluation work. As discussed in Chapter 2, one of the fundamental characteristics of successful faculty development programs is follow-up; no single event or experience is expected to be sufficient to foster genuine change. Such an Institute would not be a part of this National Academies project and would require new funding, although it could take advantage of the ties already created with institutions such as the Bibliotheca Alexandrina, The World Academy of Sciences (TWAS), or the Jordan University of Science and Technology (JUST).
The option for a follow-on Institute most favored by the committee would bring together several facilitators from the first Institute with some alumni. This arrangement could enhance their engagement with the methods and concepts promoted by the program. In the course of this second Institute, the Bibliotheca Alexandrina would have the opportunity to develop and host a website with materials from the Institutes as well as other resources to provide information and promote application of responsible science and active learning in universities and other research settings throughout the MENA region. Translating the Institute’s relevant materials into Arabic would offer an outreach opportunity for interested scientists, policymakers, and others in the region.
It is also essential to help build participants’ capacity to work more independently in their home countries. To begin the process, two participants from the first Institute, from Yemen and Egypt, will attend one of the 2013 regional Summer Institutes in the United States.59 This weeklong immersion in active learning techniques will significantly increase their skills and abilities to implement active programs in responsible science in their own countries as well as to serve as facilitators at future Institutes. A number of participants already have envisioned one-half to two-day “mini-Institutes” to provide basic content and active learning
experiences to a particular department or faculty. These might be logical projects for former Institute participants to create, perhaps in collaboration with a larger, continuing MENA participant network that involves the NationalAcademies in parallel with other comparable programs on responsible science and dual use issues.60
60 For example, with regard to dual use issues, the programs on “dual use bioethics” operated by Bradford University and a new two-year, EU-supported project to create an “International Network of universities and institutes for raising awareness on dual-use concerns in bio-technology” that began work in January 2013 have connections to some of the countries in the MENA region. Further information about the Bradford activities is available at www.brad.ac.uk/bioethics/about/ and about the EU project at www.cbrn-coe.eu/Projects.aspx.