Develop methods to efficiently design and measure the efficacy of Internet teaching technologies.
CHALLENGE SUMMARY
Over the last one-hundred years, a series of new technologies have each promised to revolutionize teaching. Slides, film and videos were all plausible technologies for replacing the classic form of instruction, in which a teacher lectures a class. Despite the many new technologies that might have changed teaching, conventional classrooms are still dominant. The World Wide Web—with the ease of posting videos and designing interactive online tests—is an attractive mélange of technologies that offer up considerable potential for influencing and improving education. Will things be different this time? How will we know the efficacy of the new Web-based teaching and learning technologies?
One reason why the Web technologies may be different is that many, many more people are authors. Pelli and Bigelow estimate that the number of authors is growing tenfold each year. The new transformation extends widespread literacy to widespread authorship. This will have an enormous impact on the range of available educational material or at least material that could be useful in educational settings.
A second reason is that Web usage, unlike slides or videos, is integrated into the typical day of many people. By integrating educational materials into daily communication and social networking tools, the new technologies have advantages that were not part of the earlier innovations.
Education is based on testing, credentialing, and building affinity groups. The Web is likely to be able to perform each of these functions, but the way in which it does so will differ from traditional classroom teaching.
How can we measure whether the new ways are effective? How can we use these measures to guide and improve the efficacy of these technologies over time? The variety of uses of Web technologies for teaching, for learning, and for research across the disciplines is impressive, but how can we assure that there is sharing and promulgation of those uses for re-use and re-mixing among the disciplines and learning environments?
Key Questions
• What can we learn from the history of introducing new technology into Education? Is there any difference with the new technologies that suggest we will have different outcomes?
• Are we prepared to measure the efficacy of the new technologies and use this ability to nudge them into useful directions?
• New software applications and new possibilities arising from them are being promoted to help authors produce educational material. Will this increase authorship and change the range of materials? What is the reasoning at academic and textbook publishers and in the software industry?
• In the US and many other Western countries, education and research are completely intertwined. What are the implications for research Universities if the education model is transformed?
• What are the implications for middle school and high school if we rely on instructional materials that are distributed, used, and take advantage of Web technologies? What is the new role for teachers and teacher education? Does it change dramatically? What is the uptake of such technologies in our public schools? Has the use of modern information technology improved teaching and learning in the K-12 grades?
Reading
Bhattacharjee Y, Clery D, Dede C, Greenfield PM, Hines DJ, Jasny BR, Mervis J, Miller G, Normile D, et. al. Special science online collection: Education and technology. Science (Special Issue) 2 January 2009;323(5910). Gates Foundation. Next generation learning: The intelligent use of technology to develop innovative learning models and personalized educational pathways. 2010.
Khan S. Let’s use video to reinvent education. You Tube (Video) March 2011.
Nochese F. Khan Academy: My final remarks. Action-reaction: Reflections on the dynamics of teaching 10 May 2011.
Pelli DG and Bigelow C. Nearly universal literacy is a defining characteristic of today’s modern civilization; nearly universal authorship will shape tomorrow’s. Seed Magazine 20 October 2009.
Robinson K. Schools kill creativity. TED Presentation 2006 June.
Stanford University. Stanford University EPGY.
Thompson C. How Khan Academy is changing the rules of education. Wired Magazine 15 July 2011.
Thrun S. University 2.0. Livestream (Video) 2012.
Watters A. The wrath against Khan: Why some educators are questioning Khan Academy. Hack Education 19 July 2011.
Widom J. From 100 Students to 100,000. ACM Sigmod 24 February 2012.
IDR TEAM SUMMARY
Zahra Hirji, NAKFI Science Writing Scholar Massachusetts Institute for Technology
IDR Team 2 was charged with the herculean task of developing methods to efficiently design and measure the efficacy of various Internet teaching technologies. Today’s teachers and self-learners have more tools at their disposal than at any previous point in history, in large part due to the proliferation of the World Wide Web. Out of all the Internet tools now available—video-audio lecture surrogates, adaptive tutorials, games, online quizzes/assessments, wikis, blogs, chat systems—how does one possibly decide which tool or tools to adopt?
To make informed decisions, teachers and self-learners need access to technology evaluation data. This information would ideally be housed in a single location and consist of standardized assessments that allow for technology comparisons. Unfortunately these data and infrastructure are largely nonexistent. As a result, users are forced to select tools based on the hyperbole of marketing rather than on scientifically rigorous assessment.
IDR Team 2 outlined the actions needed to give teachers and users pursing online education, whether formal or informal, whether for themselves or to teach others, access to the assessment information they deserve and need. The envisioned end goal also includes equipping technology users with the resources to conduct their own qualified assessments.
The Internet Is Different
The current mania surrounding Web-based teaching technologies is nothing new. Back in the 1870s, Boston schools thought the school slate— personalized chalkboards—would revolutionize education. Similar claims were then made about the film projector, radio, television, and laptop.
Educators are always looking to improve student performance, and the untapped potential of new technologies is deliciously appealing.
Schools have adopted these various technologies to different degrees, but there is little proof that such tools actually improve student performance over prolonged periods of time. As a result, the way education is generally conducted has not been dramatically altered in modern history. From K-12 to higher education, teachers and professors still commonly lecture in front of the classroom, hand out paper exams, and employ textbooks for homework.
So, is the Internet actually any different? The team definitively decided yes. Unlike previous technologies, Web-based ones can build upon the strengths of older technologies. Televisions brought pre-recorded visual material into education, but unless you had a TV and school videos at home, that technology was confined to the classroom. But using the Web, videos can be made accessible at any time—in the classroom or outside the classroom—to anyone using a computer, smartphone, or tablet.
These modern technologies also break the barrier between the classroom and world with the capacity to “inreach,” or bring untapped information or people (parents, alumni, other teaching professionals) into the classroom in a virtual context. In addition to stimulating better learners, these tools may benefit teachers by automating normally exhaustive tasks, such as grading homework or tests, which then frees up the teacher’s time for more constructive personal interactions with students. These tools can also help teachers track the progress or tailor workloads of individual students more readily. Finally, these technologies benefit those outside formal education too, where it is now easier to refresh old skills or pick up new ones on your own.
Is the Internet a Threat?
Internet tools are flexible, interactive, and adaptive technologies—and can be accessed from the privacy of your own living room. Moreover training teachers in new technologies is time and cost intensive. If these are so great, accessible, seemingly independent, do we even need teachers or traditional classrooms? Now this may seem like a bold question, but many state universities in the process of slashing funding are turning to massive open online class [MOOC] alternatives, such as Coursera and Open Learning Initiative, to replace, or at least supplement, traditional lectures. While this has not yet corresponded with the widespread layoffs of associate
or adjunct professors, the general mood among some college campuses has been described as tense, to put it mildly.
Despite the appeal of MOOCs, there is little evidence that such technologies are just as effective, or better than traditional lecture classes. For the few subjects where evidence does exist and is relatively positive, mainly for some math and science courses, it is not yet possible to extrapolate whether this teaching format will work for other subjects, such as the social sciences and humanities.
Many of the team members are professors, and a few even had experience with MOOCs. For the MOOC users, the online portion of the class served primarily as a supplement to in-class time rather than a complete alternative. In other words, these technologies were job-helpers, not job-eaters. Regardless of the future role of MOOCs in the greater education systems, the ability to assess online technologies comprehensively is paramount. The last thing society want is the complete eradication of in-class learning for sub-par online technologies.
What Criteria, Information, and Tools Are Necessary for Assessment?
When approaching the task of building useful web teaching technology assessments, it is necessary to review what criteria, information, and tools are necessary for assessment—and how much of that material is currently known or available. The team members realized there is much we do not know or have at this point in time. In response, the team identified the three major challenges: (1) knowledge of diverse stakeholder needs, (2) a lack of accessible student performance data (at an individual or even aggregate level) for different technologies, (3) and no existing cohesive framework and infrastructure for assessing and evaluating learning outcomes for such technologies.
Assessment criteria
Ideally, Internet education technology assessments should meet the criteria of diverse stakeholders—K-12 teachers, professors, self-learners, and even employers. Some general student performance assessment points include fact learning, life skills, student engagement, student retention and completion, student diversity and access, course cost, and attitudes toward lifelong learning. Is it possible for any one assessment to actually evaluate all the criteria listed, or will separate specific assessments be more useful, practical?
Data collection
In order to assess diverse performance measures, there needs to be available data on student performance, ideally measured continually or at least routinely throughout the learning process. Many existing tools do not collect or share these data with the public. Of the ones that do both, educators can still only access very aggregated information. This is one of the major charges against the popular online software Khan Academy. Despite its popularity and plethora of testimonials, the company does not make its data accessible to the public for external evaluation.
On the flip side, the mentality that all data are good data is not necessarily true (unless you are a researcher). If educators could identify metrics for which they specifically want data, such as student quiz and assessment scores and time spent per online homework assignment, then existing Internet technologies could be reprogrammed to collect tailored information that meet stakeholder specific needs.
Assessment framework & infrastructure
There is no existing framework and infrastructure for assessing and evaluating learning outcomes for these modern technologies. In fact, there are few relevant frameworks for non-Internet based education technologies. One notable example, however, is the online resource What Works. This website is a database of researched and reviewed educational programs, products, practices, and policies for the K-12 school system. The website’s existing information is high quality, but there is relatively little site content. Moreover, this resource does not currently account for Web-based teaching technologies.
Recommendations
The research team came up with two sets of recommendations to pursue the advancement of World Wide Web teaching technology assessment: the development of assessment infrastructure and the proposal of research topics to take advantage of this new infrastructure.
Building appropriate infrastructure
The team decided a future Internet teaching technology assessment infrastructure should be modeled after What Works. This resource would be
a more scaled-up comprehensive version of What Works, including a public database of teaching technology assessments, relevant for K-12, higher education, and life-learning needs. All available assessments would be a certain high quality and have to go through some level of review. Moreover, assessments would ideally be standardized to allow for technology comparisons. In addition, there should be automated tools available for good experiment and assessment design so that website users could contribute their own experiences and feedback.
Future research goals
Once this new database is constructed, various research studies using the assessment data could provide valuable information. Below are five potential research questions:
• What is the right technology for a given goal or learner?
• How do learner characteristics, such as media proficiency, mediate the effectiveness of Internet teaching technologies?
• What is the right mixture of technology and social interactions, whether face-to-face or virtual to benefit individuals and society as a whole?
• How can we anticipate the effect of new technologies?
• What are novel designs for technologies that promote learning?
Internet technologies do not just offer an alternative to classroom learning, but a potential to substantially change education. Only with additional research on technology assessment and learner preferences might the idea of Internet-driven education become a significant part of the educational system.
