National Academies Press: OpenBook

Assessing 21st Century Skills: Summary of a Workshop (2011)

Chapter: Appendix A: Agenda and Participants for the January Workshop

« Previous: References
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

Appendix A


Agenda and Participants for
the January Workshop

 

Assessment of 21st Century Skills
Workshop
January 12–13, 2011

University of California, Irvine
Beckman Conference Center
Huntington Room

AGENDA

Wednesday, January 12

 

9:30-9:40

Opening Remarks Welcome

 

Stuart Elliott (Director, Board on Testing and Assessment)

 

Bruce Fuchs (National Institutes of Health, cosponsor of the project)

   
  Overview of Workshop
  Joan Herman (CRESST, Chair of Workshop Steering Committee)
9:40-12:15 Session 1: Background Information
  Moderators: Joan Herman and Pat Kyllonen (ETS and
  Workshop Steering Committee)
   
  (9:40-10:00) Why Are 21st Century Skills Important?
  Richard Murnane (Harvard University)
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
This presentation will address the following issues:
   
 

•   What is unique in the 21st century that makes these skills especially valuable in the labor market and/ or in other life domains (learning, family life, civic engagement)?

 

•   How does the growing use of computers and technology affect the labor market and the demand for 21st century skills?

 

•   What does more recent research suggest about the skills needed to be successful in the 21st century?

   
  (10:00-10:15) How Will You Know If Your Students Are 21st Century Ready?
  Deborah Boisvert (Boston Area Advanced Technical Education)
   
  The presenter will respond to the opening presentation, reflecting her work with employers to define, teach, and assess 21st century skills of computer technicians.
   
  (10:15-10:45) The Teaching and Learning of 21st Century Skills
  Eric Anderman (Ohio State University)
   
  This presentation will address the following issues:
   
 

•   What is known about the extent to which the three skill clusters and/or the skills within them can be taught and learned?

 

•   To what extent are learning, teaching, and assessment of the three skill clusters domain specific or domain general?

   
  (10:45-11:00) Discussion
  Moderators will lead a question-and-answer session with the presenters and audience members.
   
11:00-11:15 Break
   
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(11:15-11:35) Approaches to Developing Assessments of 21st Century Skills
  Deirdre Knapp (HumRRO, Workshop Steering Committee)
   
  This presentation will address the following issues:
   
 

•   What are the different approaches to assessment of these skills and what steps are involved in carrying out these approaches?

 

•   What processes are used for identifying the skills to be measured, operationalizing the skills through the test blueprint, and creating assessment tasks and scoring procedures?

 

•   How should the intended uses of the assessment results guide the test development process?

 

•   What steps should be taken to ensure that the assessments are reliable and valid?

   
  (11:35-11:55) Unique Challenges and Opportunities in the Assessment of 21st Century Skills
  Steven Wise (Northwest Evaluation Association, Workshop Steering Committee)
   
  This presentation will address the following issues:
   
 

•   What are the unique challenges and opportunities for defining and measuring these constructs, when compared to more traditional academic skills and knowledge?

 

•   How might the results of these assessments be used? Should they be used for high-stakes purposes?

 

•   What issues may arise in relation to the validity, reliability, and fairness of assessments of these skills?

   
  (11:55-12:15) Questions and Discussion
  Moderators will lead a question-and-answer session with the presenters and audience members.
   
12:15-1:15 Lunch in the Beckman Center Cafeteria
  Continued discussion of ideas presented during the morning sessions
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

1:15-3:45 Session 2: Assessing Cognitive Skills
  Moderators: Greg Duncan (University of California, Irvine, Workshop Steering Committee) and Paul Sackett (University of Minnesota, Workshop Steering Committee)
   
  (1:15-1:45) Defining and Measuring Cognitive Skills
  Nathan Kuncel (University of Minnesota)
   
  This presentation will address the following issues:
   
 

•   What are 21st century cognitive skills? To what extent do they differ from each other and from general cognitive ability? What are the conceptual differences that are proposed to exist between these constructs?

 

•   What are the existing measures of these constructs, and to what extent do these existing measures match their conceptual specifications?

 

•   What are the relationships between the existing measures of these constructs?

   
  (1:45-2:00) Questions and Discussion
  Moderators will lead a question-and-answer session with the presenters and audience members.
   
  (2:00-3:15) Panel Discussion: Examples of Assessments of Cognitive Skills
   
  For each example, the panelists will address the following issues:
   
 

•   What skill or skills are measured? Why are these skills important?

 

•   What is the purpose of the assessment?

 

•   What strategies were used to develop the assessment and why were these selected?

 

•   What assessment methods are used and why were these selected?

 

•   How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations?

 

•   What data are available on the cost and practical feasibility of the assessment?

Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(2:00-2:20) Interactive Problem Solving for PISA 2012
  Joachim Funke (University of Heidelberg), by video conference
   
  (2:20-2:40) Operation ARIES!: Learning Critical Thinking about Science with Intelligent Conversational Agents in a Game Environment
  Art Graesser (University of Memphis) and Heather Butler (Claremont McKenna College)
   
  (2:40-3:00) Intrusive and Unobtrusive Assessment of Entrepreneurial and Technical Skills through Simulation and Gaming
  John Behrens (Cisco Systems)
   
  (3:00-3:20) Assessment of Critical Thinking and Problem Solving on the Multistate Bar Exam
  Susan Case (National Conference of Bar Examiners)
   
3:20-3:30 Break
   
  (3:30-4:00) Moderated Discussion
   
  Moderators will explore the following issues with panelists and audience members:
   
 

•   What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education?

 

•   Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education?

 

•   How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia? What functions can/should the assessments serve? How might the results be used?

 

•   What equity and accessibility challenges do these assessments raise?

 

•   What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?

Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

4:00-5:00 Synthesis of Key Ideas
  Moderator: Joan Herman
   
  (4:00-4:20) Discussion: Eva Baker (CRESST)
   
  (4:20-4:40) Discussion: Richard Murnane (Harvard University)
   
  Discussants will reflect on the day’s discussions and offer their synthesis of the ideas presented. Audience members will be invited to ask questions and share their ideas as well.
   
5:00 Conclude Formal Agenda for Day 1
  Joan Herman
   
5:30 Working Group Dinner (at Beckman Center)
  Plan for the second day of the workshop
   
Thursday, January 13
   
  9:00-11:45 Session 3: Assessing Interpersonal Skills
  Moderators: Deirdre Knapp and Juan Sanchez (Florida International University, Workshop Steering Committee)
   
  (9:00-9:30) Defining and Measuring Interpersonal Skills
  Steve Fiore (University of Central Florida)
   
  This presentation will address the following issues:
   
 

•   What are 21st century interpersonal skills and why are they important?

 

•   How are these skills typically assessed? What are the challenges in assessing them?

 

•   What types of assessments are currently available to evaluate these skills?

   
  (9:30-9:40) Questions and Discussion
  Moderators will lead a question-and-answer session with the presenters and audience members.
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(9:40-11:00) Panel Discussion: Examples of Assessments of Interpersonal Skills
   
  For each example, the panelists will address the following issues:
   
 

•   What skill or skills are measured? Why are these skills important?

 

•   What is the purpose of the assessment?

 

•   What strategies were used to develop the assessment and why were they selected?

 

•   What assessment methods are used and why were these selected?

 

•   How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations?

 

•   What data are available on the cost and practical feasibility of the assessment?

   
  (9:40-10:00) Online Portfolio Assessments of the 4 Cs
  Bob Lenz (Envision Schools)
   
  (10:00-10:20) 21st Century Skills in STEM Workforce
   
  Training Assessments
  Louise Yarnall (SRI)
   
  (10:20-10:40) Using Situational Judgment Tests for Medical School Admissions
  Filip Lievens (Ghent University, Belgium), by video conference
   
  (10:40-11:00) Assessment Centers 2011: Fifty Years of Best Practice and Today’s Innovations
  Lynn Gracin Collins (SH&A/Fenestra)
   
11:00-11:10 Break
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(11:10-11:45) Moderated Discussion
   
  Moderators will explore the following issues with panelists and audience members:
   
 

•   What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education?

 

•   Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education?

 

•   How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia)? What functions can/should the assessments serve? How might the results be used?

 

•   What equity and accessibility challenges do these assessments raise?

 

•   What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?

   
11:45-12:45 Working Lunch in the Beckman Center Cafeteria
  Continued discussion of ideas presented during the morning sessions
   
12:45-3:30 Session 4: Assessing Intrapersonal Skills
  Moderators: Pat Kyllonen and Steven Wise
   
  (12:45-1:15) Assessment of Self-Regulation and Related Constructs: Prospects and Challenges
  Rick Hoyle (Duke University)
   
  This presentation will address the following issues:
   
 

•   What are 21st century intrapersonal skills and why are they important?

 

•   How are these skills typically assessed? What are the challenges in assessing them?

 

•   What types of assessments are currently available to evaluate these skills?

   
  (1:15-1:30) Discussion
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(1:30-3:30) Panel Discussion: Examples of Assessments of Intrapersonal Skills
   
  For each example, the panelists will address the following issues:
   
 

•   What skill or skills are measured? Why are these skills important?

 

•   What is the purpose of the assessment?

 

•   What strategies were used to develop the assessment and why were these selected?

 

•   What assessment methods are used and why were these selected?

 

•   How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations?

 

•   What data are available on the cost and practical feasibility of the assessment?

   
  (1:30-1:50) Integrity Testing for Employee Selection
  Paul Sackett (University of Minnesota, Workshop Steering Committee)
   
  (1:50-2:10) Targeting Context-Specific Self-Regulated Learning (SRL) Processes: An Overview and Illustration of SRL Microanalysis
  Tim Cleary (University of Wisconsin–Milwaukee)
   
  (2:10-2:30) Assessing Behavioral Problems That Predict Poor Educational and Life Outcomes
  Candice Odgers (University of California, Irvine)
   
  (2:30-2:50) Out of the Maze? In Search of Skills for Emotional Intelligence
  Gerald Matthews (University of Cincinnati)
   
2:50-3:00 Break
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

 

(3:00-3:30) Moderated Discussion
   
  Moderators will explore the following issues with panelists and audience members:
   
 

•   What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education?

 

•   Do common themes or approaches emerge from the examples? How might the noneducation examples generalize to education?

 

•   How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two-state consortia? What functions can/should the assessments serve? How might the results be used?

 

•   What equity and accessibility challenges do these assessments raise?

 

•   What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?

   
3:30-4:00 Session 5: Reflection and Synthesis
  Moderated discussion led by workshop steering committee
   
4:00 Closing Remarks, Adjourn
  Joan Herman

PARTICIPANTS

Eric Anderman, Ohio State University

John Behrens, Cisco Systems

Lola Berber-Jimenez, California Polytechnic Science Project

Paul Bloomberg, Transformative Inquiry Design for Effective Schools and Systems

Deborah Boisvert, University of Massachusetts, Boston

Liane Brouillette, University of California, Irvine

Christopher Brown, Pearson Foundation

Peggy Burke, Transformative Inquiry Design for Effective Schools and Systems

Heather Butler, Claremont McKenna College

Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

Susan Case, National Conference of Bar Examiners

Tim Cleary, University of Wisconsin–Milwaukee

Sara Clough, ACT, Inc.

Lynn Gracin Collins, Sandra Hartog & Associates/Fenestra, Inc.

Emily Dalton Smith, Gates Foundation

Tran Dang, University of California, Irvine

Greg J. Duncan, University of California, Irvine

Steve Fiore, University of Central Florida

Dennis Frezzo, Cisco Systems

Bruce Fuchs, National Institutes of Health

Joachim Funke, University of Heidelberg

Tracy Gardner, General Educational Development Testing Service

Nicole Gerardi, University of California, Los Angeles

Art Graesser, University of Memphis

Valerie Greenhill, e-luminate

Erika Hall, Pearson Foundation

Joan Herman, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles (committee chair)

Rick Hoyle, Duke University

John Jackson, National Science Foundation

Stuart Kahl, Measured Progress

Deirdre J. Knapp, HumRRO (committee member)

Art Kramer, University of Illinois

Brandi Kujala, Educational Policy Improvement Center

Nathan Kuncel, University of Minnesota

Patrick Kyllonen, Educational Testing Service (committee member)

Robert Lenz, Envision Schools

Filip Lievens, University of Ghent

María Alicia López Freeman, California Science Project

Tim Magner, Partnership for 21st Century Skills

Michael Martinez, University of California, Irvine

Gerald Matthews, University of Cincinnati

Mick McManus, University of Queensland

Beth Miller, Nellie Mae Education Foundation

Julia Rankin Morandi, Los Angeles Education Partnership

Richard Murnane, Harvard University (committee member)

Suzanne Nakashima, California Science Project

Paul Nichols, National Center for the Improvement of Educational Assessment

Candice Odgers, University of California, Irvine

Cornelia Orr, National Assessment Governing Board

Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×

Pamela Paek, National Center for the Improvement of Educational Assessment

Jason Ravitz, Buck Institute for Education

Michael Russell, University of California, Irvine

Paul Sackett, University of Minnesota (committee member)

Andrea Saenz, U.S. Department of Education

Juan I. Sanchez, Florida International University (committee member)

Mary Seburn, Educational Policy Improvement Center

Brian Stecher, RAND

Christine Tell, Achieve

Cathy Tran, University of California, Irvine

Bernie Trilling, Oracle Education Foundation

Jerry Valadez, California State University, Fresno

Marjorie Wine, General Educational Development Testing Service

Steven Wise, Northwest Evaluation Association (committee member)

Louise Yarnall, SRI

Raymond Yeagley, Northwest Evaluation Association

Linda Zimmerman, Pearson

Doron Zinger, Olive Crest Academy

Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 127
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 128
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 129
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 130
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 131
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 132
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 133
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 134
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 135
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 136
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 137
Suggested Citation:"Appendix A: Agenda and Participants for the January Workshop." National Research Council. 2011. Assessing 21st Century Skills: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/13215.
×
Page 138
Next: Appendix B: Agenda and Participants for the May Workshop »
Assessing 21st Century Skills: Summary of a Workshop Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The routine jobs of yesterday are being replaced by technology and/or shipped off-shore. In their place, job categories that require knowledge management, abstract reasoning, and personal services seem to be growing. The modern workplace requires workers to have broad cognitive and affective skills. Often referred to as "21st century skills," these skills include being able to solve complex problems, to think critically about tasks, to effectively communicate with people from a variety of different cultures and using a variety of different techniques, to work in collaboration with others, to adapt to rapidly changing environments and conditions for performing tasks, to effectively manage one's work, and to acquire new skills and information on one's own.

The National Research Council (NRC) has convened two prior workshops on the topic of 21st century skills. The first, held in 2007, was designed to examine research on the skills required for the 21st century workplace and the extent to which they are meaningfully different from earlier eras and require corresponding changes in educational experiences. The second workshop, held in 2009, was designed to explore demand for these types of skills, consider intersections between science education reform goals and 21st century skills, examine models of high-quality science instruction that may develop the skills, and consider science teacher readiness for 21st century skills. The third workshop was intended to delve more deeply into the topic of assessment. The goal for this workshop was to capitalize on the prior efforts and explore strategies for assessing the five skills identified earlier. The Committee on the Assessment of 21st Century Skills was asked to organize a workshop that reviewed the assessments and related research for each of the five skills identified at the previous workshops, with special attention to recent developments in technology-enabled assessment of critical thinking and problem-solving skills. In designing the workshop, the committee collapsed the five skills into three broad clusters as shown below:

  • Cognitive skills: nonroutine problem solving, critical thinking, systems thinking
  • Interpersonal skills: complex communication, social skills, team-work, cultural sensitivity, dealing with diversity
  • Intrapersonal skills: self-management, time management, self-development, self-regulation, adaptability, executive functioning

Assessing 21st Century Skills provides an integrated summary of the presentations and discussions from both parts of the third workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!