Analysis of AP and IB Curriculum, Instruction, Assessment, and Professional Development
This chapter reviews the AP and IB curricula and associated instructional strategies, assessments, and professional development opportunities to identify those features that are consistent with the design principles set forth in Chapter 7. Also examined are the ways in which the four program elements interact to support or undermine students’ attainment of the deep conceptual understanding that is the goal of advanced study. In presenting this analysis, the committee acknowledges that there are many teachers in the United States who, by virtue of their experience and pedagogical skills, are able to overcome many of the program deficiencies discussed. We also recognize, however, that an improved program would only serve to enhance the capabilities of these teachers. Additionally, for the many teachers who are not as skilled or experienced, it is important that the curricula and related assessments and teacher preparation for these courses be aligned more completely with what is known about learning to ensure high standards and quality instruction for all AP and IB students.
Depth Versus Breadth
The committee’s analysis of the AP and IB programs reveals some fundamental characteristics that are incompatible with a curriculum designed to foster deep conceptual understanding. The first major discrepancy, which applies equally to AP and IB, relates to the “less is more” concept in curriculum design. The idea is simple: only if a curriculum is not overly broad in terms of the number of topics to be studied is it possible to study topics in
sufficient depth to develop deep conceptual understanding. This idea is consistent with national standards in mathematics and science for grades K– 12 (American Association for the Advancement of Science [AAAS], 1993; National Council of Teachers of Mathematics [NCTM], 2000; National Research Council [NRC], 1996). While the written materials produced by the College Board and the International Baccalaureate Organisation (IBO) acknowledge the importance of depth and focus, the daunting scope of the curriculum guides and the associated assessments in some subject areas sends a very different message. One notable exception is the revised AP calculus syllabus, which reflects an appropriate balance between breadth and depth.
AP course outlines (and assessments) are designed to mirror the content of typical college-level introductory courses (see Chapter 3, this volume). AP courses are intended to be an acceptable substitute for the introductory courses offered at more than 2,000 colleges and universities. Because AP courses cannot duplicate all of these college offerings, AP development committees try to make the courses maximally similar to as many of their college counterparts as possible. If one makes the commonsense assumption that college-level courses vary substantially in quality, such an approach will limit the development of high-quality AP courses. Average and excellent are incompatible. By focusing on average rather than exemplary programs, the development process results in an almost certain regression to the mean. A far better goal for the development of AP courses would be to design them in accordance with the principles articulated in this report. Further, suggested instructional strategies should reflect the range of best practices in both high school and college courses instead of emulating college courses across a range of quality and content. Throughout the development process, the goal of fostering learning with deep conceptual understanding must always be pursued.
AP course content is determined using data supplied by college and university department officials who respond to a survey distributed by the College Board. According to College Board officials, the response rate to these surveys is rather low—approximately 40 percent. Thus the responses of a small number of college and university departments may have an undue influence on what is taught in advanced courses across the nation. Designing AP courses to reflect a typical college course also means that until curricular changes become common in introductory college courses, these changes will not be included in AP. On the other hand, because IB courses are developed differently (see Chapter 4, this volume), they can be more responsive than AP courses to changes in the disciplines.
The committee notes that in the 1990s, the College Board began experimenting with other development strategies. In 1994, 23 nationally recognized experts in the area of calculus reform were brought together to work with the development committee in redesigning the AP calculus curriculum. The give and take and varying perspectives of these experts helped create a course that did not simply replicate a typical college course, but reflected current consensus on best practices in mathematics teaching and curriculum design.1 The collaborative model used to develop AP calculus offers insight into how course development procedures could be modified to improve programs. The best possible outcome is that a focus on quality will result in university and AP courses having reciprocal effects on one another.
As noted earlier in this volume (see Chapters 3, 4, and 8), AP and IB courses in the same discipline vary from school to school and even from classroom to classroom. Individual teachers are given substantial leeway in implementing AP or IB courses in their classrooms. While flexibility in curricula is an attractive feature for many experienced teachers, allowing them to be creative in their instructional approaches, the result is that college and university faculty face the same problems as AP and IB teachers in being unable to determine what incoming students know and are able to do.
Research on the nature and effectiveness of various practices in AP and IB classrooms is scant. Little is known, except anecdotally, about how AP and IB classrooms are organized, how the courses are implemented, and how they relate to existing school and district curricular offerings. School differences also affect how a course is implemented. The length of class periods, available facilities, and existing state standards, for example, all help shape the courses differently in each locality. Research is needed to understand these differences and their effects on student learning and achievement.
Both the AP and IB programs depart significantly from the committee’s framework for designing effective programs for advanced study by failing to address the element of instruction adequately. Neither the College Board nor the IBO gives sufficient attention to identifying and modeling aspects of
For a complete history of the reform of AP calculus, see http://apcentral.collegeboard.com/article/0,1281,150-155-0-8019,00.html (February 2, 2001).
instruction that foster learning with understanding. Further, neither program presents explicit models of the nature and range of excellent teaching in AP or IB classrooms. Sample syllabi, Internet discussion groups, professional development activities, and other resources provided by the College Board and the IBO are of varying quality, represent limited perspectives on excellent instruction, and do not adequately take into account variability among students and schools.
Because both AP and IB courses are designed to prepare students for external, standardized final examinations, instruction in AP and IB classrooms is typically directed toward that goal. Unfortunately, the broad scope of the AP and IB curricula, the significant number of examination questions that appear to require only rote learning, and the predictability of the questions included on some examinations may lead teachers to focus their attention on presenting information and teaching problem types, rather than facilitating active student involvement in learning.
One of the primary responsibilities and challenges facing teachers of AP and IB courses is the selection and development of worthwhile problems, exercises, tasks, projects, and investigations that can simultaneously develop conceptual understanding, prepare students for successful performance on the examination, and address the varying abilities and preferences of students in the classroom. Currently, AP and IB materials provide little assistance for meeting this challenge.
Promotion of High-Quality Instruction
High-quality instruction in AP and IB science and mathematics courses depends on the availability of qualified teachers. The committee found that neither the College Board nor the IBO has clearly defined the preparation and qualifications of AP or IB teachers; doing so could contribute significantly to the quality of instruction in the programs.
Schools and districts also have an important role to play in promoting high-quality instruction in AP and IB classrooms; the College Board and the IBO cannot do it alone. High-quality instruction depends on many things, such as adequate instructional resources and laboratory facilities; a structure that allows enough time for instruction; ongoing availability of high-quality professional development opportunities; and support from administrators, parents, and educational policymakers.
Most important, schools and districts must recognize that effective teaching in AP and IB classrooms requires time. Advanced study teachers, like other
effective teachers, need time to remain abreast of changes in their subject areas and in the pedagogy related to their disciplines. AP and IB teachers need time to prepare their classes, carefully grade and comment on student work, and engage in collaboration and lesson planning with other teachers (see Chapter 2, this volume). For example, Ma (1999) indicates that teachers in China spend more time preparing lessons than teaching them. Time for collaborative planning is an important part of teachers’ school hours in China. In the United States, teachers average approximately 13 minutes of planning time for every instructional hour (Darling-Hammond, 1999b; National Commission on Teaching and America’s Future, 1996).
AP and IB courses are designed to prepare students for standardized final examinations. Teachers select curriculum materials and gear instruction to prepare their students for success on these examinations, and students focus their learning on the concepts and content they believe they will encounter on the assessments. It is important, therefore, to better understand whether these assessments are aligned or in conflict with the principles of learning with understanding and the design principles of assessment.
Examination Design and Development
As noted earlier (see Chapter 3, this volume), AP examinations are designed by development committees for each of the disciplines in consultation with statisticians and psychometricians to create examinations that meet accepted standards for technical quality (American Educational research Association [AERA]/American Psychological Association [APA]/National Council on Measurement in Education [NCME], 1999).2 Thus, the development process incorporates the judgments of both disciplinary and psychometric experts. IB examinations are designed differently, in a manner similar to that which might be used by teachers who have a sophisticated understanding of their discipline (see Chapter 4, this volume). Thus, the development of IBO assessments depends heavily on the expertise and professional judgment of master teachers and less on psychometric calibrations. These differences between the two programs’ assessments should be considered in light of the ways students’ examination scores will be used and the kinds of inferences users expect to support with the scores.
Support for Inferences Drawn from Assessment Results
As noted in Chapter 6, validity research is a vital component of any high-quality assessment program. Validity involves what an examination measures and the meaning that can be drawn from the scores and the actions that follow (AERA/APA/NCME, 1999; Cronbach, 1971). For each purpose for which the scores are used, there must be evidence to support the inferences drawn. In the case of AP, for example, the College Board wants users to draw the inference that students’ performance on AP examinations is indicative of their mastery of material taught in a typical college course in the subject. The IBO wants users to draw the inference that students who earn an IB Diploma through their performance on six IB examinations3 are adequately prepared for postsecondary work in many countries.
“The process of validation involves accumulating evidence to provide a sound scientific basis for the proposed score interpretations” (AERA/APA/ NCME, 1999, p. 9). The AP Technical Manual describes two common interpretations of AP scores: (1) a good AP grade indicates that the student would benefit from entering a course more advanced than the usual first-year course, and (2) an AP grade indicates that the student should receive credit for a college course that he or she has not taken.4 The IBO does not describe the appropriate interpretations of IB grades other than to say that they reflect students’ mastery of course content that is designed to prepare them for postsecondary learning.
Given these desired interpretations, validation studies for the AP and IB assessments should include systematic evaluation of such factors as whether the right skills and knowledge are being measured and in the right balance; whether the cognitive processes required by the test are representative of the ways knowledge is used in the discipline; the extent to which the test measures students’ knowledge of the broader construct that is the target of instruction, as opposed to their knowledge of specific test items; whether the scoring guidelines focus on student understanding; and whether the test scores accurately represent different levels and kinds of understanding.
The committee’s analyses of the test items and the course syllabi on which the tests are based yielded information about content coverage. However, no data were available for evaluating whether the tests measure important cognitive skills. This is because neither program has systematically gathered data to document that test items on its examinations measure the skills they purport to measure. In making determinations about the validity of
inferences, it is necessary to know, for example, that test items intended to measure problem solving do in fact tap those skills and do not just elicit memorized solutions or procedures. There are a number of ways to collect such data. For example, cognitive laboratories might be conducted during which examinees would “think aloud” as they responded to test questions. Information from such laboratories gives test developers insight into the thought processes used by students to derive answers to test questions.
Validity evaluation should consist of more than a haphazard set of research studies. A sound validity evaluation should integrate “various strands of evidence into a coherent account of the degree to which existing evidence and theory support the intended interpretations of test scores for specific uses” (AERA/APA/NCME, 1999, p. 17). Yet the committee’s analysis indicates a surprising lack of information about what the AP and IB assessments actually measure. To date, neither program has implemented a strong program of validity research. The committee found only a few validity studies conducted with AP or IB data. For example, Morgan and Ramist (1998) examine college course-taking patterns for AP students; Morgan and Maneckshana (2000) compare performance in advanced-level courses for AP and non-AP students; and Eugene Carson reports on a small study at Virginia Polytechnic Institute and State University that compared the overall grade point averages of IB students with those of AP students and students who participated in neither program.5 While these studies may be useful, they do not represent pieces of an integrated research program (see Chapter 10, this volume, for a discussion of these studies).
Consequential Validity of AP and IB Assessments
Very little evidence is available for evaluating the consequential validity6 of the AP and IB assessments. With regard to this aspect of validity, relevant questions might include whether typical examinees become more or less interested in the domain as a result of their experience and whether they are more or less likely to pursue further study than those who do not sit for the examination. In the context of this report, one might ask how students who participate in AP and IB fare in college relative to other students and what the effect is on colleges of the ever-increasing numbers of students entering postsecondary institutions with college credit earned on AP or IB examinations. Equity concerns should also be considered, that is, whether the assessment unintentionally produces inequities of achievement or attitude in either the short or long term. While Morgan and Ramist (1998) begin to
Available at http://www.rvcschools.org [August 2000]..
Consequential validity refers to the consequences or uses of assessment results.
address some of these questions, additional studies are needed. Information about the consequential effects of AP and IB assessments are especially important as these programs expand and become a more significant component of U.S. education.
A common theme of the four panel reports that also emerged throughout the committee’s deliberations is the crucial importance of professional development for enhancing teachers’ content knowledge in their subject areas. As noted in Chapter 7, this volume, a strong grounding in subject matter knowledge is fundamental to good teaching and should be a primary goal of professional development activities. It is unrealistic to ask teachers to cultivate a deep, principle-based understanding among their students unless they themselves have such understanding. However, the ideal role of professional development for advanced study goes far beyond ensuring that teachers know their subject areas well. Effective professional development also enhances teachers’ subject-specific pedagogical knowledge and provides them with multiple perspectives on students as learners. It is grounded in situations of practice; takes place in professional communities; actively involves teachers; and is an ongoing, long-term component of their professional lives. The committee believes that without well-designed and ongoing professional development, it is difficult for teachers to provide high-quality instruction that is aligned with curriculum and assessment to support the goal of learning with understanding. The committee notes that neither AP nor IB professional development opportunities are sustained and that teacher participation in both programs is voluntary.
The mathematics and science panels note that the summer programs offered by AP are highly variable in focus and organization, as well as in quality. The mathematics panel expresses concern that many AP and IB mathematics teachers have insufficient professional development opportunities to derive both mathematical content and pedagogical knowledge or to receive help in implementing modifications to the programs. Having reviewed a significant sampling of AP summer institute materials (1998–2000), the panel finds that many of the workshops did not adequately address substantive issues relevant to teaching calculus. The science panels note that few of the AP workshops, even those that emphasized laboratory experiences, focused on inquiry.7 The panels commend the IBO for its professional development activities related to internal assessments that focus on inquiry.
The College Board and the IBO are just beginning to fully develop vehicles for establishing professional communities of learners among geographically separated teachers. The primary means being used are discussion groups and databases that can be accessed on the Internet. However, the committee notes that both the College Board and the IBO have regional organizations for teachers. Some are more active than others, and each provides different types of activities. For example, the IBO has a vibrant group in Florida that runs its own conferences, and the College Board, through its regional offices (notably in the southwest), provides opportunities for teachers in the regions to work with colleagues in professional development activities throughout the school year. Nonetheless, there are too few opportunities for teachers to interact with colleagues on issues important to teaching. This is especially true in high schools where AP or IB teachers in a particular discipline may be the only members of their departments who teach these courses.
The National Commission on Mathematics and Science Teaching for the 21st Century (2000, pp. 27–28) provides some suggestions that could become part of AP and IB Web-based efforts. These include the creation of (1) an online professional journal that would encourage teachers of advanced mathematics and science to engage in publishable research and to share new teaching strategies with colleagues, both nationally and internationally; (2) a dedicated database of teaching ideas, plans, and other resources; (3) an interactive online resource for conversations, meetings, and idea sharing; and (4) interactive videos for observing good teachers and critiquing teachers’ own efforts, for mentoring, and for online instruction. Research is needed to determine the effectiveness of such Web-based mechanisms for teacher professional development.
The committee has advocated the use of a model that can help sponsors of advanced study programs make decisions on how their programs can best improve in the years ahead. This model includes both principles of learning and design principles for curriculum, instruction, assessment, and professional development. The committee believes that if programs of advanced study, such as AP and IB, were to place principles of learning at the center of their own implicit models, the programs would improve in quality and effectiveness with regard to fostering deep understanding. Program quality would also be enhanced if the programs were to recognize the systemic and mutually interactive nature of curriculum, instruction, assessment, and professional development. With care, persistence, and a strong guiding model, advanced study programs could become worthy beacons for all of American education.