Uses, Misuses, and Unintended Consequences of AP and IB
This chapter examines a number of ways in which the Advanced Placement (AP) and International Baccalaureate (IB) programs and their assessment results are used. Some of these uses are appropriate and intended by the program sponsors; others are not. Unintended uses are making these programs high stakes in terms of their consequences for students, for teaching and learning, and for schools.
In 1998, Newsweek published a list (Mathews, 1998a) entitled “The 100 Best High Schools in America.”1 This list, which ranks public high schools according to the number of AP and IB tests taken, divided by the number of graduating seniors, was drawn from the book Class Struggle: What’s Wrong (and Right) with America’s Best Public High Schools (Mathews, 1998b). The ostensible purpose of the list, now published annually for Washington area schools in the Washington Post, is to quantify the level of challenge high schools provide to their students. The author of the list acknowledges that such rankings have their limitations, but believes it is better to have such a tool, even one that may be misleading, than to have no means of comparing schools and motivating parents and students to demand more from their schools (Mathews, 1998b).
The practice of ranking schools by the number of AP or IB tests administered, as Newsweek does, or using students’ scores on AP or IB exams for evaluating teachers or comparing the quality of teachers and schools, as some parents, school administrators, or policymakers may do, is making these programs high stakes in ways the program developers never intended.
Such misuses of AP and IB assessments are both educationally inappropriate and counter to the Standards for Educational and Psychological Testing (American Educational Research Association [AERA]/American Psychological Association [APA]/National Council on Measurement in Education [NCME], 1999). Moreover, neither the College Board nor the International Baccalaureate Organisation (IBO) has to date provided sufficient guidance to counter many of these misuses. This section considers some of the unintended consequences of these practices.
Effects on Students
Making decisions about the effectiveness of teachers or the quality of their teaching primarily on the basis of AP or IB test scores may have unintended consequences for students. For example, if teachers are evaluated on the basis of AP or IB test results, they may discourage certain students from taking AP or IB courses because they believe those students will not perform well, or they may counsel students for whom they anticipate a low grade to skip the final examination. Such practices can deny students the opportunity to try an AP or IB course or to validate what they have learned in comparison with nationally or internationally established standards.
Effects on Teaching and Learning
Teachers who are evaluated on the basis of student test results are more likely to teach to the test than to teach what they deem important (see, for example, Koretz, 1988, 1996; Koretz, Linn, Dunbar, and Shepard, 1991; Linn, Graue, and Sanders, 1990; Shepard, 1990; Shepard and Cutts-Dougherty, 1991). The committee acknowledges that the nature of the AP and IB programs makes teaching to the test the norm; the test specifications are the curriculum (see Chapter 3, this volume). However, the committee does not believe the AP and IB tests reviewed for this study sampled broadly and deeply enough across the full range of content knowledge, conceptual understanding, processes, and skills valued in the respective disciplines to make teaching to the test an appropriate strategy for developing the level of conceptual understanding that should be the goal of advanced study.
Therefore, the panels and the committee view teaching to AP and IB tests as potentially interfering with teaching what is truly important for enhancing student learning. For example, the mathematics panel notes that the AP and IB examinations have overemphasized procedure and under-emphasized application and conceptual understanding. The chemistry panel also identifies a lack of contextual problems in the AP examinations. The biology and physics panels find that both the AP and IB assessments cover too much content, and the chemistry panel decries the lack of attention to
important interdisciplinary connections. The biology and chemistry panels note a lack of test items that measure laboratory skills on the AP examinations, and the physics panel indicates that neither AP nor IB assessments require students to apply what they learn in class to solve new or novel problems. In sum, teaching to AP and IB tests for the purpose of raising test scores can lead to superficial coverage of a broad base of content knowledge, to teachers ignoring the importance of meaningful inquiry-based and laboratory experiences (AP only), and to students feeling that what they are learning in school has little application to the real world.
Effects on Teachers
Inferences about teacher quality and effectiveness cannot be supported by data drawn solely, or even primarily, from AP and IB test scores. One important reason for this is that students come to AP and IB classrooms with different levels of skill and mastery of content that make it difficult to determine the effects of a single teacher’s work on student achievement in any particular year. A well-prepared group of students might, for example, earn high scores on their AP or IB final examinations even if the quality of the teaching in their class was low. Similarly, an underprepared group of students might do quite poorly on AP or IB assessments because of their previous preparation, despite the valiant efforts of a highly qualified and dedicated AP or IB teacher. Although this is the case for all subjects, it appears to be especially true for advanced mathematics because of the hierarchical and sequential nature of the subject.
Effects on Schools
Ranking Schools by the Number of AP or IB Tests Taken
Policymakers and others often rely on numbers, rankings, comparisons of standardized test results, and other quantifiable data to draw conclusions about the state of education in the United States. The media report these data as valid and scientific, and the public uses them as a basis for making important decisions and judgments.
For example, the annual list described at the beginning of this chapter has been used by many as a measure of overall school quality. The list has taken on a life of its own. It is now so important to be included among the top 100 schools on the list that some competitive high schools not included have posted disclaimers on their Web sites indicating why this is the case.2
An example is City Honors High School in Buffalo, New York; see http://cityhonors.buffalo.k12.ny.us/city/news/1999-00/0003/news0003newsw/news0003newsw.html (January 11, 2002).
Many others that are listed among the top 100 openly publicize their ranking as a sign of excellence.
Using the number of AP and IB examinations or number of AP and IB courses offered in a school as a measure of school quality also penalizes in the arena of public opinion schools that have chosen, because of different educational values or priorities, to offer other rigorous options to their students. Ignoring the value of these alternative approaches can stifle creativity and innovation in the development of new programs. Such alternative approaches could be as rigorous as AP or IB and also be more suitable to the student body served by a particular school.
Additionally, ranking schools by the proportion of students who take AP or IB examinations ignores the central issue of how many of those students are adequately prepared through high-quality courses to succeed on the exams. The committee notes that among states and school districts where all AP students are required to take the examinations, a notable number of students may not complete the examination or answer test questions conscientiously.
Evaluating School Quality by the Numbers
The fundamental objective of education is to promote the academic growth of each student. Evaluations of school quality should help identify those schools that are accomplishing this goal and those that are not. When assessment scores are used to evaluate schools, efforts must be made to determine whether the scores are indicative of growth (or lack of growth) that is due to the quality of the instructional program, or merely reflect students’ home environments, external learning experiences, and available resources. There are many schools that provide high-quality instruction in school districts where resources are scarce whose students continue to grow academically. There are other schools in which test scores are high, but the quality of instruction may not be as good as in schools with lower test scores. The higher assessment scores may be more reflective of the students’ other opportunities than of the quality of the school itself. In sum, relying solely, or even primarily, on AP or IB test scores to evaluate school quality reflects a failure to recognize that there are substantive differences in educational institutions across the nation (see Chapter 2, this volume), while also ignoring the varying characteristics of the students who attend these schools.
With increasing calls for educational quality and accountability, educators and policymakers alike have turned to AP or IB to improve the quality
of their schools and curricula. The pressure to introduce or expand AP or IB offerings in high schools can have a number of unintended consequences. For example, schools that claim to offer advanced study programs may or may not be able to support such programs adequately, and the resulting courses may be far from what is intended by the program sponsors. Ensuring the quality and integrity of the AP and IB programs is a complex endeavor that it is frequently affected by factors beyond the control of either the College Board or the IBO.
Standards and Regulation of Courses
The IBO carefully regulates which schools can offer IB courses and how those courses must be structured. As a result, schools are unable to offer IB courses or the IB Diploma without the imprimatur of the IBO.
In contrast, the College Board has no clear standards for what constitutes an AP course or the schools that offer them. This lack of consistency invites misuse of the AP name. For example, schools have been known to label non-AP courses as AP in an effort to make their students more competitive for admission to college. Others may have instituted AP courses without ensuring that they have the facilities and personnel needed to offer a college-level program. Consequently, all four of the panels call on the College Board to certify or regulate AP programs and teachers.
The committee found that schools’ efforts to prepare students for AP and IB science and mathematics courses often help stimulate improvements in the prerequisite courses, and that this is an important and positive effect of these programs. However, there also may be some unintended consequences. For example, some students may be adversely affected when schools compress preparatory courses in order to prepare as many students as possible for AP or IB courses. Compression can occur by reducing coverage of specific subjects in a curriculum to provide adequate time for advanced study or by beginning the high school sequence of courses earlier in a student’s career.
To illustrate the point, strong preparation is important in mathematics because of the hierarchical and cumulative nature of the subject. However, the mathematics panel notes that there is considerable anecdotal evidence that some students intending to take calculus are rushed through prerequisite courses without thoroughly learning the preparatory material. Mathematical sophistication takes time to develop, and knowledge of a catalog of mathematical facts and techniques is not sufficient. The mathematics panel
notes that most of the students who do poorly in AP or IB calculus did not learn their algebra well enough. These students probably would have been better served by spending more time learning algebra and saving calculus for college. At the same time, there are students who are ready for a rigorous calculus course in high school. Schools need to maintain a delicate balance between meeting the needs of this latter group and pushing too many students into advanced mathematics before they are ready. Achieving this balance is not easily accomplished.
Compression of the curriculum also can occur when students are allowed to skip prerequisite courses and take an AP course as a first course. Among the sciences, AP physics is the course students most frequently select as a first course in the discipline. Data obtained from the College Board (College Entrance Examination Board [CEEB], 2000d) indicate that almost half of all AP physics test takers had had no prior experience with physics before enrolling in the AP course. Thus, the AP course had to cover both a year of high school physics and a year of college physics, making in-depth examination of any topic nearly impossible.
Participation in Examinations
The IBO expects that all students who take IB courses will take the associated examinations. The IBO uses student performance on its examinations to monitor and improve the overall quality of IB programs in different countries and schools. In contrast, the four panels note that the College Board has not developed examination policies or expectations for students’ participation in the AP examinations. Because the examinations provide the only external evidence that schools are preparing students in a manner consistent with the College Board’s expectations, the panels suggest that a clearly articulated policy is necessary if the College Board is to maintain quality control of the AP name.
As discussed in Chapter 5, advances in technology have made it possible for colleges, universities, technology companies, and other nonprofit and for-profit organizations to create and distribute AP courses and other AP support services, such as professional development for AP teachers, online. Although the reach of these online courses has to date been small relative to other AP opportunities, their potential for growth is unlimited. The biology and chemistry panels are particularly concerned that students who take AP courses online and earn a qualifying score on the examination could earn college credit or placement without having had any hands-on laboratory
experience.3 They call on the College Board to set quality standards for online laboratory components of science courses. It may be noted that, because only authorized schools can provide IB courses, laboratories, and examinations, third-party providers currently cannot offer programs that carry the IB name.
ACCESS AND EQUITY
Disparities in access to AP and IB courses for students who live in inner-city and rural areas are a serious educational and social problem that is discussed throughout this report. However, other students in the United States also lack meaningful access to these courses because their schools deny them the opportunity to enroll.
Limiting Students’ Access to AP and IB
Limiting students’ access to advanced study occurs in all kinds of educational settings, including the most competitive high schools in America—schools with adequate resources, qualified teachers, and well-prepared students (Mathews, 1998b). These schools, while typically advocating college preparation for everyone, have created layers upon layers of curricular differentiation such that only a select group of students are allowed entrance into certain AP and honors courses; other students are placed in less prestigious courses4 (Attewell, 2001; Mathews, 1998b; Oakes and Wells, 2001).
In a recent study, Attewell (2001) finds that many high-achieving students from elite high schools are not able to take the AP courses they would have been able to take had they attended less prestigious schools. He attributes this phenomenon to the fact that many elite high schools provide their best students with opportunities to participate in AP and honors courses while denying access to others. He finds further that, as a function of these placement policies, many highly able students who attend elite high schools are less likely than similar students in other high schools to take advanced mathematics or science courses or examinations. Schools limit participation in part, Attewell contends, because restricting access to AP to the strongest students guarantees that the school’s overall pass rate on the final examina-
Strategies for providing laboratory experiences to online students are discussed in Chapter 5.
Mathews (1998b) estimates that each year, 25,000 interested and adequately prepared students in the United States are told they cannot take AP or IB courses. He further speculates that another 75,000 or more students who have the ability to do well in such courses do not elect to take them because no one encourages them to do so.
tions will be very high, boosting their reputation with the top colleges they care most about influencing.
The committee agrees that AP and IB are not appropriate for all students. Some are not prepared for or do not want to take college-level courses in high school, and such courses are not appropriate for all high school students. However, research shows that there are few rigorous options outside of these classes, especially in high schools with well-developed AP and IB programs (Callahan, 2000). Thus, students who are denied access to or choose not to take these classes are forced to enroll in what are often far less challenging options.
Prerequisites are frequently used as gatekeepers to regulate enrollment in AP and IB courses. Sometimes this practice is deemed necessary because there is limited space. In other cases, the school simply wants to control which students take the courses to maintain quality (as represented, for example, by high test scores). The committee reviewed more than 100 curriculum guides prepared by individual high schools for their students, parents, and teachers. This review revealed that prerequisite requirements for enrollment in AP or IB courses range from open admission to highly restrictive criteria such as PSAT scores above a certain threshold, all A’s in courses leading up to the AP courses, uniformly high teacher recommendations, and evidence of consistently high levels of motivation or excellent work habits. Many of these guides also contain statements to the effect that students who take courses for which they are not recommended must have signed permission from a parent absolving the teacher and the school from responsibility if the student does not succeed. If measures are to be used to extend or restrict access to AP and IB courses, the committee urges that schools demonstrate that their criteria for entry to advanced study are valid predictors of student success.
COLLEGE CREDIT AND PLACEMENT
Throughout this report, the committee has challenged the assumption that AP courses uniformly reflect the content coverage and conceptual understanding that is developed in good college courses. We now ask whether students who place out of introductory courses in college on the basis of their AP scores are as well prepared for further study as their peers who take the introductory courses in college.
The College Board has conducted several studies to investigate this question. Recently, Morgan and Ramist (1998) examined the performance of AP and non-AP students in second-level courses in 25 subjects at 21 colleges. These subjects included sciences and mathematics, as well as many others. Breland and Oltman (2001), using data published by Morgan and Ramist (1998), examined the performance in upper-level courses of students who took AP comparative government and politics and AP economics. The conclusion of both of these studies was that students who earned qualifying scores on AP examinations appeared to earn grades in second-level courses that were as good as those earned by students who took the first-level course in college. However, this conclusion is true only for an overall average. Further, it was implicitly assumed in these studies that any student with a score of 3 or better on the AP examination would be allowed to take the second-level course. The committee knows from its department survey (see Chapter 2, this volume) that this is not the case. Thus a more precise conclusion for these studies is that students with qualifying AP scores who were exempted from first-year courses in college appeared, on average, to earn grades in second-year courses that were no lower than those earned by students who took the introductory courses in college.
With the assistance of an educational statistician,5 the committee examined the assumptions, methods, and conclusions of these two studies. This examination revealed that the methodology used in conducting the studies makes it difficult to determine how often and under what circumstances there is a positive advantage for AP students relative to non-AP students in second-level courses. There are two principle reasons for this difficulty:
The investigators based their conclusions on averages of grades that were computed across classes and institutions to obtain overall averages. No effort was made to control for differences in the types and quality of the institutions involved in the study or for how well the AP courses matched the introductory courses students had placed out of in the different institutions.
No attempt was made to control for differences among students that could be related to their performance in second-level courses. Examples of such differences include SAT scores (both SAT I and SAT II in the relevant subjects), high school grade point average, whether or not the students intended to major in the subject, and the quality of their overall high school curricula. No evidence is provided in the study reports as to whether stu-
dents with the same credentials, apart from the AP examination score, also could skip the introductory-level courses with the same positive results in the next-level course.
In conclusion, the methodology used by the investigators to gather and analyze their data makes it difficult to determine whether any apparent advantage held by AP students over non-AP students is a function of the colleges they attend, the classes they enter, their own academic backgrounds and abilities, or the quality of the AP courses they took in high school. Further, there is no way to determine, from the data provided, the number of classes among the various colleges in which non-AP students outperformed AP students. It is possible that AP students were at a disadvantage in some classes or at some colleges.6
The committee’s analysis of the College Board studies indicates that colleges and universities should not automatically award advanced placement to students with specified AP scores and assume that they will be successful.7 Rather, it is more consistent with the evidence developed by the College Board for colleges to adopt policies that consider students individually and in combination with the requirements of the courses into which they seek placement.
The practice of awarding university credit and placement for IB Higher Level (HL) courses dates back to at least 1973, when the National Council on the Evaluation of Foreign Student Credentials recommended that U.S. colleges and universities accept IB HL8 examination grades for college credit
It might have been more useful to treat the study as a meta-analysis, with each class in which there were AP and non-AP students being treated as a datum. In this scenario, each class would furnish an advantage or disadvantage for AP students with respect to their non-AP classmates. If each class were treated as a datum, standard deviations could be reported, and other class-level characteristics, such as size, instructional methodologies, and the degree to which knowledge gained in a first-level class was important for success in the second-year course, could be analyzed with respect to the performance of AP and non-AP students. Perhaps AP students do better only in small classes, or at selective colleges, or when inquiry is emphasized. Perhaps they do not do better when material from first-level courses is emphasized or when there is a poor match between the AP courses and the introductory courses taught at their colleges.
The Morgan and Ramist and Breland and Oltman studies tell us only that on average, students who are exempted from introductory courses get grades that are as good as those of students who take the first course in college. They do not tell us whether AP students at a particular institution who are exempted from that college’s introductory courses have the same depth of understanding of a subject area as students who take the first course at that college.
IB HL courses are taken over a 2-year period (see Chapter 4, this volume).
when foreign students present such grades. Since that time, more than 750 colleges have established policies to accept IB HL credits from students worldwide. According to Paul Campbell, associate director of the International Baccalaureate Organisation of North America (IBNA) (personal communication, March 2000), the number of institutions accepting IB examination scores for credit or placement has increased as a result of individual students petitioning colleges on a case-by case, college-by-college basis.
In seeking credit and placement, many students cite evidence that IB HL courses are comparable to similar AP courses for which colleges and universities routinely grant credit and placement. However, the committee could find no systematic studies that have examined the validity of the claim that AP and IB courses are comparable.9 Additionally, the IBO does not design its courses to be similar to introductory college courses, so it is difficult to determine how comparable the two are. Additionally, we know of no systematic studies that compare how well students who score differently on IB examinations perform in upper-level college courses or whether their performance varies by discipline.10 It is therefore unclear to what extent advanced placement in upper-level college courses is merited on the basis of IB examination scores alone.
More evidence is needed to understand the consequences for students and colleges of awarding advanced placement for AP or IB examination scores. The evidence appears to support the interpretation that a student who earns a high AP score will earn a good grade in an upper-level course in that discipline. However, no evidence has been developed to demonstrate that students who earn qualifying scores on AP or IB examinations have achieved a depth of understanding of the subject comparable to that obtained by students taking the introductory college-level courses. Consequently, the disadvantages that accrue to AP and IB students who decide not
The only study purporting to address the issue of AP and IB comparability was conducted at Case Western Reserve University, where the scores of students who took both AP and IB examinations in the same subject area were compared. The findings indicated that students who scored a 6 or 7 on IB HL examinations scored a 4 or 5 on the AP examination in the same subject area; the reverse was not the case. It may be assumed that all of these students took the IB course and not the AP course since students cannot take an IB examination without having taken the corresponding IB course. The methodology used in the study was not specified.
The committee found a summary of isolated studies that examine the overall performance of IB students in college. Some of these studies were conducted at Virginia Polytechnic Institute and State University, University of Florida, and Marquette University. One study that looked at IB students’ performance in upper-level science and mathematics courses revealed that IB students outperformed all other students in terms of overall grade point average and/or grades in upper-level courses (Guild of IB Schools, www.rvcschools.org [August 2000]). However, it is difficult to determine the validity of the study results because information about the methodology used by the investigators is incomplete.
to take an introductory college course in a subject before going on to a second-level course are not known.
The potential for misinterpreting and misusing test results and other aspects of the AP and IB programs is high and likely to become higher unless countermeasures are taken. The consequences of such misuses extend to students, teachers, high schools, and institutions of higher education.
The misuse of scores and programs appears less widespread for the IB than for the AP program. This difference likely results from the IBO’s maintaining tighter control over its program. In addition, the IB program is offered in far fewer schools in the United States than the AP program, making it more difficult to use data on the former program to draw inappropriate comparative inferences about the quality of schools or teachers. Until the College Board makes a concerted effort to educate the media, policymakers, and the public about correct and incorrect interpretations and uses of its examination results, the kinds of abuses described here, and the consequences associated with them, will almost certainly continue and will probably increase.
As this report demonstrates, advanced study programs have an enormous influence on virtually all other components of the education system in the United States. As the programs are currently structured, some of these influences have worked to improve education. However, serious shortcomings persist. It is incumbent upon all individuals and institutions with a stake in improving advanced study and making it accessible to many more students to do so systematically, collaboratively, and in ways that are consistent with emerging research about learning and effective program design. The next chapter offers a series of recommendations for accomplishing these goals. This set of recommendations emerges from the committee’s analysis of existing programs of advanced study; the way these programs influence and are affected by other components of the education system in the United States; and whether the programs’ current structures are consistent with the principles of learning and the design principles of curriculum, instruction, assessment, and professional development set forth in this report.