Assessing Accomplished Teaching: Advanced-Level Certification Programs (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary The National Board for Professional Teaching Standards (NBPTS) was created in 1987 in response to recommendations made by the Carnegie Task Force on Teaching as a Profession. These recommendations, reported in A Nation Prepared: Teachers for the 21st Century, called for large-scale reforms to improve the quality of the U.S. teaching force, including the formation of a national board whose task was to establish standards for exemplary teaching practice and to develop a means to award advanced- level certification to teachers who meet these standards. The NBPTS has been offering advanced-level certification for teachers since 1994. The mission of the national board is to establish “high and rig- orous standards for what teachers should know and be able to do, to certify teachers who meet those standards, and to advance other education reforms for the purpose of improving student learning in American schools.” The Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards was established at the National Research Council (NRC), at the request of the U.S. Congress and with sup- port from the U.S. Department of Education, to evaluate the impacts of the national board’s efforts. The U.S. Congress asked the NRC to develop a framework for evaluat- ing programs that award advanced-level teacher certification and to apply that framework in an evaluation of the impacts of the NBPTS. Congress specified that the framework should be general enough to be applied to  

 ASSESSING ACCOMPLISHED TEACHING other programs when data are available to permit such an evaluation and should address the following issues: 1. The impacts on teachers who obtain board certification, teachers who attempt to become board certified but are unsuccessful, and teachers who do not apply for board certification; 2. The extent to which board certification makes a difference in the academic achievement of students; and 3. The cost-effectiveness of advanced-level certification as a means for improving teacher quality. This report presents the committee’s framework for evaluating pro- grams that award advanced-level teacher certification and applies it to an evaluation of the impacts of the national board. Our principal findings are summarized below, and our conclusions and recommendations appear in their entirety in Chapter 12. We note that these recommendations are di- rected at the NBPTS, as our charge specifies, but they also highlight issues that should apply to any program that offers advanced-level certification to teachers. The Evaluation framework The evaluation framework developed by the committee is structured around eight sets of questions based on hypotheses about the way a pro- gram for certifying accomplished teachers might improve teaching: 1. Specification of the Content Standards and Development of the Assessments: To what extent does the certification program for accomplished teachers clearly and accurately specify advanced teaching practices and the characteristics of teachers (the knowl- edge, skills, dispositions, and judgments) that enable them to carry out advanced practice? Does it do so in a manner that supports the development of a test that is well aligned with the content standards? 2. Technical Characteristics of the Assessments: To what extent do the assessments associated with the certification program for accom- plished teachers reliably measure the specified knowledge, skills, dispositions, and judgments of candidates and support valid inter- pretations of the results? To what extent are the performance stan-   The American Board for Certification of Teacher Excellence (ABCTE) currently has a program under development for awarding advanced-level certification to distinguished teachersSM. 

SUMMARY  dards for the assessments and the process for setting them justified and reasonable? 3. Participation: To what extent do teachers participate in the program? 4. Impact on Outcomes for Students: To what extent does the advanced-level certification program identify teachers who are ef- fective at producing positive student outcomes, such as learning, motivation, school engagement, breadth of achievement, educa- tional attainment, attendance, and grade promotion? 5. Impact on Participating Teachers’ Professional Growth: To what extent do teachers improve their practices and the outcomes of their students by virtue of going through the advanced-level certi- fication process? 6. Impact on Teachers’ Career Paths: To what extent and in what ways are the career paths of both successful and unsuccessful can- didates affected by their participation in the program? 7. Impact on the Education System: Beyond its effects on candidates, to what extent and in what ways does the certification program have an impact on the field of teaching and the education system? 8. Cost-effectiveness: To what extent does the advanced-level certifica- tion program accomplish its objectives in a cost-effective manner, relative to other approaches intended to improve teacher quality? The NBPTS has been the topic of much discourse in the measurement, teacher education, and education policy literature; nearly 200 articles dis- cuss the board’s work. However, the majority of these documents do not report on empirical research, and only a handful yield valid findings re- lated to the questions in our charge. Thus, we relied on an evidence base that was neither broad nor deep, which we supplemented with additional investigations. Development of the STandards and Assessments Over a seven-year period, the board worked to identify the essential characteristics of accomplished teaching and to develop a method for iden- tifying teachers who demonstrated these practices. Their product is a set of standards for 25 teaching specialty areas. The standards in each area describe the ways accomplished teachers demonstrate that they know their students, their subject matter, and how to teach it; think systematically about their practice; and learn from their experience. Assessments for each specialty were designed to allow teachers to demonstrate their proficiency in classroom settings. To earn NBPTS certification, teachers must respond to six computer-based constructed-response exercises that measure subject 

 ASSESSING ACCOMPLISHED TEACHING matter knowledge and must assemble a portfolio consisting of videotapes of their teaching, written reflections on their goals and the outcomes of the lesson submitted, and student work. We reviewed the processes used by the NBPTS to develop content standards for each specialty area as well as exercises to assess them. The board convened a diverse group of experts to develop standards for high- quality teaching, and it solicited considerable feedback before adopting the standards. Its development of standards and assessments for a wide array of teaching specialty areas is a significant accomplishment. Overall, we conclude that the board’s approach was reasonable and, for the most part, conforms to professional standards for certification tests (Conclusions 5-1 and 5-2). We highlight two concerns, however. First, we initially encountered difficulty in obtaining documentation that was sufficiently detailed, al- though we note that the board eventually provided the information we needed to conduct our review. The board did not have a technical manual readily available, and the version it eventually provided was incomplete and still in draft form. Professional standards call for a testing program to maintain documentation about the technical characteristics of its assess- ments, and we recommend that the board make improvement in this area (Recommendation 5-1). Ongoing evaluation of an assessment program is critical to maintaining its quality and credibility, and providing thorough documentation that is easily accessible to outside evaluators is a critical element of this process. Our second concern relates to the translation of the standards state- ments into assessment exercises. While the content standards are written in a readable style, the language is imprecise. Translating the general state- ments of the standard to specific assessment exercises requires a significant amount of judgment on the part of the test developer. We recommend that the board develop more precise explanations of the standards to facilitate the work of the test developer and to ensure that the assessment exercises measure the intended skills (Recommendation 5-2). Technical Characteristics of the Assessments We evaluated the procedures for scoring the assessment exercises and setting the passing score, the reliability and validity of the scores, and the extent to which the assessments fairly appraise the skills of all teachers applying. Overall, we judge that the board has taken appropriate steps to ensure that the assessments meet professional standards and results from   Recommendation and conclusion numbers refer to the report chapter in which they are made and the sequence in which they appear in the chapter. 

SUMMARY  validity studies document that the assessments are effective in identifying teachers who demonstrate accomplished teaching practices. Improvement may be possible in two areas, however. First, the portfolios that the NBPTS uses have the advantage of provid- ing an authentic representation of a teacher’s skills. At the same time, the scoring process for portfolio responses is less reliable than that for more objective forms of assessment. The reliability of the scores from the NBPTS assessments is consistent with expectations for a largely portfolio-based process but lower than that desired for a high-stakes testing program. We recommend that the NBPTS explore ways to improve the reliability of the scores, possibly by increasing the number of exercises, but we caution that efforts to improve reliability should not compromise the authenticity of the assessment or substantially increase the costs associated with scoring the exercises (Recommendation 5-3). Second, a key responsibility of a high-quality testing program is regular evaluation. Our review reveals that the board has not devoted the same energy that went into the original assessment design to ongoing evalua- tion of how that design has worked over time, nor has it found ways to improve on it. Regardless of the assessment methods used, we think that the board should devote more effort to continuously improving its assess- ments. With tests that rely on multiple-choice items, developers are able to use statistical data to evaluate and refine the items before they are used operationally. Although this is not usually feasible with tests that consist of performance assessment exercises, data can be collected after their initial operational use and analyzed to identify exercises that exhibit relatively low reliability or disparate impact. It is not clear how closely the board tracks such data and uses them to improve assessment exercises. We recommend that the NBPTS collect and use the available operational data about the individual assessment exercises to improve the validity and reliability of the assessments for each certificate, as well as to minimize adverse impact (Recommendation 5-4). Teacher Participation Carnegie task force members envisioned that national board certifi- cation would become a widely recognized credential, that districts and states would value board-certified teachers, and that the numbers of board- c ­ ertified teachers would grow. They expected that board-certified teachers would become a significant presence, helping to spread the influence of teaching standards by serving as leaders and mentors. From 1993, when the program began operation, through 2007, roughly 99,300 teachers applied for board certification, and 63,800 teachers earned the credential. While these numbers represent less than 3 percent of the country’s current force 

 ASSESSING ACCOMPLISHED TEACHING of 3.7 million teachers, it is noteworthy that participation has increased over the life of the program, from about 540 applicants in the first year to about 12,200 in the 2006-2007 school year. Overall, the number of board- c ­ ertified teachers translates to about three for every five schools. Participation rates are not even across the country, however. There are higher concentrations in some states and districts, and in a few districts participation rates are approaching levels likely to be sufficient for the pro- gram to have the intended effects. Not surprisingly, the popularity of board certification appears to be related to the degree to which states and districts encourage it. Some states offer financial incentives to teachers—covering the $2,500 test fee and offering sizable salary increases to those who are suc- cessful—and have higher participation rates than states that offer minimal or no incentives. With regard to participation, we highlight several issues as concerns. First, existing data about teachers who have gone through the board certi- fication process are scant. Little is known about what teachers have done after completing the certification process, what has happened to teachers who did not pass the assessment, how many board-certified teachers are currently employed, where board-certified teachers currently work, and what jobs they do. We recommend that the NBPTS implement and maintain a database of information about NBPTS applicants and their career paths (Recommendation 6-1). In addition, there are significant disparities in applicant participation rates, with teachers from advantaged schools more likely to participate than others. Furthermore, the absolute numbers of racial/ethnic minority teacher participants are low. We think these are issues deserving of additional atten- tion, especially considering that one of the board’s goals is to place board- certified teachers in schools with high-needs students. The board has efforts under way to recruit minority teachers to pursue board certification, and we encourage it to continue its work in this area. EFFECTS OF BOARD-CERTIFIED TEACHERS ON Student Outcomes The question of how the program is related to student outcomes can be considered in two ways. First, passing the certification process may act as a signal of preexisting teaching effectiveness. Second, the process of becoming board certified may cause a teacher’s classroom effectiveness to improve. Questions related to student outcomes have generated the largest number of research studies, with most focusing on the question of whether board certification acted as signal of (preexisting) teaching effectiveness. Nearly all of these studies compare the achievement test scores of students taught by board-certified and nonboard-certified teachers; few compare other student 

SUMMARY  outcomes, such as motivation, breadth of achievement, or promotion rates. We reviewed 11 studies that measured student outcomes in terms of their achievement test performance. They focus primarily on North Carolina and Florida, states that have substantial numbers of board-certified teachers and have maintained longitudinal databases of teachers and students. Findings from these studies show that, in both states, students taught by board-certified teachers had higher achievement test gains than did those taught by nonboard-certified teachers, although the differences were small and varied by state. North Carolina, a state with a long history of encouraging teachers to pursue board certification, showed slightly larger differences between the students of board-certified and nonboard-certified teachers, while group differences in Florida were smaller. We see a relation- ship between board certification and student achievement, although the relationship is not strong and is not consistent across contexts. We recommend that additional research be conducted in this area, but we do not think that all research resources should be devoted to studies focusing on student performance on achievement tests, in part because such tests measure limited aspects of student learning. To the extent that existing data sets allow, we recommend replication of studies that investi- gate the effects of board-certified teachers on student achievement in states besides North Carolina and Florida, in content areas beyond mathematics and reading, and in grades beyond the elementary levels (Recommendation 7-1). Researchers pursuing such studies should work with the national board to obtain the information needed to study the effects of teachers who successfully obtained board certification, in comparison with effects for those applicants who were unsuccessful. We also recommend that re- searchers conduct studies of the effects of board certification on outcomes beyond scores on standardized tests, such as student motivation, breadth of achievement, attendance, and promotion (Recommendation 7-2). The choice of outcome measures should reflect the skills that board-certified teachers are expected to demonstrate. Impacts on Teachers’ Professional Growth One potential benefit of the NBPTS program is that the process of be- coming familiar with the board’s standards and completing the assessment could have a positive effect on a teacher’s classroom practice. Considering the time required by the assessment and the depth and complexity of the tasks involved, it seems reasonable to expect some impact on the practices of those who complete the process. Two studies directly investigated what teachers learn during the certi- fication process. While the results suggest that the process contributes to their professional growth, the studies were small in scale and need replica- 

 ASSESSING ACCOMPLISHED TEACHING tion. Several other studies compared the effectiveness of teachers in North Carolina and Florida in terms of their students’ reading and mathematics achievement before, during, and after their teachers earned board certifica- tion. The findings from these studies are mixed. Thus, in our judgment, the existing research neither proves nor refutes hypotheses about the effects of the certification process on teachers’ prac- tice. We think that additional research should explore this issue, and we make recommendations for three kinds of research to pursue. First, we recommend that the NBPTS and other researchers investigate the effects of the process on the candidates using pretest/posttest and lon- gitudinal designs; these studies should be designed to permit comparisons of responses for successful and unsuccessful candidates (Recommendation 8-1). We also recommend research on the effects of board certification on teachers’ practices; these studies should utilize both quantitative and qualitative methods to examine a variety of measures of teachers’ practices and a variety of student outcomes (Recommendation 8-2). Finally, we rec- ommend that researchers work with the NBPTS to obtain the information needed to study the relationships between board certification and student achievement across the various stages of board certification; these studies should examine the impacts of the certification process on teachers’ ef- fectiveness in increasing their students’ test scores, and specifically should examine effects for the years subsequent to the receipt of board certification (Recommendation 8-3). Impacts on Teachers’ career paths The goals of the NBPTS include helping to professionalize the field, motivating districts and states to raise salaries for accomplished teachers, motivating districts and states to expand opportunities for leadership in the field, and increasing accomplished teachers’ satisfaction with their careers. Little information is available to evaluate progress toward these goals. One study examined teachers’ longevity in the field. The results suggest that board-certified teachers are more likely than other teachers to indicate that they plan to remain in teaching, but the findings were based on teach- ers’ responses to only a few survey questions. This finding was corrobo- rated by results from our own analyses, which indicate that board-certified teachers actually do stay in teaching at higher rates than other teachers. However, our findings were based on a small national sample of teachers and need further corroboration. Neither the existing study nor our analyses permit causal inferences; that is, they do not indicate whether the NBPTS process causes teachers to stay in the field longer or whether the teachers who choose to become board certified are already more likely to remain in the field, regardless of whether they earn certification. 

SUMMARY  Another study addressed the question of whether acquiring board certification affects the mobility of teachers in the field. Data from North Carolina show that those who obtain board certification tend to change teaching jobs at higher rates than do unsuccessful applicants. These data also indicate that when they move, board-certified teachers are likely to move to teaching assignments with more advantaged conditions, such as schools with higher student achievement levels or fewer students in poverty. However, it is not clear that this tendency is any more prevalent for board- certified teachers than for other teachers with excellent qualifications. Additional research is needed on the career paths of board-certified teachers before any firm conclusions can be drawn. We recommend that the NBPTS and other researchers study the subsequent career choices of teachers who have applied for board certification. These studies should use methodologies that permit comparisons of teachers’ career choices before and after becoming board certified, and they should compare the choices of unsuccessful applicants for board certification, teachers who successfully obtained the credential, and teachers who did not apply for it (Recommen- dations 9-1 and 9-3). impacts on the Education System The Carnegie task force envisioned that the board’s influence would reach beyond any impact that individual board-certified teachers might have on their students. The task force hoped that the board’s standards would be widely influential and the demand for board-certified teachers would lead to improvements in working conditions for all teachers. Board- certified teachers would influence the way their colleagues taught, schools and districts would use the board standards as a guide and work to provide teaching environments conducive to the board’s approach, and teacher preparation and professional development would spread the influence of the board’s standards to future generations of teachers. Little research is available on these kinds of spillover effects. Two stud- ies examined the impact of board certification on teachers’ roles in their schools, focusing on the six states with the largest populations of board- certified teachers. The results suggest that school systems are not making the best uses of their board-certified teachers and that board-certified teach- ers often work in unsupportive environments. The studies reveal instances of administrators who discourage board-certified teachers from assuming responsibilities outside the classroom and who worry about showing favor- itism toward board-certified teachers. In some cases, administrators down- play the significance of the credential, and some board-certified teachers conceal their credential so as not to seem to be showing off. Despite these negative findings, the studies described a few school sys- 

10 ASSESSING ACCOMPLISHED TEACHING tems in which board-certified teachers are rewarded, used effectively, and offered new opportunities. In these instances, board-certified teachers are used as mentors, team leaders, and organizers of professional development activities; board certification is viewed as part of a broader commitment to improving professional development and meeting higher standards for teachers. With regard to influences on teacher preparation, the National Coun- cil for Accreditation of Teacher Education has aligned its accreditation standards for teacher education programs with the NBPTS standards. The curriculum standards for programs that prepare beginning teachers offered by the Interstate New Teacher Assessment and Support Consortium are also aligned. These efforts lay the groundwork for the NBPTS standards to impact teacher preparation, but there is no research to document the extent of the board’s influence on the content of teacher preparation programs or the standards of individual programs. From the small research base, we found little evidence that the national board is having the intended spillover effects, but we highlight the fact that much of the needed research has not been conducted. We think that board-certified teachers are unlikely to have a significant impact without broader endorsements by states, districts, and schools of the NBPTS goals for improving professional development, setting high standards for teach- ers, and actively utilizing the board-certified teachers in leadership roles. Furthermore, we think that the certification program is unlikely to have broad systemic effects on the field of teaching unless greater numbers of teachers become board certified and the Carnegie task force’s other recommendations—for creating a more effective environment for teach- ing and learning in schools, increasing the supply of high-quality entrants into the field, and improving career opportunities for teachers—are imple- mented. However, our review of the evidence led us to conclude that there is not yet sufficient research to evaluate the extent to which the NBPTS is having systemic impacts on the teaching field and the education system (Conclusion 10-1). The Cost-Effectiveness of Certification as A Means of Improving Teacher Quality Our review revealed that the research base needed to support a cost- effectiveness evaluation of the NBPTS is inadequate. Making a rough cal- culation of the costs of the program is relatively straightforward, but evaluating how these costs compare with those of other approaches for teacher professional development presented significant problems because of a lack of data. 

SUMMARY 11 Advanced-level certification of teachers has the potential to offer three kinds of benefits: (1) it can provide a systematic way of identifying high- quality teachers, that is, a signal of quality; (2) the process itself can provide a means for teachers to improve their practices; and (3) it can improve the quality of teachers throughout the education system, keeping accomplished teachers in the field and attracting stronger teacher candidates in the future. The available evidence suggests that NBPTS certification does provide a means of identifying highly skilled teachers; however, this evidence does not provide sufficient information about the other two benefits. Simply identifying high-quality teachers provides no direct benefit un- less this signal of quality is used. One of the most important benefits that might result from the program—keeping high-quality teachers in the field—could not be evaluated because the necessary data have not been col- lected. While there is some evidence that policy makers have used the signal provided by board certification to improve teaching quality (i.e., by offering salary bonuses to teachers who earn board certification as an incentive to remain in teaching), the policies were not implemented in a way that allows an examination of their impacts. Furthermore, except in isolated instances, there is no evidence that districts or schools are using the signal of quality provided by board certification to encourage board-certified teachers to work in difficult schools or to mentor other teachers. We identified three interventions designed to improve the practices of experienced teachers that could serve as comparisons in an evaluation of the cost-effectiveness of the NBPTS: (1) the ABCTE’s proposed program to certify distinguished teachersSM, (2) encouraging teachers to pursue master’s degrees, and (3) providing relevant in-service professional development. Very little information useful for a full cost-effectiveness analysis is avail- able about these interventions. Our cost analysis suggested that the annual per-teacher costs associated with board certification are probably lower than the annual per-teacher costs of obtaining a master’s degree. However, the evidence about the benefits of master’s degrees is too mixed to be able to derive a cost-effectiveness estimate that could be compared with that for board certification. For the other two possible comparisons, even less is known. Thus, we conclude that, at this time, it is not possible to conduct a thorough cost-effectiveness evaluation of the NBPTS certification because of the paucity of data on its benefits and on both the costs and benefits of other mechanisms intended to improve teacher quality (Conclusion 11-2). Such an evaluation should be undertaken if and when the necessary evi- dence becomes available. 

12 ASSESSING ACCOMPLISHED TEACHING OVERALL ASSESSMENT The board set out to transform the teaching field, and it has been in- novative in its approach. The standards captured a complex conception of advanced teaching and stimulated thinking about what accomplished teachers should know and be able to do. The portfolio-based assessment that it developed to measure teachers’ practice according to these standards pushed the measurement field forward. The NBPTS has the potential to make a valuable contribution to efforts to improve teacher quality, together with other reforms intended to create a more effective environment for teaching and learning in schools, increase the supply of high-quality entrants into the field, and improve career oppor- tunities and working conditions for teachers. Our review suggests, however, that much of the research needed to evaluate these intended impacts has not been conducted. Moreover, we point out that revolutionary changes of the kind the board’s founders envisioned would be expected to develop over decades, not years. This evaluation thus provides an opportunity to take stock of what has worked well and to suggest changes needed to respond to the current policy environment. For the national board to realize its potential, several key changes in its operation and approach are needed. We think that, if the board is to build on its accomplishments and thrive as a means of improving teacher quality in the United States, it will need to attend to the following: • The NBPTS should conduct its work according to the highest standards for credentialing programs, make its operations acces- sible to external scrutiny, and conduct regular evaluations of its assessments to ensure continuous improvement. • The NBPTS should pursue an ongoing research agenda to evaluate progress toward its goals. • The NBPTS should periodically review its assessment model, both to evaluate how it has worked in practice and to adapt to changes in the policy environment and advances in research. As part of such ongoing evaluation efforts, the board should consider whether adjustments are needed in the types of information used as the basis for certification, which might include classroom observa- tions, objective tests of content knowledge, or measures of student performance.  • The NBPTS should continue to invest in its larger mission of influ- encing the teaching field in broad, comprehensive ways. The NBPTS offered a thoughtful approach to serious problems with the way this country’s education system selects and prepares its teachers 

SUMMARY 13 and the conditions in which teachers work. Given the magnitude of the problems the board addressed, the fact that there is only limited evidence of its impact does not prove that this approach cannot be successful. For the program to have the intended impacts on the teaching field, improve- ments will be needed, both in the operational aspects of the program and in the evidence collected, as we have recommended in this report. The board cannot achieve these goals alone, however. Meeting these ambitious goals will also require a serious commitment by education policy makers to the other recommendations made by the Carnegie Task Force on Teaching as a Profession.