Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
12 Overall Evaluation Our review of the National Board for Professional Teaching Standards (NBPTS) required that we consider the program from a variety of perspec- tives and explore several complex technical questions in detail. In Chapters 5 through 11, we examined each of the questions in our evaluation frame- work on its own merits. In this final chapter, we step back to consider all of the pieces of evidence together and to offer several overall conclusions about the program. We begin by summarizing the key conclusions and recommendations related to each element of the evaluation framework. We then present our overall conclusions about the board and the role it can play in improving teacher quality. findings and Recommendations Psychometric Characteristics of the Assessment From our review of the assessments themselves and the development process, we find that, in general, high standards have been followed. The initial design and development process was extensive, and some of the most renowned measurement experts in the country had considerable input. The process was carried out carefully and in a transparent manner. The develop- ment of standards and assessments for a wide array of teaching specialty areas is a significant accomplishment. Since the program has become operational, however, attention to psy- chometric matters seems to have become routinized, and somewhat less 248
OVERALL EVALUATION 249 attention is being paid to issues that were critical during the development stage. This might be expected as a testing program matures and evolves. However, perhaps in part because of staff turnover and a change in loca- tion, historical documentation about the assessment has been difficult to locate. We initially encountered significant difficulty in obtaining documen- tation that was sufficiently detailed to allow us to evaluate the development of the standards and the design of the assessments, although we note that the board eventually provided most of the information we needed to con- duct our review. We found this deficiency to be particularly troublesome as we explored the content-related validity evidence for the national board assessment. Or- dinarily, the primary focus in an evaluation of a credentialing assessment is content-related validity evidenceâthat is, the evidence that the assessment measures the knowledge and skills it is intended to measure, based on the content standards that guide the development of the assessment. Content- related validity evidence, such as documentation of how the content stan- dards were established, who participated in the process, what the process involved, and how the content standards were translated into test items, was the most difficult for us to obtain from the NBPTS. The NBPTS is unusual in that its mission includes policy reform goals as well as the operation of an assessment program. However, in our opin- ion, the assessment is its primary responsibility. Ongoing evaluation of an assessment program is critical to maintaining its quality and credibility, and providing thorough documentation that is easily accessible to outside evaluators is a critical element of this process. The NBPTS should be able to readily provide documentation that demonstrates that its assessments are developed, administered, and scored in accord with high standards, such as those laid out in the standards documents for credentialing assessments. We note that during the course of our evaluation, the NBPTS has begun developing a technical guide, and we encourage the NBPTS to finalize this document and make it available to researchers and others interested in learning about the technical attributes of the assessments. Our key recommendations relating to the assessment itself are as follows: Recommendation 5-1: The NBPTS should publish thorough technical documentation for the program as a whole and for individual specialty area assessÂments. This documentation should cover processes as well as products, should be readily available, and should be updated on a regular basis. Recommendation 5-2: The NBPTS should develop a more structured pro- cess for deriving exercise content and scoring rubrics from the content
250 ASSESSING ACCOMPLISHED TEACHING standards and should thoroughly document application of the process for each assessment. Doing so will make it easier for the board to maintain the highest possible validity for the resulting assessments and to provide evidence suitable for independent evaluation of that validity. Recommendation 5-3: The NBPTS should conduct research to determine whether the reliability of the assessment process could be improved (for example, by the inclusion of a number of shorter exercises in the computer- based component) without compromising the authenticity or validity of the assessment or substantially increasing its cost. Recommendation 5-4: The NBPTS should collect and use the available operational data about the individual assessment exercises to improve the validity and reliability of the assessments for each certificate, as well as to minimize adverse impact. Recommendation 5-5: The NBPTS should revisit the methods it uses to es- timate the reliabilities of its assessments to determine whether the methods should be updated. Recommendation 5-6: The NBPTS should periodically review the assess- ment model to determine whether adjustments are warranted to take ad- vantage of advances in measurement technologies and developments in the teaching environment. Teacher Participation in National Board Certification The boardâs founders envisioned that NBPTS certification would be- come a widely recognized credential, that districts and states would value board-certified teachers, and that the numbers of certified teachers would grow. The founders expected that board-certified teachers would become a significant presence, helping to increase the influence of the board stan- dards by serving as leaders and mentors to other teachers. From the 1993- 1994 school year, when the program began operation, to the 2006-2007 school year, 99,300 teachers have attempted to earn board certification, and 63,800 teachers have been successful. These numbers represent approximately 3 percent of the 3.1 million NBPTS-eligible teachers in the country, and it is likely that some of those who obtained board certification will have retired or allowed their certifica- tion to lapse. While NBPTS participants represent a small fraction of the teachers in this country, the absolute volume of teachers who have pursued board certification is considerable. Moreover, the numbers of participants
OVERALL EVALUATION 251 have increased over the life of the program, from 550 applicants in the first year to 12,200 during the 2006-2007 school year. Still, it is worth noting that the original target set by the board was to identify the top 10 percent of teachers. If the board had met that goal, approximately 400,000 teach- ers would currently be certified. Assuming that all of the 63,800 teachers who had obtained board certification by 2006-2007 were still teaching, the NBPTS would be about a sixth of the way toward achieving this goal. Participation rates are not even across the country. Overall, the number of board-certified teachers translates to three for every five of the 96,513 schools in the country. However, there are higher concentrations in some districts, such as Wake County, North Carolina, with an average of seven board-certified teachers per school, and Broward County, Florida, with an average of two per school. There are other disparities as well. More teachers from advantaged schools participate, and the absolute numbers of minority teachers participating are low. The popularity of board certification varies dramatically from state to state, as does the degree to which states and districts encourage it. We were not able to find any research on the factors that influence the thinking of state policy makers about encouraging teacher participation. However, some states offer financial incentives to teachersâcovering the $2,500 test fee and offering sizable salary increases to those who are successfulâand they have higher participation rates than states that offer no incentives. In four states that have consistently offered financial incentivesâFlorida, Mis- sissippi, North Carolina, and South Carolinaâbetween 10 and 21 percent of NBPTS-eligible teachers have attempted to become board certified, well over the national rate of about 3 percent. In the seven states that have not offered incentives over the past few yearsâAlaska, Arizona, Massachusetts, Minnesota, Tennessee, Texas, and Utahâthe participation rate ranges from 0.2 to 1.5 percent of NBPTS-eligible teachers. On the basis of our review of the available data, we have drawn two conclusions and make one recommendation. Conclusion 6-1: Although the number of teachers who have obtained certi- fication is small relative to the general population of U.S. teachers, the total has grown since the program began and is now over 63,800. Participation varies significantly by state and district; however, in a few districts, partici- pation rates are approaching levels likely to be sufficient for the program to have the intended effects. Conclusion 6-2: States that offer financial incentives for attempting and achieving board certification are likely to have more teachers that apply and succeed in the program.
252 ASSESSING ACCOMPLISHED TEACHING Recommendation 6-1: The NBPTS should implement and maintain a da- tabase of information about applicants and their career paths. This effort should include routine, annual data collection as well as specially designed studies. The data collected should provide information about what teachers have done after going through the certification process, what has happened to teachers who did not pass the assessment, how many board-certified teachers are currently employed, where board-certified teachers currently work, and what jobs they do. Impacts of the National Board-Certification Process Our framework examines the various kinds of impacts the NBPTS could have on students, in terms of their learning; on the teachers who participate, in terms of their professional growth and their career paths; and on the education system itself, through board-certified teachersâ influ- ence on their colleagues, school systems, and teacher training programs. This task was more difficult than we had anticipated because, as we have reiterated throughout this report, little valid evidence is available. The dearth of evidence was somewhat of a surprise because numerous studies have been conducted on the impacts of the NBPTS. However, many were based on such small sample sizes, or suffered so severely from selection bias, attrition, or other methodological problems, that it was impossible to draw solid conclusions from their findings. These problems were evident in studies conducted by the board itself, some funded by its research grant program, and some conducted independent of the board. The more quan- titatively sophisticated large-scale studies were narrower in scope, focusing solely on one of the issues in which we were interested, student achievement as measured by standardized tests. This left us little evidence with which to answer the questions in our framework. Nevertheless, we scoured the studies for findings that we judged to be valid, given the methodology used; supportable, based on the evidence collected; and reasonable, given the limitations of the study. Impacts on Outcomes for Students The question of how the program is related to student outcomes can be considered in two ways. First, passing the certification process may act as a signal of preexisting teaching effectiveness. Second, the process of becoming board certified may cause a teacherâs classroom effectiveness to improve. These questions related to student outcomes have generated the largest number of research studies, with most focusing on the question of whether board certification acted as a signal of preexisting teaching effectiveness.
OVERALL EVALUATION 253 Nearly all of these studies compare the achievement test scores of students taught by board-certified and nonboard-certified teachers; few compare other student outcomes, such as motivation, student engagement, breadth of achievement, attendance, or promotion. We focused on studies that controlled for school and student variables related to student achievement. As a group, these studies show that the students of board-certified teachers performed better than students taught by nonboard-certified teachers (the magnitude of the differences is on the order of 0.02 to 0.08 of a standard deviation). The studies demonstrate that board certification is a signal that teachers with this credential are more effective than other teachers at raising their studentsâ test scores. Few studies examined the extent to which the certification process caused teachersâ effectiveness to improve, and the findings from these studies were mixed. We note that certification programs are not typically designed to improve the performance of those who apply (i.e., passing a cer- tification test typically does not in and of itself improve performance), and certification programs are not typically evaluated on this issue. However, the impact of the certification process is a relevant issue and it is included in our framework. While the studies examining the effects of board-certified teachers on their studentsâ achievement are generally scientifically sound, there are some caveats to consider. First, much of the research draws on data from two states (Florida and North Carolina) and one district (Los Angeles), and the studies focus primarily on achievement in reading and math for third through fifth graders. We do not know the extent to which these findings can be generalized to other jurisdictions, content areas, and grades. Second, the studies define student learning in a narrow way. Standard- ized tests of student achievement are not designed to assess the sorts of higher order critical thinking skills that teachers following the boardâs content standards would be encouraged to focus on. The NBPTS content standards are based on a view of learning in which the focus is on engag- ing students as active learners. Teachers do this by building on studentsâ experiences and interests and engaging them in activities that are purpose- ful and meaningful. Given the diversity in studentsâ backgrounds, teachers must continually adjust their plans in order to meet studentsâ needs while simultaneously building on their strengths. This kind of teaching demands thoughtful decision making, which depends, in turn, on a teacherâs ability to reflect on his or her practice. This approach to teaching may be very ef- fective and yet not be reflected as higher scores on tests designed to measure basic math and reading skills. On the basis of our review of the impacts on outcomes for students, we make the following recommendations:
254 ASSESSING ACCOMPLISHED TEACHING Recommendation 7-1: To the extent that existing data sets allow, we en- courage replication of studies that investigate the effects of board-certified teachers on student achievement in states besides North Carolina and Florida, in content areas beyond mathematics and reading, and in grades beyond the elementary levels. Researchers pursuing such studies should work with the national board to obtain the information needed to study the effects of teachers who successfully obtained board certification as well as those who were unsuccessful. Recommendation 7-2: We encourage studies of the effects of board-certified teachers on outcomes beyond scores on standardized tests, such as student motivation, breadth of achievement, attendance rates, and promotion rates. The choice of outcome measures should reflect the skills that board-certified teachers are expected to demonstrate. Such research should be conducted using sound methodologies, adequate samples, and appropriate statistical analyses. Impacts on Participating Teachersâ Professional Development The evidence pertaining to this question is scant. Only two studies directly investigate what teachers may learn in the course of the process. While the results suggest that teachers learn from the process, the studies were small in scope and the findings are in need of replication. Several other studies compare the effectiveness of teachers in North Carolina and Florida in terms of their studentsâ reading and mathematics achievement test scores before, during, and after earning board certification. As noted above, the findings from these studies are mixed. Results from surveys and our own discussions with board-certified teachers indicate that teachers are positive about the experience. Teach- ers who successfully completed the process report that it is a profession- ally rewarding experience, and that learning about the boardâs notion of reflective practice alters their approach to instruction. We note, however, that this evidence is both subjective in nature and collected after the fact. There are no studies that collected baseline data about teachers before go- ing through the process, making it impossible to attribute any findings to the process itself. In fact, while several surveys found that the majority of board-certified teacher respondents say that they participate in leadership activities and mentor other teachers, they also found that these teachers participated in these activities prior to earning board certification. Others reported that administrators discouraged board-certified teachers from as- suming responsibilities beyond their primary role of classroom instruction. Furthermore, there are no studies that evaluate the impact of the process on teachers who are unsuccessful.
OVERALL EVALUATION 255 On the basis of this review, we find that there is not sufficient evidence to draw firm conclusions. We make the following recommendations for additional research: Recommendation 8-1: We encourage the NBPTS and other researchers to undertake research to investigate the effects of the process on the candi- dates. The studies should use pretest-posttest and longitudinal designs and should allow for comparison of responses from successful and unsuccessful candidates. Recommendation 8-2: We encourage the NBPTS and other researchers to pursue more mixed-method studies, using both quantitative and qualitative research methods, to examine the effects of board certification on teachersâ practices. These studies should examine a variety of measures of teach- ersâ practices and a variety of student outcomes. Such research should be conducted using sound methodologies, adequate samples, and appropriate statistical analyses. Recommendation 8-3: Researchers should work with the NBPTS to obtain the information needed to study the relationships between board certification and student achievement across the various stages of board certification. These studies should examine the impacts of the certification process on teachersâ effectiveness in increasing their studentsâ test scores and specifically should examine effects for the years subsequent to the receipt of board certification. To the extent that existing data sets allow, we encourage replication of studies in states besides North Carolina and Florida and in subjects beyond elementary reading and mathematics. Impact on Teachersâ Career Paths A significant goal for an advanced-level certification program is to make the teaching field more appealing to the best teachers and encourage them to stay in it. Goals for the national board include helping to profes- sionalize the field; motivating districts and states to raise salaries for ac- complished teachers; motivating districts and states to provide expanded opportunities for leadership in the field; and increasing accomplished teachersâ satisfaction with their careers. Very little information is available to answer questions about this kind of impact. Only one existing study examined teachersâ longevity in the field, and the findings were based on teachersâ responses to a few survey ques- tions. We conducted analyses of data from the Baccalaureate and Beyond data set, but the sample of teachers included in these analyses was small. Nevertheless, the survey findings suggest that board-certified teachers are
256 ASSESSING ACCOMPLISHED TEACHING more likely than teachers in general to indicate that they plan to remain in teaching, and the results from our own analyses indicate that board-certi- fied teachers do actually stay in teaching at higher rates than other teach- ers. However, neither the existing study nor our analyses permit causal inferences; that is, they do not indicate whether the NBPTS process causes teachers to stay in the field longer or whether the teachers who choose to become board-certified are already more likely to remain in the profession, regardless of whether they earn certification. A third study considered whether acquiring board certification in- creases the mobility of teachers within the profession. Data from one state (North Carolina) show that those who successfully obtain board certifica- tion tend to move from one teaching job to another at higher rates than do unsuccessful applicants. These data also indicate that when they move, board-certified teachers are likely to move to teaching assignments with more advantaged conditions, such as schools with higher student achieve- ment levels or fewer poverty-level students. However, it is not clear that this tendency is any more prevalent for board-certified teachers than for other teachers with excellent qualifications. We caution that only tentative conclusions can be based on the limited evidence. The available research and existing databases did not allow us to answer many of our questions about teachersâ career paths. However, we think that there are many ways that data could be collected to address this question. Two viable approaches are through the national board itself and through the School and Staffing Survey managed by the National Center for Education Statistics, which has attempted to obtain such information but needs to revise and clarify its questions on this issue. Our primary finding is that research is needed in this area. Recommendation 9-1: The NBPTS and other researchers should study the subsequent career choices of teachers who have applied for board certifica- tion. The information they collect should be analyzed for successful and unsuccessful candidates separately so the correlation between board certifi- cation and career choice can be evaluated. Studies that track teachers over long periods should also be used to examine whether the process alters career choices. The data collected should include information about the extent to which state or district policies influenced the respondentsâ career choices. Recommendation 9-2: The National Center for Education Statistics should amend the Schools and Staffing Survey so that it collects information about respondentsâ board certification status. In designing the survey questions on this topic, The National Center for Education Statistics should pilot-test alternate versions to ensure that respondents will accurately understand the questions.
OVERALL EVALUATION 257 Recommendation 9-3: Researchers should use the data available from state- level data systems to expand the evidence on the mobility of board-certified teachers. These studies should use methodologies that permit comparisons of teachersâ career choices before and after becoming board certified and should compare the choices of unsuccessful applicants for board certifica- tion, teachers who successfully obtained the credential, and teachers who did not apply for board certification. Impacts on the Education System The Carnegie task force envisioned that the boardâs influence would reach well beyond any impact that individual board-certified teachers might have on their students. However, there is very little basis for conclusions about whether or not the board has had impacts on the education system, such as improved working conditions for all teachers, influence on the practice of nonboard-certified teachers, or changes in teacher preparation or professional development. These sorts of far-reaching effects are difficult to measure or evaluate in any context, in part because they tend to occur very slowly and to involve complex interactions among the elements of the system. The foundation for these kinds of impacts has not yet been established, however. Results from qualitative studies indicate that school systems are not making the best uses of their board-certified teachers. Principals and other school administrators sometimes discourage board-certified teach- ers from assuming responsibilities outside the classroom. Principals worry about showing favoritism toward board-certified teachers and downplay the significance of the credential. Some board-certified teachers report that they conceal their credential so as not to seem to be showing off. These kinds of findings indicate that board certification is simply not widely accepted as a signal of excellence or as an expected way for a teacher to progress professionally. Despite these negative reports, there are isolated cases in which board- certified teachers are rewarded, used effectively, and offered new opportu- nities. In these instances, administrators and other teachers are aware of and respect the board-certification process, and board-certified teachers are used as mentors, team leaders, and organizers of professional development activities. In these situations, board certification appears to be viewed as part of a broader commitment to improving professional development and meeting higher standards for teachers. We think that board-certified teachers are unlikely to have a significant impact without broader endorsements by states, districts, and schools of the NBPTS goals for improving professional development, setting high stan- dard for teachers, and actively using the board-certified teachers in lead-
258 ASSESSING ACCOMPLISHED TEACHING ership roles. Furthermore, we think that the board certification program is unlikely to have broad systemic effects on the field of teaching unless greater numbers of teachers become board certified and the Carnegie task forceâs other recommendationsâfor creating a more effective environment for teaching and learning in schools, increasing the supply of high-qual- ity entrants into the profession, and improving career opportunities for teachersâare implemented. Our review of the evidence led us to draw the following conclusion: Conclusion 10-1: There is not yet sufficient research to evaluate the extent to which the NBPTS is having systemic impacts on the teaching field and the education system. Cost-Effectiveness The final aspect of our evaluation was to examine the cost-effectiveness of the national boardâs certification program as a means of improving teacher quality. Our review revealed that, at present, the research base needed to support a cost-effectiveness evaluation of the NBPTS is inad- equate. Making a rough calculation of the costs of the program is relatively straightforward, but evaluating its benefits presented significant problems because of a lack of data. Advanced-level certification of teachers has the potential to offer three kinds of benefits: (1) it can provide a systematic way of identifying high- quality teachers; (2) the process itself can provide a means for teachers to improve their practices; and (3) it can enhance teaching as a career, keeping better teachers in the field and attracting better teacher candidates in the future. The evidence that we have to date suggests that NBPTS certification does provide a means for identifying highly skilled teachers. However, the existing evidence base does not provide sufficient information to assess the latter two benefits. Simply identifying high-quality teachers provides no direct benefit un- less the signal of quality is used in some way. For example, administrators and policy makers could implement incentives for teachers who are identi- fied as highly skilled, either to encourage them to remain in teaching or to encourage them to work in traditionally difficult-to-staff schools. Board certification could also be used as a way of identifying instructional leaders who could then support other teachers and thus pass on their skills to them. While some of these policies have been implemented (e.g., salary bonuses provided to teachers who obtain board certification, providing financial incentives to teach in schools with high needs students), the policies were not implemented in a way that allows an examination of their impacts. One of the most important benefits that might result from the program, keep-
OVERALL EVALUATION 259 ing high-quality teachers in the profession, could not be evaluated because the necessary data have not been collected. With the exception of isolated instances, there is no evidence that the signal of quality provided by board certification is being used to encourage board-certified teachers to work in difficult schools or to mentor other teachers. Cost-effectiveness estimates are best understood by comparing them to estimates for other similar interventions. Three interventions that could serve as comparison with the cost-effectiveness of NBPTS are: (1) the American Board for Certification of Teacher Excellenceâs current plans for certifying distinguished teachersSM, (2) encouraging teachers to pursue masterâs degrees, and (3) providing inservice professional development. Our cost analysis suggested that the annual per-teacher costs associated with board certification are likely to be lower than the annual per-teacher costs of obtaining a masterâs degree. However, the evidence about the benefits of masterâs degrees is too mixed to be able to derive a cost-effectiveness esti- mate that could be compared with that of board certification. For the other two possible comparisons, even less is known. Thus we conclude: Conclusion 11-2: At this time, it is not possible to conduct a thorough cost-effectiveness evaluation of the NBPTS because of the paucity of data on the benefits of the program and on both the costs and benefits of other mechanisms intended to improve teacher quality. Such an evaluation should be undertaken if and when the necessary evidence becomes available. Concluding Observations The board set out to transform the teaching field in the United States and has been innovative in its approach to this challenge. Its effort to ar- ticulate standards for accomplished teaching brought diverse voices to the table, individuals who had never before sat together and discussed the com- ponents of excellent teachingâincluding policy makers, education research- ers, teacher union leaders, teachers, and others. In this sense, the process through which these standards for accomplished teaching were developed was innovative, as were the standards themselves. The standards captured a complex conception of accomplished teaching and stimulated thinking about what teachers should know and be able to do. The portfolio-based assessment developed to measure teachersâ practice according to these stan- dards pushed the measurement field forward. The board faced significant obstacles. To accomplish its goals, the board needed to alter deeply entrenched norms and views in the teaching field. Several well-established traditions in teaching were directly antitheti- cal to the NBPTS goals. Structural elements of the field, including typical modes of teacher preparation and professional development as well as state
260 ASSESSING ACCOMPLISHED TEACHING and local policiesâand union positionsâregarding hiring, compensation, and tenure, for example, in some ways fit poorly with the approach advo- cated by the NBPTS. Perhaps more important, traditions of egalitarianism and conceptions of instruction as an autonomous and independent activity have been deeply ingrained among teachers. Yet the boardâs goals called for recognition and rewards for teachers who demonstrate their skills at col- laborating with others and make an effort to distinguish themselves from their colleagues by meeting a high standard. Moreover, measuring the outcomes of teaching (e.g., student learning) plays a much more visible role in education policy today than it did when the national board was established, with the result that expectations for reform efforts have been framed in new ways. The Carnegie task force intended that a transformed teaching profession would naturally improve student learning, but they did not envision this improvement solely or even primarily in terms of increases in studentsâ scores on standardized achieve- ment tests (Carnegie Task Force on Teaching as a Profession, 1986). The NBPTS was part of the strategy for addressing the significant disadvantages to students, teachers, and schools brought about by the perceived second- class status of teaching. The design of the national boardâs assessment re- flected a view of teachers as professional practitioners. The desired outcome was framed as producing teachers who: â¢ are committed to students and their learning, â¢ know the subjects they teach and how to teach them, â¢ take responsibility for managing and monitoring student learning, â¢ think systematically about their practice and learn from experience, and â¢ serve as members of learning communities. Over the years since the board began its work, the policy climate has shifted. Increasingly, studentsâ scores on standardized assessments have become the key components of accountability systems with rewards and sanctions for schools and teachers. The current policy focus on concrete measures of accountability, while not inherently at odds with the national boardâs original goal of professionalizing teaching, reflects a markedly dif- ferent conception of what constitutes excellent teaching. The NBPTS has the potential to make a valuable contribution to efforts to improve teacher quality, together with other reforms intended to create a more effective environment for teaching and learning in schools, increase the supply of high-quality entrants into the profession, and improve career opportunities for teachers. Our review of the research, however, suggests that there is not yet compelling evidence that the existence of the certifica- tion program has had a significant impact on the field, teachers, students, or
OVERALL EVALUATION 261 the education system. We note, however, that much of the research needed to evaluate these impacts has not been conducted, in part because the necessary data have not been collected. Moreover, revolutionary changes of the kind the boardâs founders envisioned would be expected to develop over decades, not years. The founders also intended, as we have repeatedly stressed, that the certification program would be supported by an array of other reforms, many of which have not been implemented. This evaluation thus provides an opportunity to take stock of what has worked well and what has not and to consider changes that are needed to respond to the current policy environment. We summarize our findings with the following conclusion: Conclusion 12-1: At its outset, the National Board for Professional Teach- ing Standards was innovative. Its contribution in three areas is particularly noteworthy: (1) its vision and overall plans for professionalizing the teach- ing field, (2) the nature of the assessment it developed and the impact it has had on the measurement field, and (3) the development of standards for 25 certificates and assessments for each. However, participation rates are low and few of the other elements of the Carnegie task forceâs plan, which were to have worked with the certification program, have been carried out. Moreover, for the board to realize its potential, several key changes in its operation and approach are needed. We judged it to be beyond the scope of our evaluation to make policy recommendations regarding the boardâs future, so we present these as suggestions to the board leadership. We think that, if the board is to build on its accomplishments and thrive as a means of improving teacher quality in the United States, it will need to attend to the following: 1. The NBPTS must be sure that it conducts its work according to the highest standards for assessment programs, and that its opera- tions are accessible to external scrutiny. Our review reveals that the board has not devoted the same energy that went into the original assessment design into ongoing evaluation of how that design has worked over time or found ways to modify that design in response to problems, such as low reliabilities on some assessments. 2. To be a trusted institution that can have widespread influence, the NBPTS needs to carefully distinguish between objective research and advocacy. In conducting and reporting on its own research and in presenting the research of others, the board should be careful to adhere to scholarly standards. 3. The NBPTS should pursue an ongoing research agenda to evalu- ate progress toward its goals. The board has ready access to data
262 ASSESSING ACCOMPLISHED TEACHING relevant to some of the questions we pose in this report (e.g., Does board certification increase retention in the field? What effects does the program have on teachers who are unsuccessful in earn- ing board certification?). We encourage the board to work with researchers to undertake studies of these kinds of questions using scientifically sound, methodologically strong procedures. 4. The NBPTS should periodically reviewÂ its assessment model, both to evaluate how it has worked in practice and to adapt to changes in the policy environment and advances in research. As part ofÂ such ongoing evaluation efforts, the board shouldÂ consider whether adjustments are needed in theÂ types ofÂ informationÂ used as the basis for certification, which might include classroom observa- tions, objective tests of content knowledge, or measures of student performance.Â 5. The NBPTS should continue to invest in its larger mission of influ- encing the teaching field in broad, comprehensive ways. The national board has produced a viable program for assessing teach- ers and certifying those who meet its high standards. Data have not been collected to permit evaluation of the extent to which the board has met the other goals it identified, such as creating a more effective environment for teaching and learning in schools, increasing the supply of high-quality en- trants into the profession, and improving teacher education and continuing professional development. However, it is important to point out that these are goals that depend on other actors. The board cannot compel states and districts to encourage teachers to participate, to structure teaching sched- ules to encourage collaboration, or to provide teachers with opportunities to advance professionally as they develop their expertise. The board has no means of influencing teachersâ salaries, which have not been brought in line with those in other fields with comparable demands, nor any means of altering teacher preparation or professional development programs. The board may have done an exemplary job of trying to engage others in these goals, but we had no basis on which to evaluate this aspect of their work and we did not attempt to do so. We do think, however, that is not too late to implement studies to evaluate progress toward these goals. The standards for accomplished teachers have been established, and they were developed to reflect broad consensus regarding what constitutes exemplary teaching. A significant investment has been made in developing the as- sessment program, and it is operational. It is a ready tool at a time when concern about improving teacher quality is among the issues at the top of the education policy agenda. In our opinion, the national board has offered a thoughtful approach to serious problems with the way the U.S. education system selects and pre-
OVERALL EVALUATION 263 pares its teachers and the conditions in which our teachers do their work. Given the magnitude of the problems the board addressed and the lack of systematic data collection systems on such issues as teacher mobility and the career paths of board-certified and nonboard-certified teachers, the lack of evidence of its impact does not necessarily indicate that the board is not having an impact. For the program to have the intended impacts on the teaching field, improvements will be needed, both in the operational aspects of the program and in the evidence collected, as we have recommended throughout this report. The board cannot achieve these goals alone, how- ever. Meeting these ambitious goals will also require a serious commitment by education policy makers to the other recommendations made by the Carnegie Task Force on Teaching as a Profession.