11
The Cost-Effectiveness of Certification as a Means of Improving Teacher Quality

The congressional bill that authorized the National Research Council to conduct this evaluation contained specific language requesting consideration of the extent to which certification by the National Board for Professional Teaching Standards (NBPTS) is a cost-effective method for improving teacher quality. While this is a challenging question to address, we understand why it has been posed. Although the board got its start from private funding, since 1991 it has received considerable federal money to support its work. Given the federal dollars invested in the program, it is reasonable for Congress to ask if the investment has been wise.

To respond to this aspect of the committee’s charge, our evaluation framework includes the following question:

Question 8: To what extent does the advanced-level teacher certification program accomplish its objectives in a cost-effective manner, relative to other approaches intended to improve teacher quality?

As before, we refer to Figure 2-1 for our model of the kinds of impacts that an advanced-level certification program for teachers might have. The question we address in this chapter does not explicitly appear in the model, but we regard it as an overarching question about the net effect of the various impacts of an advanced-level certification program, when considered in the context of the costs of the program. Addressing Question 8 requires us to summarize the benefits of the NBPTS, consider its costs, and compare the resulting cost-effectiveness with that of other interventions designed to



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 223
11 The Cost-Effectiveness of Certification as a Means of Improving Teacher Quality The congressional bill that authorized the National Research Council to conduct this evaluation contained specific language requesting consider- ation of the extent to which certification by the National Board for Profes- sional Teaching Standards (NBPTS) is a cost-effective method for improving teacher quality. While this is a challenging question to address, we under- stand why it has been posed. Although the board got its start from private funding, since 1991 it has received considerable federal money to support its work. Given the federal dollars invested in the program, it is reasonable for Congress to ask if the investment has been wise. To respond to this aspect of the committee’s charge, our evaluation framework includes the following question: Question 8: To what extent does the advanced-level teacher certifi- cation program accomplish its objectives in a cost-effective manner, relative to other approaches intended to improve teacher quality? As before, we refer to Figure 2-1 for our model of the kinds of impacts that an advanced-level certification program for teachers might have. The question we address in this chapter does not explicitly appear in the model, but we regard it as an overarching question about the net effect of the vari- ous impacts of an advanced-level certification program, when considered in the context of the costs of the program. Addressing Question 8 requires us to summarize the benefits of the NBPTS, consider its costs, and compare the resulting cost-effectiveness with that of other interventions designed to 22

OCR for page 223
22 ASSESSING ACCOMPLISHED TEACHING improve teacher quality. We identified the following issues to investigate and to provide evidence about the cost-effectiveness of the national board’s certification program, specifically: a. What are the benefits of the certification program? b. What are the costs associated with the certification program? c. What other approaches have been shown to bring about improve- ment in teacher quality? What are their costs and benefits? It is important to note that the existing research base for such an inquiry is inadequate. The cost side is not the issue. Although there have not been extensive examinations of the costs associated with the NBPTS, a relatively coarse consideration of costs is sufficient for the task at hand. Rather, it is the benefits side of the analysis that is the problem. Further- more, while the evidence about the benefits of the NBPTS is inadequate for a thorough cost-effectiveness evaluation, even less is known about the benefits of other interventions to improve teacher quality. As a result, the kind of cost-effectiveness comparison one would like to perform, and as stated in our charge, is not possible at this time. Despite the inadequacies in the evidence base, we lay out the issues to the extent that available research and data allow. In the sections that follow, we first consider the benefits, the costs, and the resulting cost-effectiveness of board certification as a route to improving teacher quality. We then examine the available information about the cost-effectiveness of four comparison interventions. BENEFITS ASSOCIATED WITH THE PROGRAM Before considering the specific benefits of the NBPTS, we step back to consider the ways in which an intervention intended to improve teacher quality might operate. There are three kinds of benefits such an interven- tion might produce: 1. Identifying highly skilled teachers. 2. Improving the practices of teachers who go through the program. 3. Improving the quality of teachers throughout the education system, keeping accomplished teachers in the field, and attracting stronger teacher candidates in the future. We note here that simply identifying highly skilled teachers provides no direct benefit, and therefore the first benefit requires that some action be taken once highly skilled teachers are identified. For example, administra- tors and policy makers could implement incentives for teachers who are identified as highly skilled, either to encourage them to remain in teaching

OCR for page 223
22 THE COST-EFFECTIVENESS OF CERTIFICATION or to encourage them to work in traditionally difficult-to-staff schools. Teachers identified as highly skilled could also be used as a way of iden- tifying instructional leaders who could then support other teachers and thus pass their skills onto other practitioners. We also point out that this benefit is one that is often claimed by programs that offer advanced-level certification in other fields, such as nursing or medicine: identification of highly skilled practitioners is the first step in realizing the benefits offered by a program that recognizes advanced practice. It is not necessarily a benefit in and of itself, but it serves as the foundation for other potential benefits. Moreover, the actual process of defining advanced practice can make a significant contribution to the field. While the three benefits are interrelated, they differ in critical ways. For example, it is possible for an intervention to produce one of these benefits without providing the other two. In the context of a program like the NBPTS, it is easy to see that the program may produce the benefit of identifying highly skilled teachers without improving the teaching ability of the candidates as they go through the certification process or improving the quality of teachers throughout the education system. This result can occur if the certification process itself does not provide candidates with new skills or if the resulting certification is not used by the system in a way that changes what teachers are taught, who enters and stays in teaching, and who leaves, thus having no impact on overall quality. Other interventions designed to improve teacher quality may focus en- tirely on one of these kinds of benefits and not at all on others. For example, inservice professional development is intended to improve teacher quality directly (Benefit 2) without providing a means for identifying highly skilled teachers (Benefit 1). However, increasing teacher pay is an intervention in- tended to improve teacher quality throughout the education system (Benefit 3) without directly identifying highly skilled teachers (Benefit 1) or directly im- proving the teaching ability of any particular teachers (Benefit 2). Of course, improving the teaching quality of teachers throughout the system (Benefit 3) is presumably the ultimate goal of an intervention fo- cused on teacher quality, and it is reasonable to assume that identifying highly skilled teachers (Benefit 1) or improving the teaching abilities of teachers going through a program (Benefit 2) are just two intermediate routes to achieving that ultimate goal. However, it is important to consider these two intermediate benefits separately, because their mechanisms for influencing teacher quality throughout the system differ. A certification pro- gram that improves the practices of teachers who participate in it (Benefit 2) will directly increase the quality of those teachers who participate, as long as the participants continue to be teachers, which will have a larger system impact to the extent that many teachers participate. However, a program that identifies highly qualified teachers without directly improving their

OCR for page 223
22 ASSESSING ACCOMPLISHED TEACHING teaching practices will require that the certification be used by the education system in some way that produces the benefit of increasing teacher qual- ity throughout the system. In the next sections, we examine the evidence related to each of these benefits. Benefit 1: Identifying Highly Skilled Teachers With respect to the NBPTS program’s ability to identify high-quality teachers, the available evidence shows that the board’s certification program does identify skilled practitioners, whether defined in terms of teachers’ skills or students’ achievement. The content- and construct-based validity evidence, discussed in Chapter 5, indicates that the assessment is measuring the knowledge and skills it is intended to measure, which were judged to represent accomplished teaching. The findings from value-added analyses addressed in Chapter 7 demonstrate that the assessment is identifying high-quality teachers with respect to their effectiveness at raising tested student achievement in mathematics and reading—an important, though incomplete, indicator of teacher success. Most of the investigations described in Chapter 7 report results based on comparisons of board-certified teachers with all other teachers in the system, a comparison that confounds the ability of the assessment to iden- tify high-quality teachers with the particular quality mix of their nonboard- certified colleagues. These investigations show that board-certified teachers produce gains in student achievement that are, on average, about a 0.04 standard deviation larger than their nonboard-certified colleagues (ranging from 0.01 to 0.08 in our analyses of North Carolina and Florida). The size of this difference is roughly one-half to one-fifth of the difference in value- added between teachers in the top and bottom halves of the distribution (see Chapter 7). Thus, while based on a limited conception of student achievement re- flected by standardized test scores, the findings from value-added analyses show that the NBPTS certification process does in fact identify teachers of higher quality. We emphasize that NBPTS certification identifies highly qualified teachers as determined by value-added analyses of standardized test scores, without the certification decision being determined by those test scores themselves. We highlight this point here because this is in contrast to a certification process, like that being considered for the American Board for Certification of Teacher Excellence’s (ABCTE) Distinguished TeacherSM program, in which value-added measures are a component of certifica- tion decisions. In this latter case, it would not be unexpected to find that teachers who earn board certification are more effective at raising their students’ achievement test scores, since that is part of the basis for the cer- tification decision. The fact that teachers who earn NBPTS certification are

OCR for page 223
22 THE COST-EFFECTIVENESS OF CERTIFICATION effective at improving their students’ achievement test scores beyond those who do not earn certification is an important finding, because student test score gains are not considered in awarding certification. These measures, board certification and results from value-added analyses, are independent, imperfect proxies of high-quality teaching, but their overlap provides some evidence that they are both capturing some aspect of this quality. Benefit 2: Improving the Practices of Teachers Who Participate The findings discussed in Chapter 8 show that there is some evidence that teachers’ practices improve after going through the certification pro- gram, although at present this evidence is weak and the results are mixed. Survey results indicate that candidates who go through the process report it to be a valuable experience. Some empirical research shows that candidates, even those who fail, may improve their ability to perform tasks similar to those used for the assessment. Other studies suggest that teachers are less effective at raising students’ test scores while going through the process and, in some cases, may continue to be less effective after earning certifications. In our estimation, the evidence with regard to this benefit is not yet conclu- sive, and we hesitate to draw firm conclusions from the available studies. Existing research is in need of replication in other states with other samples, other criteria, and using a variety of quantitative and qualitative methods. Benefit 3: Improving the Quality of Teachers Throughout the Education System As shown in Figure 2-1, there are several different ways in which the NBPTS certification program could lead to improvements throughout the education system. Specifically, the program could 3a. improve the teaching practices of individual teachers who go through the process, whether or not they pass (a benefit that was also addressed above); 3b. encourage skilled teachers who become board certified to continue practicing longer; 3c. lead to assigning board-certified teachers to leadership roles that allow them to help improve the teaching of nonboard-certified teachers; 3d. improve the sorting of teachers across job assignments by identify- ing highly skilled teachers and targeting incentives to encourage teachers to take positions in difficult schools; 3e. encourage potentially effective teachers to enter the teaching field

OCR for page 223
22 ASSESSING ACCOMPLISHED TEACHING (i.e., because of the recognition and monetary rewards offered by advanced-level certification); and 3f. change the teaching profession via the process and result of defin- ing excellence in teaching. Aside from improving the practices of individual board-certified teach- ers (Benefit 2/3a), these mechanisms for improving teaching quality through- out the education system require that advanced-level certification be used in effective ways that capitalize on the skills recognized by the credential. The evidence summarized in Chapters 6, 9, and 10 suggests that this is not currently being done. It has been hypothesized that board certification—and the recognition and extra pay associated with it—would encourage high-quality teachers to continue for a longer period of time in the classroom (Benefit 3b) and would lead to more effective use of their skills through assignment of board-certi- fied teachers to work in hard-to-staff schools (Benefit 3d). The evidence discussed in Chapter 9 is simply too limited to draw any firm conclusions about this. Although there is the possibility that the presence of board-certified teachers could improve the practices of their nonboard-certified colleagues (Benefit 3c), the studies discussed in Chapter 10 suggest that this is cur- rently not happening. Not only are there few examples of schools in which board-certified teachers are used in a formal way as mentors to improve the practice of their nonboard-certified colleagues, but there is also disturbing (if anecdotal) evidence that in some schools the ethos of equality across the faculty leads board-certified teachers to conceal their status from their colleagues. Furthermore, although it seems possible that the vision of accomplished teaching put forth by the NBPTS has influenced some aspects of the teacher preparation system, it is likely to be impossible to demonstrate how signifi- cant this influence has been. This does not mean that this impact has not occurred only that it is difficult to measure. In considering the impact of the NBPTS on the education system, it is important to distinguish the effect that board certification can have on the system on its own from effects that require other actors in the system to use board certification as a lever for change. If it were established that board certification acted to improve teacher quality directly (Benefit 2/3a), then no help from other actors in the system would be necessary. But mechanisms that require pay increases, recognition, mentoring roles, or tailored teaching assignments (i.e., assigning board-certified teachers to hard-to-staff schools) all require the participation of other actors besides the NBPTS. It is clear that board certification has been used as a policy lever in some states, such as North Carolina, by adopting an institutional orientation toward sup-

OCR for page 223
22 THE COST-EFFECTIVENESS OF CERTIFICATION porting teachers to acquire board certification and by providing financial support for going through the process and financial rewards for teachers who earn the credential. However, there are no examples of states that have systematically used board certification status as a way to identify quali- fied teachers to act as mentors to improve the teaching practices of other teachers in their school or district, and there are no examples of states that have systematically assigned board-certified teachers to work in the more difficult schools. We note that considerable effort went into the process of identifying the standards for the NBPTS assessment. As described in Chapter 3, this effort brought a wide and diverse set of perspectives together, and currently the board’s standards are reflected both in the standards for undergraduate teacher training and in the accreditation standards for schools of educa- tion. However, there have not been any systematic studies to evaluate the impacts of these efforts, particularly in a way that could be used in a cost-effectiveness analysis. As a result, it is important to acknowledge not only that there is limited research related to the benefits of board certifica- tion, but also that there has been inadequate experience with using board certification as a policy lever for improving teacher quality. The following conclusion summarizes our synthesis of the effectiveness of board certifica- tion as a route toward improving teaching quality: Conclusion 11-1: There is evidence from both a psychometric review of the assessment process and analysis of student achievement test results that board certification identifies highly qualified teachers. There is no conclu- sive evidence that teachers improve their practices by going through the certification process, and there is essentially no evidence that certification or the existing recognition and financial incentives awarded to board-certified teachers in some states are sufficient to substantially increase their tenure as teachers. However, the ability of board certification to identify highly qualified teachers suggests that it offers a potential policy lever for increas- ing teaching quality throughout the system if it were used in ways that have not yet been tried on a large-scale systematic basis, such as by using board certification in hiring, promotion, and assignment decisions; systematically using board-certified teachers as mentors or as teacher leaders; or by tar- geting incentives to encourage board-certified teachers to work in the more difficult schools. COSTS ASSOCIATED WITH THE PROGRAM The costs associated with the NBPTS program include the following, which are incurred for each teacher who applies for board certification: (1) the cost of running the assessment program; (2) the time for the candidate

OCR for page 223
20 ASSESSING ACCOMPLISHED TEACHING to prepare the materials and go through the assessment process; (3) the time (or cost) for any mentors or other assistance to candidates as they are going through the process; and (4) the bonuses that are paid to successful candi- dates by the states and local school districts that provide such bonuses. Ideally, one wants to measure all of the above costs, but there are poor or virtually no data on some of these items, such as the costs borne by mentors or other teachers who help the applicant. However, these costs are probably minor compared with the main costs, which include the costs of running the assessment program, the time of the applicant to prepare the materials, and the bonuses for board-certified teachers. In addition, for the purposes of comparing the cost-effectiveness of different mechanisms for improving teacher quality, it is likely that the available data on both the effectiveness and the costs will be fairly coarse for most of the options, so it is likely that a coarse cost analysis for board certification is not only the best we can do but perhaps also all that is needed. Below we provide estimates of each of these costs. Costs of Running the Assessment Program The test fee, which is currently $2,500, provides one estimate of the costs of the NBPTS assessment program per applicant. However, this esti- mate does not account for the full costs of running the program; in 2005, with roughly 12,000 applications and a test fee of $2,300, the test fee gener- ated roughly $28 million out of total income for the NBPTS of $42 million (personal communication, Joseph Aguerrebere, August 7, 2006). Another estimate of the cost of the assessment program per applicant would be to divide the total costs for the organization across all the applicants, with the justification that the assessment program is essentially the organization’s only product. Adding in the institutional costs for maintaining the orga- nization and regularly updating the assessments spreads those fixed costs over all current applicants. This gives the average per-applicant cost for running the assessment program at its current size, but it is important to note that that average cost would be smaller if there were more applicants over whom to spread the institutional costs (or higher if there were fewer applicants). In 2005, using the average per-applicant cost that includes these institutional costs, this would produce an estimate 50 percent higher than the test fee alone. To allow for some uncertainty in the appropriate costs to assign, we use the range of $2,500-$4,000. In some cases the cost of the test fee ($2,500) is covered by the ap- plicants and in other cases it is covered by the state, the local district, or (indirectly) by the federal government. That is, the federal government cur- rently provides funding to states for teacher improvement efforts, and some states draw from this funding to support bonuses for teachers who earn

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION board certification. During the 2005-2006 school year, 36 of the 51 states (including the District of Columbia) provided some sort of fee assistance. Nine of these states covered the full fee for applicants, although some im- pose limitations, such as the number of applicants allowed per year or a requirement that the candidate pass. Ten of the states paid most of the fee ($2,000-2,250), and 16 offered partial assistance of $1,250 or less. Note that the cost of running the assessment process does not include the development costs of the NBPTS, which were roughly $200 million (Hannaway and Bischoff, 2005)1 and covered by a mix of public and pri- vate sources. Since that cost is money already spent, it is inappropriate to include it in an analysis of the ongoing cost-effectiveness of the program as a means for increasing teacher quality. However, in a later section, we discuss the nature of the country’s $200 million investment in the research and development leading to board certification and consider the cost- effectiveness of that investment. Cost to Applicants for Preparing for the Assessment Cohen and Rice (2005) estimate that candidates spend approximately 400 hours during the assessment process—preparing their portfolio and preparing for and taking the assessment center exercises. In addition, they estimate that candidates spend approximately $350 on supplies related to the preparation of their portfolio submission. In most cases, the time cost to the applicants is likely to be largely un- reimbursed so that the applicant bears the cost of preparing her or his own portfolio. However, in some states or districts, candidates can obtain release time for preparing their materials, which shifts the costs from the applicant to the state or local government. Assuming an average salary with benefits of $60,000 and a 1,600-hour work year, the 400-hour time cost for appli- cants to prepare their portfolios would translate into $15,000 of salary and benefits, if the preparation time were fully reimbursed.2 In the 2005-2006 school year, six states offered teachers release time while preparing for the assessment. However, even in these states, the amount of time allotted for assessment preparation ranges from two to five days, which is far below the 1Hannaway and Bischoff (2005) estimate the costs of research and development to be $200 million, which includes both direct and indirect costs. The NBPTS estimates the costs to be somewhat less, approximately $125 million, which excludes some of the costs for outreach, recruitment, support, and other overhead. 2 The average salary without benefits for 2004-2005 was $47,750 in current dollars (http:// nces.ed.gov/programs/digest/d06/). Hess (2004) cites an unreferenced figure of 26 percent of salary for the cost of benefits for teachers and an estimate of 38 weeks of work per year at 45 hours per week (http://www.hoover.org/publications/policyreview/3438676.html).

OCR for page 223
22 ASSESSING ACCOMPLISHED TEACHING average time teachers spend in preparation. Thus, essentially the full cost of the preparation time is currently borne by the applicants themselves. Costs of Mentors and Support Programs The NBPTS estimates that approximately 80 percent of candidates participate in a support program of one kind or another, although the basis for this figure is unclear and the kind of support offered by these programs varies considerably across jurisdictions. In some school systems, support is provided in a relatively informal manner, such as weekly meetings among the candidates with mentorship provided by teachers who are already board certified. These informal support programs are relatively inexpensive. Other school systems have more formal support programs, which can be much more costly. To some extent, the availability of board-certified teachers in the juris- diction influences the kind of support that is offered. In districts in which board certification is encouraged, there tend to be more board-certified teachers, which means there is an available resource of teachers who can both lobby for funding for support programs and offer their services in as- sisting teacher candidates. In districts with fewer board-certified teachers, candidates may be on their own as they assemble their portfolios. In cases in which candidates obtain informal mentoring and support from their colleagues in preparing their materials, it is plausible that the cost of such support is likely to be in the form of unreimbursed time for the mentors. Without having any firm basis for evaluating this cost, we speculate that it might range from 10 to 40 hours. If reimbursed, the above estimates of salary and hours suggest that this mentoring time would cost roughly $400- $1,500, and so we use a cost of $1,000 for informal support. Cohen and Rice (2005) conducted an analysis of the costs of four formal support programs. The services offered by these programs varied widely as did the number of participants served. The authors report that program-related costs per participant were $1,000 (60 participants), $2,600 (100 participants), $5,600 (70 participants), and $11,200 (9 participants). They indicated that some of this variability in costs is explained by the economy of scale realized by the larger programs and some is a function of design. Not all candidates participate in such formal programs of sup- port, and only scant data are available on the prevalence of such programs. We judged the highest cost ($11,200) to be an outlier and not typical of the kinds of support offered throughout the country. Based on the above, we estimate the typical cost of support to be between $1,000 and $5,000, whether formally or informally provided.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION Costs of Salary Bonuses to Board-Certified Teachers The salary bonuses for successful candidates that some states provide come in a variety of forms.3 Sometimes they are expressed as a percentage of the base salary, sometimes as a promotion on the state’s career ladder, and sometimes as a specific amount of money. During the 2005-2006 school year, 36 states (including the District of Columbia) provided such bonuses, and they ranged from a low of $1,000 per year (in five states) to a high of $7,500 per year (in one state). The average bonus generally ranged between $3,000 and $5,000 per year, with a median value of $2,100. Overall Cost per Applicant Table 11-1 summarizes the overall costs associated with applying for board certification. Because the time costs of candidates and mentors may not be reimbursed, they are included in the table as both dollar costs and hours. The costs for a successful applicant in the year of application thus total roughly $20,000, if all costs are converted into dollars. Since roughly half of all candidates are successfully certified and since certification lasts for 10 years, this means that the application costs for every two applicants can be spread over 10 years of teaching by a single board-certified teacher, resulting in a cost of $4,000 per year of certified teaching.4 To obtain the total cost per year of certified teaching, we need to add the per-year cost of the salary bonuses, roughly $3,000-$5,000, to the per-year cost of a suc- cessful application, resulting in a cost of $7,000-$9,000 per year of certified teaching. Roughly half of this cost is paid for by the candidates themselves, on average, mostly in the form of their unreimbursed time in preparing for the assessment. If the cost of the unreimbursed time for candidates is ex- cluded, the costs are roughly $4,000-6,000 per year of certified teaching. 3 It is important to distinguish between the cost to society and the cost to the public. For the cost to society, we include the teachers’ unreimbursed time to prepare but not the costs of the salary incentives that are a benefit to the teacher; for the cost to the public, we exclude the teachers’ unreimbursed time to prepare but include the costs of the salary incentives. As a result, the two different types of costs should be roughly the same. 4 ($20,000 × 2)/10. The calculation in the text understates the eventual pass rate, which is closer to 60-65 percent, and overstates the period of time the board-certified teachers teach, which is unknown but is certainly less than the full 10 years that the NBPTS certificate lasts. For the purposes of the rough cost calculation that we are performing here, we assume that the understated pass rate and the overstated period of teaching will approximately cancel each other out, with the net result that roughly 10 years of certified teaching results from every two applications. In addition, we ignore the effects of time discounting even though the costs and benefits occur over an extended period of time; given the coarseness of the figures that are available about both the benefits and the costs, this additional refinement would not appreciably affect the result.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION It is not clear how a value-added approach would be adapted to include advanced-level certificates in these areas. These two contrasting features of the original ABCTE Distinguished TeacherSM program allow us to lay out some logical implications of this design. With respect to the potential benefits of the program, there are some critical implications. The restriction of the program to the grades and subject areas that are routinely tested would severely limit efforts to use this advanced-level certification to identify high-quality teachers and to use certification as a policy lever to improve teacher quality throughout the system. Given the existing grades and subject areas in which standardized tests are routinely available, the original approach of ABCTE would limit its coverage to less than half of the current teaching pool, depending on the state.8 Not only would this directly limit the ability of the program to identify high-quality teachers, but it would also make it very difficult po- litically to offer incentives because they could not be made available to all teachers. Note that these limitations would apply to any model for teacher certification or selection that relied on value-added approaches. While there are no existing approaches that focus solely on value- added methodologies, we point out that such an approach downplays the importance of teachers’ practices. That is, it is possible for teachers to use practices, such as explicit “teaching to the test,” that may produce increases in test scores but do not represent exemplary practice. At the same time, the original ABCTE Distinguished TeacherSM pro- gram offered a potential advantage on the cost side by substituting a less expensive testing process that would probably have resulted in a lower test fee than for NBPTS certification. However, it is important to consider the implications of this cost reduction in relation to the overall costs of NBPTS certification shown in Table 11-1. The NBPTS cost of processing an assessment ($2,500-$4,000 per applicant) results in a cost per year of certified teacher of $500-$8009 out of the total cost of NBPTS certifica- tion of $5,000-$8,00010 per year of certified teaching. The majority of the costs associated with NBPTS certification result from candidates’ time in preparing the assessment materials and from the salary bonuses offered to successful candidates. The only way to substantially reduce the costs of an advanced certification program is to reduce one of these two costs—but it is difficult to see how ABCTE could effectively reduce either one of those costs and still be a viable program. That is, teachers need a reason to pur- sue board certification, most likely some type of extrinsic incentive, such as 8 See http://nces.ed.gov/programs/digest/d04/tables/dt04_065.asp?referrer=list. × 2)/10; ($4,000 × 2)/10. 9 ($2,500 10 ($2,500 × 2); ($4,000 × 2), since it takes approximately two applicants to get one board- certified teacher.

OCR for page 223
2 ASSESSING ACCOMPLISHED TEACHING the bonuses in place for NBPTS certification. Furthermore, the assessment needs to be of sufficient substance that it serves as a reasonable basis for awarding advanced-level certification; if the assessment is challenging, can- didates will have to spend time preparing for it (although the preparation time may not be as long as the estimated 400 hours required to assemble the NBPTS portfolio). As a result, it seems likely that the total costs of the ABCTE Distinguished TeacherSM program, as originally designed, would not have been substantially lower than that of NBPTS certification. This discussion is also somewhat hypothetical, however, in that the ABCTE has made changes to its original design making it more similar to the NBPTS approach. Recently, the ABCTE Distinguished TeacherSM program has indicated that it is expanding its assessment criteria by exploring the possibility of incorporating ratings of three classroom observations in certification deci- sions as well as a supervisor’s rating. This change could affect both the costs and the benefits of the program. Classroom observations are quite costly to conduct, systematize, and score; thus, this potential change could increase costs of the ABCTE Distinguished TeacherSM program compared with the original vision. However, as noted above, most of the cost involved with certification is not in the cost for processing the assessment itself, so this increase in cost is not likely to be a critical problem. It is also possible that the ABCTE might decide that classroom observation could be used in place of test score gains for subjects and grades in which standardized tests are not available. This is, of course, purely speculative, but if it were the case, it could potentially remove a severe limitation of the ABCTE Distinguished TeacherSM program as a policy lever for improving teacher quality. Note, however, that it would also move the program to becoming more similar to NBPTS certification. Thus, as originally planned, the ABCTE presented a slightly less expen- sive alternative to NBPTS certification, but we are unsure how it would be made available to all teachers. The new version carries the same restriction and is likely to result in costs similar to those of the NBPTS. Obtaining a Master’s Degree Another alternative mechanism for improving teacher quality is to encourage teachers to pursue additional coursework beyond that completed for the undergraduate degree. Most states require graduate coursework to maintain the teaching credential, and some require teachers to earn a graduate degree within a certain amount of time after entering the school system. The salary structure typically considers both experience and gradu- ate coursework in determining a teacher’s pay, with a substantial increase associated with earning a master’s degree. Thus, we can examine the ben-

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION efits associated with earning a master’s degree in relation to the costs as a means for comparing our cost-effectiveness estimate for the NBPTS. Benefits of a Master’s Degree Over the past 30 years, approximately 25 studies have compared achievement test performance for students taught by teachers with and without a master’s degree (for a summary, see Harris and Sass, 2007). The results from these studies are mixed. The majority found that teachers with master’s degrees are not any more effective than teachers with bachelor’s de- grees at improving their students’ achievement, and, in some cases, teachers with master’s degrees were less effective. However, six of these studies did report statistically significant positive effects associated with having a master’s degree (Betts et al., 2003; Dee, 2004; Ferguson and Ladd, 1996; Goldhaber and Brewer, 1997; Monk, 1994; Nye et al., 2004), more so in comparing the effects on math performance than on reading. Only two studies (Betts et al., 2003; Ferguson and Ladd, 1996) reported statistically significant effects for reading. Two studies of teachers and students in North Carolina by Clotfelter, Ladd, and Vigdor (2006, 2007b) permit direct comparison of the effects of board certification and of master’s degrees on student achievement. In their study focused on the elementary grades, Clotfelter et al. (2006) report that teachers with a master’s degree produce achievement test gains similar to or slightly lower than teachers with bachelor’s degrees, while the effects as- sociated with national board certification were between 0.02 and 0.03 of a standard deviation. In their high school study (Clotfelter, Ladd, and Vigdor, 2007b), the authors report that teachers with a master’s degree improve their student’s performance by 0.005 of a standard deviation more than teachers with a bachelor’s degree. Again, this is lower than the effect they report for board certification status, which was 0.051. It is important to recognize that there are some likely explanations for the poor showing of benefits from master’s degrees. One explanation relates to the fact that teachers take their graduate coursework over a period of time, and thus the effects may occur gradually over the period during which the coursework is taken. Attaining the degree is just one step in the course- taking process, and there may not be large effects associated with getting past this final hurdle. Another explanation relates to the characteristics of the group of teachers with bachelor’s degrees that serves as the compari- son group in these analyses. The comparison group may include teachers who are on the path toward obtaining their master’s degree, and their own teaching may be affected by the courses they have taken. These kinds of analyses encounter the same sorts of confounds that we described in Chap- ter 7 for comparisons of board-certified and nonboard-certified teachers.

OCR for page 223
20 ASSESSING ACCOMPLISHED TEACHING A final explanation relates to the nature of master’s degree programs. There are as many conceptions of master’s degree programs for teachers as there are institutions of higher education in this country. A master’s degree may be awarded after completion of a comprehensive course of study at an accredited institution. Graduate programs are also available through alternate and less formal mechanisms, such as online courses, courses of- fered through the school system, and continuing education credit awarded for participation in various professional development activities. Thus, when a teacher obtains a master’s degree, it is not clear what body of knowledge or set of skills has been acquired. In contrast, while there are at least 50 different implementation models for the NBPTS program, the assessments are standardized. That is, all teachers pursuing a given certificate must demonstrate mastery of the same knowledge, content, and dispositions. Obtaining the title of board-certified teacher generally means the same thing regardless of the jurisdiction in which the teacher is employed. This stands in stark contrast to what is signified by obtaining a master’s degree. Costs of a Master’s Degree To summarize the cost of a master’s degree, we use cost categories that are similar to three of the categories for board certification, omitting the cost of mentoring. In place of the cost of the assessment process, there is the cost of providing the graduate program. This is the tuition for graduate school combined with the subsidy to tuition provided by public and pri- vate endowment support. The full cost of study is relatively similar across public and private universities and is roughly $40,000 per year. Thus for a one- or two-year master’s degree, the cost is $40,000 or $80,000 (Knapp et al., 1990, in Cohen and Rice, 2005).11 It is likely, however, that this is an overestimate of the actual annual costs because teachers typically pursue graduate degrees on a part-time basis, attending courses in the evenings and during the summer. The second cost category is the cost of the time for the candidate, in this case the candidate for a master’s degree. Here the required time might be estimated as 30 weeks of full-time study for each year of the master’s degree, corresponding to roughly 1,200 hours for the candidate. Given a median cost of a teacher’s annual salary of approximately $48,000 (http:// nces.ed.gov/programs/digest/d06/) and benefits at approximately 26 percent for 1,600 hours of work (Hess, 2004), this translates into a cost of roughly 11 Cohen and Rice (2005) estimate that the cost of a full-time masters program is $71,000 over two years of time. This estimate is in 2003 dollars. When converted to 2008 dollars, the cost increases to $83,500.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION $45,000 per year of graduate study.12 The third cost category is the annual salary bonus for teachers with a master’s degree. Nationally, this bonus is roughly $7,000 per year.13 The up-front cost of obtaining a master’s degree is $85,000 per year. If we assume that the degree takes an average 1.5 years to obtain and that teachers teach 25 years after obtaining it, then the resulting cost per year for obtaining the degree is roughly $5,000 if the time for teachers to study is reimbursed, $2,500 if it is not.14 Adding to this the cost of the average annual salary bonus, the result is a rough estimate of $12,000 per year of teaching for a teacher with a master’s degree if the time for teachers to study is reimbursed, $9,500 of it is not. For master’s degrees, the evidence about benefits is too mixed to allow us to produce a cost-effectiveness figure that can be compared with the $1,000-$2,250 cost per 0.01 standard deviation increase in student achieve- ment that was based on our hypothetical example of requiring all teachers to become board certified. The cost per class-year of a teacher with a mas- ter’s degree is roughly the same as the cost per class-year of a teacher who is board certified. As noted in the previous section, however, the available evidence does not consistently indicate that teachers with master’s degrees are more effective than teachers without them. Inservice Professional Development of Practicing Teachers Inservice professional development programs are generally intended to help teachers implement new curriculum, new pedagogy, new procedures, and conceivably a combination of all three. For example, if a district were putting in place a new middle school math curriculum with an online for- mative assessment component, teachers are likely to need to learn the new curriculum, learn and practice the teaching methods it required, and learn how to manage the assessment system. This sort of professional develop- ment is different from the NBPTS process, which might be considered as a point-in-time demonstration of professional competence. In contrast, most inservice professional development has an instrumental purpose. As described in the benefit section above, inservice training offers a means for improving the quality of all teaching without going through the process of identifying highly skilled practitioners. + ($48,000 × .26)] × (1,200/1,600). 12 [$48,000 13 Thiscost is estimated as the median difference, controlling for experience level, between the average salary for teachers with a BA degree and the average salary for teachers with an MA degree. Salaries for teachers with a BA degree can be found at: http://nces.ed.gov/programs/ digest/d07/tables/dt07_073.asp?referrer=list. Salaries for teachers with an MA degree can be found at: http://nces.ed.gov/programs/digest/d07/tables/dt07_074.asp?referrer=list. 14 ($85,000 × 1.5)/25 years if time is reimbursed, or ($40,000 × 1.5)/25 years, if it is not.

OCR for page 223
22 ASSESSING ACCOMPLISHED TEACHING However, there is little research that provides the kind of information needed for an evaluation of the cost-effectiveness of inservice professional development. This is not to say that there is no research on inservice train- ing for teachers, only that it has not produced the information needed for a cost-effectiveness analysis that can be compared with the cost-effectiveness of NBPTS certification. A common way to evaluate the benefits of profes- sional development is in terms of participants’ satisfaction or other quali- tative measures. We were able to locate only three studies that evaluated professional development in terms of quantitative measures: a recent study by Harris and Sass (2007) that used Florida’s state data about inservice training in value-added analyses, as well as two earlier studies (Angrist and Lavy, 2001; Jacob and Lefgren, 2004). The results from all three studies suggest there are only weak value-added benefits (i.e., the strongest effect in Harris and Sass analysis was 0.001 standard deviation, which was sta- tistically significant) associated with teachers’ professional development activities. Similarly, there is little research that documents the costs of inservice professional development in a way that can be used in a cost-effectiveness evaluation. Currently, the federal government provides funding to school districts that can be allocated for professional development. The funding is provided through the No Child Left Behind legislation, which stipulates that 15 percent of any district’s Title I allocation be directly invested in teacher professional development. In addition, districts receive federal Title II grants that are specifically targeted for professional development. Because the federal portion of a district’s budget is not likely to exceed 5 to 6 percent of the total, the likelihood is that the federal contribution for professional development would be less than 1 percent of a district’s operating budget. And in many cases a substantial portion of those funds could be directed to salaries for those who provide professional development. It is difficult, however, to directly connect this funding to specific inservice training activi- ties in a way that would be useful for a cost-effectiveness evaluation. Cost estimates of other types of professional development activities are also provided in Cohen and Rice (2005). They evaluated nine other profes- sional development programs. Their findings indicated that per-participant costs (in 2003 dollars) ranged from a low of $1,438 for the Connecticut Beginning Educator Support and Training program to a high of $14,000 for the Leadership Institute in St. Paul, Minnesota, with a median value of roughly $3,100. The studies that Cohen and Rice utilized are fairly dated, ranging from 1994 to 2001, and they provide no estimates of benefits. Given the state of the literature base at the current time, we are unable to derive a cost-effective estimate associated with inservice professional train- ing for teachers.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION Evaluations of the Effects of Class-Size Reduction A considerable amount of research has focused on the costs and ben- efits associated with class-size reduction policies, and we draw on those estimates to compare with the effectiveness of the NBPTS. The two inter- ventions are, of course, not directly comparable. Class-size reduction in and of itself provides a means to produce improvement in student learning. National board certification provides a means to identify the more effective teachers, but, as described earlier, additional actions are required to realize the benefits. Nevertheless, the comparison can be instructive when consider- ing situations in which policy makers must decide how to allocate funding. The funding could be used to hire additional (presumably effective) teachers so that students can be assigned to smaller classes. Alternatively, the fund- ing could be used to encourage teachers to pursue board certification so that the more effective teachers are identified and their skills used. A comparison can help to understand which is the better investment of funds. There have been several attempts to implement class-size reduction policies. In 1985, the state of Tennessee initiated a policy targeted at reduc- ing the student-teacher ratio in classes. The state implemented the policy as an experiment designed to examine the effects on achievement of assigning students to classes with smaller numbers of students. Students entering kin- dergarten were assigned at random to either a small class (13-17 students) or a regular class (22-26 students). Teachers were randomly assigned to the classes. Students remained in their original experimental assignment (small versus large class) over the course of four years (K-3), with a follow-up data collection in seventh grade (Finn and Achilles, 1999). The results from this experiment indicated that students in the smaller classes performed better than those in the larger classes, with effect sizes ranging from 0.15 to 0.25. Given an estimated average annual cost of $60,000 per teacher (for salary plus benefits), this is roughly a $30,00015 cost per year for a lower class size (e.g., reducing the class size from 24 to 16 students) for a benefit of roughly 0.20 standard deviation. This converts to a cost of $1,50016 per 0.01 standard deviation increase per class per year. This can be compared with our estimates of the effectiveness of na- tional board certification (under the hypothetical situation in which all teachers are required to become board certified). Superficially, the two estimates are approximately equal: NBPTS certification results in a smaller effect (0.04 versus 0.20) but also at a lower cost ($2,000-$8,000 versus 15 With teacher pay at $60,000, a class size of 24 costs $2,500 per child. A class size of 16 costs $3,750 per child. The difference ($1,250) is the additional cost per student of class size reduction. The total cost is then $1,250 × 24. 16 $30,000/(0.20/0.01).

OCR for page 223
2 ASSESSING ACCOMPLISHED TEACHING $30,000 per classroom). The resulting cost-effectiveness is similar for the two interventions: $500-$2,00017 for mandatory NBPTS certification ver- sus $1,500 for class-size reduction per 0.01 standard deviation increase per year per class.18 We caution, however, that the two interventions are not directly comparable because class-size reduction actually caused the increase, whereas the NBPTS example is hypothetical and would require additional policy actions to produce such an increase. The effectiveness estimates from Tennessee’s controlled experiment ultimately helped persuade a number of policy makers to adopt class-size reduction as an educational intervention in other states. Continuing studies of the effects of class-size reduction have led to substantial revisions in the estimates of the effectiveness of this intervention when implemented as a statewide policy. For example, results from the study of class-size reduction policy in California revealed that the effects were statistically significant but quite small, in the range of 0.05 to 0.08 of a standard deviation (Bohrnstedt and Stecher, 2002; Stecher, Bohrnstedt, Kirst, McRobbie, and Williams, 2001). In part, this was a consequence of the limited supply of experienced, high-quality teachers that were required to implement the policy. Evaluating the Research and Development Investment in the NBPTS Approximately $200 million (see footnote 2) was spent on the research and development that went into developing the assessments that now consti- tute the NBPTS certification program. Eleven private organizations funded the board during its developmental years. The Carnegie Foundation was the largest supporter, providing approximately $1 million per year for 11 years. Other funding sources included the Ford Foundation, the DeWitt Wallace- Reader’s Digest Fund, the Lilly Endowment, the Pew Charitable Trusts, the Atlantic Philanthropies, Xerox, IBM, DuPont, AT&T, and Chrysler (Han- naway and Bischoff, 2005). The board began receiving federal support in 1991, under the first President Bush, which increased substantially under President Clinton. According to Hannaway and Bischoff, between 1987 and 2002, the board received over $100 million in federal funding and an equivalent amount from other sources. In evaluating this research and development investment of $200 million (see foonote 2), it is useful to provide some other figures related to teach- ing to put the investment in perspective. As noted in Chapter 6, there are roughly 4 million teachers in grades K-12, of whom approximately 64,000 have become board certified. The annual salary and benefits for these 17 ($2,000/4); ($8,000/4). 18 These costs for class-size reduction do not include the facility costs of providing additional classrooms.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION teachers total roughly $24019 billion each year. When administrative and other costs for K-12 education are added to the cost of teachers’ salaries, the total cost of K-12 education is slightly more than $500 billion per year (http://nces.ed.gov/programs/digest/d04/tables/dt04_029.asp?referrer=list). Compared with the annual cost of K-12 education, the $200 million (see foonote 2) investment in the NBPTS is small, representing less than 0.04 percent of the cost of K-12 education for a single year. Considering the larger goal of NBPTS, which was to transform the profession of teach- ing by articulating a conception of accomplished teaching and developing an assessment for identifying accomplished teaching, the $200 million (see foonote 2) cost of the investment does not seem like a high price to pay. To date the NBPTS has not been able to bring about the transformation in teaching that was hoped for two decades ago, although it has produced an innovative certification process that does in fact identify accomplished teachers. However, in evaluating research and development costs, it is im- portant to remember that any single research and development effort is a calculated gamble with an uncertain chance of success. It is inappropriate to evaluate the initial investment decision with the knowledge available now that the NBPTS has not brought about the transformation it hoped to achieve. Instead, we have to look at the decision that was made at the time when no one knew whether or not the NBPTS would be successful in transforming the profession of teaching. If we consider that choice, it is probably fair to say that from the beginning the likelihood was probably low that the project would have truly been able to transform the profes- sion of teaching. At the same time, however, the educational payoff, if the project had achieved the unlikely result of transforming teaching, could have been very high indeed. And the gamble in the NBPTS was not under- taken lightly but represented a bold and serious effort by many leaders in education research and policy. For a serious research and development gamble that had a chance to transform teaching, even with a low prob- ability, it does not seem excessive to have invested 0.1 percent of the cost of K-12 education for a single year. It is also useful to compare the $200 million (see foonote 2) invest- ment in the NBPTS with the size of the annual research and development investment in K-12 education. Although the cost of K-12 education is quite high, the level of investment in research and development for K-12 educa- tion is quite low, relative to the rate of research and development spending in many other sectors of the economy. The annual investment in research and development for K-12 education is on the order of $1.3 billion per in salary and benefits × 4 million teachers. 19 $60,000

OCR for page 223
2 ASSESSING ACCOMPLISHED TEACHING year.20 Thus, although the investment in the NBPTS is small compared with the overall annual cost of K-12 education, it is large compared with the annual investment in research and development for K-12 education. Still, if the investment in research and development investment for the NBPTS represented 15 percent of all research and development for K-12 education for a single year—that does not seem a high price to pay for a program that was a serious effort to bring about a transformation in teaching quality. For comparison, the Gates Foundation has invested roughly $650 million in its small high schools program (Hendrie, 2004). The above calculation considers the costs that the federal government and private foundations invested in the NBPTS to get the program up and running. A more expansive consideration of costs might treat the entire program as an experiment and count the costs of certifying roughly 64,000 teachers as part of the ultimate cost of conducting that experiment. Earlier we saw that the cost per year of certified teaching is roughly $6,000, re- sulting in a total cost of $300 million over the 64,000 teachers. This cost clearly dominates the initial $200 million investment itself (see footnote 2). However, if the entire $500 million cost is seen as representing nothing more than a serious research and development gamble to bring about a transformation in teaching quality, it is probably fair to say that the gamble was still a reasonable one to take. It is important to note that this discussion about the research and de- velopment investment in the NBPTS has been stated from the perspective of the original decision 20 years ago to make a serious commitment to developing this particular vision of transforming teaching quality for K-12 education. Evaluating whether or not that initial decision two decades ago was an appropriate investment in research and development is entirely dif- ferent than evaluating the current decision about whether or not to continue to invest in the program. The decision about whether to continue to invest in the NBPTS going forward must rest on evidence that its continuing pres- ence is helping to improve K-12 education. CONCLUSIONS In this chapter, we have attempted to address the final aspect of our charge, which was significantly complicated by the limited evidence base. There was insufficient evidence on which to base a thorough cost- 20 This figure is the sum of the 2008 budget allocations for the Department of Education’s Institute of Education Sciences ($546.1 million) and for the National Science Foundation’s Education and Human Resources ($725.5). Details about the Department of Education’s bud- get can be found at: http://www.ed.gov/about/overview/budget/budget09/summary/appendix4. pdf. Details about the National Science Foundation’s budget can be found at: http://www.nsf. gov/about/budget/.

OCR for page 223
2 THE COST-EFFECTIVENESS OF CERTIFICATION effectiveness evaluation of the NBPTS, primarily due to a lack of research documenting benefits. We note that it was not necessarily the case that there is evidence of no benefits, but that the evidence base is simply too thin, and findings from research that has been conducted are in need of corroboration. For example, research on the effects of board certification on teachers’ longevity in the field and studies of the extent to which the certification process improves their effectiveness (as described in Chapters 8 and 9) have the potential to yield estimates of benefits that could be used in future cost-effectiveness evaluations. It was also not possible to compare the effectiveness of the NBPTS with other mechanisms for improving teacher quality—such as alternative kinds of advanced-certification programs for teachers, encouraging teachers to pursue master’s degrees, and providing inservice professional development—because of a lack of information on both the costs and benefits of these activities. While our cost analysis sug- gests that the annual per-teacher costs associated with board certification are probably lower than annual per-teacher costs of obtaining a master’s degree, a sufficient number of rigorous studies was not available to allow us to compare the benefits of these two interventions in a meaningful way. Thus, we conclude: Conclusion 11-2: At this time, it is not possible to conduct a thorough cost-effectiveness evaluation of the NBPTS because of the paucity of data on the benefits of the program and on both the costs and benefits of other mechanisms intended to improve teacher quality. Such an evaluation should be undertaken if and when the necessary evidence becomes available. Because of the lack of evidence for a thorough cost-effectiveness evalu- ation, we undertook a somewhat speculative approach to considering the cost-effectiveness of the NBPTS. We laid out three kinds of potential ben- efits. To date, the existing research provides evidence of only one of these benefits: identification of high-quality teachers. We pointed out that this benefit cannot be realized without some additional action that makes use of the skills of board-certified teachers, and we explored the hypothetical example of requiring all experienced teachers to become board certified. While this is a policy that has not yet been tried, tested, or debated, we think it is worth considering, possibly on a localized basis for all teach- ers or teachers in some schools. Given the substantial investment that has already been made in the NBPTS certification program, it is important to consider not only the cost-effectiveness of NBPTS certification as a realized mechanism for improving teacher quality, but also its potential if it were to be used more actively by states as a policy lever for improving teacher quality throughout the education system.