Appendix A
Reviews of Studies That Provided Evidence for the Evaluation

This appendix provides additional details about the studies that provided the bulk of the evidence for our evaluation. Box A-1 presents the primary questions on our evaluation framework, and for each study, we describe the specific relevance to the framework as well as the general purpose, the participants, and the findings. Also, for each study we provide a comment section that highlights the primary contributions of the study as well as any concerns the committee had about the methodology that affected the weight we placed on the findings. We hope that these comments will assist researchers with future investigations intended to build on this body of research.


Barfield, S.C., and McEnany, J. (2004). Montana’s national board-certified teachers’ views of the certification process. Unpublished article, Montana State University-Billings.


Relevance to evaluation framework: Question 3 (Chapter 6).

Purpose: The authors sought to determine why more teachers in Montana had not pursued board certification.

Subjects/participants: National board-certified teachers (NBCTs) in Montana.

Methodology and findings: Barfield and McEnany queried NBCTs in Montana about their certification experience. In spring 2003, the authors distributed surveys to the 31 NBCTs in the state and received responses from 22 (71 percent response rate). The survey instrument was adopted



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 275
Appendix A Reviews of Studies That Provided Evidence for the Evaluation This appendix provides additional details about the studies that pro- vided the bulk of the evidence for our evaluation. Box A-1 presents the primary questions on our evaluation framework, and for each study, we describe the specific relevance to the framework as well as the general pur- pose, the participants, and the findings. Also, for each study we provide a comment section that highlights the primary contributions of the study as well as any concerns the committee had about the methodology that af- fected the weight we placed on the findings. We hope that these comments will assist researchers with future investigations intended to build on this body of research. Barfield, S.C., and McEnany, J. (2004). Montana’s national board-certified teachers’ views of the certification process. Unpublished article, Montana State University-Billings. Relevance to evaluation framework: Question 3 (Chapter 6). Purpose: The authors sought to determine why more teachers in Mon- tana had not pursued board certification. Subjects/participants: National board-certified teachers (NBCTs) in Montana. Methodology and findings: Barfield and McEnany queried NBCTs in Montana about their certification experience. In spring 2003, the authors distributed surveys to the 31 NBCTs in the state and received responses from 22 (71 percent response rate). The survey instrument was adopted 275

OCR for page 275
2 ASSESSING ACCOMPLISHED TEACHING BOX A-1 The Committee’s Evaluation Framework Question 1: To what extent does the certification program for accomplished teach- ers clearly and accurately specify advanced teaching practices and the character- istics of teachers (the knowledge, skills, dispositions, and judgments) that enable them to carry out advanced practice? Does it do so in a manner that supports the development of a well-aligned test? Question 2: To what extent do the assessments associated with the certification program for accomplished teachers reliably measure the specified knowledge, skills, dispositions, and judgments of certification candidates, and support valid interpretations of the results? To what extent are the performance standards for the assessments and the process for setting them justifiable and reasonable? Question 3: To what extent do teachers participate in the program? Question 4: To what extent does the advanced-level certification program iden- tify teachers who are effective at producing positive student outcomes, such as learning, motivation, school engagement, breadth of achievement, educational attainment, attendance rates, and grade promotion? Question 5: To what extent do teachers improve their practices and the out- comes of their students by virtue of going through the advanced-level certification process? Question 6: To what extent and in what ways are the career paths of both success- ful and unsuccessful candidates affected by their participation in the program? Question 7: Beyond its effects on candidates, to what extent and in what ways does the certification program have an impact on the field of teaching, the educa- tion system, or both? Question 8: To what extent does the advanced-level certification program ac- complish its objectives in a cost-effective manner, relative to other approaches intended to improve teacher quality? from one used by the National Board for Professional Teaching Standards (NBPTS) in their survey of candidates in fall 2001 (National Board for Professional Teaching Standards, 2001e). The authors sought to determine why more teachers in Montana had not pursued board certification and thus added questions to the instrument to ask the respondents if they knew other teachers who were interested in earning board certification but had not yet done so.

OCR for page 275
2 APPENDIX A Reasons for not pursuing board certification included the time com- mitment, the cost, the support required to complete the process (e.g., the videotaping and portfolio assembly), and lack of administrator support. The respondents also said teachers were concerned about the consequences. For example, they said some teachers thought it was very public and too risky, some were fearful of not being successful, and some had observed harassment of teachers who do become certified. Comments: This study is one of the few that address nonparticipation in the program, and the findings are useful in that regard. However, the sample for this study was very small, and the participants may not have been the most appropriate to query about the questions in which the in- vestigators were most interested. That is, it would have been better to ask nonparticipants why they had not pursued board certification instead of asking NBCTs to speculate about their nonparticipating colleagues. This provides a first step in learning about reasons for not participating, and the findings could serve as a basis for future studies with nonparticipants. Belden, N. (2002). California teachers’ perceptions of national board certi- fication. Santa Cruz, CA: Center for the Future of Teaching and Learning. Relevance to evaluation framework: Question 3 (Chapter 6); Question 6 (Chapter 8); Question 7 (Chapter 10). Purpose: The author sought to gather information about teachers’ rea- sons for pursuing board certification. Subjects/participants: NBCTs in California. Methodology and findings: Beldon surveyed all NBCTs in California (n = 785) in summer 2001 and received responses from 519 (68 percent response rate). A focus group discussion was also held in June 2001 in Sacramento; comments from this activity are incorporated into the paper. The survey asked NBCTs about their motivations for pursuing certification and the effects of the process on them and their teaching. It also gathered information about the type of school in which the NBCTs work. Most respondents said that they pursued certification because it was a personal challenge (84 percent) and provided an opportunity to strengthen their teaching (79 percent). Between 54 and 59 percent reported that they pursued certification to receive the state’s monetary compensations. The opportunity for career advancement was also important to more than half (53 percent), as was the prospect of receiving recognition of one’s teaching qualities (50 percent). Comments: Strengths of this study are its large sample size and response rate, relative to other studies of this nature, as well as the use of focus groups to follow up the survey results. An issue that should be considered in in- terpreting the findings is that all survey questions are all worded positively;

OCR for page 275
2 ASSESSING ACCOMPLISHED TEACHING there is no opportunity for a respondent to say the process did not have an effect on his or her teaching. The only exception is the question: “Do you feel that the certification process made you a much better teacher, somewhat better, or did not impact your practice?” The inclusion of both positively and negatively worded questions would have increased the objectivity of the survey and would have helped the researchers to detect problems with response sets (i.e., the tendency for respondents to respond in a given way or to select what he or she regards as an acceptable response). Bond, L., Smith, T., Baker, W.K., and Hattie, J.A. (2000, September). The certification system of the National Board for Professional Teaching Stan- dards: A construct and consequential validity study. Greensboro: University of North Carolina, Center for Educational Research and Evaluation. Relevance to evaluation framework: Questions 1 and 2 (Chapter 5). Purpose: This study was a validity investigation that sought to evalu- ate the extent to which teachers who achieved board certification exhibited the assessed knowledge, dispositions, skills, and judgments as part of their actual classroom practices. Subjects/participants: Participants were first-time candidates who had attempted certification in one of two areas: early adolescence English lan- guage arts and middle childhood generalist. The sample included 65 teach- ers working in Delaware, Maryland, North Carolina, Ohio, or Virginia, 31 NBCTs and 34 non-NBCTs. Methodology and findings: The authors used a two-pronged approach to investigate the validity of the NBPTS. They first attempted to validate the qualities assessed on the NBPTS assessments through a literature review. They reviewed the literature and tried to identify the dimensions of accom- plished teaching. Then they observed teachers who had participated in the NBPTS assessment and rated them on the identified dimensions. Through their literature review, they identified 15 dimensions: (1) use of knowledge; (2) identifying essential representations: deep representations; (3) identifying essential representations: problem solving; (4) setting goals for di- verse learners: improvisation; (5) setting goals for diverse learners: challenge of objectives; (6) guiding learning through classroom interactions: classroom climate; (7) guiding learning through classroom interactions: multidimen- sional perception; (8) guiding learning through classroom interactions: sen- sitivity to context; (9) monitoring learning and providing feedback; (10) monitoring learning and providing feedback: test hypotheses; (11) respect for students; (12) passion for teaching and learning; (13) motivation and self-ef- ficacy; (14) outcomes of lessons: surface and deep; and (15) outcomes of les- sons: achievement. They then developed protocols for evaluating each of the dimensions. Dimensions 1 through 13 were evaluated by observing teachers

OCR for page 275
2 APPENDIX A in their classrooms; Dimensions 14 and 15 involved review of student work and achievement. To identify the sample, the authors used performance results for teach- ers who had taken the NBPTS assessments. Using the NBPTS score data, they grouped teachers as follows: (1) total score at least 1.25 standard deviations (SDs) below the cut score; (2) total score between 0.25 and 0.75 SDs below the cut score; (3) total score between 0.25 and 0.75 SDs above the cut score; and (4) total score at least 1.25 SDs above the cut score. They used this strategy to maximize the possibility of detecting differences among the groups. The list of teachers who fell into each group was randomized, and teachers were recruited until a sufficient number was obtained for the particular group. Ultimately, between 15 and 17 candidates were recruited for each group. Teachers’ performance with regard to these dimensions was evalu- ated through a variety of mechanisms, including classroom observations, reviews of teacher assignments and students’ work, interviews with stu- dents, student questionnaires that asked about classroom environment and climate and evaluated students’ motivation and self-efficacy, and students’ performance on a writing assessment. With regard to Dimensions 1 through 13, the results from classroom observations revealed that NBCTs scored higher on all of these dimen- sions than did the non-NBCTs; the differences were statistically significant (p < .05) on 11 of the 13 dimensions. Analyses of student work indicated that 74 percent of the work sam- ples of students taught by NBCTs reflected deep understanding, while 29 percent of the work samples of non-NBCTs were judged to reflect deep understanding. On the writing tasks, the mean was slightly higher for stu- dents taught by NBCTs than by non-NBCTs, but the differences were not significant (p > .05). Differences between NBCTs and non-NBCTs were negligible with re- gard to student motivation and self-efficacy levels. The authors also com- pared teachers on their participation in professional activities, including (1) collaborative activities with other professionals to improve the effectiveness of the school and (2) to engage parents and others in the community in the education of young people. Again, differences between NBCTs and non- NBCTs were negligible. Comments: This is a comprehensive study that examines construct- based validity evidence for the assessments for two NBPTS certificates. It draws from the literature on effective teaching to develop protocols for evaluating teachers and then compares teachers’ ratings on these protocols with their performance on the NBPTS. Studies of this nature are challenging to carry out. One issue with this study is that the authors’ description of the sampling procedures is somewhat vague. The report indicates that they

OCR for page 275
20 ASSESSING ACCOMPLISHED TEACHING recruited teachers via phone calls, and they provide the verbatim protocol for recruiting teachers, but they do not provide any details about this pro- cess. They do not specify how many calls were required in order to obtain the necessary numbers of participants for each of the four NBPTS score groups or whether it was more difficult to fill any of the groups. It would have been useful to know how representative the participating samples of teachers were of the full set of individuals identified for each score group. Cantrell, S., Fullerton, J., Kane, T.J., and Staiger, D.O. (2007, April 16). National board certification and teacher effectiveness: Evidence from a random assignment experiment. Unpublished paper. A paper developed under a grant from the Spencer Foundation and the U.S. Department of Education. Available: http://harrisschool.uchicago.edu/programs/beyond/ workshops/ppepapers/fall07-kane.pdf [accessed May 2008]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The authors examined the relationships between board certi- fication and student achievement. Subjects/participants: NBPTS applicants and non-NBPTS applicants teaching grades 3-5 in the Los Angeles Unified School District during the 2003-2004 and 2004-2005 school years. Methodology and findings: In this study, the authors were able to implement random assignment of classrooms to teachers. To accomplish this, NBPTS provided them with a list of applicants for board certification. Each applicant in the sample was matched with a nonapplicant comparison teacher in the same school and grade (comparison teachers had at least three years experience); 99 pairs of teachers participated. Classrooms were randomly assigned to teachers in this “experimental sample” (although students were not randomly assigned to classrooms). Another “nonexperi- mental sample” of NBPTS applicants and nonapplicants was also identified to allow the researchers to study the effects of random and nonrandom assignment. Analyses used the covariate model (lagged achievement test score) and the gain score model, as well as a set of student characteristics, classroom peer characteristics, and fixed effects for school by grade by administrative track by year. The NBPTS certification status variables included passed, failed, or withdrawn. The researchers obtained assessment results for the teacher applicants, which included both the pass/fail score and the numeric scores on each of the components of the assessment. They examined the extent to which different weightings of the component scores altered the relationship between certification status and students’ test score gains. The authors found that teachers who applied for board certification but were unsuccessful were less effective than nonapplicant teachers. The

OCR for page 275
2 APPENDIX A coefficients for the unsuccessful group were always negative—specifically, with the covariate model, –.17 for math and –.13 for reading; and with the gain score model, –.36 for math and –.21 for reading. Comparison of the coefficients for teachers who achieved board certification and teach- ers who were unsuccessful revealed that the differences were statistically significant (or approached significance with p = .05). They summarize this finding saying that the board-certified teachers outperformed the unsuccess- ful applicants by 0.2 standard deviations in math and language arts. No statistically significant differences were found in comparisons of NBCTs and nonapplicants. The results for the nonexperimental group demonstrated the same pat- terns, but the effect sizes were smaller. Comments: This study makes a significant contribution in its use of random assignment of students to teachers, which helps to control for preexisting differences among the groups of students assigned to board- certified and nonboard-certified teachers. By randomly assigning students to teachers, the study removed many of the potential threats to the valid- ity of inferences about the effectiveness of NBCTs. In addition, the study restricted the comparison teachers to those with at least three years of teaching experience to make them more like the NBCTs. However, it was not able to match more closely in terms of experience and it did not include years of experience as a control variable in the analyses. So it is possible the NBCTs and their matched pair teachers might differ in terms of experience. The authors also report some student switches after assignment (less than 15 percent) that might also have affected the results. Cavalluzzo, L.C. (2004, November). Is national board certification an effective signal of teacher quality? Alexandria, VA: CNA Corporation. Available: http://www.nbpts.org/resources/research/browse_studies?ID=11 [accessed November 20, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The author examined the relationships between board certifi- cation and student achievement. Subjects/participants: Students of NBCTs and non-NBCTs teaching 9th and 10th grade mathematics in Miami–Dade County, Florida, during the 2000-2001 to 2002-2003 school years. The sample included 107,997 students and 2,137 teacher-years. It includes all the NBCTs teaching the selected grades during the chosen school years. Student scores on the state’s end of grade accountability test (the Florida Comprehensive Assessment Test, Sunshine Standards Tests) provide the measure of student outcomes. Methodology and findings: The author used education production function methods to study differences between the outcomes of students

OCR for page 275
22 ASSESSING ACCOMPLISHED TEACHING taught by NBCTs, current NBPTS applicants, teachers who applied for but did not receive NBPTS certification, and other teachers. The covariates used in the models are detailed student background variables, including grade level, age, gender, race/ethnicity, English language proficiency, participation in free or reduced price meal programs, grade retention, gifted status, spe- cial education status, school suspensions, days absent, grade point average, math effort and conduct, and whether the students’ math class was above, below, or at grade level. The models also included teacher background variables: whether or not the teacher is teaching in the subject area of certification, the salary step (as a measure of years of experience), certification status, whether or not the teacher has a graduate degree, and the selectivity of the teacher’s undergraduate college or university. The production function used a linear model that included these vari- ables and the student’s prior year math score to predict current scores. Some models also included variables measuring school attributes, and others included school fixed effects—that is, indicator variables for each school that equal 1 if the student attended the school and 0 otherwise. Scores from 9th and 10th grade students were combined into a single data set and fit to a single model. The author found that after adjusting for all the variables mentioned above, the test scores of students whose teachers were NBCTs were statisti- cally significantly higher than students whose teachers had no participation with NBPTS. Similarly, students whose teachers were currently NBCT ap- plicants scored statistically significantly higher than students whose teach- ers had no participation with NBPTS; students whose teachers applied for NBPTS certification but failed to be certified scored statistically significantly lower than students whose teachers had no participation with NBPTS. For the author’s preferred model, the effect sizes were about 0.07 (standard deviations) for NBCTs, 0.02 for current applicants, and –0.02 for teachers who failed to receive certification. These results were relatively insensitive to variations in the model, including the use of student fixed effects instead of using prior year scores as a covariate. Comments: This study is one of the few that focus on high school stu- dents and teachers. The study used a large sample of students and explored several different models, which helps to evaluate the consistency of findings across models. One concern with this study is that the analysis does not account for the fact that student test scores are nested within classes, within schools. Given that the effect sizes are small, it is very likely that many would not be statistically significant if this clustering was accounted for. A second concern is that the model does not account for the course content, and NBCTs might not be teaching courses with the same content as other teachers. This could result in the confounding of content and NBCT

OCR for page 275
2 APPENDIX A effects. There is no discussion of course content, so the possible extent of bias cannot be assessed. Course content is particularly important with high school mathematics students because the content is highly differentiated across courses but the tests are not course specific. Clotfelter, C.T., Ladd, H.F., and Vigdor, J.L. (2007, March). How and why do teacher credentials matter for achievement? Working paper 2. National Center for Analysis of Longitudinal Data in Education Research. Avail- able: http://www.caldercenter.org/PDF/1001058_Teacher_Credentials.pdf [accessed November 27, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7); Question 5 (Chapter 8). Purpose: The study examines the relationships between student achieve- ment and board certification status. Subjects/participants: North Carolina students in grades 3, 4, and 5 for the 1994-1995 to the 2003-2004 school years and their teachers. Methodology and findings: In this study, the authors model the rela- tionship between a variety of teacher characteristics and student achieve- ment test scores. The study used a production function approach and a series of alternative specifications for the model. To motivate their model, the authors first introduce a simplified model for student achievement with the assumption that the effects of teacher quality on student achievement were the same at every grade level and were constant across all years of the study. In addition, they assumed that these effects decay at a constant rate every year. This yields a structural model for current-year test scores as an additive linear function of the prior achievement score and current-year teacher inputs. The authors used this model to motivate five more com- plex models that they then fit to the data to estimate the effects of various teacher attributes on student achievement. The first model was a simple value-added model with current-year score as the outcome or the dependent variable, and the explanatory variables in the model included prior-year score and time-invariant and time-varying teacher, classroom, and student characteristics. The authors extended this model by adding school fixed effects, so that the effects of teacher charac- teristics were measured by variation within schools and differences in the student populations across schools were not confounded with the estimates of the effects of teacher characteristics. The third model used student gain scores as the dependent variable, rather than using level score as the depen- dent variable. This model did not include prior-year score as a covariate. The fourth model returned to using current-year achievement level as the dependent variable but replaced student prior-year test score and student time-invariant variables with student fixed effects. The fifth model used

OCR for page 275
2 ASSESSING ACCOMPLISHED TEACHING student fixed effects with gain scores. All models were fit separately for mathematics and reading using all available student data. The primary model specification included an indicator variable for whether or not a teacher is currently an NBCT. For mathematics, the coef- ficient is statistically significantly positive for every model. The coefficients range from 0.018 to 0.028, but most estimates are close to 0.02. Using the models with student fixed effects, Models 4 and 5, the authors compared teachers’ effectiveness across years. They considered teachers two years prior to certification, one year prior to certification, the year of certification, and one or more years after initial certification. They found that for mathematics using Model 4, that the effects were largest two years prior to certification and postcertification, with a dip in effects the year before certification (the application year) and the first year of certifica- tion. However, with Model 5 the effects were largest for teachers prior to certification and smallest in the two years postcertification. For reading, Model 4 suggests that for teachers who are certified some- time during the study, their students scored highest relative to other stu- dents when the teachers were two years prior to certification. The effects get smaller with every year of certification staging, so that the effects for NBCTs postcertification were less than half as large as the effects two years prior to certification. This pattern did not repeat with Model 5. With Model 5, students of certified teachers did best when the teacher was two years prior to certification and during the year of certification. Thus, for both reading and mathematics, the results of this secondary analysis were highly sensitive to model specification and inconsistent with the simpler model for- mulation that included a single indicator for current NBCTs. These analyses thus yield unstable estimates that need further investigation. Comments: This is a comprehensive study that evaluates the relation- ship between board certification and student achievement for three elemen- tary grade levels across nine years. The researchers examine the results for different models, providing information about the robustness of the findings to model specification. The consistency of effects for NBCTs across multiple models for both mathematics and reading provides compelling evidence that the cohort of NBCTs in North Carolina between 1995 and 2004 raised achievement test scores more than other teachers. One shortcoming of the paper is the fact that the authors do not use longitudinal data on the teachers to study how the same teacher’s students score as the teacher’s NBPTS status changes. This could provide more in- terpretable measures of NBCT effects than the comparisons that compare teachers prior to certification with other teachers.

OCR for page 275
2 APPENDIX A Cohen, C.E., and Rice, J.K. (2005, August). National board certification as professional development: Design and cost. Arlington, VA: National Board for Professional Teaching Standards. Relevance to evaluation framework: Question 8 (Chapter 11). Purpose: The authors evaluate the costs associated with support pro- grams to prepare teachers for the NBPTS assessments. They compare costs with other mechanisms for providing professional growth to teachers. Subjects/participants: Eight sites that provide a preparatory program for teachers going through the NBPTS assessment process. Methodology and findings: The authors examine how the certifica- tion process and candidate support programs provide opportunities for teacher learning and how this model of professional development relates to principles of high-quality professional development found in the litera- ture. They looked at these issues in relation to the costs of the certification process and support programs and who bears these costs. They focused specifically on eight sites that offer support programs for teachers preparing for national board certification: Cincinnati, Miami–Dade County, Missis- sippi Gulf Coast, North Carolina A&T, San Antonio, San Diego County, Stanford, and Winston-Salem. The report provides detailed information on the costs associated with four of these sites. The authors estimate that the program-related costs per participant for these four sites ranged from $1,000 (for a program with 60 participants) to $11,200 (for a program with nine participants). They indicated that some of the variability in costs is explained by the economy of scale realized by the larger programs. They compare the costs of NBPTS support programs with the costs of obtaining a master’s degree and the costs of several state- or district-level professional development programs. Comments: This extensive study is useful for states and localities that are considering implementing a support program for teachers pursuing board certification. The authors give detailed cost estimates for four pro- grams that provide various levels and kinds of supports. For our purposes, this study was relevant to one part of our cost-effectiveness analyses, and we drew from the authors’ cost estimates for our analyses. The study might have been extended to provide information about the effectiveness of the programs. For example, it would have been useful to know the pass rate for candidates who went through each program. This would have helped states and localities in making design choices about such programs. Darling-Hammond, L., and Atkin, J.M. (2007, March). Influences of na- tional board certification on teachers’ classroom assessment practices. Un- published paper, Stanford University.

OCR for page 275
0 ASSESSING ACCOMPLISHED TEACHING it contributes to understanding nonparticipation and yields information that could be used to increase teachers’ involvement in the program. One issue that likely bears on the findings is that there were no NBCTs in the two counties at the time of the study. Only one teacher had attempted the process but had not been successful. (This was learned through personal conversation with the first author.) As the respondents noted, most were minimally familiar with the certification process, and it is quite likely that they may never have met a board-certified teacher. Thus, their responses patterns may identify issues that could be pursued in educating teachers about the NBPTS and the certification process. National Board for Professional Teaching Standards. (2001a, November). I am a better teacher. Arlington, VA: Author. Available: http://www.nbpts. org/resources/research/browse_studies?ID=23 [accessed November 20, 2007]. Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: To evaluate the impact of the certification process on teachers who have gone through it. Subjects/participants: National sample of teachers who had completed the certification process, both successful and unsuccessful candidates. Methodology and findings: This is a report of a survey conducted by the NBPTS. Surveys were sent to 10,700 candidates who had recently completed the assessment process. The survey contained 27 questions. Within four weeks, 5,641 responses (53 percent) were received, and find- ings are based on these responses. The results in the report are based on 10 questions, which were grouped in a section of the survey titled “Ben- efits of the Process for You.” The questions are worded as statements that participants are asked to agree/disagree with (on a five-point scale from strongly disagree to strongly agree). All questions are worded positively (e.g., “participating in the NBPTS process helped me develop stronger cur- ricula”; “participating in the NBPTS process helped me develop improved ways to evaluate student learning”; “as a result of participating in the national board-certification process, I believe I am a better teacher”; “I found that the National Board’s assessment process enhanced the quality of my interactions with my students”). No negatively worded statements are included. The report cites the following as findings: • 92 percent of the candidates surveyed said that they believed the national board certification process made them better teachers. • 96 percent of respondents rated the national board certification

OCR for page 275
0 APPENDIX A process as a(n) “excellent,” “very good,” or “good” professional development experience. • Participation in the national board certification process equips teachers to create stronger curricula (89 percent), improves their abilities to evaluate student learning (89 percent), and helps them to develop a framework in which they can use state content stan- dards to improve teaching (80 percent). • Participation in the national board certification process enhances teacher interaction with students (82 percent) and parents and guardians (82 percent), and helps to improve collaborations with other teachers (80 percent). • There is now a high level of awareness in schools (68 percent) and school districts (81 percent) of teachers who are candidates for national board certification and of those who have achieved certification. • Candidates for national board certification are receiving high levels of support from their teaching colleagues (86 percent), principals (80 percent), and district administrators (63 percent). Comments: This survey collected important information about the ex- periences and attitudes of NBPTS participants. However, the report of the findings is written as an advocacy piece, not a research report. There is no information on nonrespondents and no evaluation of the extent to which respondents are representative of the test-takers. The survey had the poten- tial to yield information about teachers who passed and who failed, but the results are not reported separately by group. In addition, all of the survey questions are worded positively. Inclusion of both positively and negatively worded questions would have increased the objectivity of the survey and would have allowed the researchers to examine the presence of response sets (individuals who tend to always select the same response or who tend to provide what they perceive to be an acceptable response). National Board for Professional Teaching Standards. (2001d). The impact of national board certification on teachers: A survey of national board- certified teachers and assessors. An NBPTS research report. Arlington, VA: National Board for Professional Teaching Standards. Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: To evaluate the impact of the certification process on teachers. Subjects/participants: Random sample of all NBCTs. Methodology and findings: This is a report of a survey conducted by the NBPTS. Surveys were sent to a random sample of 600 of the 4,804

OCR for page 275
0 ASSESSING ACCOMPLISHED TEACHING teachers who achieved national board certification from 1994 through 1999. Competed surveys were received from 235 respondents (41 percent). The main research questions investigated were (1) How do NBCTs and as- sessors rate the certification process as a professional development experi- ence? and (2) What effect does the certification experience have on NBCTs and assessors and other stakeholders? The authors cite the following findings: (1) the national board certifica- tion process is an excellent professional development experience; (2) NBCTs indicate that the certification experience has had a strong effect on their teaching practices; and (3) the certification process has had a positive effect on students and has led to positive interaction with teachers, administra- tors, and communities. Comments: This survey collected important information about the expe- riences of NBPTS participants. However, the report of the findings is written as an advocacy piece, not a research report. The report provides only an over- view of selected results from the study. There is not enough detail provided to make independent judgments about the validity of the results. Sanders, W.L., Ashton, J.J., and Wright, S.P. (2005, March). Comparison of the effects of NBPTS-certified teachers with other teachers on the rate of student academic progress. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/resources/research/ browse_studies?ID=15 [accessed November 27, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The study examines the relationships between student achieve- ment and board-certification status. The authors also evaluate the impact of model specification on the findings. Subjects/participants: Students in 3rd through 8th grade for the 1999- 2000 through 2002-2003 school years in Charlotte–Mecklenburg and Wake County school districts in North Carolina and their teachers. Methodology and findings: For this study, the authors separated teach- ers into four groups: NBCTs, future NBPTS candidates, NBCT applicants who failed to be certified, and teachers with no NBPTS involvement. They then compared student achievement in mathematics and reading for teach- ers in the four groups. The authors fit four models for each subject area. In two models they used the current-year score on the state’s end-of-year test as the outcome or dependent variable, and in the other two models they used the gain score (prior-year score less the previous-year score) as the outcome variable. For both subject areas and each outcome (level score or gain score), one model included random effects for teachers and the other did not. Including

OCR for page 275
0 APPENDIX A teacher random effects accounted for the nesting of students within classes and should provide substantially more accurate standard errors than the models that ignore this nesting. The two models without teacher random effects were designed to be similar to those used in the studies by Cavaluzzo (2005) and by Goldhaber and Anthony (2007). The results for these models were compared with results for models that included teacher random effects, so as to examine the effect of model specification on the results. Models with the current score as the outcome included prior-year math- ematics and reading scores as covariates. All models also controlled for students’ gender and race/ethnicity, and teacher’s years of experience. The authors fit models separately by grade and subject area, comparing NBCTs with each of the other NBPTS groups (failed applicants, future applicants, and nonapplicants). The authors reported statistically significant differences between the student outcomes for NBCTs and nonapplicants for grades 5, 6, 7, and 8 in both mathematics and reading for at least one outcome specification. For mathematics, none of the differences was statistically significant in models that include random teacher effects. For reading, differences were statistically significant in models that do and do not include teacher random effects, with the exception of grade 8, in which the effects were significant only in models that included teacher random effects. In addition, the au- thors reported that the models with random effects indicated that there was typically more variance within group than across groups. That is, there was more variance among teachers with board certification than between NBCTs and each of the other groups. Comments: The major contribution of this study is the illustration of the sensitivity of results to model specification. Specifically, the models that accounted for the nesting of students in classrooms (the models that included random effects) generally resulted in substantially larger standard errors, and thus the effects were less likely to be statistically significant. This provides evidence that analyses that do not account for such nesting produce downwardly biased standard errors, which raise the probability of reporting statistically significant effects in error. One limitation of this study is that it gives up power to detect differ- ences by analyzing the grades separately. This is particularly problematic given the likely small numbers of teachers in grades 6, 7, and 8 (sample sizes are not reported). However, the differences between NBCTs and nonappli- cants vary considerably across grades, with some of the largest differences between grades 4 and 5, in which the sample sizes were largest. Thus, even if the data were pooled across grades, it is unlikely that a strong and sig- nificant difference would exist for NBCTs.

OCR for page 275
0 ASSESSING ACCOMPLISHED TEACHING Smith, T.W., Gordon, B., Colby, S.A., and Wang, J. (2005). An examina- tion of the relationship between depth of student learning and national board-certification status. Arlington, VA: National Board for Profes- sional Teaching Standards. Available: http://www.nbpts.org/UserFiles/File/ Applachian_State_study_D_-_Smith.pdf [accessed June 2008]. Relevance to evaluation framework: Question 2 (Chapter 5). Purpose: To compare work samples for students of NBCTs and of teachers who failed to earn board certification. Subjects/participants: Teachers in 17 states who had attempted to earn board certification; roughly half of the sample consisted of NBCTs, and half were teachers who had failed the assessment. Methodology and findings: This study builds on prior work by Bond et al. (2000) and uses some of the same methodologies. The study involved comparison of instructional practices and students’ work for 64 teachers from 17 states. The sample included NBCTs and teachers who had at- tempted to become board certified but were unsuccessful. The teachers were randomly selected from information provided by the NBPTS. Initial contact with potential participants was via a mail survey. Recruitment proceeded by telephone for teachers who returned their surveys expressing interest in participating. A letter of invitation was mailed to 705 teachers (280 NBCTs, 425 unsuccessful candidates) who had pursued board certification in one of four areas: (1) middle childhood generalist, (2) early adolescence English lan- guage arts, (3) adolescence/young adulthood science, and (4) adolescence/ young adulthood social studies–history. The authors had some difficulty with recruitment, originally trying for 200 participants, 50 in each of the four certificate areas. They were not able to recruit that many participants. Initially, 202 teachers verbally agreed, but there was considerable attrition at various stages of the recruitment process. The final sample included 35 NBCTs and 29 teachers who had failed the assessment. No intermediate details are provided about the sampling methodology. Data evaluated for each teacher included (1) the teacher’s description of a unit of lessons; (2) student work samples for 6 randomly selected students from each teacher’s classroom; and (3) for the generalist and language arts teachers, students’ responses to a writing task. The teachers’ instructional materials and the students’ work samples were evaluated for deep versus surface features using a taxonomy developed by Hattie and described in Bond et al. (2000). Analysis of student work samples showed that students in classrooms of NBCTs demonstrated deeper responses more often than students in classrooms with teachers who had failed the assessment, although the dif- ferences were not statistically significant. The authors noted that sometimes

OCR for page 275
0 APPENDIX A the assignment was an issue, not the students’ level of understanding, in that the assignment was not designed to elicit “deep” thinking. For the writing assessment, 18 teachers (nine NBCT, nine non-NBCT) submitted 377 writing assessment responses. The writing samples were given a holistic score as well as six analytic scores based on specific writing/ composition features (controlling idea, organization, elaboration of ideas, voice, sentence formation). Discriminant function analysis was used to determine the relationships between certification status and writing perfor- mance. The results were statistically significant (p < .05), indicating differ- ences in the writing performance of students taught by NBCTs and teachers who had failed the assessment. Analysis of teachers’ assignments to students showed that the major- ity of the teachers (64 percent) aimed instruction and assignments toward surface learning. However, NBCTs were more than twice as likely to aim in- struction at deeper learning than teachers who had failed the assessment. Comments: This study adds to the construct-based validity evidence provided by Bond et al. (2000). It expands the sample to teachers in 17 states (the sample in Bond et al. was drawn from five states) and used a dif- ferent sampling strategy. The findings generally concur with those reported by Bond et al. One issue with this study was that the authors experienced significant difficulties recruiting participants, and as a result, the sample sizes are small. Most of the attrition seemed to have occurred at the recruitment stage, rather than during the course of the study. The exception is for the writing assessment piece; only 18 of the 64 participating teachers submitted materials to be evaluated. No details are provided to compare the char- acteristics of the final sample with the initial group of recruits. This may be particularly important when considering the representativeness of the study participants who had failed the NBPTS. The teachers in this group who agreed to participate may have been quite selective and different from those who did not agree. It is not known how this potential selection bias affected the results. Sykes, G., Anagnostopoulos, D., Cannata, M., Chard, L., Frank, K., McCrory, R., and Wolfe, R. (2006). National board-certified teachers as organizational resource. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/resources/research/ browse_studies?ID=174 [accessed November 27, 2007]. Relevance to evaluation framework: Question 3 (Chapter 6), Question 6 (Chapter 9), Question 7 (Chapter 10). Purpose: To evaluate the impact of board certification on teachers’ experiences in their schools.

OCR for page 275
0 ASSESSING ACCOMPLISHED TEACHING Subjects/participants: NBCTs and non-NBCTs in South Carolina and Ohio. Methodology and findings: This report uses the results from three data collections: a school-level survey, a state-level survey, and a four-school field study that focus on samples of teachers in South Carolina and Ohio, two states with high concentrations of NBCTs. The state-level survey was distributed to random samples of teachers in the two states. The authors used stratified random sampling methods, with school level (elementary, middle, and high school) and school location (ur- ban, suburban, and rural) serving as the strata. Surveys were distributed to the 1,500 teachers for whom both a mailing address and an e-mail address were available. Usable responses were obtained from 1,153 (77 percent), roughly half from each of the two states (566 from South Carolina and 587 from Ohio). The researchers designed some of the questions so that the responses could be compared with those from the School and Staffing Survey (SASS), which is based on a national sample of teachers. Using the SASS data, they make comparisons between NBCTs and all teachers in Ohio and in South Carolina. The schools participating in the school-level survey were selected in a multistage process. First, two urban districts were identified in each state based on the district policies about board certification. Then districts neigh- boring the urban districts were selected (one neighboring district for each urban district in South Carolina; multiple neighboring districts for each urban district in Ohio due to smaller numbers of NBCTs). Schools in each district were grouped by the density of NBCTs. Six schools were selected in each urban district: a case-study school, a school with no NBCTs, and four additional schools with varying numbers of NBCTs. Six schools in each neighboring district were also selected: one with no NBCTs and five with varying numbers. This resulted in a final sample of 47 schools (one declined after data collection began), all elementary schools. The school-level survey was administered in person to the entire faculty in each of the 47 schools. A total of 1,583 surveys were completed with an average school response rate of 84 percent. The field study involved interviews with faculty and staff at four schools in the sample of 47, two in Ohio and two in South Carolina. The schools were selected from the urban districts and on the basis of the percentages of NBCTs relative to the district average, with a goal of identifying schools with a “critical mass” of NBCTs. Survey Results. Based on the state-level survey, the authors found that in both states, NBCTs tended to perceive that they have more influence over schoolwide policies than do all teachers (based on comparisons with the SASS). Difference between NBCTs and all teachers were statistically

OCR for page 275
 APPENDIX A significantly with respect to their perceptions of influence on curriculum, the content of inservice professional development, evaluation of teachers, and hiring new full-time teachers. Comparisons between NBCTs and non-NBCTs with regard to the same issue again shows that NBCTs perceive higher levels of influence on school- wide policies, but differences were statistically significant only in the areas of establishing curriculum and evaluating teachers. The state-level survey queried NBCTs about the activities in which they have participated since becoming certified. Over half indicated that they mentor other teachers, serve as team leaders, develop or select curriculum materials, support other national board candidates, and provide profes- sional development to teachers at their schools. Some NBCTs also reported participating in district-level or state-level activities, and this seemed to vary with the number of years they had held their board certification. For example, over half of the teachers who held their certification for seven or more years, reported serving as mentors, serving as team leaders, providing professional development, supporting national board candidates, and de- veloping curriculum materials at the district level. The authors did not ask respondents if they were involved in these activities prior to receiving board certification and point out that it may be that teachers who participate in leadership activities self-select into the national board process. Field Studies. This portion of the study focuses on two elementary schools in each of the studied states. At all four schools, roughly one-fifth of the teachers were board certified. The team of researchers spent roughly a week in each school and conducted interviews with all of the teachers. They generally found that teachers did not interact with each other about their instructional practices and not too much was made of board certification. The teachers who had obtained board certification were generally positive about the experience, although they were reluctant to state that board certification signaled a level of competence that set them apart from their colleagues. The non-NBCTs tended to think there was no difference be- tween the NBCTs and themselves, sometimes citing stories of well-qualified teachers who tried and did not pass or less qualified teachers who passed. Principals also noted that they were careful about how they meted out as- signments, not wanting to seem to overly favor the NBCTs or to engender envy or resentment from the non-NBCTs. The authors provide descriptions of the ways in which NBCTs are viewed at each of the four schools and the type of leadership activities in which they are involved. In one school, “Stevenson,” the district facilitated NBCT leadership by enabling NBCTs to qualify for grade-level team leader positions within the school. The teams served as important opportunities for collaboration and the sharing of technical expertise. The team leader was seen as a potentially highly influential position. District policy allowed

OCR for page 275
2 ASSESSING ACCOMPLISHED TEACHING NBCTs to leave their classrooms to assist other schools in instructional improvement. In this school, over half of the non-NBCTs identified the leadership roles that NBCTs held within the school, although they still said that NBCTs were “no different” than other teachers in terms of their com- mitment to schoolwide issues, initiatives, and concerns. In the other schools, less experienced teachers tended to seek advice and support from the schools’ NBCTs; however, often this was by informal mechanisms. Formal mechanisms in which NBCTs could provide advice or leadership were not present, and there was evidence that principals down- played board certification and that NBCTs themselves “concealed” the fact that they were board certified. Comments: This is a comprehensive study that corroborates findings reported in Koppich et al. (2006) with regard to the unsupportive environ- ments faced by NBCTs. This study and the Koppich study drew samples of teachers from some of the same states (both studied teachers in Ohio and South Carolina), so it would be useful to investigate the extent to which the reported conditions exist in other states. The use of multiple data col- lection strategies (school-level survey, state-level survey, and case studies) is a strong feature of this study. The design of survey questions so that comparisons can be made with SASS results is also very useful. The one area that the researchers might have also explored is the issue of failing the NBPTS assessment and any impacts that might have on teachers. Yankelovich Partners. (2001, April). Accomplished teachers taking on new leadership roles in schools: Survey reveals growing participation in efforts to improve teaching and learning. Arlington, VA: National Board for Pro- fessional Teaching Standards. Available: http://www.nbpts.org/resources/ research/browse_studies?ID=22 [accessed November 27, 2007]. Relevance to evaluation framework: Question 5 (Chapter 8), Question 7 (Chapter 10). Purpose: To evaluate the extent to which NBCTs assume new leader- ship roles. Subjects/participants: National sample of NBCTs. Methodology and findings: This study, sponsored by the NBPTS, in- volved a survey of teachers who had received board certification in 1999 or earlier and focused on their participation in leadership roles. Surveys were sent to all NBCTs (n = roughly 4,800) in November 2000 and accepted until mid-January 2001. The report summarizes findings based on the 2,186 who responded as of this date (46 percent response rate). Nearly all respondents (99 percent) said that they had a very favorable or somewhat favorable regard for national board certification.

OCR for page 275
 APPENDIX A Over half of the respondents indicated that they have engaged in the following behaviors since obtaining board certification: • mentoring other teachers pursuing board certification (90 percent); • mentoring struggling teachers (83 percent); • developing or selecting materials to support student learning (80 percent); • involvement in school or district leadership activities (68 percent); • developing instructional strategies or curricula (62 percent); • developing teacher professional development programs or activities (58 percent); • speaking publicly about national board certification (57 percent); • highlighted as experts by the school, press, or community (53 percent); • seeking grants to support teaching and learning (53 percent); and • working with teacher preparation programs at colleges (51 percent). A separate question asked if the respondent had been involved in the activity prior to becoming board certified, and participation is reported as a percentage of those who indicated they are currently involved in the activ- ity. For each of the activities listed above, more than half of the respondents indicated that they had been involved in these leadership activities prior to obtaining board certification. Another question asked about the impact of certification on obtaining or keeping these leadership roles. The leadership roles that appear to be most affected by obtaining board certification all involve NBPTS in some way. For example, over half say that obtaining board certification had an impact on participation in a network of NBCTs, mentoring NBPTS can- didates, advocating for board certification, speaking publicly about board certification, and helping the NBPTS to offer board certification. Very few of the respondents indicate that board certification had an impact on their engaging in other leadership roles. The majority of respondents agreed (strongly or somewhat) with state- ments about the positive effects of leadership activities. For example, they agreed that participation in leadership activities enhanced career satisfac- tion, made them feel more significant in the profession, increased effective- ness as an educator, increased desire to remain in the profession, make them feel that the profession has a lot to offer. These statements all represent positive aspects of such participation. The report does not include any negative statements, such as leadership activities are time-consuming or it is difficult to make time for leadership activities.

OCR for page 275
 ASSESSING ACCOMPLISHED TEACHING Comments: There are two versions of this report. One is an advocacy piece called “Leading from the Classroom.” The other is in the form of a memo (from Andrew Kennelly, with Yankelovich Partners, to Mary Buday, with the NBPTS). Neither provides a complete documentation of the meth- odology and findings. The memo simply provides a copy of the survey with the percentages of candidates who selected each response option, which we have summarized above. There is also no information provided to document the extent to which the characteristics of respondents represent those of the group who received surveys (only a statement that asserts the respondents were representative). The lack of details about the methodol- ogy makes it difficult to evaluate the robustness of the findings.