In the research I reviewed, a few studies stood out as the kinds of research that are needed. Yin, Noboa-Rios, Davis, Castillo, and MacTurk (2001) described a logic model developed as part of a plan to conduct a cross-site evaluation of NSF’s Urban Systemic Initiative that would explain different stages of systemic reform. The evaluation design is intended to capture the “systemicness” of each site and the program as a whole using a replication design in which each site is considered to be a naturally occurring experiment, and cross-site patterns are seen as evidence of replication. Although it was too early in the work for Yin et al. to report results, the model is a promising approach to capturing some of the policy and structural influences of the NSES on the systems that undergird the delivery of professional development. Likewise, the Eisenhower evaluations and SRI’s cross-site SSI evaluation were exemplars of high-quality, thoughtful studies that provided substantial evidence of where and why the NSES have and have not influenced the different aspects of the professional development system. Additionally, studies like Spillane’s investigation of how policy makers’ beliefs about learning influence their policy strategies provide fresh insight into the often superficial levels of understanding of those leaders charged with enacting the NSES and the profound influence of local culture and context on the implementation process.
There are also several important areas where research is largely silent. There are several professional organizations that have traditionally provided guidance to professional developers, but we know little about the influence of the NSES on the way these organizations provide leadership for their members. For example, there are several organizations that accredit universities to provide pre-service education, such as the National Council for Accreditation of Teacher Education and the Teacher Education Accreditation Council, as well as the new Interstate New Teacher Assessment and Support Consortium. It would be worthwhile to specifically study whether and how these organizations have changed their systems since the advent of the NSES. Additionally, there are also professional organizations (e.g., Association for Supervision and Curriculum Development, National Staff Development Council) that provide guidance to a large number of in-service professional developers. How have these organizations been influenced by the NSES?
At the beginning of this paper, I presented a framework for developing robust research-based evidence. Within this framework, the goal for researchers and the sponsors of research is to develop a more coordinated body of evidence in order to systematically build a strong case in support of a particular hypothesis (in this case, the influence of the NSES on policies, pre-service professional development, or in-service professional development). Building a strong evidence base requires multiple examples of quality research employing appropriate methods that together provide confirmatory findings. The evidence examined in this study suggests that the current research base is of variable quality and provides too few reinforcing results. While there are an incredible number of talented researchers across the nation, our efforts are largely unfocused and idiosyncratic. The current educational research system lacks commonly accepted standards of quality research (regardless of methodology), poor coordination, and too few incentives that would allow us to build a systematic evidence base around important questions like the influence of the NSES on the system of professional development.