10
Thinking Systematically

In this volume we have discussed the dimensions of assessment, including its purposes, the domains to be assessed, and guidelines for selecting, implementing, and using information from assessments. Beyond this, however, one cannot make use of assessments optimally without thinking of them as part of a larger system. Assessments are used in the service of higher level goals—ensuring the well-being of children and their families, ensuring that societal resources are deployed productively, distributing scarce educational or medical resources equitably, facilitating the relevance of educational outcomes to economic challenges, making informed decisions about contexts for the growth and development of children, and so on. Assessments by themselves cannot achieve these higher goals, although they are a crucial part of a larger system designed to address them. Only when the entire system is considered can reasonable decisions about assessment be made.

This chapter argues that early childhood assessment needs to be viewed not as an isolated process, but as integrated in a system that includes a clearly articulated higher level goal, such as optimal growth, development, and learning for all children; that defines strategies for achieving the goal, such as adequate funding, excellent teaching practices, and well-designed educational environments; that recognizes the other elements of infrastructure



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 301
10 Thinking Systematically I n this volume we have discussed the dimensions of assess- ment, including its purposes, the domains to be assessed, and guidelines for selecting, implementing, and using information from assessments. Beyond this, however, one cannot make use of assessments optimally without thinking of them as part of a larger system. Assessments are used in the service of higher level goals—ensuring the well-being of children and their families, ensuring that societal resources are deployed productively, distributing scarce educational or medical resources equitably, facilitating the relevance of educational outcomes to economic challenges, making informed decisions about contexts for the growth and development of children, and so on. Assessments by themselves cannot achieve these higher goals, although they are a crucial part of a larger system designed to address them. Only when the entire system is considered can reasonable decisions about assessment be made. This chapter argues that early childhood assessment needs to be viewed not as an isolated process, but as integrated in a sys- tem that includes a clearly articulated higher level goal, such as optimal growth, development, and learning for all children; that defines strategies for achieving the goal, such as adequate fund- ing, excellent teaching practices, and well-designed educational environments; that recognizes the other elements of infrastructure 0

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT instrumental to achieving the goal, such as professional develop- ment and mechanisms for monitoring quality in the educational environment; and that selects assessment instruments and pro- cedures that fit with the other elements in service of the goal. We begin by noting the multiple state and federal structures in which early childhood assessments are being implemented. These structures have emerged from different sources with different funding streams (e.g., federally funded Head Start, state- funded prekindergarten, foundation-funded intervention pro- grams) and rarely display complete convergence of performance standards, criteria, goals, or program monitoring procedures. Thus, referring to “a larger system of early care and education” is slightly deceptive, or perhaps aspirational. Furthermore, even the well-established programs in the “system” may lack key components—for example, they may assess child outcomes but not relate those outcomes to measures of the environment, or they may not have a mechanism in place for sharing child outcome data in helpful ways with caregivers and teachers. We use recent National Research Council reports, state expe- riences with the No Child Left Behind Act, and the recent work of the Pew Foundation–sponsored National Early Childhood Accountability Task Force—a national effort focused on account- ability in early childhood—as a basis for articulating the compo- nents needed in order for early childhood assessment to be part of a fully integrated system. We also provide some examples of progress toward this goal at the state level. Although we did not find any examples of fully integrated systems, in which services are provided by a single source and the assessment infrastructure is fully aligned and developed, the three states we describe are moving toward integrating early childhood assessment in a well- articulated system. WHAT DO WE MEAN BY A SYSTEM? The idea of a system comes up often in education discussions and analyses—there are education systems, instructional systems, assessment systems, professional development systems—but it is not always clear what the word actually means. Systems have a number of important features, which are enumerated in Systems

OCR for page 301
0 THINKING SYSTEMATICALLY for State Science Assessment (National Research Council, 2006).1 In particular, they are organized around specific goals; they are made up of subsystems, each of which serves its own purposes; the subsystems must work well both autonomously and in harmony with one another for the larger system to work well; and a miss- ing or poorly operating subsystem may cause a system to function poorly or not at all. In our use of the term with reference to early childhood assessment, the committee intends • that assessments be seen as a part or subsystem2 of a larger system of early childhood care and education, which addresses the multiple aspects of child development and influences discussed in this volume; • that selection of assessments be intimately linked to goals defined by that larger system; • that procedures for sharing information about and using information from assessments be considered as part of the process of selecting and administering assessments; and • that different parts of the assessment system itself (stan- dards, constructs, measures, indicators) work together. Systems need to have well-developed feedback loops to pre- vent over- or undercompensation for changes in a single part. Feedback loops occur whenever an output of some subsystem connects back to one of its inputs. For example, a fundamental feedback loop occurs in the classroom when a teacher identifies problems that children are having with an idea or skill and adjusts his or her instructional techniques and the learning environ - ment in response. When this causes the children to learn the idea or skill successfully, one would say that the feedback loop has worked effectively. Implementation of a similar feedback loop at the level of the program takes child performance as the input for identifying classrooms in which teachers need additional 1This section and the following one on infrastructure draw heavily on the content of the National Research Council’s 2006 report, Systems for State Science Assessment. 2Although assessment is here defined as a subsystem of a larger system, through- out this chapter we refer to the “assessment system” for the sake of simplicity, except when the distinction is important.

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT assistance in implementing instructional activities. These two sub- systems—the individual- and the program-level feedback from child performance to teacher supports—function well as part of a larger system if the same or consistent information is used in both loops. However, if, for example, the teacher is responding to child performance so as to enhance creative problem solving, whereas the institution is encouraging teachers to focus on children’s rote memorization capacity, then the subsystems conflict and do not constitute a well-functioning system. In a well-designed program, the assessment subsystem is part of a larger system of early childhood care and education comprised of multiple interacting subsystems. These other sys- tems include the early learning standards, which describe what young children should know and be able to do at the end of the program; the curriculum, which describes the experiences and activities that children will have; and the teaching practices, which describe the conditions under which learning should take place, including interactions among the teachers and children as well as the provisioning and organization of the physical environ- ment (National Research Council, 2006). The relationships among these four subsystems are illustrated in Figure 10-1, adapted from the “curriculum, instruction, assessment (CIA) triangle” commonly cited in the educational assessment community. Each of these subsystems is also affected by other forces, for example, laws intended to influence what children are expected to learn, professional development practices, and teacher preparation policies influenced by professional organizations and accredit- ing agencies. We argue in this chapter that all these components must be thought of as part of a larger system, and that they must be designed so as to be coherent with one another, as well as with the policy and education system they are a part of, and with the goals for child development that the entire system is meant to be promoting. We reframe these arguments as a conclusion to this chapter. INFRASTRUCTURE FOR AN ASSESSMENT SYSTEM An early childhood assessment subsystem should be part of a larger system with a strong infrastructure that is designed to

OCR for page 301
0 THINKING SYSTEMATICALLY FIGURE 10-1 A schematic relationship (the “CIA Triangle”) among early learning standards, curriculum, instructional practices, and assessment. Figure 10-1, bitmapped provide high-quality early care and education. The infrastructure R01340 is the foundation on which the assessment subsystem rests and is critical to its smooth and effective functioning (National Early Childhood Accountability Task Force, 2007). The infrastructure encompasses several components that together form the system: 1. Standards: A comprehensive, well-articulated set of stan- dards for both program quality and children’s learning that are aligned to one another and that define the constructs focused on in assessment as well as the performance levels identified as acceptable. 2. Assessments: Multiple approaches to documenting child performance and reviewing program quality that are of high quality and connect to one another in well-defined ways, from which strategic selection can be made depend- ing on specific purposes. 3. Reporting: A p rocedure, defined on the basis of the standards and the assessments, designed to maintain

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT an integrated, user-accessible database of assessment results, provide for quality assurance and integrity of data, and generate reports for the varied user audiences and purposes. 4. Professional development: Ongoing opportunities pro- vided to those at all levels (practitioners, assessment a dministrators, program directors, policy makers) to understand the standards and the assessments and to learn to use the data and data reports with integrity for their own purposes. 5. Opportunity to learn: Procedures for ensuring that the environments in which children are spending time offer high-quality support for development and learning, as well as safety, enjoyment, and affectively positive relationships. This is crucial when decisions about children or programs are based on assessment outcomes. 6. Inclusion: Methods and procedures for ensuring that all children served by the program will be assessed fairly, regardless of their language, culture, or disabilities, and with tools that provide the most useful information for fostering their development and learning. 7. Resources: Assurance that the resources needed to ensure the development and implementation of the system com- ponents are available or will be recruited. 8. Monitoring and evaluation: Procedures for continuously monitoring the system itself to ensure that it is operating effectively and that all elements are working together to serve the interests of the children. This infrastructure must be in place to create and sustain an assessment subsystem within a larger system of early child- hood care and education. Ensuring the adequacy of each of these components raises some critical challenges. A challenge to the adoption of systems-level thinking about early childhood care and education, and thus about early childhood assessment, is the absence, under current U.S. policies, of a unified structure for early care and education. The current variety of separate programs seg- regated by setting, by agency, and by funding streams, with their numerous challenges to delivering uniformly high-quality early

OCR for page 301
0 THINKING SYSTEMATICALLY care and education services, also serves as a barrier to developing a unified system of assessment. While the suggestion that these many barriers to an integrated system must be vaulted may seem unrealistic, we argue that a vision of a well-integrated, coherent system is needed to guide the development of policy for young children. We expand on the importance of each component of a well-organized system below. Standards The most fundamental aspect of the assessment system is the set of explicit goals for children’s development and learning around which the larger system is organized, thus providing the basis for coherence among the various elements. In most educa- tional settings, these are referred to as “standards,” but in early childhood education sometimes other terms, such as “guidelines” or “foundations,” have been used. Whatever they are named, these standards direct the design of curriculum, the choice of teaching practices, and the priorities of teachers in setting instruc- tional goals, planning activities and experiences, and organizing the environment. They are the starting point for developing assessments, judging performance levels, and rating children’s and the program’s growth and performance. Standards are also the framework for reporting children’s performance to educators and the public and for focusing pro- gram improvement efforts. Note that, although these standards are to be applied to children’s performance, they can be used as one input in establishing accountability for teachers, centers, and states (National Research Council, 2006). Thus, while some may see holding teachers, early care and education settings, and states to these standards for children’s performance as potentially punitive, others argue that they constitute a defense of the right of children to a high-quality and fair early childhood environment. Note that when applying the same logic to the programs in which children are to be educated, an equivalent set of statements can be made regarding program standards. For example, consider the No Child Left Behind Act (NCLB), which requires states to have reading, mathematics, and science standards for K-12 education that must be of “high quality,”

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT although the act says relatively little about what characterizes standards of high quality. While we are emphatically not recom- mending that the NCLB regime be extended to early childhood education, it is important to understand the NCLB framework, as it is the most common reference point on standards in the United States, and states are being asked by the federal government to align their preschool standards with their K-12 standards. Under the act, the word “standards” refers both to content standards and to achievement standards. The law requires states to develop challenging academic standards of both types, and a federal guid- ance document describes them as follows (U.S. Department of Education, 2004): • Academic content standards must specify what all children are expected to know and be able to do; contain coher- ent and rigorous content; and encourage the teaching of advanced skills. • Academic achieement standards must be aligned with the state’s academic content standards. For each content area, a state’s academic achievement standards must include at least two levels of achievement (proficient and advanced) that reflect mastery of the material in the state’s academic content standards, and a third level of achievement (basic) t o provide information about the progress of lower- achieving children toward mastering the proficient and advanced levels. Note that achievement standards are often also referred to as performance standards. The NCLB-driven standards apply to children in grades 3-12 and link directly to the explicitly defined academic content areas that are also assessed in determining adequate yearly progress for schools. It would be inappropriate to borrow this model unchanged and apply it to early childhood settings, in which explicit instruction in well-defined academic content areas is not characteristic of excellent care and education. The Council of Chief State School Officers defines common s tandards and assessment-related terms in language rele - vant to the early childhood community (http://www.ccsso.

OCR for page 301
0 THINKING SYSTEMATICALLY org/projects/SCASS/projects/early_childhood_education_ assessment_consortium/publications_and_products/2838.cfm). It defines standards as “widely accepted statements of expecta- tions for children’s learning or the quality of schools and other programs.” Of critical importance in this definition is the inclu- sion of program standards on equal footing with expectations for children’s learning. The report Systems for State Science Assessment (National Research Council, 2006) examines the role of standards in certain educational assessments and recommends that they be designed with a list of specific qualities in mind: standards should be clear, detailed, and complete; be reasonable in scope; be correct in their academic and scientific foundations; have a clear conceptual framework; be based on sound models of learning; and describe performance expectations and proficiency levels. State standards that have been developed for K-12 education do not meet these requirements as a whole, although some come closer than others. Recent analyses of states’ early childhood standards also suggest some misunderstanding of the difference between content and performance (Neuman and Roskos, 2005; Scott-Little, Kagan, and Frelow, 2003a). Appendix C presents a brief description of the cur- rent status of state standards for early childhood education, and includes some discussion of the efforts to align early childhood with K-12 standards. Standards should be arranged and detailed in ways that clearly identify what children need to know and be able to do and how their ideas and skills will develop over time. Learning progressions (also called “learning trajectories”) and learning performances are two useful approaches to arranging and detail- ing standards so as to guide curriculum, teaching practices, and assessment. Learning progressions are descriptions of successively more sophisticated ways of thinking and behaving that tend to follow one another as children mature and learn: they lay out in text and through examples what it means to move toward more mature understanding and performance. A useful example of the ideas of learning progressions and learning performances in the preschool years is California’s Desired Results Developmental Profiles-Revised (DRDP-R) and

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT its learning progression for interpersonal skills. This learning progression has been viewed as being composed of six areas, for each of which a measure (or observational guide) has been constructed: 1. expressions of empathy, 2. building cooperative relationships with adults, 3. developing friendships, 4. building cooperative play with other children, 5. conflict negotiation, and 6. awareness of diversity in self and others. The learning progression itself is summarized in the DRDP-R Preschool instrument (California Department of Education, 2005). Taking the interpersonal skills example further, we can examine one of the measures to see what the learning progression looks like. For example, consider the measure “building cooperative play with other children.” For the chosen measure, the progres- sion, expressed as four successive levels, is as follows (starting from the lowest): (a) interacts with other children side-by-side as they play with similar materials, (b) engages with another child or children in play involving a common idea or purpose, (c) shows preference for particular playmates but plays cooperatively with a variety of children, and (d) leads or participates in planning cooperative play with other children. This measure in the learning progression is brought to life by examples of learning performances that could illustrate the differ- ent levels. Examples for the lowest level (a in the list above) are: (i) plays blocks side-by-side with other children, (ii) hands another child a toy that he or she is looking for, and (iii) hands a bucket to a child sitting next to him or her in the sandbox.

OCR for page 301
 THINKING SYSTEMATICALLY Note that the teachers are encouraged to develop their own examples, so that these three do not become canonical. To illus- trate changes to the second level in this measure, examples for the next level (b in the list) are as follows: (i) plays with blocks with another child, (ii) plays in sand to build a castle with several other children, and (iii) joins another child to help look for a lost toy. More examples of learning performances are shown in Figure 10-2, which is a copy of the scoring guide for the measure “building cooperative play with other children.” Learning progressions should be developed around the orga- nizing principles of child development, such as self-regulation. Such organizing principles—which are sometimes referred to as the “big ideas” of a curriculum—are the coherent foundation for the concepts, theories, principles, and explanatory schemes for child development (National Research Council, 2006). Organizing standards around these big ideas represents a fundamental shift from the more traditional organizational struc- ture used in K-12 standards, in which standards are grouped under discrete topic headings. For example, instead of listing “knowledge of 10 letters” as a desirable outcome for a 4-year-old, one might list letter recognition and phonological awareness as examples of performances under a heading such as “emergent understanding of literacy forms.” A likely positive outcome of reorganizing standards from many discrete topics to a few big ideas is a shift from breadth to depth of coverage, from long lists of goals to a relatively small set of foundational values, principles, and concepts. If those values, principles, and concepts are the target of instruction, they can develop naturally and be extended over time. Specifying learning performances is a technique for elabo- rating on content standards by describing what children should be able to do if they have achieved a standard. Some examples of learning performances: children should be able to interact

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT • Hold programs accountable to program standards that support the achievement of desired results and are used to measure program quality. • Provide a data collection mechanism for evaluating the quality of individual child development programs. • Create a base of information on the relationships between processes and results that can be used to target technical assistance to improve practice in all child development programs. At the state level, educators use the desired results system to identify successes and areas for improvement so that CDE can provide support and technical assistance to increase program quality. At the program level, practitioners use the desired results system to determine the extent to which children and families are achieving the desired results, so that quality improvement activities may be effectively targeted to directly benefit program participants. The desired results system encourages differences in the structure and objectives of individual child development programs. It is culturally sensitive and linguistically responsive to the diverse populations of children and families served. Including Children with Disabilities The desired results system is also being coordinated with a concurrent project, Desired Results: Access for Children with Disabilities Project (DR Access, http://www.draccess.org/index. html). The DR Access project is funded by the CDE Special Educa- tion Division and coordinates with the DCRF system in two ways. First, DR Access staff members worked with CDE staff members and CDE’s contractors during the development of the desired results system to make the Desired Results Developmental Profile as inclusive and appropriate as possible for assessing the progress of young children with disabilities. Second, DR Access staff mem- bers have also developed a system of adaptations and guidelines for the Desired Results Developmental Profile that allows practi- tioners to assess children with disabilities in an appropriate man- ner within the structure of the desired results system. Through these two approaches, DR Access staff members

OCR for page 301
 THINKING SYSTEMATICALLY ensured that the desired results system was responsive to the needs of young children with disabilities and was applicable to all settings in which they and their families were served. The vision held by the contributors to desired results and DR Access was that, through collaboration, a continuity of outcomes would be achieved for all children in CDE programs. Components of the System The desired results system has six basic components: desired results, indicators, themes, measures, criteria for success, and measurement tools. 1. Desired results: The six desired results, to which all CDE- funded child care and development programs are expected to contribute, are that children are personally and socially competent, are effective learners, show physical and motor competence, are safe and healthy, and have families that support their learning and development, and achieve their goals. These desired results encompass the four develop- mental domains—cognitive, socioemotional, language, and physical development. 2. Indicators: An indicator defines a desired result more specifically so that it can be measured. For example, an indicator of the desired result “children are personally and socially competent” is that “children show self-awareness and a positive self-concept.” Desired results are gener- ally better measured by using multiple indicators; no single indicator gives full information on all aspects of achievement. 3. Themes: A theme describes the aspect of development that is being measured for each indicator (e.g., self-awareness: dependence and interdependence, understanding that one’s self is a separate being with an identity of its own and with connectedness to others). 4. Measures: A measure quantifies achievement of a particular indicator and developmental theme (e.g., a preschooler can communicate easily with familiar adults).

OCR for page 301
 EARLY CHILDHOOD ASSESSMENT 5. Criteria: The criteria for success define the acceptable level of achievement for each indicator (e.g., English language learn- ers who entered the program with no comprehension of Eng- lish now participate in read-alouds by repeating key words). 6. Measurement tools: A measurement tool is the actual instrument or procedure used to capture or track informa- tion on indicators and standards of achievement (e.g., the Desired Results Developmental Profile). Professional Development The training and implementation phase of desired results for center-based programs and family child care home networks is being carried out in a series of regional training sessions for local program administrators. Assisted by the California Institute on Human Services, CDE is providing comprehensive training designed to facilitate implementation of the desired results system in programs at the local level and to build the capacity of local programs to train staff members who work directly with children. Participation in the training is by invitation only, and sites are selected one year before they are due for a Coordinated Compli- ance Review or Contract Monitoring Review. Nebraska Results Matter (http://ectc.nde.ne.gov/special_projects/ results_matter/results_matter.htm) is designed to improve pro- grams and child and family outcomes for all children in Nebraska from birth to age 5, whether they are served through school dis- tricts, the Early Development Network (Part C of the Individuals with Disabilities Education Act), newly implemented infant and toddler programs funded through the Early Childhood Endow- ment, or community partners. The system grew out of earlier efforts to monitor and evaluate state-funded preschool programs. Its broader application came as a result of recent federal require- ments for reporting outcomes for children with disabilities. The system employs both program quality assessment and child outcome assessment to accomplish several purposes: improve experiences, learning, development, and lives of young children

OCR for page 301
 THINKING SYSTEMATICALLY (birth to age 5) and their families; inform program practices; demonstrate program effectiveness; guide the development of local and state policies and procedures; and provide data to dem- onstrate results. The system is administered through the Nebraska Department of Education. Major partners include the state’s Early Childhood Training Center, Health and Human Services, the Munroe-Meyer Institute at the University of Nebraska Medical Center, and multi- county educational service units. The system operates with the advice of the Results Matter Child Measurement Task Force. Child Assessment Child assessment tools were selected based on whether they employ ongoing observation of children engaged in real activities, with people they know, in natural settings; reflect evidence-based practices; engage families and primary care providers as active participants; integrate information gathered across settings; are individualized to address each child’s unique ways of learning; inform decisions about day-to-day learning opportunities for children; and reflect the belief that development and learning are rooted in culture supported by the family. The selected tools also reflect optimal congruence with Nebraska’s Early Learning Guidelines (Birth to Three and Three to Five; http://ectc.nde.ne.gov/ELG/elg.htm) and are congru- ent with the program standards found in Rule 11, Regulations for Early Childhood Programs (http://www.nde.state.ne.us/ LEGAL/RULE11.html). These tools are the High/Scope Child Observation Record (COR), the Creative Curriculum Develop- mental Continuum, and the Assessment, Evaluation and Pro- gramming System (AEPS). The state has purchased licenses for the use of these tools; programs complete the assessment online. Some districts have chosen to use more than one assessment and thus more than one online system. Districts began entering data in 2006, and the first data were reported to the Office of Special Education Programs in the U.S. Department of Education in February 2008. The use of these tools supported through the online data system provides the state with unprecedented opportunities to compile needed data,

OCR for page 301
 EARLY CHILDHOOD ASSESSMENT not only for the required state and local reporting functions, but also for ongoing program improvement and curriculum planning. Nebraska’s system is responsive to the federal mandate of the IDEA Part C (birth to age 3) and Part B, 619 (ages 3 to 5), as well as the state requirements of Nebraska Department of Education Rule 11, Regulations for Early Childhood Programs (http://www. nde.state.ne.us/LEGAL/RULE11.html), which apply to all pre-K programs operated through public schools. Program Quality Assessment The system also includes regular evaluation of programs to ensure that they achieve and maintain overall high quality, employ qualified staff, and operate in compliance with fed- eral and state guidelines. Programs receiving state funding are required to conduct an annual evaluation using one of the envi- ronment rating scales, such as the Infant/Toddler Environment Rating Scale-Revised, ITERS-R (Harms, Clifford, and Cryer, 1998); Early Childhood Environment Rating Scale-Revised, ECERS-R (Harms, Cryer, and Clifford, 1990); or the Early Language and Literacy Classroom Observation, ELLCO (Smith and Dickinson, 2002), and complete Nebraska’s Rule 11 reporting and approval processes. Data obtained from these tools are used to develop improvement plans. In addition, programs are strongly encour- aged to participate in the accreditation process of the National Association for the Education of Young Children and receive technical and financial assistance to do so. Professional Development Programs receive continuous support to ensure that their participation in Results Matter does generate the highest quality data and knowledge about how to use it to improve program quality and child and family outcomes. The state’s Early Child- hood Training Center, in cooperation with the organizations that provide the program and child assessment tools, regularly offers training in their use. The state maintains a cadre of professionals who have achieved reliability in the use of the environment rating scales. In addition, each program provider is required to submit

OCR for page 301
 THINKING SYSTEMATICALLY a Fidelity Process Plan to address how the reliability and validity of the child observational data will be monitored and recorded. These plans describe initial training and subsequent activities to strengthen the validity of the data. New Jersey New Jersey’s Abbott Preschool Program is designed to pro- vide high-quality preschool education to children ages 3 and 4 in 31 of the state’s poorest districts. The program has a mixed deliv- ery system and is conducted in school districts and community- based centers, including Head Start programs. Curriculum and Instruction The New Jersey State Department of Education (NJDOE) has developed a set of early learning standards—the Preschool Expec- tations: Standards of Quality (2004)—which outline what children should know and be able to do at the end of their preschool pro- gram across a comprehensive set of domains. Five curriculum models have been approved: Creative Curriculum, High/Scope, Tools of the Mind, Curiosity Corner, and Bank Street. Each is aligned to the Preschool Expectations. Each district is required to select one of these approved curriculum models and to provide early childhood educators with professional development related to appropriate curriculum implementation. Assessments The NJDOE designed two performance-based assessments in the areas of literacy and mathematics that were linked directly to the Preschool Expectations: the Early Learning Assessment System-Literacy (ELAS-L; New Jersey Office of Early Childhood Education, 2004) and the Early Learning Assessment System for Mathematics (ELAS-M; New Jersey Office of Early Childhood Education, 2006). In the initial years of the preschool program, the state provided professional development for teachers in the observation and documentation of young children’s learning and in administering and scoring the ELAS assessments. While these

OCR for page 301
 EARLY CHILDHOOD ASSESSMENT measures were originally intended to be used both for program evaluation and to inform instructional practice, state officials decided that they would be used only for instructional planning. In the ninth year of Abbott preschool implementation, the districts must select a commercially produced performance-based assess- ment that covers the entire range of domains in the Preschool Expectations. The ELAS instruments may still be used in the areas of literacy and mathematics. Assessment at Various Levels At the classroom level, teachers administer a performance- based assessment that covers the range of domains outlined in the Preschool Expectations. These formative assessments are intended to inform instructional practice and to give teachers direct infor- mation on the learning and development of individual children. Up one level, a sample of the community-based and school district classrooms is assessed for quality on the ECERS-R. The results of these measures are used for teacher professional development. ECERS-R scores are also reported at the district level and used to monitor classroom quality across the 31 districts. Statewide, a longitudinal study is tracking the progress of a sample of children who have participated in the Abbott preschool program on nationally normed measures of language, literacy, and mathematics. In addition, a regression discontinuity design is being used to estimate the impact of preschool on the performance of children who received it in comparison to those who did not. THINkING ABOUT ASSESSMENT AS A SYSTEM Despite the clear advantages of a systems approach to early childhood care and education, there is no doubt that the move toward systematicity will encounter many obstacles. The states and the federal government often effect change in the early childhood system by introducing new programs, local or limited innovations, and underfunded mandates. These might well con- stitute good models or useful efforts, but they undermine efforts to build coherence across programs and funding sources at the same time.

OCR for page 301
 THINKING SYSTEMATICALLY Similarly, laudable efforts to increase accountability can lead to consequences that undermine coherence. For practical reasons, accountability efforts typically involve selection of a small number of assessment instruments that carry high stakes for the program. Concentrating attention on a specific test rather than on building a system can lead to unintended consequences. When the results of that test have significant repercussions, one consequence is often that the prevailing instruction and curriculum will come to be significantly affected by the particulars of that test—specifically, by the details of material tested or the formats used. In this situ- ation, gains observed in test results may not represent true gains in learning or progress toward meeting standards. Instead, they may primarily reflect children’s improved ability to respond to items on a particular kind of test. A typical pattern is that test scores in the first years after a new test is introduced will show significant—and publicly celebrated—increases, particularly if high stakes are involved, but these improvements tend to level off after that initial uplift (see Linn, 2003, for a general survey; see also Herman and Perry, 2002, for an example from California). Further evidence of this phenomenon comes from cases in which alternate indicators of the tested skill fail to match the gains shown by the high-stakes test. If children have indeed improved in mathematics, for example, gains should be evident on other indicators of mathematical skill; if not, the gains are suspect. The disjunction between the high-stakes and alternate tests of the same skill has been observed with older children for mathematics (e.g., see Koretz and Baron, 1998, for an example from Kentucky) and is the typical pattern seen when comparing results on state tests to those on National Assessment of Educational Progress (Linn, 2003). Some observers believe that such patterns as these illustrate the limits of what can be achieved primarily through test preparation, and that continuing improvement over the long term will require more meaningful changes in the teaching, learning process, and assessment. These findings suggest the need for a systematic approach in which it is possible to validate gains and the meaning of test scores continuously over time. Assessment by itself cannot improve children’s learning—it is the correct use of assessment information that can bring about that aim. If they are to improve learning, assessments must be based

OCR for page 301
 EARLY CHILDHOOD ASSESSMENT on the early learning objectives and be set in contexts that relate to curriculum and teaching practices that are common in early childhood education. Assessment should appraise what children are being taught, and what is taught should embody the aims of learning described in the standards. Thus, all of the elements in the early childhood education system have to be built on a shared vision of what is important for children to know and understand, how teaching practices affect that knowledge and understanding over time, and what can be taken as evidence that learning and development have occurred (National Research Council, 2001). The following criteria, developed by the committee, opera- tionalize these somewhat abstract principles in important char- acteristics that child outcome measures should have if they are to provide useful evidence for the improvement of early care and education systems. 1. A clearly articulated purpose for the testing. 2. Identification of why particular assessments were selected in relation to the purpose. 3. A clear theory connecting the assessment results and quality of care. 4. Observation of quality of instruction and specification of what would need to be focused on for improvement. 5. A clear plan for following up to improve program quality. 6. Strategizing to collect the required information with a mini- mum of testing. 7. Appropriate preparation of testers to minimize disruptive effects on child responses. A ssessment systems must operate at multiple levels— individual child, classroom, center, school district, state, and national levels. An assessment system is thus sensitive to a variety of influences—some that originate from the top and spread down, and others that work from the bottom up (National Research Council, 2001). Assessments of children must be based on an appreciation of the development and learning of typically developing children and of the typical range of variation for children of any age. This knowledge must be based on the best scientific evidence available,

OCR for page 301
 THINKING SYSTEMATICALLY must be sensitive to the values inherent in choosing to concentrate on specific areas rather than others, and must be completed by sound professional expertise (National Research Council, 2001). An example of an instrument designed according to these prin- ciples is the Desired Results Developmental Profile-Revised, a part of which is illustrated in Figure 10-2. Thus, a successful system of assessments must be coherent in a variety of ways (National Research Council, 2001, 2006).3 It will be horizontally coherent when the curriculum, instruction, and assessment are all aligned with the early learning standards, target the same goals for learning, and work together to support children’s developing knowledge and skill across all domains. It will be vertically coherent when there is a shared understanding at all levels of the system (classroom, center, school or program, and state) of the goals for children’s learning and development that underlie the standards, as well as consensus about the purposes and uses of assessment. And it will be developmentally coherent when it takes into account what is known about how children’s understanding develops over time and the content knowledge, abilities, and understanding that are needed for learning to progress at each stage of the process. Developmental coherence should extend across the boundaries between preschool and K-12 schooling, to ensure that the goals for young children’s learning and development are formulated by taking into account later goals and expectations and with an understanding of how early accomplishments do and do not predict later achievement. These coherences are necessary in the interrelationship of all the subsystems. For example, the development of early learning standards, curriculum, and the design of teaching practices and assessments should be guided by the same framework for under- standing what is being attempted in the classroom that informs the training of beginning teachers and the continuing professional development of experienced teachers. The reporting of assess- ment results to parents, teachers, and other stakeholders should also be based on this same framework, as should the evaluations of effectiveness built into all systems. Each child should have an 3This section on coherence draws heavily upon the content of the National Research Council’s 2006 report, Systems for State Science Assessment.

OCR for page 301
0 EARLY CHILDHOOD ASSESSMENT equivalent opportunity to achieve the defined goals, and the allo- cation of resources should reflect those goals. We emphasize that a system of assessment is only as good as the effectiveness—and coherence—of all of its components.