| ||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 18
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
2
Committee Procedures and Characteristics of Studies Reviewed
As explained in Chapter 1, our charge as committee members was to evaluate the quality of the evaluations of the 13 National Science Foundation-(NSF-) supported and 6 commercially generated mathematics curriculum materials.
We were not charged with and therefore did not:
Evaluate the curriculum materials directly
Rate or rank specific curricular programs
We recognize that both tasks could interest a broad constituency, but we believed that the field would profit from a careful, thorough review and summary of previous evaluations and research studies in relation to how previous work might inform and strengthen future efforts. We were aware that the mathematics education field lacks a clear consensus on what constitutes an effective curriculum and how to measure it to provide adequate, valid, and timely information to decision making bodies. It is appropriate to have a range of curricula from which to choose that represent a variety of preferences and values; when this is the case, decision making on curricular materials inevitably combines values and evidence-based reasoning. We did not intend to recommend the elimination of values in curricular decision making, but instead wished to contribute to efforts to increase the quality of evidence provided to the process.
Some readers may be disappointed by our not offering a “stamp of
OCR for page 19
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
approval” on specific curricula or providing a report card, as others have done for state standards or tests (U.S. Department of Education, 1999; Achieve Inc., 2002). This decision was deliberate. As a committee of the National Research Council of The National Academies, our primary contribution was to clarify the phrases scientifically valid evaluation study and scientifically established as effective in the context of K-12 mathematics curricula. Such an analysis can elucidate the current knowledge of how these curricula were evaluated and help decision makers avoid judgment errors that are likely when the completeness or scientific rigor of evaluations of such materials is misunderstood.
Recognizing the complexity of judging curricular effectiveness, this report is designed to assist future evaluators and policy makers in designing and conducting evaluations that provide accurate, comprehensive, and valid advice to decision makers and practitioners on the efficacy of curriculum materials. Our primary goal was to advise our audiences on what could be learned from these initial efforts and how lessons learned, strategic decisions, adaptations in method, errors and weaknesses, and tentative patterns of results could further future evaluation efforts and decision making on curricular policy.
CURRICULA UNDER REVIEW
The following 13 mathematics curricula programs1 (The K-12 Mathematics Curriculum Center, 2002) were supported by the NSF, and the evaluations of these materials were reviewed by our committee:
Elementary School:
Everyday Mathematics (EM), Grades K-6 (SRA/McGraw-Hill)
Investigations in Number, Data and Space, Grades K-6 (Scott Foresman)
Math Trailblazers, Grades K-6 (Kendall/Hunt Publishing Company) Middle School:
Connected Mathematics Project (CMP), Grades 6-8 (Prentice Hall)
1
Each of the NSF-supported curricula is at least a three-year core curriculum (National Science Foundation, 1989, 1991). A condition of second-year funding for the NSF-supported curricula materials was a firm commitment by a publisher for national dissemination (National Science Foundation, 1989, 1991).
OCR for page 20
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
Mathematics in Context (MiC), Grades 5-8 (Holt, Rinehart and Winston)
MathScape: Seeing and Thinking Mathematically, Grades 6-8 (Glencoe/McGraw-Hill)
MathThematics (STEM), Grades 6-8 (McDougal Littell)
Middle School Mathematics Through Applications Project (MMAP) Pathways to Algebra and Geometry, Grades 6-8 (currently unpublished)
High School:
Contemporary Mathematics in Context (Core-Plus), Grades 9-12 (Glencoe/McGraw-Hill)
Interactive Mathematics Program (IMP), Grades 9-12 (Key Curriculum Press)
MATH Connections: A Secondary Mathematics Core Curriculum, Grades 9-12 (IT’S ABOUT TIME, Inc.)
Mathematics: Modeling Our World (MMOW/ARISE), Grades 9-12 (W.H. Freeman and Company)
Systemic Initiative for Montana Mathematics and Science (SIMMS) Integrated Mathematics, Grades 9-12 (Kendall/Hunt Publishing Company)
Given our expanded charge, we also included a few of the commercially published, non–NSF-funded curricula. We planned to select the curricula by market share; however, such data are highly proprietary and contested. An additional complicating factor was that most reports of market share are identified by publisher name rather than a particular product line. Publishers produce numerous overlapping and sometimes competing mathematics curriculum products, especially given recent acquisitions and mergers. Thus determining market share by program was problematic.
We located two sources of market share data independent of the publishers (Education Market Research, 2001; Weiss et al., 2001). In addition, we received testimonial data from other suppliers of widely used curricular materials in mathematics, including Key Curriculum Press, Saxon Publishers,2 and Texas Instruments. Among the six curricula, we sought representation from each of the four major textbook publishers:
2
Saxon Publishers suggested Simba Information Inc.’s (2002, 2003) Print Publishing for the School Market 2001-2002 Yearly Report and Educational Marketer’s monthly newsletter as sources for market share data.
OCR for page 21
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
McGraw-Hill (including Direct Instruction, Frank Schaffler Publishing, Macmillan, Glencoe, SRA/Open Court, Everyday Mathematics, and the Wright Group)
Reed Elsevier (including Harcourt, LexisNexis, Reinhard and Winston, Rigby, Steck-Vaughn, Reading Recovery, Heinemann, and Riverdeep)
Vivendi (including Houghton Mifflin, McDougal Littell, Riverside Assessments, Sunburst Technology, and Great Source)3
Pearson (including Addison Wesley Longman, Scott Foresman, Silver Burdett Ginn, Simon and Schuster, Globe Fearon, Modern Curriculum Press, Celebration Press, Dale Seymour Publications, Prentice Hall School, Waterford Early Reading, Waterford Early Math and Science, Sing, Spell, Read and Write)
We selected two publishers per grade band level (elementary, middle, and high school). Because our independent sources only identified publishers with the largest market share and not specific mathematics curriculum materials, we asked the publishers to select their curricula with the highest market share. The publishers then submitted the curricular materials and accompanying evaluation studies that they had conducted or were aware of for our review.
We analyzed evaluations of the following six commercially generated programs:
Elementary School:
Math K-5, 2002 (Addison Wesley/Scott Foresman)
Harcourt Math K-6 (Harcourt Brace)
Middle School:
Applications and Connections, Grades 6-8, 2001 (Glencoe/McGraw-Hill)
An Incremental Development, Sixth Grade, Eighth Grade (2nd edition) and An Incremental Development, Seventh Grade, Algebra 1/2 and Algebra (3rd edition) (Saxon)
3
Houghton Mifflin Company was later sold by Vivendi (in December 2002). Houghton Mifflin Company sold Sunburst Technology in October 2002.
OCR for page 22
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
High School:
Larson Series, Grades 9-12, 2002 (Houghton-Mifflin/McDougal Littell)
University of Chicago School Mathematics Project Integrated Mathematics, Grades 9-12, 2002 (Prentice Hall)
Prentice Hall4 was an exception and could not choose its curricular materials because we specifically asked for the secondary materials of the University of Chicago School Mathematics Project (UCSMP) to be part of our review. UCSMP was selected because its history and profile represented a “hybrid” between the two categories (NSF-supported and commercially generated curricular programs), and all of its development and research support for the first edition was provided by foundations.5 We chose UCSMP because, similar to the NSF curricula, its philosophy and program theory are aligned with the National Council of Teachers of Mathematics (NCTM) Standards (NCTM, 1989), although it preceded the NSF-supported curricula in its development period (Thompson et al., 2003). It also differs from the high school NSF-supported materials in that it preserves the traditional course sequence of algebra, geometry, algebra and trigonometry, and advanced mathematics, including newer topics such as statistics and discrete mathematics, whereas the other NSF-supported materials integrate across mathematical subfields at each grade level. UCSMP’s development was situated at a university, unlike any other commercially generated curricula. As a result, many published studies and doctoral dissertations were conducted on it.
DATA GATHERING
Information on evaluation studies of the 19 mathematics curricula projects was gathered in several ways. First, we found contacts for all
4
UCSMP, recently acquired by Prentice Hall, received broad NSF support and its secondary program was first headed by Zalman Usiskin and Sharon Senk. It eventually included five components, including an elementary component that produced Everyday Math (headed by Max Bell, with NSF support, and included in this study), a professional development component with NSF support, and an evaluation component headed by Larry Hedges and Susan Stodolsky. In this review, UCSMP refers to the secondary program only, and Everyday Mathematics is coded as EM. Following our charge, UCSMP is categorized as a secondary commercially generated project, whereas EM is categorized as NSF supported. Both had private foundation funding, and for grades 4 through 6 materials, EM received NSF funding.
5
Amoco Foundation, Carnegie Corporation of New York, and General Electric Foundation.
OCR for page 23
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
curricula under review and requested copies of curricular materials and evaluation studies. We received the requested curriculum materials from all publishers except Harcourt Brace. Seventeen of the 19 curricula submitted public evaluation materials to our committee (except Math K-5, 2002 [Addison Wesley/Scott Foresman] and Harcourt Math K-6 [Harcourt Brace]). We requested that principal investigators from the NSF-supported mathematics curriculum projects send reports they had submitted to the NSF, as well as their own evaluation studies of their materials or others of which they were aware. We also gathered evaluation studies from all four mathematics NSF Implementation Centers (http://www.ehr.nsf.gov/esie/resources/impsites.asp). We then checked citations and bibliography entries from these studies for possible additional evaluations and acquired copies of new studies. Finally, we conducted library and web searches, and emailed several mathematics and mathematics education listservs requesting evaluation studies. We then obtained copies of pertinent studies. A total of 698 evaluation studies were found or submitted for consideration.
We held two evidence-gathering workshops in 2002. The two workshop panels addressed the following questions:
How would you define or evaluate effectiveness of a K-5, 6-8, or 9-12 NSF-supported or commercially generated mathematics curriculum?
What evidence would be needed? Be specific in terms of (1) primary and secondary variables, (2) methods of examining or measuring those variables, (3) research designs, and (4) other relationships under investigation.
The first workshop consisted of panels addressing specific topics:
Evaluation and cross-disciplinary frameworks on curriculum implementation in complex settings;
Developer, researcher, and evaluator perspectives of curriculum effectiveness;
The role of content analysis and research on learning in evaluating curricula effectiveness;
Consideration of professional development needs in curricular implementation; and
Curriculum decision making and evaluation in school settings.
The second workshop on commercially generated materials asked the same general questions, with two additional requests for comments:
How do you evaluate materials in relation to the quality and effectiveness of the materials themselves, including content analysis, theories of
OCR for page 24
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
learning, and teaching? Discuss the role of authors and developers in the process of evaluation.
How does your company consider the issues of implementation in relation to effectiveness, such as professional development, high-stakes tests, standards, technology, equity, and the adoption of materials and marketing issues?
Much of the debate around curriculum quality in the mathematics and mathematics education community resulted in part because content analysis is an ill-defined concept; therefore, we solicited statements on this topic. Sixteen prominent mathematicians and mathematics educators from a variety of perspectives on content analysis were identified. We sent a written request and received statements from eight: Richard Askey, University of Wisconsin, Madison; Eric Gutstein, University of Illinois, Chicago; Roger Howe, Yale University; William McCallum, University of Arizona; R. James Milgram, Stanford University; Luis Ortiz-Franco, Chapman University; Deborah Schifter, Education Development Center; and Hung Hsi Wu, University of California, Berkeley. We asked for their advice on content analysis by addressing the following questions:
What should be included in a content analysis?
How would you judge the quality of a content analysis?
What is the definition of content analysis?
Does your response represent the intended and enacted curriculum?
What references are available in the field on this topic?
THE STUDY MATRIX
We included evaluation studies that focused on one or more of the 13 NSF-supported or 6 commercially generated mathematics curricula, and whose authorship and affiliation were identified. Evaluation studies also had to fall into one of the following categories: (1) Comparative Analysis, (2) Case Study, (3) Content Analysis, (4) Synthesis Study, (5) Background Information, Historical Documentation, or Report to the NSF; and (6) Informative Study (Chapters 4 through 6 provide category descriptions.) We did not wish to limit its initial review to published studies because the topic is relatively current and some papers may not yet have been published. Dissertations would have been excluded if only published studies had been chosen, and we believed these studies could contain useful information. Finally, we sought studies from the following classifications pertaining to curriculum implementation:
OCR for page 25
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
Studies with specific student outcomes
Content analysis studies
Studies of classroom implementation and school environment
Studies of teacher knowledge, teacher characteristics, and professional development
We decided to add to these classifications as we identified additional relevant categories. None were found. The decision tree (Figure 2-1) illustrates the process for categorizing the evaluation studies.
We considered all 698 studies that were found or submitted. If the study met the criteria listed, it was added to the bibliography for review. If it did not meet these criteria, it was placed on a list along with the documented reasons for exclusion. A study whose inclusion was difficult to determine was submitted for committee review. The bibliography of studies that are included in our analysis appears in Appendix B. The 417 studies that met the inclusion criteria for categories 1 through 6 were entered into our database for review.
STUDY CHARACTERISTICS
Table 2-1 shows the distribution of studies by methodology and identifies them as NSF-supported, commercially generated, or UCSMP.
We identified studies that fit the categories of content analysis (n=36), comparative analysis (n=95), case studies (n=45), and synthesis (n=16) as particularly salient. These 192 studies formed the core of the review because they provided direct information pertinent to reviewing the evaluation on materials’ effectiveness. Therefore, a large percentage of studies initially considered did not meet the criteria and were excluded from further review. The categories of background information, historical documentation, reports to the NSF, or informative studies were eliminated from further review, though they remain a valuable source of information about program theory and decision making that affected evaluation study designs.
The number of studies in the commercial category was far smaller than the number of studies on the NSF-supported materials or UCSMP.6 Two factors seem to account for this disparity. First, many NSF- or foundation-supported curricula were required to provide evaluations. The NSF also funded some of these curriculum projects to conduct further evaluation studies. Second, the NSF and UCSMP materials were written primarily by
6
The committee separated UCSMP from the NSF-supported and commercially generated materials because of its hybrid nature.
OCR for page 26
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
OCR for page 27
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
FIGURE 2-1 Decision tree for categorizing evaluation studies.
OCR for page 28
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
TABLE 2-1 Distribution of Types of Studies
Type of Study
Number of Studies
Percentage of Total
1. Comparative analysis
95
13.6
NSF-supported curricula
66
69.5
Commercially generated curricula
16
16.8
UCSMP
11
11.6
Not counted in above
2
2.1
2. Case
45
6.4
NSF-supported curricula
45
100.0
Commercially generated curricula
0
0.0
UCSMP
0
0.0
Not counted in above
0
0.0
3. Content analysis
36
5.2
NSF-supported curricula
17
47.2
Commercially generated curricula
1
2.8
UCSMP
12
33.3
Not counted in above
6
16.7
4. Synthesis
16
2.3
NSF-supported curricula
15
93.8
Commercially generated curricula
0
0.0
UCSMP
1
6.3
Not counted in above
0
0.0
5. and 6. Background information and Informative studies
225
32.2
7. Do not include
281
40.3
TOTAL
698
100.0
university faculty whose graduate students often conducted the studies as part of research toward their degrees. Finally, unlike the NSF-supported materials, commercial publishers often conducted market studies, which emphasize how potential purchasers will view the materials. Thus, many commercially generated studies were only marginally useful in evaluating curricular effectiveness.
The evaluation studies were distributed unevenly across the curricula (Table 2-2). Three of the five curricula with the most evaluation studies under review received additional NSF funding to conduct revisions.7 The elementary, Everyday Mathematics, and secondary components of the UCSMP materials followed.
A database was developed to summarize the studies. Each study was
7
Connected Mathematics, Mathematics In Context, and Core-Plus.
OCR for page 29
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
TABLE 2-2 Distribution of Curricula by Study Type
Type of Study*
Number of Appearances in Any Study
Curriculum Name
1.
2.
3.
4.
5. and 6.
NSF-supported curricula
Elementary
Everyday Mathematics
17
9
7
2
16
51
Investigations in Number, Data and Space
5
1
2
2
9
19
Math Trailblazers
1
1
1
2
6
11
Middle school
Connected Mathematics Project (CMP)
10
18
8
2
42
80
Mathematics in Context (MiC)
1
8
7
5
52
73
Math Thematics (STEM)
2
6
4
2
13
27
MathScape
0
2
1
1
5
9
Middle School
Mathematics Through Applications Project (MMAP)
0
0
0
1
7
8
High school
Contemporary Mathematics in Context (Core-Plus)
13
5
3
3
19
43
Interactive Mathematics Program (IMP)
17
2
4
2
12
37
Systemic Initiative for Montana Mathematics and Science (SIMMS)
5
1
2
2
10
20
Math Connections
2
0
2
2
6
12
Mathematics: Modeling Our World (MMOW/ARISE)
0
0
2
1
5
8
Commercially generated curricula
Elementary
Addison Wesley/Scott Foresman
0
0
2
0
1
3
Harcourt Brace
0
0
1
0
0
1
Middle school
Saxon
13
0
6
0
21
40
Glencoe/McGraw-Hill
1
0
2
0
4
7
High school
Prentice Hall/UCSMP
13
0
14
3
46
76
Houghton-Mifflin/ McDougal Littell
2
0
0
0
1
3
Number of evaluation studies
95
45
36
16
225
417
Number of times each curriculum is in each type of study
102
53
68
29
275
528
*Study types: (1.) Comparative Analysis, (2.) Case Study, (3.) Content Analysis, (4.) Synthesis, (5. and 6., respectively) Background Information and Informative Study.
OCR for page 30
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
read and analyzed by two or more committee members, the Mathematical Sciences Education Board staff, or graduate students trained to search for and record the study characteristics listed in Box 2-1.
After initial review, we studied in depth the first four categories in Table 2-2 because these studies provided detailed evaluation data.
STUDY CHARACTERISTICS FOR CATEGORIES 1 THROUGH 4
Table 2-3 shows distribution of the published studies by type of study. Of the studies reviewed, only 22 percent were published in journals or books. Approximately 28 percent of comparative studies and 31 percent of case studies were unpublished doctoral dissertations. Although dissertations are unpublished, these studies have been vigorously screened and provided valuable insight into current evaluation data on the curricula under review.
TABLE 2-3 Distribution of Published Studies by Type of Study
Type of Study
Number of Studies
Published Study
Unpublished Study
Unpublished Thesis or Dissertation
1. Comparative analysis
95
15
53
27
NSF-supported curricula
66
12
39
15
Commercially generated curricula
16
0
7
9
UCSMP
11
3
6
2
Not counted in above
2
0
1
1
2. Case
45
11
20
14
NSF-supported curricula
45
11
20
14
Commercially generated curricula
0
0
0
0
UCSMP
0
0
0
0
Not counted in above
0
0
0
0
3. Content analysis
36
3
33
0
NSF-supported curricula
17
3
14
0
Commercially generated curricula
1
0
1
0
UCSMP
12
0
12
0
Not counted in above
6
0
6
0
4. Synthesis
16
14
2
0
NSF-supported curricula
15
13
2
0
Commercially generated curricula
0
0
0
0
UCSMP
1
1
0
0
Not counted in above
0
0
0
0
TOTAL
192
43
108
41
OCR for page 31
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
BOX 2-1
Study Characteristics
Author(s)
Title and date of publication
Sample program(s) of interest
Comparison curriculum used program
Design of experiment
Author(s) background regarding study
Version of material
Published? Where?
Unit of analysis
Study of duration
Research question
Outcome measures: Student level
Standardized tests
Other measures (attitudes, absentee rates, or dropout rates)
Outcome measures: Teacher level
Content knowledge
Attitude
Student population: Sample and comparison
Total number of students
Gender
Race/ethnicity
Socioeconomic: Free or reduced lunch
Other
Teacher population: Sample and comparison
Total number of teachers
Hours of professional development received
Use of supplemental materials
Mathematics certified
Average number of years of teaching experience
School population: Sample and comparison
Total number of schools
Staff turnover rate
School location (urban, suburban, rural)
Enacted curriculum, measurement, and findings
Author(s) findings/claims
OCR for page 32
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
TABLE 2-4 Distribution of Author’s Background by Type of Study
Author’s Background Regarding Studies
Type of Study
Number of Studies
Internal
External
Graduate Student
Unknown
1. Comparative analysis
95
35
27
29
4
NSF-supported curricula
66
20
24
19
3
Commercially generated curricula
16
9
1
6
0
UCSMP
11
5
2
4
0
Not counted in above
2
1
0
0
1
2. Case
45
10
14
14
7
NSF-supported curricula
45
10
14
14
7
Commercially generated curricula
0
0
0
0
0
UCSMP
0
0
0
0
0
Not counted in above
0
0
0
0
0
3. Content analysis
36
16
20
0
0
NSF-supported curricula
17
6
11
0
0
Commercially generated curricula
1
0
1
0
0
UCSMP
12
10
2
0
0
Not counted in above
6
0
6
0
0
4. Synthesis
16
16
0
0
0
NSF-supported curricula
15
15
0
0
0
Commercially generated curricula
0
0
0
0
0
UCSMP
1
1
0
0
0
Not counted in above
0
0
0
0
0
TOTAL
192
77
61
43
11
Table 2-4 shows the distribution of the author’s background by type of study. Across the four types of evaluation studies reviewed, 40 percent were done by authors internal to the curriculum project studied. If the study had more than one author, the authors were considered internal if one or more were related to the project (e.g., curriculum developer, curriculum project’s evaluator, and curriculum staff). Twenty-two percent of the studies were conducted by graduate students, who may be internal to the project because often they are the curriculum developer’s graduate students who perform the research studies. Because the relationship of the graduate student to the curriculum project is not always known, it can only be definitively stated that all authors were external to the project in 32 percent of the studies. The relationship of the author to the curriculum project is unknown in 6 percent of studies.
OCR for page 33
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
TABLE 2-5 Distribution of the Study Size by Type of Studies
Study Size (Students)
Type of Study
Number of Studies
0-299
300-999
>1,000
n/a
1. Comparative analysis
95
42
28
23
2
2. Case
45
16
3
3
23
Table 2-5 shows the number of students sampled in each of the comparative and case studies (n=140). Studies with fewer than 300 students made up the largest percentage in both comparative and case studies: 44 and 36 percent, respectively. Only 19 percent of the 140 studies had a student sample greater than 1,000 students.
Table 2-6 shows the distribution of sample school locations by type of
TABLE 2-6 Distribution of Sample School Location by Type of Study
Percentage of Studies That Reported Location*
Type of Study
Number of Studies
Percent of Studies to Report Data
Urban
Suburban
Rural
Only the State/Region
1. Comparative analysis
95
88
36
46
31
37
NSF-supported curricula
66
89
29
38
24
35
Commercially generated curricula
16
81
31
44
19
38
UCSMP
11
91
40
60
40
20
Not counted in above
2
100
100
50
50
0
2. Case
45
71
27
55
30
30
NSF-supported curricula
45
71
27
55
30
30
Commercially generated curricula
0
0
0
0
0
0
UCSMP
0
0
0
0
0
0
Not counted in above
0
0
0
0
0
0
TOTAL
140
84
33
49
31
35
*Many studies report more than one of the following three types of locations: urban, suburban, or rural.
OCR for page 34
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
study. Approximately 88 percent of comparative studies and 71 percent of case studies reported data on school location (urban, suburban, rural, or state/region). Suburban students were the largest percentage in both study types. Rural students were the smallest sample in comparative studies, which implies less information is known about curricular effectiveness in these regions. Most studies did not break down the sample by each of the school location types; thus an exact percentage of school types could not be determined. The data that were reported showed wide variation in demographics, although compared with overall representation in the country, minority populations were undersampled (U.S. Department of Education, 2001).
Content analysis studies are not included in Tables 2-5 and 2-6 because they do not report data on students or sample school locations. Synthesis studies are also excluded because they are summaries of multiple studies and typically did not report data on types of schools or students or include data from only some of the studies considered.
Only 19 percent of comparative and case studies provided detailed information on teachers (e.g., certification, years of teaching, or measures of content knowledge) shown in Table 2-7. Generally, comparison groups were based on matching student and not teacher characteristics. Therefore,
TABLE 2-7 Distribution of Studies That Reported Teacher Data by Type of Study
Percent of Reporting Studies
Type of Study
Number of Studies
Reported Number of Teachers
Reported Teacher Experience*
1. Comparative analysis
95
52
14
NSF-supported curricula
66
56
9
Commercially generated curricula
16
31
19
UCSMP
11
64
36
In 2 categories (not above)
2
0
0
2. Case
45
87
29
NSF-supported curricula
45
87
29
Commercially generated curricula
0
0
0
UCSMP
0
0
0
In 2 categories (not above)
0
0
0
TOTAL
140
63
19
*If a study reported on teacher certification or number of years of teaching, it was counted as “Reported Teacher Experience.”
OCR for page 35
On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations
some bias may be present in the studies in terms of use of volunteer teachers. A substantial percentage of the studies included some measure of implementation by including teacher logs, classroom observations, interviews, and so forth. However, few included any type of measure of quality of instruction, although case studies were more likely to provide insight into these factors than were comparative studies.
Representative terms from entire chapter:
content analysis
Items in cart [0]
