Grading the Nation's Report Card Research from the Evaluation of NAEP (2000) / Chapter Skim
Currently Skimming:
10 Putting Surveys, Studies, and Datasets Together: Linking NCES Surveys to One Another and to Datasets from Other Sources
10 Putting Surveys, Studies, and Datasets Together: Linking NCES Surveys to One Another and to Datasets from Other Sources
Pages 195-228
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 195... ...
The surveys of paramount interest are those sponsored by the National Center for Education Statistics (NCES)
|
From page 196... ...
Sponsored by the National Science Foundation, the project' s general goal was to improve the way we capitalize on data that bear on educating scientists, mathematicians, and engineers. The book's implications, inadvertent and otherwise, are important for designing NCES surveys, among others.
|
From page 197... ...
It is a fine illustration of combining different datasets so as to learn whether males and females really differ on mental abilities and how they might differ. For instance, the authors' dependence on well-defined national probability samples avoided the inferential problems encountered in earlier studies, notably depending on self-selected samples (as in SAT/ACT testing)
|
From page 198... ...
198 _' 3 o Cq a' be a' a' a' · Cq o Cq a' Cq x VO a' o Cq C)
|
From page 199... ...
We can learn about the role of privacy issues and statistical analysis implications from a related body of work (e.g., Cox and Boruch, 1988~. We learn about appraising the benefits and costs of linkage of administrative records or the difficulty of doing so on account of sloppy practice, from aggressive investigatory agencies such as the U.S.
|
From page 200... ...
The examples are put into categories that have meaning for scientists and an informed public: national probability sample surveys, longitudinal studies, studies of the quality of data, intersurvey consistency, and hierarchical studies. National Probability Sample Surveys Virtually all national probability sample surveys in this country and elsewhere are an exercise in combining information from different systems.
|
From page 201... ...
In the federal statistics arena, most studies of response quality or measurement error require linkage and are described regularly in the professional literature. Scholarly reports usually appear, for instance, in the annual Proceedings of the Section on Survey Methods of the American Statistical Association and in reports issued by the federal agency that sponsored the work.
|
From page 202... ...
Reconciliation studies that illuminate the discrepancies that might be found between two or more independent surveys are important. It would be discomfitting to find a 10 percent difference in the number of teachers in the United States based on one NCES survey, for example, in contrast to another NCES survey undertaken independently and within a year or two of the first survey.
|
From page 203... ...
All in what follows depends on learning from others about what linkage has meant in the context of work sponsored by NCES and others. Vernacular and Definitions in Education Statistics The Hilton (1992)
|
From page 204... ...
The conscientious reader might observe, for example, that Bohrnstedt makes a careful distinction between link and integrate. He refers, for example, to the "integration" of CCD information with NAEP data, and he discusses the possible "linkage" of NELS:88 and NAEP data.
|
From page 205... ...
Witness the rough consensus on using two or three definitions of "dropout," for example. Witness also the NCES surveys of how public schools ask about student's race and ethnicity and the stupefying variety in measurement that then impedes better thinking.
|
From page 206... ...
Sampling frames must be defined similarly, if not identically, for linkage to be easy. For example, a change in sampling frame from one that is entity based to location based led to the need for reconciliation studies in SASS.
|
From page 207... ...
New variables, generated by different sources and observed on the primary sample, are added to the primary sample dataset. For instance, transcripts generated by schools on the courses that students take were added to a primary study that elicits information from students such as HS&B.
|
From page 208... ...
A different sample of a different or the same population using identical measures is put together with a primary sample dataset. The studies combined by Hedges and Nowell (1995)
|
From page 209... ...
See, for instance, the Proceedings of the Survey Research Methods Section of the American Statistical Association for articles on this topic. Questions and Modes of Response: An Illustration Hilton's (1992)
|
From page 210... ...
Organizational Vehicles A national ex-ante strategy requires that an institutional vehicle be exploited to plan for linkage of surveys or other research projects across branches, dimensions, or units in an agency and to plan similarly across independent agencies. In the United States the Interagency Council on Statistical Policy is one vehicle for planning across statistical agencies.
|
From page 211... ...
They include the Interagency Task Force on Child Abuse and Neglect, to judge from recent conferences of the NRC's Board on Children, Youth and Families and the Committee on National Statistics. Federal Statistical Agencies and Federal Agencies with Other Missions In the United States and some other countries, considering ex ante policy on putting datasets together invites thinking about the institutional separation of passive statistical surveys from actively controlled experiments for planning and evaluation programs.
|
From page 212... ...
The third topic, inventing registries of linkages, has implications for enhancing understanding of the first two topics and for evaluating the products of any linkage effort or policy. Mapping Questions and Response Categories We are aware of no comprehensive effort to develop intellectual maps of the specific questions asked in surveys mounted by NCES or other federal statistical agencies in the United States.
|
From page 213... ...
has undertaken to consolidate and analyze information on background variables, how they are asked, and how they might be "harmonized." The multicountry surveys of primary interest in this context are the Eurobarometer (European Commission) , the International Social Survey Programme (29 member countries)
|
From page 214... ...
Consider, for instance, the crude number of response categories for a simple question about whether the respondent works in the four tabulated surveys: Survey Number Number of response categories 5 1 2 3 4 7+ 10 11 They might be displayed so as to emphasize common features for example, ISSP ESOMAR EURO ECHP Self-employed Never employed N N Y Y Y Y Y N Note: ISSP = International Social Survey Programme. ECHP = European Community Household Panel.
|
From page 215... ...
Analyses Analyzing how questions about the same variable differ from one another across surveys, how response categories differ whether and how marginal distributions of responses differ, and so on is a complex matter. Scholars can exploit registries or maps of questions in analyses, thereby adding value to the maps and increasing our understanding of how to measure well and how to link different surveys.
|
From page 216... ...
Without a deeper and more burdensome search, we have no idea whether the questions that address these variables are the same or whether response categories are the same from one question to the next. Nor do we know whether question and response categories are the same as those in surveys cataloged by Brooks-Gunn et al.
|
From page 217... ...
Evaluating the Products of Linkage Evaluation, regardless of its style or method, would have to take into account the purposes of putting datasets together. The purposes and dimensions outlined earlier might be used to organize the effort.
|
From page 218... ...
Linking NCES Surveys Several of the NCES datasets mentioned earlier, including NAEP, SASS, NELS:88, and CCD, contribute in distinct ways to the research and policy-making communities' understanding of a variety of important educational issues. NAEP for example, generates national and subnational estimates of achievement in core subject areas on a regular basis.
|
From page 219... ...
The task of linking CCD and NAEP data was by no means straightforward or seamless, however, because the NAEP dataset did not include the CCD unique identification code for participating school districts or schools. Yet, as Terhanian discovered inadvertently, the NAEP dataset did include the two "broken" pieces (i.e., separate variables)
|
From page 220... ...
The registration process turned out to be flawed, however, because a nontrivial percentage of CCD information was either obsolete or missing (i.e., it seemed "lost". For this reason about 10 percent of the first several hundred JASON registrants could not find their school districts or schools listed among those on the registration Web site menu.
|
From page 221... ...
Linkage and Augmentation of NCES Data and Non-NCES Data At times, states, other federal agencies, and government contractors produce information that can be linked to NCES datasets, including NAEP. For instance, the Pennsylvania Educational Policy Studies Project, which is affiliated with the TABLE 10-2 Linkages Between and Among NCES Datasets Level Data Source District SASS NELS:88 CCD NAEP School SASS NELS:88 CCD
|
From page 222... ...
to which NAEP or other NCES datasets might be linked. The TCI is a district-level index that accounts for factors that underlie differences in the cost of living among school districts (Chambers, 1995~.
|
From page 223... ...
SUMMARY Implication: Electronic Mapping NCES, and perhaps other statistical agencies, can invent a Web-based system for mapping the variables measured in each survey sponsored by the agency (and other studies) , the questions that address the variables, and the question response categories, exploiting hypertext to facilitate the acquisition of deeper information and wider searches.
|
From page 224... ...
It can be exploited by statistical agencies in the linkage context as it is, in effect, in individual survey efforts and formalized.
|
From page 225... ...
1997 Connecting NAEP Outcomes to a Broader Context of Educational Information. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
|
From page 226... ...
In 1992 Proceedings of the Section on Survey Research Methods. Alexandria, Va.: American Statistical Association.
|
From page 227... ...
Pp. 105-l lO in Volume II: 1994 Proceedings of the Section on Survey Research Methods.
|
From page 228... ...
1995 Administrative Record Opportunities in Educational Survey Research. Report prepared for the National Center on Educational Statistics.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.