The Importance of Common Metrics for Advancing Social Science Theory and Research A Workshop Summary (2011) / Chapter Skim
Currently Skimming:
3 Indicators
3 Indicators
Pages 31-52
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 31... ...
described his specific perspective on the use of standardized indicators in policy making and decision making. He has experience working for several political leaders committed to using such indicators, including former prime minister Tony Blair, Australian prime minister Kevin Rudd, and the prime minister of Greece, George Papandreou, who is currently addressing a set of issues around harmonization and standardization related to national debt.
|
From page 32... ...
Also, there is a movement from objective facts to subjective measures of experience, for example, fear of crime as well as crime volume, patient satisfaction, and other relational measures of trust and feedback as well as classic health outcomes. He observed that a broader shift to complement output and outcome measures with relationship measures is moving quickly around the world, although with less speed in the United States.
|
From page 33... ...
In addition, he continued, there are also fields in which new indicators are needed, for example, the use of the Internet for public services. Related to this topic, Mulgan reported on a review that he recently conducted on the state of knowledge about behavior change and its relevance to health policy.
|
From page 34... ...
Valuing Social Impact According to Mulgan, it is important to measure social value by creating standardized metrics or tools to compare investments in programs. While the question of measuring social value has been alive in the world of policy since before the mid-1960s, he noted that there have been several
|
From page 35... ...
Mulgan also believes that time horizons, used in standard commercial discount rates, are often very inappropriate for valuing social and environmental goods. Mulgan reported that the Young Foundation has been commissioned by the British Health Service to develop a set of tools for measuring social value and the value of health service innovations, as part of a broader effort to try to guide public services to think about the long-term productivity of specific interventions.
|
From page 36... ...
Having authoritative public bodies make judgments using standardized metrics in transparent ways is as important as having the metrics themselves, and just as important as the recognition that all of these have, in Mulgan's words, "limited half lives." Mulgan ended by sharing his belief that even the best indicator will be useful for a time but will then need to replaced and updated, because that is simply the nature of social knowledge. STANDARDIZED MEASUREMENT In his presentation, Robert Pollak raised concerns about the premature application of standards and the notion that standardization will make for successful science, rather than the idea that successful science generates standardization.
|
From page 37... ...
and the cost of living index to examine the use of standardized measurements for disability. The crucial aspect of having a theory is that it provided a way of dealing with a lot of hard problems that arose in constructing the CPI.
|
From page 38... ...
One model, which he attributed to Dennis Fryback, is an appeal to utility that attempts to identify what people actually value. Another method entails using the theory of disability to predict the probability of nursing home entry within the next year, on which an index of disability could be based.
|
From page 39... ...
Pollak reiterated that although there are different ways to build a framework for constructing an index, the main point is to choose one and to factor in the possibility of biases. HIGH SCHOOL COMPLETION RATES In his presentation, John Robert Warren (University of Minnesota)
|
From page 40... ...
According to Warren, dropout status or completion rates computed from cross-sectional sample surveys are best suited to describing levels of human capital in a population. Because the goal is to describe the share of all individuals who have obtained a credential, it is important to use data that include people who may have gotten those credentials from any number of places: public schools, private schools, GED programs, community colleges, adult education programs, prisons, or the Internet.
|
From page 41... ...
For example, in the previous presentation, Pollak described a method to treat all high school credentials as essentially equivalent; however, this is not necessarily the best approach, because economists have long questioned the relative labor market value of GEDs, and little is known about alternative credentials. To measure a school's holding power, dropout and completion rates need to directly and accurately reflect a specific location.
|
From page 42... ...
Accuracy of Data The third reason Warren outlined for the differences in high school dropout and completion rates has to do with the accuracy of the underlying data used to construct them. Even when the measures are intended to quantify the same thing and even when they agree on the technical definition of the numerator and the denominator, the estimates often differ.
|
From page 43... ...
Three entities are important in terms of determining what race is in America: legal definitions, the Census Bureau, and the Office of Management and Budget (OMB)
|
From page 44... ...
The fact that the Census Bureau has rarely published data for all 126 combinations is evidence that this system is unworkable to produce specifications for congressional redistricting, civil rights, or voting rights enforcement, for example. Other problems have resulted from these race categories: • The inability of federal agencies to agree on which categories or subsets of categories to use for decision making.
|
From page 45... ...
15 are retained and persons reporting two or more races are categorized as "two or more races." One of the flaws of this system is that about 14 percent of the total American Indian population claim Hispanic origin, but this is not reported; Cubans and Puerto Ricans of African descent are also not identified. This system, in fact, undermines the comparability of data with data from agencies adhering to the 1997 standard, such as the Census Bureau.
|
From page 46... ...
The ability to trace the continental origins of human DNA and to connect this information with other genetic traits yields a tempting schematic for measuring race in a way that can be standardized, measured objectively, and is invariant with respect to evolving attitudes and shifting public opinion. However, Snipp warned, genotypes do not necessarily correspond to phenotypes; phenotypic traits observable in the everyday lived experience of race may or may not correspond to the continental origins measured by genetic testing.
|
From page 47... ...
The cost of the census in the United States is unsustainable, and this will result in a shift from its current reliance on survey data to increased use of administrative records and perhaps eventually on digital data. A digital footprint leaves enormous amounts of data and raises questions about what are proprietary data.
|
From page 48... ...
Prewitt commented that the big issue regarding administrative data is the potential ability to cross data sets from education with those on health and social services.
|
From page 49... ...
For example, if the marriage category is defined only as being legally married and living together, then that definition can be used in any data set as an independent and dependent variable. He contended that it is better to have the raw data in order to see what independent variables are correlated with a given definition.
|
From page 50... ...
Willis argued for a two-way flow of information, noting that federally funded Research Data Centers that allow researchers access to restricted data have benefited from the exchange between Census Bureau personnel and academics. Prewitt said that ideally interaction between the producers of administrative data and social scientists would develop in such a way as to yield high-quality data, as well as better program administration from the resulting data.
|
From page 51... ...
The program is likely to be terminated for political reasons and concerns about human rights and privacy. Another example is the history of the unemployment rate in the United Kingdom, which has undergone a range of treatments, from political manipulation to a return to a theoretical measure of surplus labor supply.
|
From page 52... ...
While some Research Data Centers have already started thinking about the relationship between administrative and survey data, they have not yet addressed digital data. Prewitt raised concerns about the quality control of digital data being used by the U.S.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.