National Academies Press: OpenBook

Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy (2014)

Chapter: 3 Prioritizing Measures and Framing a Data Collection Strategy

« Previous: 2 What Should Be Measured?
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

3

Prioritizing Measures and Framing a Data Collection Strategy

3.1. CRITERIA FOR ASSESSING DATA COLLECTION OPTIONS

Several criteria can be considered for assessing the viability of the Current Population Survey (CPS) and its comparative advantage as an instrument for collecting data on social capital. In the first section of this chapter, we discuss some of those criteria—specifically: (1) How accurately and validly can a given component of social capital be measured? (2) What is the nature and strength of the evidence linking measurable elements of social capital with social, economic, and health outcomes? and (3) What is the potential of data sources other than federal surveys to yield comparable or better1 information at comparable or lower cost? Following this discussion, we consider in greater detail the role of causal and correlative evidence in establishing priorities, along with technical survey issues that create some additional data collection constraints.

Accuracy and Validity

Information must be sufficiently accurate to be viewed as credible and to allow researchers to investigate linkages among variables. Part of this criterion is embodied in the question: “Are we measuring what we think

__________________

1“Better” can involve many factors, and we do not pretend such a judgment is easy. Suppose, for example, that a data source allows for more timely and smaller area estimates but is more biased, and the bias is not precisely known. Is this comparable or better? We address some of these issues in Sections 3.3 and 5.1.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

we are measuring—has construct validity has been established?” The concept of “trust” as approached in some of the social capital literature illustrates this point. The General Social Survey (GSS) asks, “Generally speaking, would you say that most people can be trusted or that you can’t be too careful in dealing with people?” Glaeser et al. (2000) examined whether behavior in a trust game corroborates survey-based measures of trust, derived from questions such as this from the GSS, and found that it does not always do so. The authors reached three important conclusions, among others (Glaeser et al., 2000, p. 841): (1) “[S]tandard survey questions about trust do not appear to measure trust…[though] they do measure trustworthiness, which is one ingredient of social capital”; (2) to measure trust, surveys should be redesigned to include “questions about past trusting behavior”; and (3) the most promising strategy for measuring trust (and trustworthiness) is to develop instruments that combine both experiments and surveys.

Other studies (e.g., Bellemare and Kroeger, 2007; Sapienza et al., 2007) have found stronger positive correlations between responses to trust questions and actions in experiments. In an experiment using the German Socio-Economic Panel Study, Naef and Schupp (2009, p. 32) found that survey-derived trust scales tend to measure only one dimension of trust, such as trust in strangers, among the many that are possible and important, such as trust in institutions or an index of trust in known others. The position adopted in much of the experimental economics literature that attitudinal survey questions are poor predictors of trusting actions in games seems, in light of some of these recent studies, slightly premature. Nonetheless, for the central question of this report, the current evidence is suggestive that the CPS supplements are not optimal for generating data for studying complex relationships between trust and other outcomes of interest. In general—beyond trust—more research experiments are needed to interpret what is being measured by questions in surveys such as the CPS Civic Engagement Supplement and to begin understanding the accuracy of the data and their relationship to the underlying concept of interest.

Nature and Strength of the Evidence

Decisions about what data to collect should be guided by the ability of the information to reveal trends in health, crime, employment, resilience to shocks, and other outcomes of interest. Evidence on the importance of explanatory variables generated from pilots, experiments, and small-scale data collections is critical for making these decisions. That is, the utility of a measure in decision making and policy evaluation is a basic crite-

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

rion when making the case for government-supported data collection—particularly in flagship surveys where there is great competition for space. The strength of correlative or causative connections, as well as the perceived importance of the hypothesized outcome, are key criteria for setting data collection priorities. For example, if trust in others in a neighborhood is strongly associated with crime rates and weakly associated with, say, mental health, it suggests that trust may be more useful to measure in a crime and victimization survey than it would be in a health survey. However, if mental health is considered a larger social issue than crime, the weaker linkage for the latter would be offset in determining the focus of data collection resources.

The Potential of Alternative Data Sources

The U.S. Office of Management and Budget and the agencies responsible for the federal statistical system determine standards and guidelines and appropriate content for surveys on an ongoing basis. In addition to such benefits as larger sample sizes, higher standards for methodological transparency and documentation, better archiving and access, and increased likelihood of being repeated over time, government surveys also typically enjoy higher response rates than do those in the private sector. And for some elements this is critical. Information about people’s volunteering activities is an example. Abraham et al. (2009) showed that high response rates are important for measuring volunteerism because people who engage in these activities are also most likely to participate in surveys such as the American Time Use Survey (ATUS); thus selection bias (associated with nonresponse, in this case) would be exacerbated in a low response rate survey. This finding suggests an area of comparative advantage for the CPS Volunteer Supplement.

Administrative data sources—both government and nongovernment—are becoming prominent in the alternative data landscape. Sometimes these data, produced as a by-product from program or other (nonstatistical) needs, can be linked with survey and other data to allow richer analyses than would be possible with survey data alone.2 The optimal data strategy for one data set or survey therefore cannot be sensibly designed without consideration of other elements of the data infrastructure. The ability to link government data sources means that covariate information may not be limited to the fields on the primary survey vehicle. Tax data, Social Security records, and information on program participation are all

__________________

2For example, Chetty et al. (2013) combined administrative tax data from the Internal Revenue Service and local area variables to analyze patterns of intergenerational occupational and earnings mobility.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

examples of administrative data that could contribute to research of questions related to social capital.

The criteria discussed above provide the basis for our recommendations in Chapter 5 about the questions and modules to develop and include in surveys and about the role of the federal statistical system operating in a world characterized by rapidly expanding survey and nonsurvey data collection alternatives. However, these considerations do not provide an unambiguous basis on which to proceed with data collection.

In addition to the task of quantifying issues, decisions about what to include would require weighting each criterion, which is subjective and context dependent. It is not always clear, for example, what would be of greater use: data on a variable that is weakly associated with quality of life (typically considered a very important indicator of people’s well-being), or data on a variable that is a strong predictor of voter turnout (arguably less important to well-being).

Similarly, easy-to-measure indicators are not necessarily the most useful to policy makers or researchers. Current city, state, and national indices of civic health (such as those developed by the National Conference on Citizenship) include dashboards of indicators that often simply reflect what data are available rather than what would be most desirable for research, policy, and public information purposes. For example, voting rates are comparatively easier to measure accurately and regularly than are multidimensional concepts like social cohesion, but that does not mean it is the “right” thing to measure for a given purpose. It is worth asking to what extent are the currently available data elements simply a function of what is feasible to collect, rather than a reflection of what the analytically optimal metrics would be. The answers to these questions are not clear, but these are the tradeoffs that should be considered when developing data collection strategies.

3.2. EVIDENCE OF CAUSALITY AND ASSOCIATIONS—AND POLICY IMPLICATIONS

While its antecedents go back further, much of the modern literature on social capital traces back to Putnam (1993, 2000) and the work of the Saguaro Seminar (see Chapter 1). This literature extends broadly across multiple social science disciplines and into a number of research domains: the social capital of firms (e.g., Humphrey and Schmitz, 1998); the role of trust in neighborhood vitality and safety (e.g., Jacobs, 1961; Sampson and Graif, 2009); and political participation and democracy (e.g., Giugni and Lorenzini, 2010; Verba and Nie, 1972). The work has also covered a range of empirical approaches, including individual and group-level analyses (e.g., Glaeser et al., 2002) employing fixed-effect and instrumental

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

variable models (e.g., d’Hombres et al., 2010; DiPasquale and Glaeser, 1999) and observational and experimental methods (e.g., Naef and Schupp, 2009). Some researchers have focused on developed countries while the interests of others has been on transitional economies (e.g., Narayan and Pritchet, 1999) or cross-national studies (e.g., Gesthuizan et al., 2011).

The impacts of any given element of social capital on measurable outcomes are still largely unknown. Indeed, the nature and strength of the relationships vary over time and across places. In some cases, it is difficult to even distinguish where and when more (or less) of a phenomenon is clearly “good” (or “bad”) and, in turn, whether the policy objective should be to raise or lower it, or by how much. For example, it is not obvious what the optimal levels of group cohesion or of individual connectedness are, especially for situations in which activities create bonds within groups while simultaneously eroding bridges across groups. The same is true with such indicators as divorce rates or income equality. Similarly, the positive returns from being connected with neighbors, or having trust in them, almost certainly differ in remote villages and large cities.3 These complexities notwithstanding, social capital research has produced valuable insights (which we document next) and advanced understanding of a range of social phenomena covering a broad range of topics in the social, health, and economic policy domains.

Illustrative Studies

Statistical agencies, in consultation with the Office of Management and Budget and with legislators, determine the content of the CPS and other major surveys modules.4 In making those decisions about social capital content, they need to answer the question, “what data have been most usefully applied in studies of and policies related to civic engagement, social cohesion, and other aspects of social capital?” In this section we selectively review the literature to provide an indication of the breadth and quality of evidence tying various components of social capital to

__________________

3Some aspects of social capital have been shown to be higher in rural than urban areas (Coleman, 1990; Knowles and Anker, 1981; Krishna and Uphoff, 1999; Narayan and Pritchett, 1999; Putnam, 2000), even though social connections between people decrease substantially with physical distance and transportation costs (Glaeser et al., 2002). These differentials are likely changing in step with the expansion of communication modes (cell phones, Internet) that have radically reduced the costs of “connectedness,” especially in remote areas.

4Standards for new items to be included in surveys generally dictate that they have a proven track record in other (academic or smaller) surveys or be put through a rigorous testing process. However, agencies will usually accept items that have been shown to work. Prior testing of many of the elements of the questions on the CPS Civic Engagement Supplement took place in the Social Capital Community Benchmark Survey, which saved time in development.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

outcomes in social, health, and economic policy domains (essentially criterion 2, above).5 This review is suggestive of how social capital relates to measurable individual and societal outcomes; it also assesses the state of development of research on the topic and where needs exist for more data and research. The domains (each with at least some policy relevance) discussed are connectedness and social outcomes, the effects of neighborhood social capital on crime and public safety, social cohesion and community resiliency, home ownership and civic engagement, social connections and self-reported well-being, the health effects of isolation, and social capital and mental illness.

Connectedness and Employment Outcomes

Extensive research exists on the role of social contacts in obtaining jobs, much of it suggesting causal links (e.g., Granovetter 1995; Loury 2006). According to Ioannides and Loury (2004), the use of personal networks in job search is highly prevalent, with 25 to 80 percent of jobs obtained through personal networks (as opposed to applying through employment agencies or approaching employers without referral)—though jobs may more often be found through “weak-ties” (acquaintances) than “strong-ties” (family and friends) (Granovetter, 1973). A literature review by Mouw (2006, p. 82) focused on this kind of network social capital—specifically, claims that “the characteristics and resources of friends, contacts, and groups may affect individual outcomes”—because the problem of causality in this area is particularly clear. He argued (p. 80) that “much of the estimated effect of social capital simply reflects selection effects based on the myriad nonrandom ways in which people become friends” and discussed ways in which progress has been made in dealing with nonrandom selection due to homophily—the tendency of people to associate and bond in nonnegative ways with similar others.6

Mouw (2006) reviewed a number of studies that employ inventive identification strategies to generate statistical evidence of the effect of connectedness on various outcomes. For example, in order to examine the extent to which the strength of people’s social networks affects their

__________________

5For more comprehensive reviews of the social capital literature (of which there are many), see Portes (1998) on its origins and applications in sociology and Halpern (2004) on social capital of interest to policy communities. A number of reviews conducted by international agencies—especially the UK Office for National Statistics OECD—are also available.

6There is also substantial research on peer effects is outside the employment literature. For example, Kremer and Levy (2008) explored peer effects (associated with drinking) and college achievement (GPA) using data on randomly assigned roommates; Duncan et al. (2005) examined the impact of peer effects on alcohol and drug use using quasi-experimental data from randomized housing studies.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

employment opportunities, Bayer et al. (2004) used census data for Boston to show that a person’s residential proximity to others with jobs and who can easily share job information leads to employment opportunities. A block-group fixed-effects model was developed to test for reverse causality—that is, the possibility that coworkers share information when searching for houses or apartments. The authors took measures to properly identify the effect of interest by restricting data to respondents who lived in the neighborhood for at least 2 years and who worked at their current job less than 40 weeks the previous year. Such efforts involving creative use of data can begin to get at the direction of effects—in this case, between connectedness and employment opportunities.

Researchers have examined this relationship between connectedness and employment outcomes in the context of immigrant integration by looking at interactions between characteristics of destination communities and outcomes of those who have located there. Van Kemenade et al. (2006, p. 19) found that “having access to close networks of people from the same cultural origin—as well as to programs that support these networks—is associated with the social and economic integration of immigrants in the host county and with their well-being.” Munshi (2003) found that the network size of immigrant communities has a substantial effect on employment probabilities among Mexican immigrants.7

In terms of policy implication, the above findings may be interpreted as ambiguous. If—unlike public health, social trust, crime rates, or happiness—employment is a zero-sum game, such that connectedness does not increase the number of employment opportunities in the aggregate; rather it only influences who gets a job, presumably those with stronger connections. In this case, public policy seeking to increase connectedness would only alter how employment outcomes are distributed. If government intervention increased connectedness uniformly, perhaps nothing would change. If it equalized connectedness among people, the factor would merely be minimized as a meaningful variable in employment outcomes. Again, this observation may be particularly relevant to policies in the contentious immigration debate. A program to improve immigrant connectedness to new communities could lead to (or be perceived to lead to) an immigrant taking a job that could have gone to a native worker. Colussi (2013) explored the role of immigrant social networks and job search outcomes.

__________________

7Elsewhere, Ooka and Wellman (2006) found that educational attainment is positively associated with being in heterogeneous friendship networks; first generation immigrants with postsecondary education were found to be more likely to be in a heterogeneous network than those with less education. Hagan (1998) documented the role of networks in Houston’s Latino immigrant communities. Massey et al. (1993) is a seminal work that depicted the role of networks in migration.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

Effects of Neighborhood Social Capital on Crime and Public Safety

Communities or neighborhoods in which people have high levels of interaction and trust have been shown to be more immune to social ills, such as crime, and more likely to share resources for the general good (Sampson et al., 1997). The literature on determinants of crime examines many aspects of the issue: community structure and policing and crime (Sampson and Groves, 1989); the role of neighborhood-level collective efficacy—defined as “social cohesion among neighbors combined with their willingness to intervene on behalf of the common good—in reducing violent crime” (Sampson et al., 1997); social order and violence (Sampson et al., 2008); the relationship between differential social organization, collective action, and crime (Matsueda, 2006); and the role of disadvantage and institutions in neighborhood violent crime (Peterson et al., 2000). Studies about cities or regions provide a deep understanding of what can be learned about complex phenomena that shape people’s communities and cities.8 Such research underscores the need for specialized, subnational level data projects for understanding local area phenomena.

The area of crime provides an excellent case study of how social capital variables can play either the role of cause, effect, or both and of other complicating methodological factors, such as selection effects. It is easy to tell a story about how certain neighborhood characteristics create the environment for crime. However, there may be circular, feedback mechanisms at work as well. When a neighborhood carries a reputation as unsafe, having poor schools, and lacking social amenities, higher income households have the means to look elsewhere to live (or to leave), which can in turn lead to further deterioration as measured by some set of social capital indicators. Sampson et al. (2002) addressed the most pressing methodological problems encountered in the study of neighborhood effects—most notably selection bias—and concluded that approaches for dealing with them require experimental designs and observational approaches that deal directly with spatial and temporal dynamics of social processes.

Halpern (2004) pointed out that high crime is not just limited to poor neighborhoods, but also to areas of low social capital and high mobility—that is, a “high degree of accessibility” created by the presence of major thoroughfares and permeable boundaries. High crime areas, he continued tend to be characterized by less social cohesion, as is the case when fewer neighbors know or trust one another. Halpern acknowledged

__________________

8One outstanding example is a major study, funded by the Russell Sage Foundation, of evidence about the social, cultural, political, and economic lives of second-generation residents of New York City, comparing how they fare relative to their first-generation parents and native-born counterparts (Kasintz et al., 2008).

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

and explored the difficulty of determining the extent to which low social cohesion leads to higher crime, and vice versa, and the more subtle question, “could it be that the accessibility, and perhaps social mix, of certain neighbourhoods cause both higher crime and lower social cohesion independently?” (p. 124). He conceded that the direction of these effects is difficult to disentangle while acknowledging that the work of Sampson et al. (1997) on Chicago neighborhoods using localized surveys and other data sources, along with multilevel modeling methods, gets closest to doing so—specifically showing convincingly that collective efficacy does reduce crime through a number of mechanisms.

Social Cohesion and Community Resiliency

A relatively new research field is emerging to address relationships between a community’s characteristics and infrastructure, both physical and social, and its preparedness for disasters and other exogenous shocks. Implicit in such reports as Disaster Resilience: A National Imperative (National Research Council, 2012) is the idea that nations and communities have much to gain (or avoid losing) by investing in infrastructure—both physical and social—that enhances resilience to natural and human-caused disasters. Much of this research involved recognizing the role and importance of social capital in the process of a community’s reaction. For example, this factor has been hypothesized as playing a key role in why New Orleans suffered so much graver and persistent consequences post-Katrina than did Vermont after the damaging 2006 floods.9

Although social capital indicators are often correlated with income, inequality, marital status, socioeconomic status, and other objective measures related to people’s well-being, Sampson (e.g., 2012) and others have shown that community resilience and flourishing is “not wholly a dependent variable of the income and education of the community’s residents” and that there are examples of low-income communities that demonstrate more collective efficacy than high-income communities.10 Disentangling these effects is the challenge in this research. The work done in connection with the 1995 Chicago heat wave is a good example of convincing evidence generated through a well-documented natural experiment. The research found that neighborhoods showed differential resiliency; death

__________________

9For an overview of this research, see Klinenberg (2013).

10“Collective efficacy” is a term that can be applied beyond the context of neighborhoods; it can be relevant to collective interests based on class, race/ethnicity, gender, citizenship, or age. Also, as described in the Introduction, there are cases in which highly fractured pursuits of collective efficacy undermine social cohesion; civic engagement and social cohesion do not always go together. For instance, the civil rights and women’s movements were forms of civic engagement that were accused of undermining social cohesion.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

tolls varied dramatically across neighborhoods with similar per capita incomes but with different social structure characteristics.

Home Ownership and Civic Engagement

A number of researchers have investigated the hypothesis that home ownership gives people higher stakes in a community and more incentive to invest time and effort to its functioning and livability, although the results from this research have not been consistent. Data from the General Social Survey (GSS) and the American National Election Survey (ANES) revealed that homeowners report higher rates of voter participation, political knowledge, and associational memberships than do renters (Blum and Kingston 1984; DiPasquale and Glaeser, 1999; Rossi and Weber 1996). And a study of “the influence of home ownership and mobility on civic engagement among low-to-moderate income households” found evidence that homeowners are more likely to participate in some types of civic engagement, but that the relationship between home ownership and hours of volunteering was not significant (Paik, 2013). Using CPS data, McCabe (2013) showed weak links—relative to education, residential stability, and income—between ownership and voting or civic engagement, calling into question tax policies favoring home ownership, as well as programs that promote low-income home ownership.

Social Connections and Self-Reported Well-Being

Self-reported (subjective) well-being has been shown to correlate strongly with people’s connectedness with friends and family and with their neighborhood’s characteristics. Stiglitz et al. (2009, p. 183) assessed the evidence:

Much evidence at both the aggregate and individual level suggests that social connections are among the most robust predictors of subjective measures of life satisfaction. Social connections have a strong independent effect on subjective well-being, net of income. Moreover, the available evidence also suggests that the externalities of social capital on wellbeing are typically positive, not negative (Helliwell, 2001; Powdthavee, 2008). In other words, increasing my social capital increases both my own and my neighbors’ subjective well-being, and thus represents a coherent strategy for improving QoL [quality of life] for the country as a whole.…The analysis of the effects of social connections on subjective well-being is in its infancy. Much of it does not account for unmeasured individual characteristics, and most of it relies on cross-sectional data. That said, recent analyses have strengthened the case that the link between at least some forms of social connections and subjective well-being is causal. Krueger et al. (2009) report that, when controlling for individual fixed

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

effects (such as personality traits), most pleasurable activities involve socializing—religious activities, eating/drinking, sports, and receiving friends. Similarly, in a recent large-scale U.S. panel survey on religious attendance and subjective well-being, Lim and Putnam (2008) found that religious attendance at time 1 (or time 2) predicted subjective well-being at time 2, controlling for levels of subjective well-being at time 1, as well as many other covariates; the essential mechanism involved in this relation is neither theological nor psychological, but rather the strong effect of “friends at church” on well-being. Fowler and Christakis (2008) also report evidence suggesting that subjective well-being can spread in a beneficially “contagious” way from one person to another.

The authors concluded that, “for no other class of variables (including strictly economic variables) is the evidence for causal effects on subjective well-being probably as strong as it is for social connections.”

The evidence is far from complete on these questions, however. There have been some highly visible critiques in the literature regarding causal claims—such as those by Fowler and Christakis (2008) that were based on their analysis of Framingham Heart Study participants—about the relationship between personal networks and self-reported happiness or other outcomes.11 Much of the debate about the Fowler and Christakis article was on the effects of social networks on propensity toward obesity. Lyons (2011) found evidence of this transmission mechanism—for example, if a person’s close contact became obese, the person himself was more likely to become obese—to be weaker than initially claimed. Lyons’ interpretation of the data led to the conclusion that shared environments and self-selection both explain the clustering of obesity in social networks—that is, people with lifestyles conducive to obesity may well gravitate toward one another. While debates about both descriptive inferences and the causal implications are extremely important, the central point here is that analyses such as the one by Fowler and Christakis are particularly valuable for investigating causal effects because of their longitudinal structure.

The Health Effects of Isolation

The links between cohesion, connectedness, and other aspects of the social environment and population health outcomes are among the best established by research, and the evidence accumulating from this research is expanding rapidly and convincingly. This research goes further than in many other domains in that it is suggestive of pathways between social contacts (or isolation) and health, particularly for elderly people

__________________

11This survey indirectly generated data on social networks in that it asked participants to name a friend who could help researchers locate them in the case that they moved.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

(Wilkinson and Marmot, 1998). Longitudinal data (such as those exploited by Steptoe et al., 2013) on individual characteristics and behavior are needed to distinguish between codeterminants and effects; for example, if isolation leads to depression and illness or if less healthy people choose more isolated lives.

Elements of social capital may also be used to deter unhealthy activities, such as drug use and alcoholism (Frank et al., 2006; Sampson et al., 1997). However, this work is complicated because the analyses has to be able to separate out material and economic determinants of health, which may be highly correlated with the presence of high social capital characteristics in a society.12 A recent meta-review examined 148 research studies on social relationships and mortality risk (Holt-Lunstad et al., 2010). The authors noted that rapid growth in research on the links between social relationships and mortality was triggered by House et al. (1988, p. 541), who proposed a causal association between the two: “Social relationships, or the relative lack thereof, constitute a major risk factor for health—rivaling the effect of well-established health risk factors such as cigarette smoking, blood pressure, blood lipids, obesity and physical activity.” Holt-Lunstad et al. (2010, p. 14) ultimately interpreted the evidence as supporting the 1988 claim by House et al.:

Data across 308,849 individuals, followed for an average of 7.5 years, indicate that individuals with adequate social relationships have a 50% greater likelihood of survival compared to those with poor or insufficient social relationships.…The overall effect remained consistent across a number of factors, including age, sex, initial health status, follow-up period, and cause of death, suggesting that the association between social relationships and mortality may be general, and efforts to reduce risk should not be isolated to subgroups such as the elderly.…This meta-analysis also provides evidence to support the directional influence of social relationships on mortality. Most of the studies (60%) involved community cohorts, most of whom would not be experiencing life-threatening conditions at the point of initial evaluation. Moreover, initial health status did not moderate the effect of social relationships on mortality. Although illness may result in poorer or more restricted social relationships (social isolation resulting from physical confinement), such that individuals closer to death may have decreased social support compared to healthy individuals, the findings from these studies indicate that general community samples with strong social relationships are likely to remain alive longer than similar individuals with poor social relations.

__________________

12The intertwined social capital, distribution of resources, and economic effects on health are discussed in Altschuler et al. (2004) and Islam et al. (2006).

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

They conceded, however, that:

[C]ausality is not easily established. One cannot randomly assign human participants to be socially isolated, married, or in a poor-quality relationship. A similar dilemma characterizes virtually all lifestyle risk factors for mortality: for instance, one cannot randomly assign individuals to be smokers or nonsmokers. Despite such challenges, “smoking represents the most extensively documented cause of disease ever investigated in the history of biomedical research.” The link between social relationships and mortality is currently much less understood than other risk factors; nonetheless there is substantial experimental, cross-sectional, and prospective evidence linking social relationships with multiple pathways associated with mortality. Existing models for reducing risk of mortality may be substantially strengthened by including social relationship factors.

Holt-Lunstad et al. (2010, p. 14) drew a parallel to research on high mortality rates among infants in custodial care (i.e., orphanages):

Even when controlling for pre-existing health conditions and medical treatment…lack of human contact predicted mortality.…This single finding, so simplistic in hindsight, was responsible for changes in practice and policy that markedly decreased mortality rates in custodial care settings. Contemporary medicine could similarly benefit from acknowledging the data: Social relationships influence the health outcomes of adults.…Efforts to reduce mortality via social relationship factors will require innovation, yet innovation already characterizes many medical interventions that extend life at the expense of quality of life.

In a study of the effects of individuals’ social relationships and their physical health and the mechanisms through which influences may work, Cohen (2004, p. 677) concluded that social support is integral to stress buffering:

[It] eliminates or reduces effects of stressful experiences by promoting less threatening interpretations of adverse events and effective coping strategies.…[Social integration] promotes positive psychological states (e.g., identity, purpose, self-worth, and positive affect) that induce health-promoting physiological responses; provides information and is a source of motivation and social pressure to care for oneself.

However, he pointed out that relationships can also create negative interaction that “elicits psychological stress and in turn behavior and physiological concomitants that increase risk for disease” (Cohen, 2004, p. 677).

A study of international differences in mortality at older ages (National Research Council, 2011) illustrated the difficulty of establishing the relationship between health and social factors more generally. The

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

study used data from the English Longitudinal Study of Ageing (ELSA) and the U.S. Health and Retirement Survey (HRS), but the differences in societal characteristics are small in the two countries. Ideally, to uncover effects, one would need to look at countries with bigger differences that also have high quality and comparable data. A major element linked to health outcomes seems to be whether or not elderly people are connected strongly enough to friends and family to have a support structure, that is, to avoid isolation.

For measuring isolation, the question content in HRS and ELSA includes a sufficiently deep set of variables to allow multidimensional “indexes of isolation and loneliness” to be calculated. In a study of social isolation, loneliness, and all-cause mortality in older men and women, Steptoe et al. (2013) constructed such an index for individuals in the sample based on their responses to questions about three factors: marital or cohabiting status; contact with children, other family members, and friends; and their participation in various clubs, organizations, and groups. They concluded (p. 5797) that “both social isolation and loneliness are associated with increased mortality, but it is uncertain whether their effects are independent or whether loneliness represents the emotional pathway through which social isolation impairs health.” In a similar study for the United States using HRS data, Coyle and Dugan (2012) found that the proportion of Americans who reported they had no one to talk to about important matters rose from 10 percent in 1985 to 25 percent in 2004. The authors suggest that this finding argues for policies to increase social connection and support for the elderly, especially as populations have become more solitary.

Finally, work on the social environment as a health determinant is also proceeding in the physical sciences. Dobbs (2013) summarized research demonstrating measurable effects on the human immune system associated with people’s social lives, quoted biologist Steve Cole: “We typically think of stress as being a risk factor for disease.…And it is, somewhat. But if you actually measure stress, using our best available instruments, it can’t hold a candle to social isolation. Social isolation is the best-established, most robust social or psychological risk factor for disease out there. Nothing can compete.” Continuing, Dobbs wrote:

This helps explain, for instance, why many people who work in high-stress but rewarding jobs don’t seem to suffer ill effects, while others, particularly those isolated and in poverty, wind up accruing lists of stress-related diagnoses—obesity, Type 2 diabetes, hypertension, atherosclerosis, heart failure, stroke. Despite these well-known effects, Cole said he was amazed when he started finding that social connectivity wrought such powerful effects on gene expression.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

Social Capital and Mental Illness

In an examination of links between social capital and mental illness in 21 studies, DeSilva et al. (2005, p. 619) found that the evidence was strongest for “an inverse association between cognitive social capital and common mental disorders”; evidence was less convincing for establishing associations between cognitive social capital and child mental illness and combined measures of social capital and common mental disorders. Some of the studies reviewed use individual-level measures of social capital (e.g., respondents’ rating of trust in others, or their self-reported participation in organized activities); others use “ecological” indicators of social capital taken from an aggregated statistic (e.g., the crime rate in a neighborhood or turnout in an electoral ward). DeSilva et al. concluded that “the strength of the current evidence, in particular that from studies measuring ecological social capital, is inadequate to inform the need for or development of specific social capital interventions to combat mental illness.” They recommended (p. 626) that the current methodological and empirical weakness could begin to be addressed by a research program that includes the following steps: “(1) Measure all dimensions of social capital—that is, cognitive, structural, bridging, bonding, and linking; (2) Use validated social capital measures; (3) Be explicit about causal pathways between social capital and mental illness; (4) Examine associations longitudinally; (5) Research developing world and rural populations.”

Since the DeSilva review, Welsh and Berry (2009, p. 588), using the Household, Income and Labour Dynamics Survey in Australia, found that “structural and cognitive components of social capital were each related to both mental health and satisfaction with a wide range of aspects of life…[and that] social capital was better at predicting mental health scores for men than for women, but the opposite was true for satisfaction.” Similarly, Berkman and Glass (2000) found that mental health may be affected through such pathways as provision of social support and promotion of healthier behaviors. Given the current state of evidence, one could reasonably conclude that the relationship is unlikely to be unidirectionally causal from social capital to mental illness; thus, at this point, the policy implications are still unclear.

Social Capital and Educational Outcomes

Research on the relationship between social capital and educational outcomes has a long tradition, dating back at least to Coleman (1988) who studied the effects on communities when social networks are “closed.” One of his key findings was that test scores were better in schools where teachers knew many of the students’ parents and vice versa—that is,

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

where both were part of students’ networks. The networks closed when teachers and parents knew each other.

Analogous with other research areas described in this section, it is difficult to decipher the extent to which social capital in students’ communities leads to school success and the extent to which stronger social ties tend to emerge in more successful schools. However, promising causal modeling methods are becoming more commonplace. Lopez Turley et al. (2012, p. 9), for example, tested the effectiveness of the Families and Schools Together (FAST) Program, “designed to develop relations of trust and shared expectations among parents, school staff, and children” and to improve children’s outcomes, specifically the reduction of behavioral problems. Their study follows a cluster-randomized design in which the researchers were able to assign half of a sample of 52 schools (drawn from San Antonio and Phoenix) to participate in FAST and the other half to operate as usual, without the program. Results from the experiment’s multilevel models revealed (Lopez Turley et al., 2012, p. 1):

…strong positive effects of treatment assignment on parent social capital and more modest but statistically significant effects on reducing children’s behavioral problems. Complier average causal effect (CACE) models show that the strongest effects on parent social capital occurred for families that participated fully in the intervention, whereas the CACE models were less consequential for child outcomes. Instrumental variables models suggest that the social capital effects may be regarded as causal, and causal mediation models suggest that the intervention effects on child outcomes are mediated by social capital.

Compiler average causal effect (CACE) modeling techniques build on the Angrist, Imbens, and Rubin (Angrist et al., 1996) instrumental variable methods and are designed to generate unbiased estimates of the difference in outcomes for a group of compliers of an intervention with those who could have but did not engage in a treatment. These methods have been used extensively in randomized controlled trials to examine effects for children engaged or not engaged with interventions. This and similar techniques can be extended to other applications; the effects of job training on job search outcomes for the unemployed is one example explored by Yau and Little (1996). While CACE models involve challenging statistical assumptions, the inherent structure is often of policy interest because it allows examination of the effects of an intervention for groups of individuals who receive treatment services.

The important point for the discussion here is that methodological advances in statistical techniques, such as CACE and mixture modeling methods, create opportunities for research on social capital to make advances in addressing causality. Experimental manipulation, such as in

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

the studies cited above, offers a methodological pathway for testing the causal effects on outcomes from various dimensions of social capital.

Implications from the Research

Our interpretation of this literature is that—with the exception of social isolation as a risk factor for health—compelling evidence of causal relationships between social capital indicators and outcomes of policy interest has not yet been established, though insightful information about correlative associations often has been. Conceptual ambiguity of the term “social capital,” as described above, and the fact that empirical work on the topic has primarily been limited to correlational analyses, make it difficult to distinguish whether “social capital is a reflection of unobserved variables, a matter of selection (individuals who are alike tend to associate with one another), or a matter of influence (social capital and behavioral outcomes are causally related)” (Lopez Turley et al., 2012, p. 1). A central example of the chicken-and-egg problem is the question: Do successful groups succeed because they have lots of social capital or do successful groups surround themselves with social capital because they have the means to do so? Or, as posed by Durlauf (1999, p. 3): “[D]o trust-building social networks lead to efficacious communities, or do successful communities generate these types of social ties?”

Although the study of social capital seems particularly difficult, understanding causal properties is challenging in many areas of social science. Heckman (2000, p. 91) described the economics case:

Some of the disagreement that arises in interpreting a given body of data is intrinsic to the field of economics because of the conditional nature of causal knowledge. The information in any body of data is usually too weak to eliminate competing causal explanations of the same phenomenon. There is no mechanical algorithm for producing a set of ‘assumption free’ facts or causal estimates based on those facts.

The problem of establishing causality is found in Putnam’s work as his measures of social capital were highly correlated with good educational outcomes (higher income), good health, and well-functioning government (Sobel, 2002, pp. 141-142). Putnam acknowledged this, but much of his work took the tone that higher levels of social activities led to good outcomes. For example, he wrote (Putnam, 2000, p. 328): “e.g., if one wanted to improve one’s health, moving to a high-social capital state would do almost as much good as quitting smoking.”

Durlauf (2002, p. 464) examined the way in which empirical evidence has been developed in investigations of the link between social capital and socioeconomic outcomes. His focus was on the econometric issues

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

that arise in studies of social capital, which “typically compare outcomes for individuals or aggregates who have social capital versus those who do not.” These studies, he argued, are hamstrung by the problem that, “without a theory as to why one observes differences in social capital formation, one cannot have much confidence that unobserved heterogeneity is absent in the sample under study.”13

Durlauf was clear that empirical studies in social science—he used Furstenberh and Hughes (1995), Narayan and Pritchett (1999), and Knack and Keefer (1997) as exemplars—are not typically “right” or “wrong”; rather, they offer evidence of causal links of varying strength. This, he argued, is also the case for research on social capital and socioeconomic outcomes which, for the most part, fails to distinguish between social capital effects and those associated with other individual and contextual or endogenous effects such as income, mobility, and education. He added that the definitional ambiguity underlying “social capital”—which makes identification impossible and has led to questionable validity of instrumental variables and untenable exchangeability assumptions—has exacerbated the causality problem for this field of research (Durlauf, 2002, p. 474):

…the literature seems to be particularly plagued by vague definition of concepts, poorly measured data, absence of appropriate exchangeability conditions, and lack of information necessary to make identification claims plausible. These problems are especially important for social capital contexts as social capital arguments depend on underlying psychological and sociological relations that are difficult to quantify, let alone measure. These problems suggest…in using observational studies…that researchers need to provide explicit models of the codetermination of individual outcomes and social capital, so that the identification problems that have been analyzed may be rigorously assessed.

Durlauf (2002) concluded that studies have not yet established empirically the importance of social capital in explaining various socioeconomic outcomes (p. 459) and that observational data does not go far in establishing an evidence base tying social capital variables to important social, economic, and health outcomes (p. 477). On the second point, he noted (p. 477):

…in light of the vagueness of the concept, I believe that the use of observational data to identify substantive forms of social capital is unlikely to be successful. The relatively more compelling evidence from the social

__________________

13For a discussion of the obstacles in econometric modeling of social interactions, see Manski (2000).

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

psychology literature, in contrast, suggests that economic experiments may be a more promising way to obtain empirical insights.

To establish causal links, Durlauf, Sobel, and others argued that social psychological experiments, such as that reported by Glaeser et al. (2002) in a study of trust, hold more promise for establishing social interaction effects related to trust and other social capital elements. Durlauf (2002, p. 475) cited, as a good example of the kind of detailed data needed to truly understand how social capital (which is concentrated mainly at localized geographic units), the Project on Human Development in Chicago Neighborhoods:

[The project is] designed to produce a rich data set on attitudes among Chicago residents on a wide range of issues. In 1995, over 8,000 individuals were surveyed across over 300 neighbourhood clusters. What is critical in the study is the rich set of information that is produced which allows for the integration of information about individual characteristics with information on individual attitudes in order to study how these relate to communities, i.e., the social environment. This data set has provided insights into a very wide range of phenomena.…Sampson et al. (1999), for example, find that even if one restricts attention to poorer neighbourhoods, there is wide variation in the residents’ expectations of the behaviour of their neighbours and that this variation helps predict differences in neighbourhood social problems. For example, for poor neighbourhoods where individuals feel unable to rely on neighbours to report truancy or call the police in response to observing illegal activity, various social pathologies will be more serious. This sort of finding in turn is very suggestive of the role of community institutions in ameliorating social problems and indeed fulfils the authors’ objective of moving beyond the typical vague formulations of social capital….

And:

Relative to standard empirical analyses of social capital, this work has several advantages. First, the data set gathered in this project provides much richer controls for individual heterogeneity than are typically available. Second, the detailed attitudinal measurements in the study extend social capital analyses in directions that are far more conducive to the description of the causal mechanisms by which social capital is created. The expectation of neighbours’ behaviour which Sampson et al. describe gives a far more compelling vision of the role of community networks in influencing group outcomes than a cross-country regression of growth rates on vague measures of trust. Third, the detailed nature of the study may provide ways to characterise the endogenous formation of social capital, something that is critical for establishing identification of social capital effects.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

Studies based on highly granular, ongoing, and multisource datasets appear to offer the greatest promise for untangling the circularity of causal pathways—e.g., to what extent does deterioration of job growth in a city lead to social problems and desolation, and vice versa; to what extent does connectedness lead to reduced crime, and to what extent does reduced crime lead to connectedness—and to consider the extent to which engagement and cohesion are just symptoms. This kind of intensive empirical analysis allows for investigation of the causes of social capital and not just the effects of social capital on outcomes, an issue raised by Glaeser et al. (2000), who considers the theoretic and empirical evidence on the formation of capital.

Our assessment of implications for data collection from the above literature can be summarized as follows:

  • Although the social capital literature is extensive and provocative, it has yielded numerous compelling observations and correlations and has produced claims very much worth studying. The evidence tying its essential components to specific social, economic, or health outcomes in a causal way is a work in progress. Research findings continue to accumulate, however. Work on the causal effects of social capital on children’s outcomes is indicative of how advanced modeling methods are being used in this research. Multiple casual modeling approaches are used, which “provide stronger evidence than previous studies that social capital improves children’s outcomes and that these improvements are not simply a result of other factors that explain the selection of social relations but rather that these improvements result from the social relations themselves” (Lopez Turley et al., 2012, p. 23).
  • Among the areas for which social capital concepts have been applied most convincingly are health research looking at the relationship between social isolation or loneliness and the mental and physical health of older populations; and the role of community characteristics in creating resilience to economic downturns or to disasters. Another important example is the work noted above on child outcomes that demonstrates how newer statistical modeling methods can be brought to bear in an experimental context to establish causal links and, because it deals with interventions, in a policy-explicit setting.
  • Data collected in the CPS Civic Engagement Supplement have not yet been successful in strengthening evidence of the casual links between various dimensions of social capital and important economic, social, and health outcomes, nor have these data been used extensively in academic research. CPS supplement data
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
  • have typically been used in publications that summarize the data (such as the various civic health index reports), but they cannot support research that models codeterminants of individual outcomes and social capital in a way that address identification and other econometric problems.

  • The real promise for developing a deep understanding of how neighborhood and community-level factors interact to affect aspects of people’s lives requires study of a rich set of variables from diverse data sources that allows for the integration of information about individual characteristics and on individual attitudes in order to study how they relate to communities and to the social environment, and over long periods of time.

CONCLUSION 4: The study of social capital, though a comparatively young research field, is sufficiently promising to justify investment in data on the characteristics of communities and individuals in order to determine what factors affect their condition and progress (or lack thereof) along a range of dimensions. Improved measurement, additional data, and resulting research findings are likely to find uses in policy making.

And—though data collected from large population surveys have not been widely used in research attempting to advance understanding of the causal links between various elements of social capital and outcomes that can be affected by policy—such data are still essential because of their value in providing descriptive information and because evidence continues to accumulate that phenomena described as social capital play an important role in the functioning of communities and the nation.

3.3. TECHNICAL SURVEY ISSUES

Data quality and practical survey methodology issues are also important in constructing an overall data collection strategy—that is, when considering what aspects of social capital should be given priority for measurement using the CPS supplements and which ones should be left for other surveys or for nonsurvey instruments. The measurement and survey issues discussed in this section are not unique to social capital and are well covered in a very deep research literature. And, given its long history dealing with surveys, the U.S. statistical system is well equipped to handle most of them or to judge the extent to which, for a module to be used for measuring social capital, these factors constrain what can realistically be accomplished.

Following the list in Hudson and Chapman (2002), the survey issues

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

identified below—survey length, time, and structure; item appropriateness and sensitivity; item development and quality; sample size, and use of proxy interviews—should factor into any evaluation of elements being considered for inclusion in the CPS Civic Engagement and Volunteer Supplements (or other survey options). These issues become even more critical if the civic engagement and volunteer supplements were to be combined into a single module.

Survey Length, Time, and Structure

The CPS allows about 10 minutes for respondents to complete a survey supplement. This time limit necessitates decisions about tradeoffs in terms of the frequency with which questions can be asked—for example, more questions, but not included in every year of a supplement versus fewer questions asked with greater frequency. Alternatively, the sample can be divided so that random subgroups are asked different questions. This method has been used in federal data collections, though it reduces item precision by lowering the effective sample sizes for each question. Split samples can also be used to experiment with questionnaire designs. For studies of social capital, as with other topics for which causality is difficult to establish, the importance of longitudinal data, or at least regularly repeated cross-sectional questions, is clear.

Item Appropriateness and Sensitivity

Many people view certain topics as inappropriate for government surveys, and there are questions that people are uncomfortable answering (of course, the sensitivity of topics varies across individuals). Questions about religion, attitudes about race relations (and other aspects related to “bridging” social capital), or about numbers or kinds of friendships are just a few examples of questions that are sensitive for some respondents. And different survey modalities may lead to different levels of positive (or negative) response bias. Participants may be less forthcoming on surveys administered by interviewers relative to more impersonal Internet instruments. It is also possible that survey mode has a differential impact on responses to “objective” questions (e.g., did you vote in the last election) and “subjective” questions (e.g., trust in neighbors, quality of friendship ties). As described by Hudson and Chapman (2002, p. 8): “Some agencies also shy away from opinion items. This restriction may make it difficult to measure some aspects of social capital—e.g., the norms and trust that are engendered by community-building—forcing instead a greater focus

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

on measurable activities as a proxy for underlying attitudinal concepts.”14 Another factor is the questionnaire context, which plays a role in determining the scope of appropriate questions. The CPS is primarily a labor force survey; questions on volunteering were added because of their relationship to paid work.

Item Development and Quality

Through the U.S. Office of Management and Budget, the federal statistical agencies maintain standards for items to be included in their surveys. Rigorous testing of questions is part of the process. Prior testing of the Social Capital Community Benchmark Survey by the Saguaro Seminar (see Chapter 1) allowed many of the questions on social capital to be included in the CPS. Similarly, questions being incorporated into the Neighborhood Social Capital Module of the American Housing Survey were developed and tested over a long period by Robert Sampson and colleagues for the Project on Human Development in Chicago Neighborhoods. But, as pointed out by Hudson and Chapman (2002), “not all ‘proven’ items are automatically acceptable for inclusion;” they still must be determined to be relevant to the survey’s subject matter and justifiable on other grounds.

Sample Size

We have repeatedly made the point that phenomena associated with civic engagement, social cohesion, and other dimensions of social capital are often most interesting when studied at neighborhood and community levels or for specific subpopulations; this has obvious implications for data collection. Again, from Hudson and Chapman (2002, p. 8):

For aggregate national estimates, a survey with a sample size of as few as 1,000 individuals would be sufficient. However, from a policy perspective, the underlying issues of equality and access embedded in social capital necessitate disaggregation among policy-relevant social groups, such as racial/ethnic groups; residents of urban, suburban, and rural communities; socio-economic groups; and adults of various ages. The greater the degree of disaggregation desired, the larger the sample must be in order to produce reliable data; oversampling of small groups also becomes an important sampling feature.

______________

14Whether or not agencies should dismiss attitudinal measures out of hand is a matter of opinion. One could reasonably argue that, if such questions are critical to understanding the outcomes of interest, it may be justifiable. Still, government-funded academic surveys may provide a comparative advantage in such cases.

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×

For researchers studying the impact of local events (plant closings, natural disasters, etc.) and for understanding why or predicting which localities are better prepared to recover from a natural or other shock, sources other than national surveys are required, unless those surveys can be funded at levels to support very large sample sizes. When national surveys are not possible or efficient, planning is needed so that information can be collected consistently on features of communities. This kind of planning will increasingly rely on unstructured and uncoordinated data sources. As discussed in some detail in Chapter 5, combining individual- and community-level information that goes well beyond survey data, as was done in the Chicago Neighborhoods Study, will become increasingly important.

Use of Proxy Interviews

A drawback of the CPS is that, in order to obtain information for every adult household member, it uses proxy responses (i.e., a person answering the survey for a household can answer questions about other household members).15 Proxy responses are particularly problematic for questions about attitudes. Two questions on the 2011 CPS Civic Engagement Supplement (“can you trust people in your neighborhood” and one asking about confidence in institutions) specify that they are not to be asked for proxy respondents, which is good for accuracy but results in empty data fields. This characteristic reduces the value of the CPS as a vehicle for measuring dimensions of social capital; this point was made by Hudson and Chapman (2002, p. 9):

For the typical factual questions included in many surveys (basic demographic information, work status, earnings, etc.) proxy interviews are usually acceptable. However, some dimensions of social capital involve typically private, subjective judgments (e.g., questions about trust, interest in politics). It is doubtful that these dimensions can be validly assessed using proxy interviews. Disallowing proxy interviews necessarily restricts the surveys that can be considered as carrier instruments.

These considerations will become ever more important in the current environment in which agencies are reluctant to increase the number of survey instruments or survey questions they administer without a federal mandate or indications of clear, central policy relevance.

__________________

15The accuracy of proxy reporting has been well studied; see, for example, Tourangeau et al. (2000).

Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 57
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 58
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 59
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 60
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 61
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 62
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 63
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 64
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 65
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 66
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 67
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 68
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 69
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 70
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 71
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 72
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 73
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 74
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 75
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 76
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 77
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 78
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 79
Suggested Citation:"3 Prioritizing Measures and Framing a Data Collection Strategy." National Research Council. 2014. Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy. Washington, DC: The National Academies Press. doi: 10.17226/18831.
×
Page 80
Next: 4 Competing and Complementing Data Strategies: The Role of the Federal Statistical System »
Civic Engagement and Social Cohesion: Measuring Dimensions of Social Capital to Inform Policy Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

People's bonds, associations and networks - as well as the civil, political, and institutional characteristics of the society in which they live - can be powerful drivers affecting the quality of life among a community's, a city's, or a nation's inhabitants and their ability to achieve both individual and societal goals. Civic engagement, social cohesion, and other dimensions of social capital affect social, economic and health outcomes for individuals and communities. Can these be measured, and can federal surveys contribute toward this end? Can this information be collected elsewhere, and if so, how should it be collected?

Civic Engagement and Social Cohesion identifies measurement approaches that can lead to improved understanding of civic engagement, social cohesion, and social capital - and their potential role in explaining the functioning of society. With the needs of data users in mind, this report examines conceptual frameworks developed in the literature to determine promising measures and measurement methods for informing public policy discourse. The report identifies working definitions of key terms; advises on the feasibility and specifications of indicators relevant to analyses of social, economic, and health domains; and assesses the strength of the evidence regarding the relationship between these indicators and observed trends in crime, employment, and resilience to shocks such as natural disasters. Civic Engagement and Social Cohesion weighs the relative merits of surveys, administrative records, and non-government data sources, and considers the appropriate role of the federal statistical system. This report makes recommendations to improve the measurement of civic health through population surveys conducted by the government and identifies priority areas for research, development, and implementation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!