6

Data Collection Strategies

6.1 OVERALL APPROACH

Research has shown that it is possible to collect meaningful and reliable data on subjective as well as objective well-being. Subjective well-being encompasses different aspects (cognitive evaluations of one’s life, happiness, satisfaction, positive emotions such as joy and pride, and negative emotions such as pain and worry): each of them should be measured separately to derive a more comprehensive appreciation of people’s lives…. [SWB] should be included in larger-scale surveys undertaken by official statistical offices. (Stiglitz et al., 2009)

The charge to this panel was, in a sense, to deliver an assessment of the extent to which it agrees with the above conclusion of the Commission on the Measurement of Economic Performance and Social Progress, with a primary focus on experienced well-being (ExWB). And, for the most part, the panel does agree: information about the evaluative and experience dimensions of subjective well-being (SWB) is extremely promising for contributing to a fuller understanding of people’s behavior and life conditions, and such information should be collected by national statistics offices to the extent that it is practically and financially feasible. However, the panel also recognizes that measurement approaches are not yet fully mature, which generates concerns about their unqualified adoption at this time.

Going further, the panel may appear at odds with the actionable part of the Sarkozy Commission’s conclusion that “Despite the persistence of many unresolved issues, these subjective measures provide important information about quality of life. Because of this, the types of questions that



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 103
6 Data Collection Strategies 6.1  Overall approach Research has shown that it is possible to collect meaningful and reliable data on subjective as well as objective well-being. Subjective well-being e ­ ncompasses different aspects (cognitive evaluations of one’s life, happi- ness, satisfaction, positive emotions such as joy and pride, and negative emotions such as pain and worry): each of them should be measured sepa- rately to derive a more comprehensive appreciation of people’s lives. . . . [SWB] should be included in larger-scale surveys undertaken by official statistical offices. (Stiglitz et al., 2009) The charge to this panel was, in a sense, to deliver an assessment of the extent to which it agrees with the above conclusion of the Commission on the Measurement of Economic Performance and Social Progress, with a primary focus on experienced well-being (ExWB). And, for the most part, the panel does agree: information about the evaluative and experience dimensions of subjective well-being (SWB) is extremely promising for con- tributing to a fuller understanding of people’s behavior and life conditions, and such information should be collected by national statistics offices to the extent that it is practically and financially feasible. However, the panel also recognizes that measurement approaches are not yet fully mature, which generates concerns about their unqualified adoption at this time. Going further, the panel may appear at odds with the actionable part of the Sarkozy Commission’s conclusion that “Despite the persistence of many unresolved issues, these subjective measures provide important inform­ ation about quality of life. Because of this, the types of questions that 103

OCR for page 103
104 SUBJECTIVE WELL-BEING have proved their value within small-scale and unofficial surveys should be included in larger-scale surveys undertaken by official statistical offices” (Stiglitz et al., 2009, p. 10). We certainly agree that SWB data have proved their value and are worth pursuing by statistical agencies. The challenge is in interpreting which “larger-scale surveys” are appropriate. In the U.S. context, the bar for getting questions onto the Current Population Survey (CPS) or American Community Survey (ACS) has typically been very high,1 and it would be advisable for questions to be thoroughly tested and under- stood before risking using space on those surveys.2 In contrast, inclusion of SWB questions is warranted in some larger-scale, government-funded academic surveys such as the Health and Retirement Study (HRS) in the United States or the English Longitudinal Survey on Ageing. The panel also agrees with Stiglitz et al. (2009) that, where feasible, inclusion of SWB questions on the largest population surveys will produce useful information. However, there are many data needs to inform policy in many domains, and it is not obvious yet that the need for SWB data is more critical than the need to include or improve data programs covering other areas where society, economy, and health must be monitored. A necessary first step is to begin (in the case of the United States) or continue (in the case of the United Kingdom) experimenting with question design and module structure. The UK Office for National Statistics (ONS) is progressing with just such an experimental mode, and it reports remain- ing open to refining the questions. However, as implied by the OECD Guidelines (OECD, 2013, p. 20), there is some commitment to align SWB measurement internationally, and “experimental” status may inhibit roll- out and efforts to harmonize. Alternatively, pressure to harmonize may stunt design experimentation and innovation. At this point, the panel sees the risk, asso­ iated with greater experimentation, of inhibiting harmoniza- c tion as the one worth taking for the foreseeable future. The state of the art has progressed sufficiently far to provide a basis for survey question structure and wording and for experimenting with different 1 Though not always. A civic engagement supplement was added to the November CPS in 2008 and 2010 with somewhat sketchy evidence of how the data would be used and with still-unsettled knowledge of the links between the elements (the module included questions on trust, connectedness, engagement, and other constructs) and social, economic, and health outcomes. Although both social capital and SWB are fertile and important research areas, the panel would argue that the evidence base for structuring SWB data collection is better developed than it is for social capital. 2 For survey vehicles, such as the CPS and ACS, where space is scarce and highly sought-after, questions typically have only one chance to get things right. Once questions go on and off, they are unlikely to ever be put back on, so the justification for placement must be as solid as possi- ble before the first implementation. Investigation of SWB questions for the major surveys could be a project for the interagency statistics group to consider; a precedent is the process whereby race question were refined by the U.S. Census Bureau—an issue that cuts across surveys.

OCR for page 103
DATA COLLECTION STRATEGIES 105 survey vehicles. These directions can be continued even while uncertainty remains about exactly what kinds of measures will prove most useful to researchers and policy makers and what statistics should be published. In short, this measurement domain is, in the panel’s judgment, very much still in the “let a thousand flowers bloom” stage of development. Evidence about SWB links to behavior, outcomes, and policy levers (causal and otherwise) will continue to accumulate based on research using­ a range of data sources, public and private. The approach should be to get SWB content on surveys where possible and where it makes sense and to exploit all opportunities to learn more about the data.3 If, ultimately, a particular SWB metric evolves to a point that warrants “official statistics” status, it may become preferable to have it measured on one official survey and not many; otherwise, different values will be estimated, and the public will be confused. Until then, as a fuller understanding of their properties develops, it is important to try various measures of SWB dimensions and components on a range of surveys. Further, for ExWB specifically—where, relative to measures of evaluative well-being, it is more difficult to envision reporting of some aggregated number—it will be useful to include questions in a range of contexts (time-use surveys, health surveys, housing surveys that include questions about neighborhood amenities and conditions, sur- veys of the elderly, and other targeted assessments) because it would be beneficial to have information about different sets of covariates for different applications. It is unlikely that an identical module could be simply plugged into different surveys to suit the many envisioned purposes for SWB data; rather, the questions will need to be tailored to the purposes for which a given survey is put. An example is HRS, which includes questions about the connectedness of the elderly to their children and friends, a trait hypoth- esized to be correlated with happiness and with health outcomes. 6.1.1  The Measurement Ideal Because SWB has multiple dimensions and its measurement sheds light on people’s behavior and life conditions at different levels of aggregation, from the individual up to national and international group comparisons, an ideal data infrastructure would require a multipronged approach. Large-scale population surveys—such as the four-question module in the UK Integrated Household Survey or the Gallup World Poll—make up one component of a comprehensive measurement program. Data from these surveys, typically drawn from global-yesterday measures of ExWB and 3 At this stage of development, the task of improving measurement methods still lies mainly with academic researchers (many with funding from grant-making government institutions) as opposed to statistical agency staff.

OCR for page 103
106 SUBJECTIVE WELL-BEING from life-evaluation questions, provide the large sample sizes essential for repeated cross-sectional analyses capable of identifying and tracking suffer- ing or thriving subgroups and for research on special populations such as the unemployed (for whom life expectancy is falling). Such data may prove useful for informing policy at the macro level.4 The Gallup survey data have also been used for analyses that narrow the focus to specific populations and for city, state, and international comparisons and global hypothesis testing. It is not yet known whether ONS and the Gallup Organi­ a­ ion use ­ z t the right adjectives, or enough adjectives, or if they include the optimal c ­ovariates; such an assessment is of course conditional on purposes to which the data will be put. For example, the CPS (in which the American Time Use Survey [ATUS] module resides) is designed to optimize employ- ment measures at specific levels of geographical specificity. Beyond eco- nomic contextual needs, there is, for SWB assessments, a need for datasets that include health and geographical covariates. It is important to establish the right context for a SWB module before attempting to make it permanent. As was learned from the experiences with an official poverty measure, it is difficult to change and improve measure- ment systems once they become entrenched. Reducing suffering may be an analogous policy goal to reducing poverty, so perhaps lessons can be drawn in terms of how the measurement approach should be developed to best serve that goal. The OECD Guidelines (OECD, 2013) will be influential for those countries wanting to go forward with the large population dataset approach, whether this is undertaken by developing new surveys or finding ways to continue ongoing efforts;5 others will pursue more experimental, smaller-scale approaches before committing—or choosing not to commit. An ever-present consideration for a national statistical office is whether to begin data collection with incomplete knowledge of appropriate structure and its likely value—presumably starting modestly with a small module of questions—or waiting until more is known, when possibly a more expen- sive, more multidimensional approach can be supported. The second prong of a comprehensive measurement program is smaller or more specialized data collections. One option is to construct experiments or pilots within existing large survey programs (for example, the ATUS time-use module, which is part of the CPS)—often using outgoing samples that are rotating out of the survey. The American Housing Survey’s new 4 The Gallup Organization achieves large sample sizes by surveying people (1,000) every day in its daily U.S. poll. This data collection approach allows things like weekend and holiday effects to be captured. However, for some questions, it may be useful to achieve a similar sample size by surveying much larger groups in a single data collection period but fielding the survey less frequently. 5 In addition to the United Kingdom, Brazil, Chile, Mexico, and a number of other countries have SWB data collection initiatives up and running.

OCR for page 103
DATA COLLECTION STRATEGIES 107 Neighborhood Social Capital module is another example of a possible home for SWB content; adding ExWB questions to this module would allow researchers to explore links to community characteristics, connectedness, and resilience (an association specifically cited by Stiglitz et al. [2013] as potentially very important). For measuring the role of positive experi- ences, health surveys provide an increasingly secure foothold, as research s ­ trengthens the knowledge base about the links between healthy emotional states and healthy physical states. The advantage of targeted studies and experimental modules is that they can be tailored to address specific questions of interest to researchers and policy makers—whether about health care, social connectedness of the elderly, city planning, airport noise management, or environmental moni- toring. The end objective of these efforts should not be a perfect measure, which does not exist, but measures that generate information that can be usefully combined with other sources in a range of applications. One clear advantage of smaller-scale, specialized surveys is that they can often be supported by funding agencies—such as the National Institute on Aging’s support for HRS and the ATUS SWB module—that can ensure the underly- ing purpose is well thought out. A third prong to an ideal data infrastructure would consist of panel studies designed to document changes in SWB over time. How individuals’ ExWB and evaluative well-being change over time and in reaction to events and life circumstances cannot be fully understood without longitudinal information, which may also help to make progress on causality questions (e.g., does getting married make people happier, or are happier people more likely to get married?). More emphasis should be given to development of longitudinal data sources and within-subject panel design, both to develop optimal measures methodologically and to begin making progress in sorting out causality across measures and events. The policy relevance of monitor- ing SWB changes over time is clear where, for example, it is important to know the full impact on people of new legislation, such as the Patient Protection and Affordable Care Act, or the full impact on outcomes of experiments such as the Oregon Health Care Study.6 Changes in a population’s aggregate-level SWB associated with specific events, even very dramatic ones such as the financial collapse or the Sep- tember 11, 2001, terrorist attacks, can be difficult to detect using broad surveys, even with very large samples (Deaton, 2012). In the case of the 2001 terrorist attacks, Metcalfe et al. (2011) examined consequences for 6 For the Oregon Health Care Study, 6,387 low-income, nonelderly, nondisabled adult par- ticipants were selected to be eligible to apply for Medicaid coverage. The comparison group consisted of 5,842 counterparts (with respect to income, age, and disability) who were not eligible for Medicaid coverage.

OCR for page 103
108 SUBJECTIVE WELL-BEING the well-being of UK citizens using measures of mental distress from the 12-item General Health Questionnaire. They found an impact on happiness roughly equivalent to one-fifth of the average magnitude associated with becoming unemployed. Anticipated events offer natural experiments and opportunities to test how measures of SWB react to them or to changing conditions. One exam- ple is a project to study the SWB impact associated with the 2012 London Olympics. Ongoing research by a team headed by Paul Dolan followed a group in the United Kingdom throughout 2011, 2012, and 2013, using online surveys, supplemented with additional telephone interviews during the 2012 Olympic Games, to track how satisfied subjects were with their life overall, as well as how happy or anxious they were on certain days. The study is also comparing the SWB of people in Paris (which lost the bid to host the 2012 Olympic Games) and Berlin (which did not bid). The motivation for the study is to improve understanding of the impact of big events—for example, is the effect short-lived or is there a longer legacy effect?—in a way that is useful for decision makers considering bringing these types of big events to their localities. Specialized studies of this kind can often draw supportable inferences from smaller samples than are used in general surveys. Just as panel data have allowed researchers to learn more about the characteristics of pov- erty (revealing less chronic poverty and more movement in and out of poverty than was once thought), they may be useful for learning about the duration of depression or suffering and whether these conditions are more chronic or if there is extensive movement by individuals in and out of groups defined by these states. It is difficult to study these phenomena without panel data that are collected fairly frequently. Schuller et al. (2012) reviewed the contribution of longitudinal data in analyzing SWB responses for a range of key well-being domains, such as relationships, health, and personal ­ nance. A panel structure also creates pitfalls. For example, ask- fi ing a panel of individuals SWB questions on, say, a quarterly basis might give rise to a focusing effect, where the first (or previous) response acts as a reference point for subsequent ones because individuals might recall their previous response (Dolan and Metcalfe, 2010). A final prong of an ideal data collection is real-time experience sam- pling. As described in Chapter 3, momentary sampling methods have been central to SWB research but largely out of practical reach for adoption by national statistical offices. However, rapid changes in technology and in the way the public exchanges information have brought the world to a point where momentary assessment techniques may now be on the horizon for national statistics. Precisely knowing how people are doing emotionally and what they are doing in the moment can shed light on the effects of commuting, air pollution, child care, and a long list of areas with clear ties

OCR for page 103
DATA COLLECTION STRATEGIES 109 to policy. As the ways in which government agencies administer surveys change (with response rates declining and survey costs continuing upward for conventional data collection methods) and as monitoring technologies continue to rapidly evolve, new measurement opportunities will arise. Considered in terms of comparative respondent burden, it may soon be- come less intrusive to request a response to a modern electronic Ecological Momen­ ary Assessment (EMA) device, or perhaps a smartphone beep, than t to ask respondents to fill out a long-form survey. So, while EMA may not be practical for the ACS or CPS for the foreseeable future, real-time analyses may be (or become) practical for a number of other surveys, particularly in the health realm. Which elements to pursue of the four-pronged approach outlined above is a matter of national and specific program priorities. ONS is moving forward on a broad survey of SWB measures, and it may later begin add- ing granular time-use and targeted survey components. The plan would be quite different for the United States, where ATUS has been temporarily in place as an “experimental module” but where a SWB module for the largest population surveys is lacking. In the meantime, it is a boon to researchers ­ that different organizations are taking the lead in different areas so that the relative merits of different approaches can be assessed. 6.1.2  Next Steps and Practical Considerations Appropriate next steps for the statistical agencies will be dictated to a large extent by perceptions of the state of maturity in the evolution of SWB measures. Evaluative well-being has for some time been measured with one or two questions in many large-scale surveys, and that approach can and will continue to be applied at a relatively low cost in national and international surveys (as done now by ONS or by the Gallup Organization). ExWB is less well understood and less well tested—though, certainly, some questions still remain about evaluative well-being as well—and therefore its measurement is more challenging from a survey methodology standpoint. Given that detailed time-use surveys are expensive and burdensome to conduct and that simple ExWB measures (e.g., global-yesterday measures) used in larger-scale surveys such as the Gallup World Poll seem to track well with the more detailed measures, one approach would be to include simple measures of ExWB in a set of large-scale surveys. The results could then be rounded out using more detailed surveys of time use, such as ATUS, on a more targeted, small-scale basis. As acceptance and validity become more established, a more aggressive move to add content to U.S. federal surveys can be supported. At this point, however, more research and testing are needed before the federal statistical system should settle on a specific approach or create an “official series” for ExWB comparable to, say, the

OCR for page 103
110 SUBJECTIVE WELL-BEING unemployment rate.7 There simply is not enough known yet about ExWB over time to present it in an official government series. CONCLUSION 6.1:  SWB is an exciting and potentially very impor- tant construct that adds content to and could influence the direction of policy debates. Recent research has rapidly advanced our understand- ing of the properties of ExWB measures and their determinants. This promise notwithstanding, more research and assessment are needed before ExWB is included as a regular and permanent component on flagship U.S. surveys, such as the ACS and CPS. ExWB metrics are not yet ready to be published and presented as “official statistics.” Although the level of confidence needed for an official series (which becomes less methodologically flexible than satellite or experimental data) has not yet been established, SWB modules, including questions on both ExWB and evaluative well-being, are appropriate for inclusion in more targeted surveys, such as the ATUS or those administered by various health statistics agencies. The issues described in this report can likely be resolved (or better understood) through experimental pilots and targeted surveys and from further study of results from ONS, the Gallup Organization, ATUS, and other current activities. RECOMMENDATION 6.1:  ExWB measurement should, at this point, still be pursued in experimental survey modules. The panel encourages inclusion of ExWB questions in a wide range of surveys so that the properties of data generated by them can be studied further; at this time, ExWB questions should only be considered for inclusion in flag- ship surveys on a piloted basis. Numerous unresolved methodological issues, such as mode and question-order effects, question wording, and interpretation of response biases need to be better understood before a module should be considered for implementation on a permanent basis. More of the research recommended in this report should be completed (not all by statistics agencies) before committing to a particular version for national time series. The above statements raise the difficult question of what the criteria are for establishing the level of confidence needed for an official series. As described in Box 6-1, national and international statistical offices take some care to define what official statistics are and to specify the roles that they serve. Assessments of reliability, accuracy, data interpretation, proven policy 7 The panel notes that there are still multiple series for unemployment measures, and there is still methodological debate about them.

OCR for page 103
DATA COLLECTION STRATEGIES 111 BOX 6-1 Fundamental Principles of Official Statistics, from the United Nations Statistics Division Principle 1. Relevance, Impartiality, and Equal Access Official statistics provide an indispensable element in the information system of a democratic society, serving the government, the economy and the public with data about the economic, demographic, social and environmental situ- ation. To this end, official statistics that meet the test of practical utility are to be compiled and made available on an impartial basis by official statistical agencies to honor citizens’ entitlement to public information. There are many elements to this principle. First, official statistics are one of the cornerstones of good government and public confidence in good government. Official statistics, by definition, are produced by government agencies and can inform debate and decision making both by governments and by the wider com- munity. Objective, reliable and accessible official statistics give people and orga- nizations, nationally and internationally, confidence in the integrity of government and public decision making on the economic, social and environmental situation within a country. They should therefore meet the needs of a range of users and be made widely available. Second, to meet the test of practical utility, statistics must be relevant, of a quality suitable for the use made, and in a form that facilitates easy and correct use. The key to achieving this is maintaining an understanding of what statistical information users want and how they want it. SOURCE: United Nations Statistics Division, see http://unstats.un.org/unsd/goodprac/­ bpaboutpr.asp?RecId=1 [September 2013]. relevance, and credibility among data users certainly figure into the decision to establish an official series, but it is essentially an iterative process whereby data are first deemed worth collecting, then used to produce pilot or test statistics, and sometimes rising to be published as an official series (as in the case of the consumer price index or unemployment rate).8 There is no objec- tive, bottom-line criterion indicating when statistics become qualified to be an official series. However, the criteria listed above are part of building a strong case for taxpayer support and for the potential sustainability of a measure.9 8 See Principles and Practices for a Federal Statistical Agency: Fifth Edition (National Re- search Council, 2013), a report periodically updated by the Committee on National Statistics, for a thorough discussion of these issues. 9 For example, a “civic engagement” module was added as a supplement to the U.S. CPS in 2008 and 2010. There was some political and researcher support for the module, but the

OCR for page 103
112 SUBJECTIVE WELL-BEING In thinking about plans for the United States specifically, an important distinction is that between an official statistical series and government data collection more generally.10 In the view of this panel, the concept of ExWB is certainly ready for the latter but not yet the former. In the meantime, as data-driven research results accumulate in ways that support (or do not) official SWB statistics, there are actions that the statistical agencies can take to help move things forward. There is, of course, a danger in being too timid with recommenda- tions for moving into new measurement areas. After all, how can research and development occur without data creation and without risk? In this spirit, Conclusion 6.1 and Recommendation 6.1 should not be interpreted as a knock against the ambitious work undertaken, and the progress be- ing made, by ONS. Many of the fixed resources—both intellectual and fi ­ nancial—for adding the four-question module to the Integrated Household Survey (and others) have already been expended, and the results to date have created an excellent opportunity to begin analyzing data properties, interpreting the results, and generally using them as a test bed for further development of SWB measurement. As researchers take advantage of this emerging data source, much may be learned about the SWB of the UK population and about next steps in developing effective and useful SWB modules. Further, ONS has stated the view that “National Statistics” status does not preclude further refinement.11 6.2  How to leverage and coordinate existing data sources Although researchers have benefited enormously from data collection by the Gallup World and Daily Polls, the World Values Survey, and others, there is clear value (complementary at the very least) in anchoring data col- lection work in government statistical systems. Government surveys often supplement was dropped for 2012, perhaps in part because the case for its continuation had not been made clearly enough by these criteria. In contrast, the research support and use for the ATUS module of the CPS has been quite broad, and the case for its continuation has been easier to make. The bar for an official series (say of time-use patterns or civic engagement) would be much higher still. 10 Although it is not within the scope of the panel’s Statement of Task, the panel cannot ignore the current political and budget climate, which makes the practical hurdles to introduc- ing new survey content quite high. Furthermore, the decentralized nature of the U.S. statistical system creates additional complications for launching a well-coordinated effort analogous to what ONS has done for the United Kingdom. 11 For statements on the experimental nature of the ONS data collection initiative, see: http:// www.ons.gov.uk/ons/rel/wellbeing/measuring-subjective-wellbeing-in-the-uk/first-­annual-ons- experimental-subjective-well-being-results/first-ons-annual-experimental-subjective-well-being- results.html [October 2013].

OCR for page 103
DATA COLLECTION STRATEGIES 113 include rich sets of covariates, large sample sizes, and comparatively high response rates, along with the potential to link with administrative and other data sources. Dwindling budgets may eventually call some of these advantages into question, yet government surveys are stable and likely to be an important public good for many years to come. To be realistic, agitating for entirely new, large surveys of SWB seems unlikely to pay off in the foreseeable future. In the current budget cli- mate, the statistical agencies will have to be more opportunistic, which likely means formulating a strategy to embed SWB question in existing instruments. ONS has already followed this course, adding modules to the A ­ nnual Population Survey in 2012 and the Opinions Survey in 2011. The strategy described above is not too different from that taken by ONS, which is mixing large and small survey instruments. In the United States, a compa- rable strategy would be to add modules to the ACS or CPS—but the panel has already discussed the practical difficulties of doing that. Nevertheless, beyond these broad population surveys, options do exist to add questions to more targeted instruments. 6.2.1  SWB in Health and Other Special-Purpose Surveys Several surveys provide a platform for SWB measurement in health domains. One of these, the HRS, is a nationally representative longitudinal survey of more than 26,000 Americans over the age of 50. Conducted every 2 years, HRS has included satisfaction-of-life questions as well as ExWB questions (there was a hedonic well-being module as recently as 2012). The funding agencies—the National Institute on Aging and the Social Security Administration—determined that these questions were useful for generat- ing insights into the health and work transitions of older Americans. HRS, which is conducted by the University of Michigan, includes a wealth of contextual information—on demographics, income, wealth, employment status, health, and disability—making it all the more attractive as a home for SWB research. It is an excellent example of how inclusion of SWB in a more targeted way can lead to rich investigation of well-defined questions, such as how disability in older populations relates to their emotional states (Daly and Gardiner, in press) or how their health is affected by family con- nectedness and support (National Research Council, 2010). The English Longitudinal Study of Ageing creates similar research opportunities for UK studies. The National Longitudinal Study on Youth, conducted by the U.S. Bureau of Labor Statistics, is another survey that would be useful for studying ExWB alongside work and other factors at the younger end of the age spectrum. While these kinds of longitudinal datasets are extremely useful for studying impacts that accompany changing life circumstances, repeated

OCR for page 103
114 SUBJECTIVE WELL-BEING cross-sections are sometimes needed to provide information about the evolution of the population. There are also sample-size trade-offs. The questions of interest will dictate whether it is preferable to survey more respondents cross-sectionally or to survey fewer over multiple periods. For national (aggregate-level) statistics, large samples are needed to pick up change so that subpopulations affected by events (e.g., the unemployed; people in New Orleans post-Katrina) can be captured. Krueger and Schkade (2008) provided a straightforward assessment of how to assess the reli- ability of some SWB measures. A related issue is whether to field sur- veys intermittently (as in employment surveys) or continuously (as in the Gallup Daily Poll). Other than resource constraints, there seems little rea- son against a continuous mode. Among other strong candidates for SWB data collection in the U.S. statistical apparatus are the National Health Interview Survey and the National Health and Nutrition Examination Survey. These surveys in- clude the necessary covariates to study the influence of health and health care on SWB (and possibly vice versa). For these surveys, there is a clear policy rationale related to health care delivery (for patients and care g ­ ivers) for embedding SWB questions. Similarly, the Behavioral Risk Factor Surveillance System—a repeated cross-sectional survey, which includes a county-level identifier and questions about SWB—has been used to study life-style choices and SWB (focusing mainly on evaluative well-being). For example, Brodeur (2012) examined the impact of smoking ban policies (at the county level) on self-reported life satisfaction, using the Behavioral Risk Factor Surveillance System and the Needham Life Style Survey, both of which include a broad set of variables such as household income and smoking behavior. The Survey of Income and Program Participation (SIPP), which has historically reflected an interest in self-assessments of well-being, represents another option. SIPP is a natural fit for SWB measurement because of the wealth of income and program-activity questions that form the core of the survey. Kominski and Short (1996) noted the relationship of income to SWB versus other factors and recognized the possibility that some members of a population may have objectively low levels of income (and commodi- ties that can be purchased with that income), yet still be relatively “well off” if other aspects of their lives act to compensate in some way. As an ex- ample, they noted that an extensive social support system had been shown in other research (e.g., Helliwell and Putnam, 2004) to significantly offset some of the disadvantages associated with low income and low wealth. Developers of the SIPP recognized that these kinds of research questions require subjective self-assessments of one’s quality of life. A 1978 pilot module of SIPP did ask about respondents’ normative self- assessments, using a seven-point “delighted-to-terrible” scale to describe

OCR for page 103
DATA COLLECTION STRATEGIES 115 their “life as a whole.”12 Later, during the early 1990s, an interagency group of researchers considered how to develop a set of questions that could be added to the SIPP to elicit a broader (than just income) concept of well-being from survey respondents. The working group was tasked with developing an “extended well-being” topical module for inclusion on the 1991 and 1992 panels of the survey (Kominski and Short, 1996). This module did not ask now-conventional questions for measures of evaluative well-being or ExWB but instead asked about various domains: housing con- ditions, crime conditions, neighborhood conditions, presence of help when in need, food adequacy, etc. The point here is that the SIPP is an appro­ riate p context for targeted research questions about benefits trade-offs that could be fruitfully supplemented with SWB information. The policy application here is quite clear: How to value the bundle of “goods” (including non­ market goods and services) provided to low-income families. Another candidate for ExWB data collection is the large American Housing Survey (AHS), which is overseen by the Department of Housing and Urban Development and conducted by the Census Bureau.13 Inspired by the research of Robert Sampson on Chicago neighborhoods, the survey will include a new module in 2013 called the “Neighborhood Social Capital Module,” which was created as a “rotating topical module that collects data on shared expectations for social control, social cohesion, and trust within neighborhoods, and neighborhood organizational involvement.”14 The AHS survey is conducted with a large, geographically diverse sample, which will enable detailed neighborhood social-capital assessments to be produced for 25 metropolitan areas. Adding SWB questions to the AHS would allow researchers to explore the relationship of SWB measures with community characteristics (the magnitude of income disparities, provi- sion of social services, etc.). Social context is an association that has been studied in some detail by Helliwell and Putnam (2004), among others; it is cited specifically by Stiglitz and colleagues (2009) as central to population well-being. The Panel Study of Income Dynamics is another option for ExWB ques- tions; it would be particularly useful for researchers studying the relation­ ships between care-giving arrangements, connectedness, health, and SWB. It offers a large, representative national sample of U.S. households and uses subsets of respondents assessed in multiple waves. The 2001 and 2003 12 The seven-point scale, together with the first two items, was originally developed and extensively tested by Andrews and Withey (1976). 13 One attractive feature of this survey is that it is quite large; 179,000 responses are e ­ xpected, which is substantially larger than the CPS supplements, and it is longitudinal. 14 This text is from the Office of Management and Budget supporting statement for this data collection initiative, which can be found at http://www.reginfo.gov/public/do/­ DownloadDocument?documentID=369083&version=0 [October 2013].

OCR for page 103
116 SUBJECTIVE WELL-BEING waves included items on SWB. Also, Child Development Supplements that have included evaluative well-being questions about how often during the past month respondents had felt “(1) happy, (2) interested in life, and (3) satisfied”—have been attached to past waves of the PSID. Recommendation 6.2: ExWB questions or modules should be included (or should continue to be included) in surveys where a strong case for subject-matter relevance can be made—those used to address targeted questions where SWB links have been well researched and where plausible associations to important outcomes can be tested. Good candidates include the Survey of Income and Program Participa- tion (which offers income, program participation, and care-giver links); the Health and Retirement Study (health, aging, and work transition links); the American Housing Survey’s Neighborhood Social Capital module (community amenities and social connectedness links); the Panel Study of Income Dynamics (care-giving arrangements, connect- edness, and health links); the National Longitudinal Survey of Youth (understanding patterns of obesity); and the National Health Interview Survey and the National Health and Nutrition Examination Survey (health and health care links). If harmonized modules were developed that were short enough, they could in principle be included in a range of surveys. However, for surveys with a specific orientation (e.g., understanding the conditions of retirees or the time use of individuals) it would typically be preferable to tailor ques- tions to research objectives. One possible benefit of an initiative to design a standard ExWB module or instrument (perhaps developed by a research network, the National Bureau of Economic Research, the Russell Sage Foundation, the Roybal network, or through a pilot study competition) is that it would encourage discussion of where the measures are useful and where not, and it may help to reframe the discussion about what are the clearest policy applications. Also, if inclusion of a uniform ExWB question or module into a number of surveys were considered, global-yesterday measures would be the likely default instruments, as they are short by design and flexible in terms of sur- vey mode (i.e., the time of day when the question can be asked). However, they are more limited in the scope of detail that can be collected relative to something like the ATUS SWB module, or certainly the EMA-type methods. 6.2.2  Taking Advantage of ATUS The ATUS SWB module is, at this time, the most important U.S. gov- ernment data collection on ExWB. Funded by the National Institute on

OCR for page 103
DATA COLLECTION STRATEGIES 117 Aging and overseen by the Bureau of Labor Statistics, an SWB module was included in ATUS in 2010 and 2012, but there are no plans currently in place to field it in 2014 or beyond. The ATUS SWB module is the only U.S. federal government data source of its kind—linking self-reported informa- tion on individuals’ ExWB to their activities and time use. As described in Chapter 3, time-use data derived from the Day Reconstruction Method (DRM) or a modified DRM is essential for linking ExWB to activities and, in turn, to policy levers. The fact that ATUS itself is a supplemental module to the CPS, which is focused on labor market and other economic informa- tion, adds further value. Research (e.g., Krueger and Mueller, 2008, 2011) has shown that long-term unemployment is strongly linked to suffering, so this relationship can potentially be studied.15 Time-use surveys are needed to determine how people change their time allocations and to indicate which activities are most enjoyable and which are most miserable.16 Questions about “overall happiness yesterday” miss much of what is interesting in this regard. Important research has already been conducted using the time-use data (for example, that cited above on the effects of unemployment and job search on people’s SWB). If attaching SWB questions to an existing instrument can be done at low marginal cost, it seems a good value (see Appendix B for the panel’s interim report on the ATUS SWB module). Work conducted with ATUS—sometimes in combina- tion with other data sources—has indicated the potential of the module to contribute to knowledge that could inform policies in such areas as health care and transportation. If a policy changes time use—typically the most valuable market and nonmarket resource in an economy—then it is easy to make the case for data collection. Conclusion 6.2: Time-use data are being collected by the U.S. government, and self-reported well-being questions add an important dimension to such data. The ATUS SWB module is practical, stable, inexpensive, and worth continuing as a component of ATUS. Not only does the ATUS SWB module support research; it also generates infor- mation to help refine SWB measures that may be considered for future additions to official statistics. 15 Effective January 2011, the CPS was modified to allow respondents to report durations of unemployment up to 5 years. Prior to that date, the survey allowed reporting of unemploy­ ent m durations of up to only 2 years; any response greater than 2 years was entered as 2 years. 16 Not all time-use policy questions require ExWB information. Increased time spent in com- muting is known to have a negative impact on people’s emotional states; one only needs to look at activity-based average scores and allocation of time to compare one state and another in a general way.

OCR for page 103
118 SUBJECTIVE WELL-BEING Extending the material developed during the course of the panel’s early deliberations and presented in its interim report on the ATUS SWB module (see Appendix B) are the following additional conclusions: • Continuation of the ATUS SWB module enlarges samples by allow­ ing pooling of data across years. This enables more detailed study and comparison than has been possible to date of population subgroups, such as people in a given region and specific demo- graphic groups (e.g., young people, the elderly). Because two new questions—one on overall life satisfaction and one on whether respondents’ reported emotional experiences yesterday were “typi- cal”—were introduced to the module in 2012, addi­ ional waves of t the survey will allow assessment of changes in response to those questions over time (although the responses over time will not be from the same respondents). • Cost and other effects on ATUS. As a supplement to an existing survey, the marginal cost of the SWB module, which adds about 5 minutes to ATUS, is small. While further study of the module’s ef- fects on response and bias in the main ATUS should be undertaken, it appears likely that these effects are modest because the module comes at the end of the survey, after respondents have already been asked to report their activities for the preceding day. • The ATUS SWB module could be the basis for a standardized set of questions that could be added to other time-use surveys around the world, which together might provide useful comparative informa- tion across different populations. ATUS provides an appropriate vehicle for experiments to improve the structure of abbreviated DRM-type surveys. Experimental modifications to consider include • Split sample surveys. Half of ATUS respondents could receive one question while the other half gets another; this would be useful for testing such things as sensitivity to different scales and question wording.17 • Finding the optimal number of activities on which to collect ExWB information. It is not obvious that three activities is the optimal number of activities to include on the ATUS SWB module. It may be useful to ask about ExWB associated with more activities in order to increase the reliability of daily estimates. Importantly, 17 In its well-being survey, ONS has used, or plans to use, split trials to test such things as sensitivity to different scales, question wording, and order and placement of questions.

OCR for page 103
DATA COLLECTION STRATEGIES 119 sampling more episodes increases the power to examine activity- specific effects, which may be particularly valuable for addressing policy questions. Doubling or even tripling the number of epi- sodes may be cost-effective, although that benefit would have to be weighed against considerations of participant burden and the potential impact on response rates. • The data characteristics that emerge from sampling three con- secutive activities, in which the first in the sequence is randomly selected, could be tested for comparison with the current structure, in which all three are chosen randomly. Questions of interest in- clude what additional things could be learned (e.g., how emotional impact of one event may carry over to others) and what would be lost from such a question structure. • Selecting the “right” positive and negative emotion adjectives for module questions. As described in section 2.1, research supports the separation of positive and negative states but, more generally, should the SWB module be focused more on suffering or happi- ness? The module could experiment with different adjectives and how interpretation varies across populations. • Additional or replacement questions for consideration. A possible example is adding a question or two about sleep, such as: “How many hours of sleep do you usually get during the week?” or “How many hours of sleep do you usually get on weekends?” The objec- tive of such questions would be to find out if respondents’ reports about behaviors and emotions—feeling happy, tired, stressed, sad, ­ pain—are influenced by (chronic) sleep deprivation or other sleep patterns.18 A methodological question is how well people recall the previous night’s sleep. • Selecting among competing measures of evaluative well-being. Is the current Cantril approach, which is perhaps the most remote from ExWB measures, optimal? Alternative versions of the evalu- ative well-being measure are common in the literature. 18 Thisidea was raised by Mathias Basner, of the University of Pennsylvania School of Med- icine, who noted that self-assessments of habitual sleep time overestimate physiological sleep time and that estimates of habitual sleep time based on ATUS overestimate self-­ ssessments a of habitual sleep times found in other population studies. Therefore, he suggested, it would be very elucidating to compare self-assessments of sleep time for the two questions sug- gested above against estimates based on ATUS responses for the day before the interview day (public comments for the ATUS SWB module: see http://www.reginfo.gov/public/do/­ DownloadDocument?documentID=120293&version=0 [October 2013]).

OCR for page 103
120 SUBJECTIVE WELL-BEING 6.3  Research and experimentation— the role of smaller-scale studies, nonsurvey data, and new technologies At this point in the conceptual development of SWB measures—and in keeping with the panel’s conclusion that it is not yet useful to construct a national measure for general monitoring purposes—data collection should be carried out using targeted or specialized tools and in experimental modules of existing surveys. Smaller-scale studies have already shown their potential to inform the development of survey measures of SWB and to be used in substantive research applications. An example is the Krueger and Mueller (2012) study of how job search affects dimensions of SWB among the unemployed using a repeated survey of 6,025 unemployed workers in New Jersey. Recommendation 6.3:  For ExWB, the data collection strategy of the statistical agencies should remain experimental until data proper- ties and correlative and causal relationships among variables are better understood. This means more research and preliminary testing before committing to particular approaches (e.g., to a given survey module structure). Beyond the statistical agencies, it is likely that researchers will increas- ingly exploit alternative, nontraditional survey sources to learn more about SWB. One example is the study of the SWB impact associated with the Lon- don Olympics using multiple survey modes (including Internet), cited in sec- tion 6.1.1. Social media data and other kinds of organic data (those, such as administrative records or company-maintained information, produced initially as a by-product of nonstatistical purposes) may become increas- ingly useful for shedding light on trends in people’s emotional states. Word- mining exercises have been used to show patterns in emotional states—for example, a Facebook happiness index showed the standard weekend and holiday effects and expected changes associated with major events, such as disasters. Additionally, analyses of data generated by social media and other Internet activities will produce insights relevant to public policy beyond those focusing primarily on aspects of negative experience such as distress or pain. As illuminated by social or political movements such as the Arab Spring and by mass protests across the world ranging from anticapitalist movements to demonstrations concerning police behavior or health re- forms, other negative feelings such as collective anger and sense of injustice may be as important in the public policy context as individual experiences of distress. Not much is known about these collective experiences, and the tools have not yet been developed for studying them carefully, but they are

OCR for page 103
DATA COLLECTION STRATEGIES 121 surely important; measuring and understanding them would be a significant benefit to public policy. Big data will play a role in this research.19 The Mappiness project (mappiness.org.uk), designed to investigate well-being effects to the public associated with open green space in the London area, delivers instant feedback on how Mappiness app-holders feel: happy, relaxed, and awake. It allows monitors to look at individual-level variation for people located in different outdoor environments. Potential applications envisioned by the creators include assessing interventions in the form of random controlled trials of such things as remediation or green exercise, assessing impacts of events such as the Olympics, and natural and recreation resource monitoring. This project provides a clear example of the emerging methods to capture SWB in the context of EMA measures and the role of portable recording—in this case the use of cellphones and global positioning system (GPS) tracking. New measurement techniques such as the geospatial cellphone responses in the Mappiness project are now making it possible to consider EMA-type data in survey contexts. The British Millenium Cohort Study is considering use of geospatial cellphone responses as a post-survey supplement. There are still major unresolved data quality and representativeness issues in this world of new data and big data. For instance, the sampling properties are largely unknown for data generated by social media, phone records, Internet usage, and the like. A bright red flag of caution needs to be attached to these data sources, acknowledging the unknown distribu- tional characteristics of various underlying subpopulations. This is sure to be a major emerging statistical research topic. Social media data need to be scaled, and the best methods are likely to change as the penetration of various media and technologies evolve. One must also be careful not to clump all kinds of new technologies or big data together. For instance, a Facebook index may not work well for objective statistical analysis, but an iPhone bleep test of a carefully sampled population might—or vice versa for some questions. In the Mappiness project, within-individual confounding is possible; that is, causal pathways may run in both directions: people may 19 While avoiding a formal definition, Capps and Wright (2013) usefully contrast official statistics and big data in terms of database size, dissemination timing, nature of data use per- mission practices, costs of production, and data collection design. Sources of big data cited by the authors include “data that arise from the administration of a program, be it governmental or not (e.g., electronic medical records, hospital visits, insurance records, bank records, and food banks); commercial or transactional digital data . . . (e.g., credit card transactions, on- line transactions; sensor data (e.g., satellite imaging, road sensors, and climate sensors); GPS tracking devices (e.g., tracking data from mobile telephones); behavioral data (e.g., online searches about a product, service, or any other type of information and online page views); [and] opinion data (e.g., comments on social media).”

OCR for page 103
122 SUBJECTIVE WELL-BEING go to certain places when they are happy, or they may be happy because they are at that location.20 Conclusion 6.3:  For now and the immediate future, the primary means for measuring and tracking ExWB, and SWB more broadly, remains population surveys. Neither the practical and economic chal- lenges to “traditional” survey methods nor the promise of alternative ways for measuring the public’s behaviors and views have reached a point where it is sensible to transition away completely from the former. Thus, the panel agrees with the view expressed in the report of the Commission on the Measurement of Economic Performance and Social Progress that “reliable indicators can only be constructed through survey data” (Stiglitz et al., 2009, p. 184). However, this constraint is likely to change going forward, partly out of practical considerations concerning the cost and viability of conducting large government surveys. Survey research is facing numerous challenges involving the impact on survey response rates of both technological factors (answering machines, mobile phones, etc.) and sociopolitical developments (respondent “burnout” from the proliferation of polls, mistrust of polls and pollsters, etc.). Lower response rates in turn affect the reliability and validity of telephone-implemented survey findings. Surveys such as the CPS (conducted through a combination of in-person and telephone instruments) that have maintained very high response rates (92-94 percent for the core CPS in 2003-2005) are extremely expensive to conduct. Their cost raises concerns about their sustainability and cre- ates a high-stakes competition for the limited space available on their questionnaires. Partly in response to these pressures, online surveys have emerged, some with promising results. Often, results from these surveys are of value not because they provide valid population-level information (though some panels are working to achieve this goal) but because they may offer a good laboratory for testing different approaches and hypotheses before embark- ing on larger, more expensive, and more burdensome programs. They may, for example, offer opportunities to study mode effects or to test differ- ent adjectives describing emotion, experience, or life satisfactions. More broadly, the emergence of big data (which consists mainly of data generated for purposes quite different from those driving government surveys) that can be captured from a variety of (largely though not exclusively) digital 20 The Mappiness developers note that between-individual confounding should not be a fac- tor because their model is estimated exclusively from within-individual variation (MacKerron and Mourato, 2013).

OCR for page 103
DATA COLLECTION STRATEGIES 123 information and communication technologies, coupled with advances in computational science analytic techniques, raises the possibility of devel­ op­ng less-obtrusive indicators of citizens’ well-being, behaviors, and opin- i ions. Researchers have, for example, accessed Twitter to study word use associated with different circumstances such as job search (Antenucci et al., 2012). Such exercises can be used to study changing word use in the population in order to better understand how respondents communicate. Researchers working at the University of Pennsylvania Computer Sci- ence Department have begun conducting research based on the idea that: The words people use on social media such as Twitter, Facebook, and Google search queries are a rich, if imperfect, source of information about their personality and psychological state. [They] are developing methods to estimate variation in subjective well-being over time and space from social media word use . . . [and] studying the variation in use of words relating to PERMA (Positive emotion, engagement, relationships, meaning, and accomplishment), and how these correlate with Gallup poll ­ nswers and CDC data at the State level.21 a Similarly, Quericia et al. (2012)—also working from a computer sci- ence background—engaged in a project to track “gross community happi- ness” for physical communities (London, in this case) from tweets. To this end, they examine, for a number of communities, the relationship between sentiment expressed in tweets and community socioeconomic well-being. They “find that the two are highly correlated: the higher the normalized sentiment score of a community’s tweets, the higher the community’s socio- economic well-being” (p. 265). Companies such as Knowledge Networks22 have made strides in online research, and online surveys are increasingly common in academic scholar- ship. However, questions remain regarding their ability to fully substitute for more traditional survey modes, and more independent comparative research is needed. Nonetheless, it is important that government agencies follow these developments so they are prepared to adjust the ways they gauge SWB and other important measures in the future. This will entail monitoring survey data collected by private and other public organizations in order to assess needs, determine the most effective and efficient use of scarce government-survey space, and develop survey measures that are both valid and reliable and that best complement and supplement existing regu- larly conducted surveys. They also must stay abreast of developments in the survey research field, including threats to traditional survey modes and 21 This webpage describing their work on “word use, personality and well-being” can be found at http://www.cis.upenn.edu/~ungar/CVs/WWBP.html [October 2013]. 22 See http://www.knowledgenetworks.com [October 2013].

OCR for page 103
124 SUBJECTIVE WELL-BEING developments in alternative survey modes such as online surveys. Finally, ­ for government data collection to stay relevant and feasible, statistical agencies will need to apportion some of their resources to following and understanding (and hopefully applying their own considerable expertise to) emerging methods of research designed to explore the use of both digital and digitized big data and other computational science methods for measur- ing people’s behavior, attitudes, and states of well-being.