An Agenda for Research in Survey Measurement of Work Disability
The committee's mandate includes advising the Social Security Administration (SSA) on the design, methods, and content of the Disability Evaluation Study (DES). The purpose of this workshop was to bring together disability researchers and experts in survey methods to discuss conceptual and survey design and measurement issues and to identify unanswered questions of measurement of persons with work disabilities. During the workshop sessions participants presented and discussed many issues and offered suggestions for further research. In the final session of the workshop, participants identified the gaps in survey methods and measurement of work disability and suggested a framework for long-term research for SSA and others in the field.
The measurement of the number of people with work disabilities is complex. The complexity stems, in part, from differences in the various conceptual models of the enablement-disablement process and alternative interpretations of the models. The measurement of persons eligible for disability benefits from the Social Security Administration (SSA) is made additionally complex by the incongruity between the various conceptual models and SSA's model based on its statutory definition of work disability. The various constructs do not necessarily identify the same population. Hence, throughout the workshop participants struggled with how to measure people with work disabilities—people who would apply for SSA benefits—and those who would be classified as persons with work disabilities as a result of the SSA benefits decision process.
In addition, examination of the empirical literature reveals significant voids in the field's knowledge of the error properties associated with the measurement of persons with disabilities or work disabilities. The limited empirical literature
raises questions about both the validity and the reliability of various disability measures currently in use in household-based surveys. Among the goals of the workshop was the identification by participants of gaps in knowledge about the error properties associated with the measurement of people with disabilities and the development of a research framework to address those gaps.
The framework outlined by the participants encompasses four broad areas of research:
Coverage Error. To address the serious shortage of research on the effects of coverage error on estimates of the population of persons with work disabilities or potential SSA applicants and beneficiaries, research is needed on the assessment and reduction of errors related to the choice of sampling frame and screening instruments.
Measurement Error. In light of the volatility evident in the limited empirical literature investigating the error properties associated with measures of disability and work disability, research is needed on the assessment and reduction of measurement error.
Nonresponse Error. To address the paucity of research on the effects of nonresponse error, research should be directed toward understanding the correlates of nonresponse and the impact of differential nonresponse on estimates of the population of persons with work disabilities or potential SSA applicants and beneficiaries.
Environment. To address the void between conceptual models of disability that identify the role of the environment in disability and the current set of measures used in surveys of persons with disabilities, research is needed on the development and assessment of effective measures of the environment.
Coverage error defines the failure to include all eligible people on the list or frame used to identify and sample the population of interest. The use of screening questions to identify the population of interest leads to an additional source of coverage error—the exclusion of persons because of inaccurate classification at the time of the screening.
Adequacy of Household-Based Surveys
Household-based data collection efforts, by definition, eliminate from the sampling frame those members of the population who are homeless, as well as those who are living in institutions. Those residing in group homes, assistedliving facilities, and other new types of living arrangements may or may not be included in the frame, depending on how the distinction between institutional
and noninstitutional residence is made. The extent of this coverage error when attempting to describe the entire U.S. population with disabilities is unknown and is likely to be a function of the type of disability, with estimates of the population with mental retardation or mental health problems most likely subject to the highest rates of coverage error. Empirical data are needed to estimate the differences in the rate of disabilities and the characteristics of the population with disabilities on the basis of household surveys compared with the rate and characteristics on the basis of data for the entire population.
Effects of Alternative Approaches to Screening
The use of a screening instrument to identify the population of interest affects coverage error. Three areas of research were enumerated as being particularly important with respect to the use of screening instruments:
The effect of alternative wording of questions on the identification of the population. Given the discrepancies in the rates of disability reported in the literature, establishing the reliability of screening items is particularly important.
Comparisons of estimates based on simultaneous screening and interviewing with those based on separate screening operations. Participants cited empirical evidence that the use of simultaneous screening and interviewing operations (compared with separate field operations to conduct only the screening) may result in a lower incidence of the characteristic of interest. This research should also focus on understanding the mechanism by which the two operations result in different estimates.
The effect on estimates when a subsample of cases classified as negative according to screening questions are included and rescreened as part of the extended interview. (Statistics Canada in its Health and Limitations Survey takes this approach.)
Estimates of the population with disabilities appear to vary as a function of the essential survey conditions under which the data are collected, specifically, the mode of data collection, the wording of the specific question, the context of the question, the overall content of the survey as well as the survey's sponsorship, and the nature of the respondent providing the information (self-response versus proxy response). However, the confounding of various design features prevents researchers from identifying the marginal impact of each.
Valid and reliable measures of participation by people with disabilities in the social and economic environments are needed, regardless of the number of questions required to do so. Valid questions must address the conceptual models that view work disability as a matter of degree, suggesting that the measurement of
disability be on a continuum as opposed to the dichotomous measures used in many surveys. Participants acknowledged that, regardless of the type of impairment, the development of valid and reliable measures of disability, and especially work disability, is a challenging undertaking. Participants stressed that the episodic nature of mental disabilities, as well as perceptions of social stigma, make the measurement of mental disabilities all the more difficult. The episodic nature of mental disabilities suggests that the reporting of a mental disability may be subjected to greater variability than is true with physical disabilities. Hence the reliability of such measures may be low, even though respondents are reporting quite accurately about their disabilities over time. Perceptions of social stigma most likely will affect the validity of reports of mental disabilities. Respondents tend to err on the side of not reporting such disabilities, especially in face-to-face interviews.
Workshop participants identified three areas of research for developing valid and reliable measures of work disability:
Assessment of the effects of specific wording and context of question. In the discussion of question wording, structure, and context, workshop participants identified a number of potential research topics.
Research directed toward understanding respondents' comprehension of the key concepts within the question. Participants suggested that investigations be targeted at understanding respondents' comprehension of the concept of disability as well as their comprehension of specific words used in the measurement of persons with disabilities, such as “difficulty,” “work,” “performance,” and “ability.”
Decomposing long questions used to screen for persons with disabilities. In light of the apparent discrepancies in the reports of disability obtained with complex screening items, participants suggested that investigators conduct experiments in which they decompose long, complex single questions into multiple items and make comparisons between the two approaches with respect to reliability, validity, and length of administration.
Assessment of context on estimates of the population with disabilities. Workshop participants called for experimentation to identify the role of context on estimates, where context is broadly defined, ranging from subjective factors such as mood to objective factors such as the survey sponsor, the questions immediately preceding the disability measures, and even such factors as the weather.
Assessment of the effects of self-reporting and proxy reporting. A limited empirical literature on the effects of self-reporting and proxy reporting of functional limitations suggests that the direction and the magnitude of response error are, in part, related to whether the report is provided by the individual or by proxy. Among proxy reporters, it is also a function of the relationship between the target individual and the reporter. Participants
identified the need for further evaluation of the validity and reliability of both self-responses and proxy responses, including, but not limited to, comparison of both self-reports and proxy reports with performance indicators for key impairment or disability measures and further empirical studies that address the direction and the magnitude of response error as a function of the relationship between the proxy reporter and the target person of interest (e.g., caregiver or noncaregiver spouse), especially proxy reports obtained for persons with mental retardation and mental health impairments.
Assessment of the effects of essential survey design features. The background papers (see Chapter 2 and Chapter 3) provided illustrative examples of how estimates of persons with disabilities or persons with work disabilities vary as a function of essential survey design features. A meta-analytic study that would establish significant design predictors of prevalence was discussed as a first step toward disentangling the marginal effects of various design features.
Participants also suggested a number of experiments designed to provide an understanding of the marginal impacts of these various design features. These experiments focused on the need to: (a) understand how the sponsorship of the survey affects both the properties of nonresponse (motivation to respond or not respond) and the measurement process (response editing and formation); (b) measure and examine the effects of the presence of others during survey administration, especially in the measurement of mental illness; (c) understand the effects of mode of interview, incorporating a design that is sensitive to the different issues associated with the measurement of those with physical, sensory, mental, and emotional impairments; and (d) investigate the incorporation of new technology (e.g., audio computer-assisted interviewing) to enhance participation and privacy among persons with disabilities.
Given that conceptual models of disablement view the process as a continuous, dynamic, interactive process, participants indicated that longitudinal measurement of disablement was imperative to understanding the relationship among individual characteristics, the environment, and work participation. Estimates based on longitudinal designs need to be compared with those based on cross-sectional data collection efforts.
Impact of Nonresponse Error on Estimates of Persons with Disabilities
Although no empirical data exist that address the impact of nonresponse on estimates of persons with disabilities, workshop participants speculated that the nature of a person's impairments or disabilities may result in differential non-
response among members of the population with disabilities. This void in the literature suggests that a priority with respect to nonresponse research is the assessment of differential nonresponse among persons with disabilities due to various physical, cognitive, and mental impairments. One approach could be the use of two-phase sampling of nonrespondents, that is, select a sample among those persons who were nonrespondents and then, using extraordinary methods, attempt to measure these individuals. The methods could include a significantly reduced questionnaire (for example, one that simply attempts to measure impairments and disabilities), the offering of incentives, or the collection of data via an alternative mode (e.g., from face-to-face interviews to telephone interviews). Such an approach would allow researchers to quantify the differences between respondents and nonrespondents as to the rates and characteristics of impairments and disabilities.
“Gatekeepers” and Interviewers
“Gatekeepers” (household members who may limit access to the sampled person) and survey interviewers may represent sources of nonresponse error unique to the measurement of persons with disabilities. Gatekeepers may limit access to persons with disabilities who, if provided with the opportunity, might be quite willing to serve as respondents. The contribution of gatekeepers to nonresponse, and the differential impacts of gatekeepers for telephone surveys compared with face-to-face interviews have never been addressed in the literature. Similarly, interviewers may classify sampled persons as incapable of serving as a respondent, because of cognitive capabilities, sensory impairments, or other impairments. Research is needed to address the extent to which such judgments by an interviewer result in nonresponse among the population of primary interest.
MEASURES OF THE ENVIRONMENT
One of the major voids between conceptual models of impairment and disability and the measurement of impairments and disabilities in surveys is the inadequacy of survey questions to measure the environment. Current data collection efforts, for the most part, fail to measure the environment and its impact either as a means of facilitating participation or as a barrier to participation in the social and economic environments.
Workshop participants underscored the need for the development of standardized objective measures of both the physical and social environments. Participants stressed that the measurement of environmental context should examine
both factors that accommodate impairments and those that serve as barriers. The development of objective measures of the physical environment may be facilitated by fostering collaboration with those in ergonomics and human factors engineering, fields in which a primary focus is the measurement of the environment. Some also noted that the classification of environmental features enumerated in the second revision of the International Classification of Impairments, Disabilities, and Handicaps (ICIDH-2) provides a well-defined architecture for the development of questionnaire items designed to capture environmental factors that affect the disablement process.
To aid in the development of objective measures of the social environment, some participants noted the need to develop and test questions concerning social climate, barriers, and stigma. These questions are especially important for those with mental illness, but they are relevant for and should be asked of all persons with disabilities. To aid in this development of objective measures of the social environment, some participants suggested collaboration with researchers who have developed measures of racial and ethnic discrimination.
One of the challenges related to the development of objective measures of the environment is the identification of a set of questions that can be asked of the general population. This objective contradicts the fact that barriers as well as accommodations in the work environment are a function of the person's impairment and the occupation in which he or she is employed. To fully understand either barriers to employment or factors that facilitate employment, questions need to be tailored so that they are relevant to the individual's situation. Ethnographic exploratory studies of workplace environments are one means by which to inform household measurement of accommodation and barriers. For those who are no longer working, questions that enumerate what accommodations would be necessary to facilitate, or barriers that prevent, participation in the workforce need to be designed and subjected to evaluation.
Similar to the discussion concerning objective measures of environment, workshop participants identified the need to develop subjective measures of both the physical and social environments that either facilitate or limit participation.
In addition to research to develop such measures, the discussion concerning the collection of subjective measures of the environment elicited two additional research topics: (1) assessment of systematic differences in evaluating the environment among those for whom the environment is benign versus those for whom the environment is hostile and (2) assessment of the difference between self-reports and proxy reports of subjective environmental conditions.
Workshop participants expressed strong sentiment that SSA and other federal agencies should undertake a systematic approach to addressing the research agenda identified during the workshop discussions. The approaches currently used in surveys to measure people with disabilities are incongruous with modern theoretical models. The empirical literature examining measurement error associated with specific questions, albeit limited, suggests that items currently used to screen or measure persons with disability are subject to low levels of reliability and are of questionable validity, especially for people with mental illness. The impacts of both coverage error and survey nonresponse on estimates of the population with disabilities and work disabilities have not been addressed in the literature. In light of these points, workshop participants indicated that the measurement of people with disabilities and work disabilities could be greatly improved with research directed toward one or more of these agenda topics.