Survey Measurement of Work Disability: Summary of a Workshop (2000)

The workshop sessions were designed to discuss issues relating to:

• the implications of different concepts for survey measurement problems;

• sampling, accessing, and measuring people with disabilities;

• questionnaire development issues for measures of work disabilities; and

• the role of environment in survey measurement of disability.

This chapter summarizes the workshop participants' discussions that flowed from the presentations of the two papers presented in Chapter 2 and Chapter 3. Participants identified many unanswered questions about measurement that need to be researched. Some of the key issues that surfaced during the discussions are summarized in this chapter.

One of the major challenges related to the measurement of persons with disabilities and persons with work disabilities concerns the translation of the various complex conceptual models into questions that can be comprehended by the general public and that produce both valid and reliable measures. The Social Security Act defines disability (for adults) as “. . . inability to engage in any substantial gainful activity by reason of any medically determinable physical or mental impairment which can be expected to result in death or which has lasted or can be expected to last for a continuous period of not less than 12 months. . .” (Section 223 [d][1]). As stated, the definition appears to presuppose a simple

relationship between a medical condition and the ability to work and one that is amenable to measurement via the survey process. In contrast, most contemporary theory concerning disability focuses on the complex nature of the relationship between medical conditions and disability and the importance of environmental factors, including the physical and social environments, in mediating the relationship between a medical condition and work. The first session of the workshop focused on a review of the two background papers, addressing the various conceptual frameworks, the complexity inherent in the measurement of disability, and the empirical evidence of measurement error associated with instruments that have been used to measure disability and work disability.

As discussed by Jette and Badley (see Chapter 2) both the Nagi model of disability and the model proposed in the second revision of the International Classification of Impairments, Activities, and Participation (ICIDH-2) emphasize that the accomplishment of particular social roles, such as work, involves not only the accomplishment of the particular activities related to the work task but also the ability to perform with respect to several other activity domains. For example, participation in work activities may also involve the ability to use transportation systems to get to and from work. Therefore, the inability to find accommodating transportation may result in a “work disability,” even though the individual is quite capable of performing the tasks associated with his or her occupation.

Both the Nagi and ICIDH-2 conceptual models view the disablement process as a function of the physical and social environments, accommodations and barriers within those environments, and personal attributes and resources. Work disability therefore is unlikely to be related either directly or only to a health condition or impairment. Rather, from the perspective of these conceptual models, work disability (or any other social role) is a function of the interaction of the physical and social environments, accommodations and barriers within those environments, together with the psychosocial and other attributes of the person. Therefore, the measurement of work disability should take into account personal attributes, as well as the social and physical environments in which the person operates.

The complexities of the conceptual models of disability lead to tension in the measurement of disability. The measurement of disability is further complicated by the often ambiguous and varying uses of the terms used in conceptual models and measurement instruments both within and outside the field. In addition, in light of the passage of the American with Disabilities Act, the language is evolving. These factors suggest the use of a survey instrument in which multiple questions concerning disability are asked to set the context for the measurement process, clarify terminology, and tap the multiple domains of interest. This conflicts with the desire of the Social Security Administration, as well as numerous other federal agencies, to identify a short battery of questionnaire items that can

identify persons with disabilities and work disabilities through the survey process. Of primary concern is the fact that the questions used to identify persons with disabilities should be both valid and reliable and should be robust when used under varying survey design features.

The empirical literature addressing the measurement error properties of disability and work disability, albeit limited, provides evidence of low reliability and questionable validity. For example, questions concerning disability were asked in the 1991 Canadian census; individuals were sampled on the basis of their responses to the census and administered the Health and Activity Limitation Survey (HALS). The sample included both individuals who indicated a disability and those who did not in response to the census questions. Of the 35,000 persons classified as disabled on the basis of answers to the census questions, 20 percent were classified as not disabled on the basis of responses to HALS. Among the 113,000 persons classified as not disabled on the basis of the census data, 5 percent were classified as disabled on the basis of the data from HALS. Similar examples of an apparent low reliability are evident in the 1990 U.S. census and the CRS (see examples provided in Chapter 3, p. 51). Factors such as the wording of the question, the context of the question, the mode of data collection, the nature of the respondent (self-response versus proxy response), and the sponsorship of the survey all appear to affect estimates of disability.

Even if it is well measured, work disability is a matter of degree, suggesting that the measurement of work disability should be on a continuum as opposed to the dichotomous measurement currently used in most surveys. The categorization of an individual as “work disabled ” or not could then be based on clearly defined and established thresholds.

The survey challenges associated with the measurement of persons with disabilities are not limited to measurement error but must also address errors of nonobservation. Particular physical or cognitive disabilities may affect an individual's likelihood of inclusion in the sample or participation in the process if sampled. For example, the use of telephone for data collection limits participation among deaf persons.

Several key issues were identified during the discussion, including the following:

• Work disability is a multidimensional, complex concept that investigators do not know how to define well. Nor is the relationship among an individual's characteristics, the environmental characteristics, and the phenomena of disability understood. Disability or limitations in participation are a result of a complex interaction among an individual and his or her attributes, the physical and social environments, and the accommodations and barriers to participation. The measurement of disability is further confounded by the fact that all characteristics, whether intrinsic individual characteristics or characteristics of the environment, change with time.

• The measurement of the environment is not limited to the physical, tangible elements but includes all aspects outside the individual, including the social environment. Attitudes held by other individuals, discrimination, and laws all affect the environment and determine the extent to which the environment accommodates or hinders participation.

• There is a significant gap between the theoretical concepts and the current set of measures that are being used. In addition, there is little information about the measurement error properties of the survey measures currently being used to enumerate persons with work disabilities.

• A meta-analysis of variations in estimates of the prevalence of disability and work disability as a function of various survey design features would be informative about the relative effects of various features on survey estimates. The meta-analysis must include studies from outside the United States. Similarly, further exploratory analyses of the data that currently exist may provide additional insight as to the mechanism(s) that leads to variability in survey estimates of disability.

• In addition to the meta-analysis and further analyses using existing data, workshop participants noted the need for a program of experimentation to identify the relative contributions of various survey design features to variability among surveys and variabilities in repeated measurements of the same individual.

The development of valid and reliable measures of persons with work disabilities requires that one identify differences in the various conceptual models and the implications of those differences for survey measurement. Workshop participants were asked to consider the following questions:

• What are the key differences among the concepts that have implications for survey measurement? How would one go about assessing, for example, the practical import of the conceptual differences in a survey measurement environment?

• For the purposes of measuring work disability, what components must or should be measured? What research should be conducted to eliminate gaps in the knowledge needed to answer that question?

• Do any of the measures presently being used represent a “gold standard?” What research should be conducted to eliminate gaps in the knowledge needed to answer questions concerning a gold standard?

• What is known about the use of terminology related to disability by lay people, and do any of the conceptual models provide a framework that relates better or worse to the terminology used by the lay public? What

research needs to be done on translating the concepts to the terminology used by the lay population? Does the use of a particular terminology vary across subgroups?

Despite apparent differences between the conceptual models, workshop participants noted that there is, in fact, a lot of commonality among the models. Of critical importance with respect to the operationalization of the various conceptual models is a clear distinction between the measurement of capacity and participation. Both need to be measured, as do the factors that illuminate the relationship between capacity and participation. Further research is needed on the development of measures of capacity and participation.

One of the gaps identified by workshop participants is the lack of research addressing the relationship between measures of quality of life and general health status and measures of disability and work disability. One means by which the relationship between measures of health status and measures of disability can be addressed and the measurement of capacity can be disentangled from performance within health status measures is to foster cross-disciplinary research between the health status research community and the disability research community.

The disablement process is described in the conceptual models as a continuous, dynamic, interactive process, implying that longitudinal measurement is imperative to understanding the relationship among the individual's characteristics, the environment, and participation in the workforce.

Participants stressed the need to develop valid, reliable measures of workforce participation, regardless of the number of questions required to do so. Once a gold standard is developed, the predictive reliability of any subset of those questions could be established.

In summary, the sessions identified the following key issues:

• The various conceptual models have commonalities that, for survey measurement, imply little or no difference in what needs to be measured.

• Regardless of the conceptual model of interest, there is a need to understand and measure both capacity and performance, contrasting self-reports of capacity and performance with objective performance measures. Many of the current health status measures sets confuse capacity for work with actual work performance. Further research is needed to better understand the relationship between more general health status measures and measures of disability.

• The conceptual models imply that a dynamic relationship between a person, a set of environmental factors, and participation exists. Such models beg for longitudinal measurement to further understand the dynamic nature of disability.

• Central to the measurement of disability or work disability is the issue of duration. Even in cross-sectional data collection efforts, the goal is to characterize an individual not simply with respect to impairments, limita-

tions, or disabilities on the day of the measurement but rather with respect to some notion of duration. Duration has been measured in terms of time since initial onset, time since most recent episode, and even expected duration. Further research is needed on the best means by which disability should be defined with respect to the dimension of duration.

• The participants called for the development of a set of valid, reliable measures of work disability, regardless of the number of question items necessary to capture the multiple dimensions of disability and the various domains outlined in the theoretical concepts. Once such a set of measures is established and the relationship between self-reports and unbiased observation or performance is documented, research could address the reliability and validity of a short form of the gold standard.

Errors of nonobservation are of potential concern for any data collection effort. Of particular interest with respect to the sampling and measurement of persons with disabilities are issues related to coverage error (i.e., the exclusion of persons with disabilities from the frame used for sampling) and nonresponse error (i.e., the extent to which those who do not participate—because of an inability to be contacted, an inability to participate, or an unwillingness to participate— differ with respect to disabilities from those sampled individuals who agree to be interviewed). Design decisions related to the mode and method of data collection, usually viewed as decisions that affect measurement error, also potentially affect both coverage error and nonresponse error.

Obtaining estimates of the population of persons with disabilities raises particularly challenging issues with respect to the development of a frame from which individuals can be sampled for inclusion in a study. The workshop participants noted that persons with disabilities are probably more likely to live in institutions or in group quarters or to be homeless; hence, reliance on a sample frame consisting of households will result in noncoverage of the population of interest.

Design decisions concerning mode of data collection, sample frame, and nonresponse are closely interconnected, perhaps more so for the measurement of persons with disabilities than for other measures. For example, if one is interested in conducting a telephone survey, then the sampling frame is not all households but rather households with telephones. Persons living in households without telephones will have a zero probability of selection. To the extent that individuals who live in households without telephones differ from those who live in households with telephones, the resulting estimates will suffer from coverage error. However, the choice of the telephone as the mode of data collection further

assumes that within the households with telephones there is at least one person who can hear well enough to participate.

Findings from the National Comorbidity Study were discussed as illustrative of the interplay among decisions concerning mode of data collection and potential nonresponse error. This study was designed to measure the prevalence of disabling psychotic disorders. The survey was conducted as a household-based face-to-face survey, viewed as the best mode of data collection. The survey achieved a respectable response rate of 85 percent. That meant, however, that 15 percent of the sample selected for participation did not participate, and the prevalence of this type of disability is estimated to be about 1 percent. To address this issue, the investigators surveyed 20 percent of the nonrespondents. They were offered incentives, alternative modes of data collection (e.g., telephone), or alternative locations for data collection (e.g., public place). The nonresponse study found relatively high rates of anxiety disorders among those respondents who were willing to complete the nonresponse interview but who were not willing to have a stranger in their homes. The findings suggest that, for this particular study, the choice of mode of data collection was related to the nonresponse error. Permitting flexible modes of data collection may have resulted in a reduction of the nonresponse rate and a reduction of the nonresponse error.

Workshop participants also discussed the following topics:

• The critical issue that has not been addressed in the research literature is the relationship between the measure of interest, persons with disability, and either coverage error or nonresponse error. The phenomena that investigators are interested in measuring will, in some cases, determine noncoverage for particular types of sampling frames and nonresponse for particular modes of data collection. To successfully sample and include the population of interest, one must consider multiple frames for sampling and multiple modes for data collection.

• Because the phenomenon of interest is a dynamic rather than static state, it is important that the design of any study in which a questionnaire is used to screen or identify the population of interest include in the second stage some portion of the respondents who were not classified as impaired or disabled.

• One dimension to be considered in the sample design is time; for example, homeless individuals are not necessarily homeless all of the time.

• The assessment of disability relies on the willingness of respondents to reveal the information. Hence, one focus for research is to address the motivation of respondents to accurately report the presence of a disability.

• Disability cannot be defined as an attribute of the person apart from the environment in which he or she resides; hence, attempts to measure the person outside the context of that environment are not relevant.

Cognitive psychology provides a theoretical framework for addressing the processes involved in the question-answer interchange between an interviewer and a respondent. This framework is informative with respect to where the process may be problematic for the measurement of persons with disabilities. The question-answer process is most often categorized as a four-stage process involving comprehending the question, retrieving the information, making a judgment concerning whether the retrieved information is relevant, and formulating a response. Given the complexity of the concept of interest, workshop participants indicated that comprehension represented the largest challenge in the development of effective questions to measure disability and work disability. Survey questions are probably most effective in screening out the large number of persons without disabilities as well as identifying persons who are current beneficiaries of Supplemental Security Income or Social Security Disability Insurance. Questionnaires were viewed as most problematic in identifying disabled persons who currently received no benefits and who may or may not be employed.

Potential comprehension problems associated with the measurement of persons with work disabilities were illustrated by using questions scheduled to be included in the long form of the decennial census for the year 2000. The proposed question is “Because of a physical, mental, or emotional condition lasting 6 months or more, does this person have any difficulty in doing any of the following activities? Working at a job or business?” The response categories are yes or no. The question includes at least three distinct elements: (1) a causal connection between the disability and an underlying medical or psychological condition; (2) the condition is enduring; and (3) the condition results in an inability to do something, in this specific case, the inability to work. To answer the question the respondent must determine what is meant by work, must determine whether the inability to work meets the criteria related to the time element, and must attribute the inability to work to an underlying condition. In addition, the question involves an implicit comparison to a standard, for example, the ability or inability to perform some set of tasks associated with a particular job.

The complexity of the question contributes to problems of comprehension. The empirical literature indicates that many of the questions designed to measure disability are subject to context effects, providing evidence of comprehension problems. Workshop participants called for research in which the reliability of complex multidimensional questions designed to measure persons with disabilities is compared with the reliability of a series of shorter items. The hypothesis is that the use of several less complex questions may improve comprehension with little or no additional administration time for the interview.

Many survey questions that attempt to measure disability include a dimension of duration (e.g., “lasting six months or more”). For some questions, dura-

tion is expressed in the past tense; other items express duration either in the past tense or as a duration that is expected to continue for a specific time period into the future. Although the intention of including a dimension of duration is to eliminate short-term disabilities (e.g., a broken leg that will limit the respondent for the next 6 weeks), workshop participants expressed concerns as to respondents ' understanding of the dimension of duration as well as concern with the determination of an appropriate duration interval. This was of particular importance for the measurement of persons with intermittent disabilities. Given the dynamic nature of disability, what is the appropriate time period to be used in the survey question? Workshop participants reiterated the need for research related to respondents ' comprehension of the questions designed to measure work disability. For example, among persons with episodic mental disabilities, questions of duration become quite difficult to answer. Should the respondent report with respect to date of first onset, total duration of episodes, or duration of most recent episode?

What information concerning disability can and cannot be retrieved during the course of an interview? For many kinds of questions dealing with autobiographical memory, retrieval of the information is the most difficult task facing the respondent. This is most likely not the case with respect to work disabilities. Persons with disabilities and their immediate families are very likely to be aware of a disability if there is one and to be able to retrieve it from memory with little difficulty, if the respondent understands what is being asked. Similarly, the process of making a judgment concerning the retrieved information or formulating a response does not present the respondent with cognitively difficult tasks, but, rather, presents the respondent with tasks that may have implications with respect to issues of social desirability. For example, in a setting in which a respondent is applying for disability benefits, there may be a strong motivation to edit responses toward disability. In household interviews, on the other hand, individuals may wish to deny or minimize their disabilities. Workshop participants suggested the need to conduct research on how to reduce question threat through the use of self-administered modes of data collection, including the use of audio computer-assisted self-interviewing procedures.

The nature of the phenomena of interest implies that information often will be obtained from proxy respondents, that is, one person reporting for another individual. Although there is a small body of empirical literature indicating that responses concerning functional assessments or disabilities obtained by proxy differ from those obtained from a respondent reporting for him- or herself, the direction and magnitude of the response error are not well understood. For items perceived to be threatening, responses obtained by proxy may be subject to lower levels of response error than responses obtained from the target individual.

The discussion resulted in the identification of several additional research activities related to questionnaire development. These include the following:

• A program of research to address the extent to which change over time reflects real change as opposed to measurement error. Although several data files that include repeated measures of disability exist, the time between the first and subsequent interviews is often 6 months or more, a period during which it is quite conceivable that, in fact, some change has occurred. A reinterview program in which respondents are reasked the critical questions of interest within a relatively short time frame would permit estimation of simple response variance. Resolution of discrepancies between reports for the two time periods would permit the separation of true change from response variance.

• An experiment or series of experiments to examine differences in comprehension and response distributions associated with a single, multidimensional disability question (e.g., the ones proposed for the decennial census) compared with those associated with a questionnaire in which a single question is decomposed into a series of shorter items.

• A research program to address the contextual factors that affect responses to questions on disability. Context would include the questions immediately before the disability items, the overall content of the questionnaire (including the sponsorship of the survey), and the environmental context (e.g., the unemployment rate or the weather).

The literature offers at least two frameworks for the measurement of the impact of environment on participation among persons with disabilities. The Quebec model (Fougeyrollas, 1998) focuses on obtaining ratings of the individual's life habits, for example, determining the level of accomplishment for a certain activity (from no difficulty to not performed) or ascertaining the type of assistance that a person uses, including assistive technology, environmental modification, and personal assistance. Respondents are also asked to rate the environment, where environment includes such factors as the physical environment, social attitudes, and rules related to the workplace.

An alternative framework is offered by the ICIDH-2, which can be used to classify the environment on several levels, including the personal level (e.g., the immediate environment of the person including, but not limited to, the home, school, and workplace), the community level (including economic and social institutions), the broader cultural and social customs and structures, the physical, human-made environment, and the natural environment.

To measure environment, researchers need to have a shared understanding of activities and participation, using the ICIDH-2 vernacular. Activity, sometimes referred to as “naked capacity,” is the ability to perform an activity without any kind of technology or personal assistance. In measuring activity, one could con-

sider a continuum ranging from “extremely well” to “not at all.” Performance could then be defined as performance of activities with all types of assistance. Respondents could be asked whether they do the activity with any type of assistance and, if yes, what kind of assistance. This type of question sequence would allow researchers to understand the types of assistance or environmental accommodation that facilitates the performance of an activity. Such an approach allows people an opportunity to express their views of their level of performance using available assistance and technology.

Environment also alters an individual's perception of disability. For example, the number of applications for SSI and SSDI benefits often varies as a function of the economy, suggesting that individual perceptions may, in part, be a function of the labor force options available. The passage of the Americans with Disabilities Act most likely will change the social climate concerning disabilities and entitlements of persons with disabilities; hence, over time, there may be an increase in the number of individuals who classify themselves as disabled.

Workshop participants expressed concern about the validity of the perspective of the respondent as reporter of the environment, as well as the validity of reports of the environment provided by proxy respondents. To alleviate these concerns, participants recommended comparison of both self-reports and proxy reports of the environment with unbiased measures of the environment to provide empirical data on the validity of the reports by the two types of respondents. For example, work environments could be sampled by trained observers who rate the environment and by comparisons of those ratings with reports obtained from self-reporters and proxy respondents.

• To understand the tasks associated with a particular occupation, rather than looking only at how able-bodied people perform the task, the manner in which people with disabilities perform the task should be examined. Such information could be incorporated into the Occupational Information Network (O*NET) to determine whether other individuals with similar scores are participating in a particular occupation.

• The presence of others during a survey changes the environment of the measurement, and may affect the reporting of disabilities.