Measuring Functional Capacity and Work Requirements: Summary of a Workshop (1999)

This session was designed to discuss the following issues:

• What are the strengths and limitations of self-reports, proxy reports, performance testing, and clinical observation?

• How do they vary across different domains and their components?

• To what extent should assistive devices be considered in measuring functional capacity?

• What are the practical implications of the issues for the Social Security Administration (SSA) (e.g., instrument practicality, ease of administration, safety, cost)?

• How should issues of reliability and validity be incorporated in the context of SSA's disability decision process?

• Do different populations have different measurement requirements (e.g., schizophrenia versus arthritis versus spinal injury versus Alzheimer's disease)?

The principal discussion question for this panel is: What are some of the strengths and limitations of different approaches to measurement—self-reports, proxy reports, performance testing, and clinical observation? What criteria should guide the selection of measures of functional capacity? A comparison of the rela-

tive strengths and weaknesses of the different measurement approaches requires a careful consideration of both the conceptual and methodological issues.

One criterion is assessment within a conceptual framework. To evaluate and compare different measurement approaches, it is important to know precisely what is meant by the concept ''functional capacity" in relation to the SSA's disability decision process. SSA's definition of disability is the inability to engage in any substantial gainful activity (SGA) by reason of any medically determinable physical or mental impairment. Clearly, the focus in work disability is on the inability to engage in SGA for specific reasons. The focus is on the inability to engage in that activity, not the inability to do specific tasks within gainful employment, not the inability to use certain muscle groups, and not the inability to do other very specific tasks.

The focus of the process here is the outcome. The process that the agency is dealing with is substantial gainful employment, a concept that is very complex and difficult to assess. As pointed out earlier, in order to assess one's ability to do gainful activity, the capacity of the individual has to be taken into account and there are several approaches to help assess that. However, the environment also has to be taken into account to reach a useful determination, not only the physical environment, but also the social environment. It is not enough to just assess capacity. Determination of work ability, therefore, requires: (1) determination of the requirements of the job and work environment, and (2) assessment of an individual's capacity to work.

From a conceptual point of view, the Nagi Disablement Model, shown in Figure 4-1, is useful to help think about measurement (Nagi, 1991; Verbrugge and Jette, 1994).

Within this disablement model, pathology may result from an injury as well as from infection, metabolic imbalance, degenerative disease processes, or other etiology. It involves the disruption of normal processes as well as the simultaneous efforts of the organism to restore a normal state. Impairments, anatomical, physiological, mental, or emotional, include residual losses or abnormalities that result from pathology (or other sources). An important characteristic of impairment is that it refers to abnormality at the organ or body system level as opposed to the entire organism. For example, impairments resulting from a lower extremity fracture include restricted range of motion and diminished muscle strength. Functional limitations or functional incapacities refer to the difficulties the individual has in his or her capacity to perform certain tasks considered normal for everyday living (e.g., walking, handling and grasping objects, climbing stairs, thinking, etc.). Functional limitations refer to the individual's capacity to do certain tasks, not to whether, how often, or how the individual actually performs the tasks in daily life. Disability is defined as limitation in performing socially defined roles and tasks within a sociocultural and physical context, including: (1) independent living (e.g., basic ADLs); (2) social interactions (e.g., church, contact with friends); (3) major usual activities (e.g., school, work, instrumental ADLs); and (4) recreational roles (e.g., hobbies, sports). Economic, social, educational, and psychological resources avail-

FIGURE 4.1. A disablement model. SOURCE: Jette, 1997. Reprinted with permission of Lippincott Williams & Wilkins.

able to the individual as well as characteristics of the physical and sociocultural environment will interact to affect the extent to which impairments and functional limitations result in actual disability.

One way of helping to clarify the issue of focus of measurement is to reflect on the difference between assessing an attribute versus a relational concept (Nagi, 1991). Attributes are disablement outcomes that pertain to the characteristics or properties of a body system or individual. Examples of impairment attributes commonly addressed in work-related injury rehabilitation include muscle strength and range of motion. Gait speed, lifting capacity, and bending ability, by contrast, reflect attributes of the individual's performance; thus they are functional limitations. To assess work-related attributes, be they impairments or functional limitations, one need not look beyond the individual or body system to identify what to assess. An assessment of function or impairment can thereby be used to predict actual work disability.

Relational concepts, in contrast to attributes, cannot be accounted for solely among the characteristics of a body system or the individual. Relational concepts include elements of the situation other than the individual's attributes. In the disablement model, the concept of disability is a relational concept. Therefore, to assess work disability directly an indicator must assess the individual's capacities in relation to work role expectations as well as the social and physical environmental conditions in which they are to be performed. To understand the reasons why a person has a work-related disability, one has to consider not only the person's own functional capacities, but also the person's capacities in relation to

relevant physical and social environmental expectations and the individual's response to his or her limitations in capacity.

A clinical example may help clarify the distinctions being made between attributes and relational concepts. Two patients with low back pain may present to a physical therapy rehabilitation program with very similar clinical profiles. They both may have moderate pain with impairments, such as limited lumbar range of motion and muscle weakness. Their pattern of functional limitation may also be similar, with difficulty rising from a chair, limitations in lifting capacity, and slow, painful gait patterns. Their work disability, however, may be quite different. One patient may have severely restricted his or her outside-the-home activities, including work, staying in bed most of the day watching television. The other may be engaged in his or her social and occupational life, albeit at a restricted level of activity. These two patients with the same pathology present very different work disability profiles (i.e., the relational component) yet have very similar profiles of attributes (i.e., their underlying impairments and functional limitations). Reasons for the difference may include degree of satisfaction with one's job, level of physical demands in the job, depression secondary to the low back injury, or issues of secondary gain.

Briefly, within the SSA disability decision process, should the focus of measurement be on specific impairments, functional capacity to work, or on work disability? Should work disability be assessed directly or predicted through the assessment of individual attributes? The answer to these conceptual questions is critical to determining the relative merits of different assessment methods (performance testing versus self-report versus proxy reports, and so on). Based on a review of the current literature on work-related disability determination, it appears that the field is unclear as to the appropriate focus of measurement for the purpose of making work disability decisions. Existing work disability determination approaches in the relevant literature include the assessment of impairments, functional capacities, or disability and, in some cases, combinations of concepts without regard to any underlying conceptual framework.

In the context of this model, SSA is interested in the concept of disability from the point of view of work. SSA is not interested directly in function or in impairment or in pathology, but only as they bear on the concept of work. So the validity question is the key one for any kind of process or measurement approach that the SSA comes up with, evaluates, and promulgates in the future. The question is to what extent the process tells us something about the individual's limitation or ability to perform the work role. In order to answer that, one has to take into account the environment. Capacity is part of it, but disability as defined by SSA is influenced by the environment. Thus it is understandable why people come up with approaches to measure function as a capacity assessment, but the actual outcome people are interested in is disability, which is more complex to measure. These measures become relevant only as they help us predict disability.

SSA's goals are to find a measure of "functional capacity" that will be reliable (reproducible) and valid, so it really tells us something about the underlying con-

cept of work disability. Furthermore, it has to be sensitive and specific, so it can deal with false positives as well as with false negatives (i.e., it has to be both sensitive and specific). In addition, it has to be practical, safe, and, ideally, inexpensive to administer. That is no easy task. In fact, anyone who has done work on assessment knows that attaining this goal is like searching for the ''holy grail." It cannot be done; it is extremely challenging to try to meet all these criteria.

A major challenge for SSA in trying to measure disability is to balance the scientific concerns (like reliability and validity) with the practical and pragmatic concerns requiring real-world tradeoffs. One has to give up some reliability and validity, sensitivity, and specificity if, in fact, the goal also is to come up with something really practical and inexpensive to administer. It is not possible to achieve all of these.

From an assessment point of view, going back to the concept presented, the simplest thing to do is to measure disability directly. The easiest way to measure a concept like work disability is to directly ask people about their ability to work. That is the simplest, most practical, and efficient way to do it. However, there appears to be a general sense that direct assessment of disability, although extremely practical, efficient, and attractive, would not be scientifically justified because of the possibility of false positives. A lot of people could be designated as eligible for disability benefits when in, fact, they were not truly work disabled. That is a validity concern. Therefore, efforts have been made to back up and develop indirect methods of assessment by measuring a person's capacity to do specific tasks that hopefully will be a bit more precise, that will allow prediction of whether or not someone has the capacity to do work, and somehow integrate these with an assessment of what the demands are in the work environment.

There are many standardized protocols that are quite reliable and valid for assessing the individual's capacity to do specific functional tasks. There are also methods that are available and being used that look at organ and body system impairments. In evaluating protocols that assess either of these concepts, the implicit assumption is that a measure of body system impairment or functional limitation will accurately predict level of work disability. The extent to which this assumption is true can be demonstrated empirically, and this challenge represents a classic validation research question. Once draft protocols are developed for this SSA initiative, empirical testing needs to be conducted to demonstrate the degree to which the chosen protocols validly predict level of work disability in this population.

What dimensions of functional capacity and/or disability should be assessed? Whether one directly assesses an individual's actual disability or predicts his or her level of disability based on an assessment of impairments and or functional capacity, one has to decide on the scope of the assessment or relevant domains to be examined.

For example, in the literature on work-related functional capacity assessment, a range of functional domains have been described by Lechner and asso-

ciates (1997). They include: lifting, standing, walking, sitting, carrying, pushing pulling, climbing, balancing, stooping, kneeling, crouching, crawling, reaching handling, fingering, talking, feeling, hearing, and seeing. Using the Functional Independence Measure, Stineman and colleagues (1997) recently published an analysis of disability domains included within this measure. They identified four different dimensions: self-care, sphincter control, mobility, and cognitive disability.

The customary ritual of scientific evaluation when one is considering the use of an assessment approach is to review the existing literature on relevant assessment protocols and obtain documentary evidence comparing empirical evidence of the various protocols.

Reliability and validity for different protocols are key elements of the process of justification. If a protocol has been designed to measure change within persons over time, its ability to detect minimally clinically important differences, a property called responsiveness , is also important (Guyatt et al., 1987). The specificity of a protocol with respect to the identified domains of function or disability to be assessed must also be considered (Feinstein et al., 1986).

Minimum psychometric criteria, as recommended by Spitzer (1987), include:

• The performance characteristics for content validity, criterion validity (if feasible), or construct validity should have been declared in advance of evaluating the protocol.

• Content validity should be enhanced at the development stage of a protocol by invoking the views of representative panels of patients, providers, and ordinary citizens.

• Reliability should be verified by those for whom use of the instrument is intended.

• When a gold standard exists, criterion validity should be done with the types of patients among whom the scale will eventually be used and under similar circumstances.

• When a gold standard does not exist, construct validation should include at least one approach of discriminant validity and one of convergent validity.

The quality of existing literature on evaluating the psychometric properties of assessment protocols used for work-related disability determination appears very mixed.

In addition to determining a protocol's psychometric properties, one needs to determine the protocol's suitability for its intended purpose and setting. This includes determination of issues such as safety, cost, practicality, respondent burden, ease of analysis, and reporting. If the intended purpose and setting are not suitably addressed, an established protocol will be unsuitable, no matter how good its psychometric properties.

Whatever process SSA decides on, the concern and challenge is that the farther one moves from a direct assessment of work disability, the more crucial will be the need for validation studies to show that an assessment of capacity to function (an assessment of organ or body system impairment), tied somehow with an assessment of the demands in the work environment, will be predictive of the individual's work disability. That is what SSA is searching for. Whether it can be done and also be practical and inexpensive is doubtful.

The easy part is to actually calculate reliability, validity, sensitivity, and specificity. These methods are well established and studies can be readily designed and implemented. The challenge is to come up with a protocol and a process that have a reasonable expectation for delivering the predictive validity and still retain those qualities of low-cost and practical implementation.

When identifying measures of functional capacity or any other ability, the measurement ideals must be kept in focus. Measures are need that are objective, that people can agree on, that are not the result of subjective whim, that are reproducible, and that are not dependent completely on the situation, the occasion, and the circumstances, but say something about the underlying quality of interest—work capacity in this case.

Measures need to be unidimensional, that is, measure one thing and not multiple things. Ideally, measures should be derived from theory. A collection of items that hopefully may add up to something may not measure what we want if we are stumbling around in the dark. A theory sheds light and is critical, though experience with different items and measures may lead to revising the theory.

Ultimately, the final measurement ideal is that the items used define a particular construct. Of course, they are subject to revision based on experience gained. The distinction between theory and method should be recognized—what is being measured (e.g., functional capacity, work ability, or something else) and how it is being measured (i.e., the actual instrument or tool).

Some of the desirable characteristics are well outlined in the psychometric literature. A measure ought to span a sufficient range of ability. It should have a sufficient number of strata of capabilities, capacities, and abilities, whether that is reading, writing, arithmetic, work, or something else. A measure that distinguishes only high and low might be good enough to determine if someone deserved disability benefits. Usually, though, more subtle distinctions are needed, at least a high-middle-low or even more refined categories.

A measure should be targeted on the sample or population of interest. For example, if the functional independence measures of the mini-mental examination are administered to a general population, clearly the instruments would not

be relevant, but they would be applicable to some of the applicants for disability benefits.

Measures also should be "equal-intervaled" as in a yardstick, that is, the distance between "1" and "2" is the same as the distance between ''14" and "15.'' The rating scale, or Rasch, analysis developed out of the educational measurement field provides a useful method for addressing many of the issues of unequal measurement and defining a hierarchy of items. Some of the key concepts that this model distinguishes fall generally within item response theory, such as distinguishing the ability of the person separate from the difficulty of the items that are administered to the person, and the ability to define a hierarchy of each of the items for better performers to poorer performers, which measure the construct of interest.

Distance between rating scales also can be quantified with some known measurement error between those points. Raters, as well as people who give opinions about the severity of functional limitations, can be calibrated on their performance, thereby removing the effects of their leniency or toughness, consistency, or erratic decisions from ratings of functional capacity or whatever ability is being measured. This has to do with the concept of "fit." The fit of persons, items, or raters can be quantified to the underlying measurement model. For example, when applying a functional status measure to a patient with spinal cord injury, the normal expectation is that the higher up the spine, the more limited and impaired is the person. Some people, however, with relatively high lesions are able to walk but not move their upper extremities. These people have what is called central cord syndrome. This group of persons does not fit the functional capacity measurement model for spinal cord injuries. The functional capacity items need to be calibrated separately for this subsample of persons with spinal injuries.

The same issues may be relevant in the mental impairment field, where issues of the way in which depression results in functional limitations versus schizophrenia versus something else may require subsamples of people to calibrate the measures of functional capacity.

Both in educational testing and particularly in certification of professionals, concepts of computerized adaptive testing and item banking have received a lot of interest in recent times. The same kind of applications might be applied profitably to work on disability issues.

Most of the problems that SSA faces relate to definition and policy decisions. Ultimately, definition has to drive the whole process. The basic questions that need to be answered before designing a measure or measurement process are:

1. What is being measured ultimately: ability or disability? Each requires a different approach and emphasizes different issues. How is this construct defined? From the SSA mandate, it would seem that the purpose is to determine disability, as defined by Congress. Also, it must be decided if the goal is to identify those who are disabled among those who apply for disability, or whether it is critical to know everyone in the country with disabilities, as they may have an entitlement and not know it.

2. How is the definition made operational? An operational definition needs to be developed for disability. This is ideally in the form of criteria answering the question: "How will we know if someone is disabled?" This definition should be based on work performance and not on an indirect functional measure.

3. Is there a "gold standard" against which to assess any measure that is developed? If not, can a substitute standard be developed which would depend on longitudinal assessment, expert opinion, and all data assessment (LEAD). This would be a process that (almost) everyone would agree could correctly classify individuals as "disabled" or "not disabled" without worrying about the resources or cost needed to make the determination. Put another way, if resources were not an issue what would be done to decide if someone is disabled or not? Some measure that approximates a gold standard will be needed in any validation study.

4. Once a LEAD standard is in place, several potential methods of determining disability can be tested against this standard. This step is critical. If "the truth" cannot be known directly, then we must come up with a consensus way of determining it.

5. Disability determination might use screening, two-stage testing, comprehensive evaluations, and other approaches to establishing who is disabled and who is not. Ultimately, there will not be a single measure that decides disability but rather a process of measurement, starting with a screening instrument to "weed in" those who are clearly disabled and to "weed out" those who clearly are not disabled. Then a second stage process will look at whomever is left to decide carefully if they are disabled.

6. How much error and in what direction is acceptable? At every step of the assessment process there will be error. Each measurement used will have its own error. The amount of error in each instrument compared to the lead standard is knowable; so is the direction of error. Is an instrument that overestimates disability preferred to an instrument that overestimates nondisability, or vice versa? Also, the frequency of disability by the LEAD standard in the population in question must be known, as it has a bearing on error. If the population in question is the universe of people who apply for Social Security Disability Insurance, the frequency of disability is high and error is less (fewer false positives). However, if the population in question is the U.S. labor force, where disability is much less frequent, then any determination process is likely

to yield a large number of false positives (i.e., determine people are disabled who really are not).

7. What resources are available to the determination process? How much should the average determination cost? There will be a tradeoff between cost and error. If the LEAD standard assessment were applied to every case, the overall cost would be astronomical. As the determination process moves away from the LEAD standard, it costs less but is less accurate. The correct balance between the two is a policy question.

Various issues in measurement development were then discussed:

• Not everyone with a given condition (e.g., schizophrenia, spinal cord injury) is disabled. The conditions—disorders-processes in which there is a likelihood of being disabled should be identified and their presence affirmed by a doctor, using standard medical procedure (medical impairment listings).

• There are certain sets of other (usually rare) conditions (e.g., coma) where disability is a definite outcome of the condition. These also should be identified in a different set of listings.

• Everybody from the above two groups should then be assessed, using a series of measures to determine their functional ability to work. It should be kept in mind that the ideal measure of functional capacity to work would be as direct a measure of work capacity as possible. Measures of functioning, cognition, and psychopathology are indirect measures.

However, some of these indirect measures can be used to screen people. For example, a battery of functional measures might be administered and, if individuals do poorly, the likelihood of their being able to work is very low and they should be considered disabled. Those who do well would go on to a more detailed assessment of their functional capacity to work using something very close to the LEAD standard developed. Examples of indirect measures that might be used include:

• Self-report. At the very least, self-report can be used to screen out those who say they are not disabled. The accuracy of self-report of being disabled against a LEAD standard requires determination. The error of self-report can be determined and then the characteristics of individuals who self-report disability in error can be determined.

• Proxy reports. The most accurate proxy reports are likely to come from the work setting. However, their accuracy also must be determined against a LEAD standard.

• Performance testing. Testing under circumstances close to work can also be carried out and have its accuracy be determined as well against the LEAD standard.

• Clinical observation. This measure needs to be defined better. Does it mean observation in a clinical setting or observation by a clinician (e.g., a doctor). This method is least likely to bear fruit as a measure of capacity to work because the clinical setting and the clinical observer are not set up to determine disability. However, it might be used to screen, so its accuracy for the LEAD standard should be assessed.

• Composite of above. Once the accuracy of each method above versus the LEAD standard is known, then combinations of the above can be used and compared to LEAD to see if synergy can be accomplished.

• Reliability of measures tells us how much agreement there is if a given measure is used at different points in time or by different people conducting the measurement. High interobserver reliability should be pursued. Test-retest reliability should also be pursued for all measures, but short time intervals of test-retest are optimal, since disability in some cases is transient.

• Validity of measures tells us how accurate a measure is of what we want it to measure, in this case, disability. There are several types of validity. Ideally, any measure developed should have:

• superior face validity before any field testing (basically, expert opinion of its ability to measure disability);

• superior concurrent-criterion validity against a LEAD standard in field testing;

• superior predictive validity, meaning it predicts the course of disability over time; and

• construct validity, such as the practice of validating functional measures against other functional measures (but not a LEAD standard) is not very important.

Basically, two separate processes are involved—determining policy and developing measures within the context of that policy. Once the policy questions are answered, the task of determining measurement methods is more straightforward. Ultimately, policy decisions will be needed early in the process on issues such as the direction and magnitude of tolerable error and the kinds of reliability and validity specifically required of the measures used in the process.

Some of the key issues that surfaced during the general discussion are:

• SSA's disability insurance program is not meant to protect all people with disabilities but only the class of people defined by its statute. A broad question is: Whom is our society prepared to protect through

public insurance policies and who are the people it is not prepared to protect? In the end, this is a policy decision. A gold standard, therefore, is elusive because there are tremendous variations among societies about people eligible for benefits. This difference is not because there is a great difference in underlying health conditions, but because there is a difference in what it is that these societies are willing to protect.

• Self-reporting of disability as distinct from performance and clinical testing was a recurring issue.

• Research is being conducted on self-reporting that needs to be considered in terms of disability, as disability relates to functional limitations in relationship to work. In the disability field, how people feel about their own job and their own capability is underplayed. Moreover, recent research has shown that it is possible to validate what a person does and does not do.

• Experience has shown that false negatives are a result of self-reporting just as much as false positives, if not more so. People with disabilities tend to underreport their disabilities, rather than overreport them. However, in the population SSA is dealing with, people seeking benefits, there are only true positives or false positives, depending on whether they meet the statutory definition, but there are no true or false negatives.

• Answers to self-report questions have a lot to do with the extent to which the respondent feels the environment is safe. Answering a population-based survey like the census is very different from answering questions when one is coming to an SSA office seeking benefits.

• The context is also important in the assessment of information used in the determination process. For example, from a research perspective self-reports often mean answers to questions in national population-based surveys. From a programmatic perspective where people come to the SSA seeking benefits and self-reporting on the existence of disability, the context is entirely different. The relationship between these two kinds of self-reporting is not fully understood. Various dimensions must be considered, including the physical, cognitive, degree of training and education, work history, and motivation. In different contexts, people's motivation will differ.

• Self-reporting should be viewed in a broader context of how it might improve the quality of the decision process. There is literature to suggest that if self-reporting is viewed as participation by the individual and is combined with the physician's report, people are more accepting of a denial from SSA.

• Self-reporting of performance is not relatively straightforward, as some suggest, but, rather, very complex. Testing measurements do not exist at the present time that give reproducible, valid, sensitive, low-cost assessments to classify persons as unable to work, particularly if assistive technology and other accommodations are also taken into consideration.

• What is being measured—functional capacity or the ability to perform work? It is questionable if one tool can be developed for all occupations. The predictive validity of measuring just the person and assuming it will tell something about his or her work capacity is doubtful.

• Should functional assessment include the work environment? The technology exists for such assessments; policymakers have to decide what to measure. This is a policy problem and not a measurement issue. Work is a combination of the individual and the environment, but no attempt is made to assess the environment. Performance of work in real terms is what SSA measures. It is generally known that other factors are involved, such as motivation and accommodation. Work evaluation then becomes a surrogate for an environmental assessment. Can SSA's regulations be rethought to adapt to the current environment?

• If disability is defined as a combination of environmental and personal factors, the law does not allow dealing with environmental factors. Work requirements and substantial gainful activity (SGA) evaluation becomes a surrogate for that environment. The law does not have to be changed, only its regulations. Interpretation of the law is in the regulations, and they are a function of the changing environment. Regulations may be one way to begin to approach the current issues.

• One has to be careful to distinguish two different meanings of environment. The environment of a particular applicant and that applicant's job is one meaning, and that is not something that can be considered under SSA's statute. It is applicable under the Americans with Disabilities Act and a whole series of service provision statutes. The SSA law, however, does consider a whole series of things that have to do with the average working environment. SGA is a social construct about what in the environment is an appropriate test of whether somebody is, in fact, engaged in SGA. To some degree it is arbitrary; it is a recognition of the social environment. The medical environment is recognized in the way the medical listings are tweaked to determine if certain conditions are more or less disabling than they once were. Measuring requirements of jobs in the national economy is a very broad question about the work environment on average in the national economy. This meaning of environmental conditions does come under the purview of the Social Security law and therefore its implementing regulations. It is important, therefore, to keep in mind which meaning of environment is relevant to the current deliberations and the SSA statute.

• SSA has to make millions of determinations. Clearly there is no gold standard, but SSA has to proceed as if there is one and make incremental changes to the current process instead of trying to create something new all at once.

• The goal of the redesign is to move closer to reality in measuring work capacity (e.g., the medical listings do not tell much about work capacity, so would functional capacity be a better way to determine if someone can work?). That is SSA's thinking in focusing more on functional capacity. But where is the evidence that the current system is broken? That information is essential to assess improvements, if any, from the new system.