Improving the Disability Decision Process
This chapter briefly discusses the key issues identified in the committee’s preliminary assessment of the Social Security Administration’s (SSA’s) research plan to redesign the disability decision process (IOM, 1998), and the subsequent decision by SSA to terminate this redesign effort and explore ways to incrementally improve the current process. The chapter then discusses and makes recommendations on the research needed to bring about fundamental improvements in the current disability decision process.
Determination of eligibility for disability benefits under the Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI) programs is an inherently difficult task. To qualify for benefits under these programs a person must have a medically determinable impairment. Although the existence of a medically determinable impairment is a necessary condition, it is not a sufficient condition for receipt of benefits. The statutory definition makes clear that this program deals with work disability. The applicant is considered to be “disabled” (as defined by the Social Security Act) not just because of the existence of a medical impairment, but because the impairment precludes gainful work (Hu et al., 1997) anywhere in the national economy, taking into consideration the person’s age, education, and work experience, which are commonly referred to as vocational factors. Disability determination is a complex
process, inescapably involving some interpretive judgment about capacity for work (GAO, 1994; Hu et al., 1997). At a minimum, making such decisions requires clinical determination of the extent of a claimant’s physical, mental, or sensory impairments; analysis of the degree to which such impairments limit the claimant’s functional capacity relevant to work roles; and consideration of the interaction of the claimant’s physical, mental, or sensory impairments with the person’s age, education, and work experience to provide an overall picture of the claimant’s future capacity for any sort of work. Finally the disability decision process requires a means for comparing those capacities with the capacities demanded by work roles in all jobs in the national economy that provide substantial gainful activity (SGA) earnings level.
While many of the factual determinations are relatively straightforward, others range from the difficult to the nearly impossible. For instance, while measures of visual acuity are reasonably well understood and can be readily translated into sensory limitations, the measurement of pain and its effect on function is much less amenable to objective determination. The real demands of jobs in the national economy are constantly shifting in ways that make straightforward measures of functional capacity problematic guides to a worker’s true capacity for success in the workplace. Therefore, it is impossible to know precisely the extent of imperfection in the determination of disability, as evidenced by the lack of agreement observed in an examination of rater reliability as measured by the variations within and between states in the allowance rates by examiners (Gallicchio and Bye, 1980; DHHS, 1982). SSA has been struggling with these issues for more than 40 years in the face of high volumes of claims for adjudication (millions of claims per year decided by more than 10,000 adjudicators at various levels of the process) and high levels of legal challenge and political oversight.
THE CURRENT DECISION PROCESS FOR INITIAL CLAIMS
The standards for evaluating disability claims are specified in SSA’s implementing regulations (20 Code of Federal Regulation, parts 404 and 416, subparts P and I) and in written guidelines. These regulations and guidelines describe a sequential process for determining whether or not a claimant meets the statutory definition of disability.
The purpose of developing the sequential decision process is to provide an operationally efficient definition of disability with a degree of objectivity that can be replicated with uniformity throughout the country. SSA’s overall objective is to adjudicate claims as consistently, expeditiously, and cost-effectively as possible. As described briefly in Chapter 1,
the disability decision process for initial claims involves five sequential decision steps (SSA, 1994a).
In the first step, the SSA field office reviews the application and screens out claimants who are engaged in substantial gainful activity.
If the claimant is not engaged in SGA, step two determines whether the claimant has a medically determinable severe physical or mental impairment.
During the third decision step, the documented medical evidence is assessed against the medical criteria to determine whether the impairment meets or equals the degree of severity specified in SSA’s Listings of Impairments (Listings). A claimant whose impairment(s) meets or equals those found in the Listings is allowed benefits at this stage. The Listings serve the purpose of allowing rapid payment of benefits to claimants whose presumed residual functional capacity (RFC), given the severity of their impairments, would preclude work at virtually any job.
In the fourth decision step, claimants who have impairments that are severe, but not severe enough to meet or equal those in the Listings, are evaluated to determine if the person has residual functional capacity to perform past relevant work. Assessment of the RFC requires consideration of both exertional and nonexertional impairments. If a claimant is determined to be capable of performing past relevant work, the claim is denied.
The fifth and final decision step considers the claimant’s RFC in conjunction with his or her age, education, and work experience to determine whether the person can perform other work that exists in significant numbers in the national economy.
The determination in the fifth step is based on the 1978 Rules and Regulations, Medical–Vocational Guidelines (referred to as the vocational grid). The vocational grid, like the Listings, is intended to lend objectivity to the determination process and facilitate uniform administration of the vocational portion of the disability determination process. The grid reflects only physical (exertional) impairments. It does not consider nonexertional (e.g., mental or cognitive) impairments. The regulations also recognize that some claimants will have multiple impairments or environmental limitations (e.g., they cannot be around fumes) that are not effectively covered by the grid regulations. These cases must be decided outside the grid.
Proposed Redesigned Decision Process
SSA has stated that the redesigned disability decision process should
be simple to administer;
facilitate consistent application of rules at each decision level;
provide accurate and timely decisions; and
be perceived by the public as straightforward, understandable, and fair.
The goal of the new decision process was to focus decision making on the functional consequences of an individual’s medically determinable impairment(s) (SSA, 1994a). Although the presence of a medically determinable impairment would remain the central requirement for eligibility as required by law, the redesigned process would focus directly, rather than indirectly, on the applicant’s functional ability to work and would rely on standardized instruments for measuring functional capacity to reach decisions. Medical and technological advances and societal perceptions about the work capacity of a person with disabilities appear to support a shift in emphasis from the current focus on disease conditions and medical impairments to that of functional inability. For example, people with disabilities are able to function today with personal assistants and assistive devices.
SSA assumed that under this proposed decision process, the majority of disability claims would be evaluated using a standardized approach to measuring functional ability to perform substantial gainful activity. Standardizing the approach to assessing individual functional ability would facilitate consistent decisions regardless of the professional training of the decision makers in the decision process. The new disability decision process, as envisioned by SSA, would assess a person’s functional ability once, relying on objective, standardized, functional assessment instruments. SSA believed that focusing decisions on the functional consequences of a person’s medical impairments would permit physicians and others who provide medical evidence, as well as decision makers, to use a consistent frame of reference for determining disability, regardless of the diagnosis and would facilitate evidence collection by reducing the need for developing extensive medical records (SSA, 1994a).
In the proposed plan, decision makers would consider whether a person has a medically determinable impairment(s), but would no longer impose a threshold “severity” requirement. Instead, they would compare the individual’s impairment(s) against an “Index of Severely Disabling Impairments.” The Index would replace the Listings of Medical Impair-
ments (SSA, 1994a). The Index was to have described, quickly and easily, impairments that are so severely debilitating that, when appropriately documented, they can be presumed to equal a loss of functional ability for SGA without assessing the individual’s functional ability and without consideration of the person’s age, education, and work experience. The medical findings in the Index were to have been as nontechnical as possible and to exclude such things as calibrations or standardization requirements for specific tests and/or detailed test results. The Index was to have been easy to understand and simple enough for laypersons to understand. SSA, therefore, believed that it would no longer need the concept of “medical equivalence” that is in the current decision process, thus eliminating one decision step in the current sequential evaluation process. If the claimant was not considered eligible for benefits based on the Index, then the person’s functional ability would be measured using standardized instruments or protocols linked to clinical and laboratory findings (SSA, 1994a). The effect of the statutorily prescribed factors of age, education, and work experience would be considered when deciding the claimant’s ability or inability to engage in substantial gainful activity.
The sequential process as it exists today and a new process as proposed in SSA’s redesign proposal are illustrated in Figure 6-1.
SSA’S RESEARCH PLAN FOR A REDESIGNED DISABILITY DECISION PROCESS
As directed by the Commissioner, SSA officials developed a research plan in 1995 for examining the feasibility of, and developing, the various components of the redesigned disability decision process. The plan had three components:1
Information gathering (comprehensive review and analysis of existing information) on
functional assessment instruments;
occupational classification systems;
disability determination processes used in other disability programs in the United States and other countries; and
the effects of age, education, and work experience (vocational factors) of the applicant.
Integration, synthesis, and development of a prototype for a new disability decision process:
analysis and evaluation of the literature reviews undertaken in the first component; and
development of prototype(s).
Testing, analyzing, and refining the prototype:
laboratory research and other small-scale testing; and
the National Study of Health and Activities (NSHA): a national sample survey to estimate the size and characteristics of the population eligible for disability benefits, determine factors permitting them to work, assess future changes in the prevalence of disability, and serve as one of the evaluation mechanisms for the decision process prototype(s).
Work on the first two components of the research plan was under way when this committee began its study. The research mostly involved contracts to conduct literature review and analysis in the areas of functional assessment instruments, occupational classification systems, disability determination processes used in other disability programs, and effects of vocational factors in the disability decision process. Some of the information-gathering activities were completed when the committee began its review. Work on the second component and parts of the third component involved the award of a task-order contract at the end of fiscal year 1997. The purpose of this contract was to synthesize and integrate the results of the literature review contracts and the NSHA; to develop, test, evaluate, and refine alternative prototypes for a redesigned disability decision process; and to undertake additional research as needed. Under SSA’s redesign research plans, additional work in the research and development of the decision process would have been undertaken in subsequent task orders under this umbrella contract and/or under separate contracts.
In summary, SSA’s research plan aimed at developing and testing the functional assessment instruments in the disability decision process, examining the effect of vocational factors on decisions, exploring what is being done in other disability programs, and developing a prototype for a revised disability decision process.
Committee’s Review of the Research Plan
Early in the study, the committee conducted a preliminary review of SSA’s workplan and individual research projects completed and under way. The committee also explored other relevant internal documents provided by SSA in response to requests for information. It heard presentations from the staff of SSA on work completed to date, plans to integrate the results of the research projects and the NSHA to develop a redesigned disability decision process, and the time line for completion of all research.
After reviewing the available documents and discussions with its contractors, the committee decided that a preliminary assessment of the adequacy of the research plan was needed to guide SSA management in determining whether the research activity undertaken was adequate and what more, or different, was needed to conclude whether or not the proposed revisions in the decision process were feasible, practical, and could be implemented nationally.
According to SSA, the goal of this research is to devise a more efficient and more accurate method for making timely determinations of disability for Social Security claimants (SSA, 1996). In the context of that goal, the committee outlined an initial conceptual framework of issues,
research steps, and methods for a research plan or design to develop and assess the proposed new decision process as a workable solution to current problems. Such a framework should delineate at the outset the nature and extent of the problem. Complaints, whether ultimately substantiated or not, often suggest that a program should be evaluated and improved. However, in order to assess the validity of the complaints, objective evaluative criteria should be established a priori, so that the various complaints about the program can be evaluated and the program’s performance can be measured. The next step would be to identify alternative solutions that might address the problems. Finally, the proposed new decision process should be tested to determine whether it is workable and whether it will alleviate the problems initially identified. To determine if a redesigned disability decision process would lead to improvements, one or more studies need to be conducted to provide information on how the current program is working relative to the established criteria. Analysis of data from such studies would identify the gaps between performance and the goals of the program. This framework is reproduced in Table 6-1.
The committee then reviewed the general features and directions specified by SSA in its research plan and the individual projects within the plan with reference to each of the research steps identified in the framework. It reviewed and commented on both the completed research projects listed earlier and the scope of work in the relevant requests for proposals for the conduct of the research. It identified critical elements of a research design that were missing from SSA’s plan, expressed serious concerns about these gaps, and made recommendations for redirected and new research effort.
The committee commended SSA for initiating the major task of redesigning to improve the disability decision process and undertaking a range of research activities related to the functional consequences of medical impairments and for recognizing the need to assess the feasibility, validity, and reliability of a proposed redesigned decision process. Nevertheless, the committee concluded in its second interim report (IOM, 1998) that the research completed, under way, and planned appeared to lack the necessary overall framework and lacked the critical elements of a well-designed research plan. Some of the key conclusions reached in that report are summarized below.
In 1995, SSA contracted with the Virginia Commonwealth University (VCU) to review systems, methods, and instruments that measure a person’s functional capacity to perform activities and tasks, to develop a matrix of categories to classify these instruments, and to evaluate them to determine their potential application in the disability decision process. VCU’s main conclusion in its report was that no government or private entity is currently using functional assessment instruments specifically
TABLE 6-1 Issues and Methods to Be Addressed in a Framework for a Research Plan for a New Disability Decision Process
Needs assessment research
Identify alternative options
Small-scale testing Field evaluation
Program evaluation and transition to implementation
SOURCE: Reprinted from IOM, 1998.
for determining work disability benefits, and a global measure of functional assessment does not exist that would be a valid indicator of disability for all populations currently served by SSA. Such an instrument will likely have to be developed and tested.
After reviewing the advantages and disadvantages of both functional and clinical assessment measurements for SSA’s needs, VCU concluded that objective functional assessment can and should be a component of the redesigned process. VCU, however, stopped short of constructing the global measure of functional capacity; instead, it recommended several steps SSA should take in moving toward its development. The committee expressed concern that SSA had not made clear the conceptual or theo-
retical basis for believing that such a standardized, global instrument existed or can be constructed before launching the literature review project on the subject. Skeptical of one global, standardized, universally accepted measure, the committee recommended that SSA develop alternative research plans for development and use of functional measures in the disability decision process in the event that the proposed global standardized functional assessment instrument is not developed and tested (IOM, 1998).
In 1996, SSA contracted with the American Institutes for Research (AIR) to conduct a comprehensive review of the literature pertaining to systems and methods of classifying occupations in terms of the physical and mental capacities required, to develop a taxonomy of occupational classification systems, and to assess the applicability of systems for SSA’s redesigned disability determination process. This review relates directly to one of the key elements in the proposed redesigned disability decision process, namely, assessing baseline work. The purpose of the review is to determine if a standard exists, and if not, whether it is feasible to develop one to describe basic physical and mental demands of a baseline of work. AIR concluded that while none of the occupational classification systems exactly or ideally matched SSA’s needs, the Occupational Information Network (O*NET) under development was the closest match to SSA’s needs. O*NET is an occupational classification system being developed by the Department of Labor (DOL) under contract with AIR to replace the Dictionary of Occupational Titles. One of the reasons AIR recommended O*NET over the other systems is because it uses level scales to measure the amount of skill needed to perform certain jobs. Incumbents choose a numeric rating based on their reading of the behavioral anchors. Cognitive and mental descriptors are also included in O*NET, but the physical ability scales that O*NET uses may not be specific enough to help SSA. Many other issues were identified by the contractor that need to be resolved before O*NET can be used for SSA’s purposes. These issues are described in the committee’s interim report (IOM, 1998) and are discussed further later in this chapter. The committee questioned how O*NET will be used. SSA’s research design did not appear to be oriented to address this question. How does SSA plan to supplement O*NET with respect to contextual or other factors that are not well covered. There were no indications in the research plan that the gaps in O*NET would be carefully considered and specific research identified to fill those gaps. The committee also was concerned about the synchronization of timing for completion of O*NET and SSA’s target completion of the research for development and implementation of the disability decision process. The committee recommended that SSA develop an interim plan for an occupational classification system in the event that the O*NET database is either not completed
or is insufficient to meet the needs of a new disability decision process. It also suggested that SSA should explore entering into an interagency agreement with the Department of Labor to initiate a version of O*NET that would collect information on minimum as well as average job requirements to better serve SSA’s needs to assess ability to engage in substantial gainful activity.
The committee made recommendations for changes in priorities and improvements in the research projects that were under way and others yet to be developed. It urged SSA to adopt a rigorous research design process and to develop, early in the research, objective validation criteria and validation plans to be able to make the ultimate judgments on whether or not the proposed changes would yield the desired results. In issuing the interim report the committee hoped that the recommendations embodied in that report would be incorporated in the contract research that was under way and in new research not yet initiated at the time. The committee’s detailed discussion, findings, conclusions on the various issues, and recommendations are embodied in its second interim report to SSA (IOM, 1998). All the recommendations flowing from the committee’s preliminary assessment are included in Appendix C.
The committee had planned to examine and comment further on the adequacy of the entire research plan when completed, the results of the completed research, and any subsequently initiated research for the redesign effort. However, after the committee issued its second interim report (IOM, 1998), SSA undertook an internal reevaluation of its disability decision process redesign initiatives. SSA concurred with several of the committee’s conclusions and some of the recommendations. However, rather than undertaking the additional research and redirection of the research as recommended by the committee, SSA decided to no longer actively pursue the new decision-making process proposed in Disability Redesign, but to improve the current process, focusing at this time on updating the Listings (SSA, 1999b). According to SSA, its new strategy is to concentrate on improving the overall adjudication process to ensure that decisions are made as accurately as possible, that those applicants who should receive benefits should get them as early as possible, and that the adjudication process is consistent throughout (SSA, 1999b).
To make improvements, SSA has stated that it has redirected and refocused disability policy development and related research activities in order to address both the longer-term goals of redesign, and the more immediate, pragmatic needs of the disability programs (SSA, 1999b). SSA expects these process initiatives to
enhance the quality of decisions at all levels;
streamline the disability process by applying the lessons of the disability process redesign efforts; and
update medical and vocational rules used in making disability determinations.
The initiatives to achieve the first two objectives relate to components of the reengineering process that are outside the purview of this committee. The third objective—updating the medical and vocational guidelines—is a direct outgrowth of the now abandoned initiative to develop a new decision process.
Updating the Listings
SSA states that although it is no longer focusing on development of the new decision process described in the disability redesign plan, it is continuing to explore the potential in some of those ideas. However, it is now devoting most of its resources to needed improvements to the current evaluation process (SSA, 1999b).
SSA further states that the proposed new decision process was intended in part to address concerns about the current Listings by replacing it with an Index of Impairments. However, SSA has concluded that the Listings serve a vital role in ensuring the cost-effectiveness of the adjudication process. Currently about 60 percent of the allowance decisions are based on the Listings without developing and conducting a complete, in-depth functional and vocational analysis. Rather than replacing the Listings, SSA is now engaged in a concentrated effort to update and improve these medical guidelines (SSA, 1999b). Medical advances in both diagnosis and treatment have made updating the Listings long overdue. As stated by SSA, the general approach to revising a section of the Listings is to begin with its adjudicative experience and program knowledge. Having identified an area of interest, medical literature is reviewed and, as warranted, experts in the field are consulted. If more extensive research is needed, a contract(s) may be negotiated to obtain the information. Medical experts from within SSA are consulted to develop an initial proposal.
Functional Consequences of Impairments
SSA informed the committee that “the proposed new decision process was to have relied on simple, objective readily available functional tools to assess an individual’s ability to work. Initial research conducted under contract by SSA has not shown a basis for believing that such a standardized, universally accepted global instrument applicable to indi-
viduals with physical and mental impairments can be constructed or that some similar approach would be possible. Further, some of the other assumptions on which the proposal for a new process was based (e.g., the assumption that these functional tools would be routinely used in clinical practice and, therefore, readily available without cost in approximately 75 percent of the cases), no longer seemed reasonable” (SSA, 1999b, p. 7). SSA further stated that this does not mean that functional ability and functional testing cannot be important components of SSA’s disability evaluation process. They can and they should.
SSA has indicated that it continues to believe in the need for good information on functional ability and testing as a key part of the decision process. It has begun to focus on an alternative plan to use functional assessments in the current decision process and is addressing the issue in three different ways.
Ensure that functioning is appropriately considered within the current evaluation process, in terms of functional criteria that are part of SSA’s standards, and in terms of providing practical policy guidelines for the use of functional testing in the current process. Many Listings include functional criteria. As the Listings are revised, SSA will seek to achieve consistency, simplicity, and administrative practicality of functional criteria.
Issue updated policy guidelines addressing the uses of functional assessments in the current decision process. The guidelines will address issues such as the evidentiary nature of functional assessment results, when to consider purchasing functional assessment tests, and the kind of tests to purchase.
SSA has asked its expert consultants to begin developing new ideas for ways to more closely investigate the use of functional testing in SSA’s disability decision process (SSA, 1999b).
Committee’s Assessment of SSA’s Post-Redesign Plans
This section addresses the necessary prerequisites for a scientifically sound approach to disability determination at SSA. Several of the key issues that the committee had identified earlier in the study in the context of the problems associated with SSA’s research plan for redesigning the disability decision process still have to be addressed with respect to the activities undertaken to improve the current process. Therefore, they bear emphasizing again in this report.
SSA has stated that the purposes of incrementally improving selected components of the current sequential disability decision process are to enhance quality of decisions, streamline the process, and update medical
and vocational rules used in determining disability. This effort then, like the previous redesign effort, calls for comparative judgments. It presumes analysis of baseline information from the current decision process to assess the effectiveness of the current decision process and compare it with similar analysis of changes in the new decision process.
However, based on the information provided by SSA, the committee assumes that the agency has not conducted such baseline analysis with predetermined criteria for evaluating the components of the sequential disability decision process leading to the decision to redesign. Moreover, such analysis does not appear to have a place in the current research plan. SSA’s current research approach focuses mostly on the new decision and therefore fails to build in tests that may be critical to answering the comparative questions and, ultimately, to the decision whether or not to adopt the changes in the current process.
Baseline Evaluative Criteria
Regardless of whether SSA attempts to redesign and develop a new disability decision process or leaves the current process in place and makes improvements within the individual components of the sequential process, SSA needs to establish objective measurable criteria against which the current process can be assessed. Studies should be conducted on the existing process and data analyzed in the context of the established criteria in order to identify the nature of the problems in the current process. Without such a capacity, proposals for “reform” may be proposals for “change,” but it is impossible to determine whether they are proposals for “improvement.”
When the committee reviewed SSA’s research plan for the redesign initiative it was unable to conclude that SSA had put a satisfactory research plan in place (IOM, 1998). The research proposals and other documents reviewed by the committee provided no indication that SSA had conducted any baseline analysis with predetermined criteria for evaluating the Listings component, or for that matter any other component, of the sequential determination process leading to the decision to redesign the system. Moreover, such analysis did not appear to have a place in the research plan. SSA’s research approach mostly focused on the new decision process and thus failed to build in tests that may be critical to answering the comparative questions and ultimately to the decision whether or not to adopt a new decision process.
The same issues appear to exist today as SSA moves toward making incremental changes in selected components of the current decision process. In the absence of evidence to the contrary, the committee assumes
that the agency again has not conducted such baseline analysis leading to the current activities to improve the existing process incrementally.
For example, in its original redesign plan, SSA had proposed to replace the current Listings with an Index of Disabling Impairments that would serve to “screen-in” the obvious cases without addressing functional capacity and vocational factors. However, nowhere had SSA specified the levels of specificity and sensitivity that would be satisfactory for this Index. There was no attempt to determine whether the current Listings satisfied the goals for specificity and sensitivity overall, or the degree to which those goals were satisfied by different Listings for different body groups and different conditions. Without such baseline evaluative criteria and analysis it seemed impossible to specify either what the problems were with the current Listings or whether some redesigned Index would do a better or worse job in relation to the agency’s goals for that particular screening instrument. Unfortunately, SSA appears to be going down the same path in its current efforts.
Throughout the documents reviewed by the committee relating to the redesign research, including the scope of work for the research contracts and in presentations before the committee, SSA has recognized the need to test the new disability decision process by applying standards of validity, reliability, sensitivity, specificity, credibility, and flexibility. In addition, the stated objectives of the redesign also include requirements such as simplicity in administration, consistency, accuracy, timeliness, equity of decisions at all levels, and fairness. However, to the committee’s knowledge no measurement criteria have been established to test the current and the redesigned process along any of these lines.
Measurement is the process of linking abstract concepts to empirical indicators (Carmines and Zeller, 1979). Various terms are commonly used to describe measurement. For instance, to determine the extent to which a particular empirical indicator(s) represents a given concept, one can examine the reliability of the concept, that is, the reproducibility of a decision for each case within and/or across decision makers. But a process or indicator needs to be more than reliable if it is to provide accurate results. It must also be valid. Although the terms reliability and validity are often used together, they are not synonymous. A decision may be reliable but it may not be valid. Both reliability and validity reflect matters of degree. Validity represents a set of criteria by which the credibility of research may be judged. For example, it measures the degree of agreement between the disability decision and actual fact of disability. Moreover, validity has several meanings. These include construct validity; content validity; criterion validity, which includes concurrent validity and predictive validity; and study validity, which includes internal and external validity (Last,
1983). The definitions among the disciplines of logic, epidemiology, social science, and statistics do not always correspond.
Effectiveness measures the extent to which the decision process is achieving its goals and purposes (Berk and Rossi, 1990), and the concept of effectiveness must always address the issue of “compared to what?” regardless of whether it is marginal effectiveness, relative effectiveness, or cost-effectiveness. Finally, efficiency measures the results of a process in terms of resources expended and time.
The committee notes that the fields of both science and law pay unusual attention to the definition of terms. When the two fields intersect, it is particularly important to be specific about the terms used. For example, although the word “disability” is used by SSA, the actual phenomenon focuses on an attribute more narrowly defined than the inability to perform the usual activities of daily living. It refers to the inability to engage in substantial gainful activity because of a medically determinable physical or mental impairment leading to death or expected to last for at least 12 continuous months. Such distinctions are key to SSA’s use of terms such as “validity,” and here the legal use of the term is important to guide the scientific criteria. The appropriate construct to use in judging whether the current or the new decision process achieves its goals must take into account the legal language.
The committee, however, recognizes that the actual fact of disability may be unobservable. For example, the number of persons with disabilities under the Social Security definition will vary as a result of judicial interpretations. Each aspect of the law is subject to differing interpretations and judgments. Problems arise at each step of the sequential decision process, whether one is determining if a person has an impairment that will result in death or that is expected to last at least 12 months or, more crucially, whether by reason of that impairment a person cannot engage in SGA. The difficulty of making such a decision, however, is not the issue. Rather there is no single true answer to the question of whether a person with that impairment should be expected to engage in SGA. Simply put, a “gold standard” does not exist. Therefore, it is necessary to substitute some criterion or target and assess how well actual determinations are meeting this target. Whatever SSA chooses as a criterion or target also must be disseminated to stakeholders and decision makers as soon as possible, along with a plan for validity assessment. Only with such openness will the validity assessments be accepted when they become available.
The brief discussion of measurement terms and issues clearly demonstrates the need for SSA to specify early in the redesign effort what it means by the terms validity, reliability, sensitivity, specificity, credibility, flexibility, and all the other related terms that it uses; and how it plans to
measure them, that is, what measurable criteria will be used to assess these standards vis à vis the disability decision process research and the level of sensitivity and specificity it is willing to tolerate. The same criteria should then be used to evaluate the quality of both the current process of disability determination and any prototype to be tested.
Listings of Impairments
As stated in the previous section, SSA has decided to devote its attention to updating the Listings but no timetable has been set for the completion of the various phases of this initiative. “The Listings was originally designed to highlight readily identifiable disabling impairments. Many of the Listings have since evolved into complex and highly detailed diagnostic requirements, demanding specialized medical evidence that may not be readily available from treating sources. Some, but not all, of the Listings consider the functional consequences of an impairment; however functional considerations vary significantly among the Listings” (SSA, 1994a, p. 11). The committee believes that it is indefensible that most Listings have not been reviewed and updated in more than 10 years. SSA has stated that limited staff resources, the need to address new legislative mandates during the 1990s, and the lack of adequate research on disability criteria to support Listings updates have been at least part of the problem. SSA had not made a comprehensive revision of the adult mental disorders listings since 1985 (OIG, 2000). The report of the Office of the Inspector General (OIG) states that by the late 1990s, the National Academy of Social Insurance (NASI) (Mashaw and Reno, 1996), the General Accounting Office (GAO, 1998), and the Social Security Advisory Board (SSAB, 1998) also were expressing concern that SSA was not updating the Listings regularly, but was simply extending the expiration dates for a number of years when the Listings expired (OIG, 2000). According to the OIG report, SSA staff has acknowledged that during the 1990s they did not always have the necessary research in place to support proposing revisions to the Listings or other disability projects.
SSA informed the committee in 1999 that it is correcting the situation; it has added 15 positions in the component responsible for Listings policy and has started to address some of the most critical needs. However, the committee is not aware of any attempt at this time to evaluate the currency and consistency of the Listings based on specific criteria, or at least those groups of conditions that account for a significant proportion of the disability rolls. SSA appears to have made the decision to modify certain Listings without any attempt to first evaluate them and use the findings to guide the update of the Listings.
The committee in its interim report to SSA (IOM, 1998) supported the
conclusion of the Disability Policy Panel of NASI to give high priority to research related to the Listings as well as to evaluate the consistency of the presumptions underlying the Listings for different body systems (Mashaw and Reno, 1996). The committee stated that SSA should conduct the necessary research, prior to making changes in the Listings, to (1) determine whether or not the current Listings satisfy the agency’s goals for specificity and sensitivity, (2) determine whether or not these goals are satisfied consistently across the Listings for the different body groups or conditions, and (3) evaluate the options to correct the problems detected by these evaluations, as it develops any new list of medical impairments. The committee has not changed its position.
It appears likely that the agency’s agenda for reform in this area will be driven as much by internal and external anecdotal concerns, including general perceptions of which Listings are the most outdated, as by any long-range strategy. Nevertheless, the committee believes that a successful process of Listings revision must be based on a systematic approach to evaluation, design, and testing. The committee has not seen any indication of a plan for determining the specificity and sensitivity parameters for any existing or proposed Listing. Developing such parameters seems critical to both the scientific and the political validation of the Listings as a decisional tool.
Because the Listings screen is meant to be used to identify clear cases of disability, one would expect this screen to be devised such that it is highly specific (seldom identifies false positives) and relatively sensitive (identifies some substantial number of true positives). The question for SSA is how specific and how sensitive. In order to undertake meaningful research on the validity of any medical listing, SSA must be able to specify the acceptable level of specificity and sensitivity by which it can validate the screen against those criteria.
SSA provided the committee with a list of ongoing projects designed to update the Medical Listings and improve their performance. The committee, however, has no information suggesting that baseline criteria were established at the outset or that any method was developed for validating the existing and proposed new Listings against those criteria. These are serious and difficult issues. As SSA moves forward to incrementally revise and reform the current decision process, it must be able to determine whether or not changes are improving the accuracy of the process. Indeed, it has to be able to make these determinations prior to the time that changes are implemented on a national basis. Whether or not specific Listings need to be improved and the direction of that improvement must await the results of the baseline evaluation and subsequent reevaluation.
The committee in its interim report to SSA (IOM, 1998) recommended that early in the redesign effort SSA should specify how it will define,
measure, and assess the criteria it will use to evaluate the current disability decision process as well as any alternatives being developed. Because of the critical importance of this issue and the assumption that not much has been done in this area, the committee reiterates its position. In any scientific process, standards of acceptance or rejection are declared before, not after, data are analyzed. Similarly in an evaluation research process, evaluative criteria and validation plans should be determined by the agency early in the research process and not, as currently planned, after the Listings are identified for updating and changes are developed.
Recommendation 6-1: The committee recommends that prior to making the changes in the current decision process, SSA should
establish evaluative criteria for measuring the performance of the decision process;
conduct research studies and analyses to determine how the current processes work relative to these preestablished criteria, and
evaluate the extent to which change would lead to improvement.
Analysis of data from such studies in the context of the established criteria would identify the nature of the gaps between what the program is supposed to achieve and its actual performance. Without these research steps and analysis, there is no objective way to conclude if the changes are more effective and more efficient than the existing process.
Since SSA is devoting its attention to updating the Listings, this recommendation is most applicable at this time to the Listings. However, the committee notes that the Listings apply only to one step (step 3) of the five-step sequential evaluation process for determining disability. The baseline evaluation recommended for the Listings should ultimately evaluate the total process and not just one component.
The committee is encouraged to note the recent cooperative agreement awarded by SSA in December 2000 to the Disability Research Institute (DRI) to undertake research for developing a process of validation of the Listings in order to assess them and to ensure that changes made actually result in improvements in the disability decisions. This research will (1) compile and list published articles in the literature pertaining to the validation of disability insurance decisions, (2) undertake a critical review of the literature on assessing validity and, primarily and most importantly, on the development of appropriate validation criteria, and (3) review methods by which these validation criteria could be operationalized. Although completion and implementation of this project may help validation efforts for future revisions of the Listings and other compo-
nents of the disability determination process, unfortunately no such input exists for the revisions currently under way or completed.
The committee recognizes that SSA faces two major challenges in providing an appropriate research base for the improvement of the disability decision process. First, it operates an ongoing program that requires continuous incremental adjustment in order to make appropriate decisions. Second, the environment of disability decision making is constantly shifting in ways that have unanticipated consequences for the current process and that generate movements for substantial reorientation of the entire disability benefit programs. Moreover, it goes without saying that SSA should recognize the cost trade-offs when it sets targets for sensitivity and specificity, and other measures.
Assessing Vocational Capacity
Determining the Demands of Jobs in the National Economy. As indicated in the previous section, the Dictionary of Occupational Terminology (DOT) is no longer being updated by the Department of Labor, leaving SSA with no replacement. The DOT has served as a primary tool for determining whether a claimant has the capacity to work. The Department of Labor is replacing DOT with O*NET. The committee, in its preliminary assessment of the redesign research plan, had expressed its concerns that O*NET as it was being developed for DOL would not meet SSA’s needs (IOM, 1998). For one thing, it focuses on average rather than minimum requirements as needed by SSA. The committee also questioned how SSA planned to supplement O*NET with respect to contextual and other factors that are not well covered. Discussions at the workshop sponsored by the committee on Measuring Functional Capacity and Work Requirements (IOM, 1999a) pointed out the problems associated with using O*NET for SSA’s purposes. The DOL expects to use O*NET, as a comprehensive database of work requirements for use in job training, job counseling, and job placement for the department’s employment and training programs and for use by individual state Employment Security Agencies in the extensive work that they do with workers who need jobs or who have recently become unemployed.
As discussed at the workshop, although O*NET is very useful for DOL’s purposes, SSA’s purpose in defining the functional capacity to work for purposes of the disability legislation is very different from the purposes of the DOL in creating O*NET. SSA’s purpose is much more difficult. Moreover, the labor market and occupational literature indicate that there are many difficult measurement problems related to occupation and job characteristics. Information developed by job incumbents is not always consistent with the information developed by job analysts,
and the information developed by job analysts is not always consistent with the views of workers’ supervisors. The Bureau of Labor Statistics conducts employer surveys that try to define the characteristics of a job that affect its pay levels, but even there measurement difficulties sometimes exist. In addition, from the perspective of the worker—as with a disabled individual—it is often a bundle of capabilities that the worker brings to the job that makes the work experience a success or a failure.
Workers with the same educational backgrounds have different skills, work ethics, and orientations to work. These in turn bring a different bundle of capabilities to a job, and their performance is affected by those capabilities. In addition, the task of developing a set of factors that capture the essence of each occupation that makes practical sense is complex and difficult. Clearly, a great deal more careful research and experimentation is required to evaluate what functional capacity to work really means and exactly how it would be applied to persons with disabilities.
When the committee reviewed SSA’s redesign research plan, there were no indications in the plan that the gaps in O*NET will be carefully considered and no specific research to fill those gaps was identified. The committee, therefore, had recommended that SSA should develop an interim plan for an occupational information classification system until a more permanent solution is found. The committee also suggested that SSA enter into an interagency arrangement with the DOL to initiate a version of O*NET that would collect information on minimum, in addition to average, job requirements to better serve SSA’s needs to assess ability to engage in SGA.
Subsequent to the committee’s interim report, SSA revised the work requirements of its original task order contract on integration, synthesis, and development of the redesign process to focus solely on a comprehensive assessment of O*NET as a replacement data source for the current decision process. SSA did not necessarily expect this work to produce a comprehensive resolution to the problem. It believed, however, that it must complete such an analysis to move forward (SSA, 1999b).
The final report of the contractor (AIR, 2000) identified several positive and negative aspects associated with O*NET’s incorporation into SSA’s disability determination process. Some of the positive aspects identified include the ability to (1) obtain consensus from a variety of sources on the set of 54 O*NET descriptors appropriate for use in SSA’s disability decision process; (2) ascertain the number and identity of occupational units that are represented at various intervals of the descriptors’ rating scales; (3) determine if the occupational units are sufficiently representative for SSA’s use; (4) identify 33 occupational units that contain at least one sedentary or unskilled job, as defined by DOT, representing approximately 3 percent of the 1,122 occupational units in O*NET; and (5) provide excellent descriptions of occupations in the task lists that decision
makers can use to help determine the specific activities that comprise the claimant’s current or previous occupations.
On the other hand, several aspects of the O*NET structure and content could lead to problems if SSA incorporated O*NET into the decision process. More than half of the occupational units had at least one domain for which the majority of descriptors were unreliable. The final report (AIR, 2000) emphasizes that a major overarching problem with O*NET is the numerical ratings. These ratings do not seem to be consistent across occupational units. The contractor’s analysis found that the ratings of more than half of the descriptors are unreliable. Moreover, the DOT titles are grouped by dimensions that are unrelated to worker characteristics or requirements of the O*NET descriptors. Several of the 54 selected descriptors contain O*NET ratings with interrater reliabilities lower than .70. The report concludes that the numerical ratings on O*NET descriptors, and therefore on any O*NET occupational unit, underlie the problems of O*NET. Therefore, SSA must exercise extreme caution in drawing inferences about the relation between specific numerical values on a rating scale and specific level of required functioning. The report further states that the foregoing concerns provide sufficient evidence to warrant SSA’s careful consideration of the quality of either analyst or incumbent ratings as conducted and proposed for O*NET. The report also suggests that O*NET’s descriptor data may not be as precise as they seem, resulting in measurement errors as well as improper interpretation of the severity of claimants’ impairments (AIR, 2000).
Without an appropriate characterization of job requirements that can be matched to the vocational characteristics of disability claimants, SSA might be cast back into the era in which it relied extensively on the testimony of “vocational experts,” or their written evaluations, as the way to integrate claimants’ functional capacities, vocational factors, and the demands of work into an objective determination of their capacity to engage in substantial gainful employment. Barring some resolution, SSA will be left with no objective basis upon which to justify decisions concerning an individual’s capacity to do jobs in the national economy (AIR, 2000).
SSA realizes that O*NET will not work for its needs without major reconstruction of the system. SSA is taking steps toward resolving the problems. The committee is informed that SSA has reopened its dialogue with DOL to explore other ways of incorporating information about the requirements of work into the decision process and is actively pursuing with DOL the issue of an occupational database on a national level to avoid two separate databases with separate funding. It is also meeting with the various associations of rehabilitation specialists, occupational and physical therapists, and workers’ compensation analysts. Private sector stakeholders have organized an interdisciplinary task force. It plans to
meet with SSA and DOL to decide what is needed and how best to go about getting the information.2
Age, Education, and Work Experience. SSA has not updated the research base on the effect of age, education, and work experience on work disability that had been used in developing the medical–vocational guidelines of 1978, known as the “grid rules.” Since then, much has changed with regard to the relative importance of each of these factors. As part of the initiative to redesign the decision process, SSA included in its research plan the evaluation of the effect of vocational factors—age, education, and work experience—on the ability to work or adapt to work in the presence of functional impairment. To assist in deciding an appropriate way to incorporate into the redesigned disability decision process the specific statutory requirement to consider an individual’s vocational factors in determining ability to work, SSA entered into a reimbursable agreement with the Federal Research Division of the Library of Congress to review and evaluate published literature and any other research pertaining to the subject. A report of the review was submitted to SSA in 1998. The findings of the review are summarized very briefly below.
Although age strongly affects decision making under the vocational grid, the literature review of the existing literature suggests that age may have little or no independent influence on ability to work (as distinguished from the likelihood of being hired or retained). Rather than chronological age being a common contributing factor to declining capacities, current studies suggest that the population actually becomes much more heterogeneous with respect to its functional capacity as it ages. Moreover, except for the relatively vague concept of “adaptability,” age does not seem to have a strong correlation with modal ability to work.
Education is clearly an important factor in employability. It affects the ability to acquire new skills, and earning power is related to education level. It is especially a problem with mental impairments. However, education above basic literacy levels has an uncertain relationship to the ability to do jobs that would produce substantial gainful employment. High levels of education are not necessary for jobs paying $8,400 per year. High levels of education may, of course, suggest that only the most debilitating injuries or illnesses would prevent substantial gainful employment by persons with such levels of educational attainment. In combination, these attributes suggest that education may be important as a vocational factor only at the upper and lower range of educational attainment, but not in the middle ranges.
Increasingly basic skills are a priority with employers in identifying new employees. Employers value basic skills that flow from education, including the capacity to learn new skills or information, much more than they value job-specific skills. As with age, the independent influence of work experience is difficult to evaluate. Work experience is certainly important in terms of capacity to return to a worker’s own occupation after an injury or illness. However, the vocational factors are used most often in evaluating the capacity of workers to do jobs other than those that they have held before.
In summary, the review raised questions about the utility of multiple gradations of educational attainment in evaluating the vocational factors in disability determination and the utility of making determinations based on a worker’s transferable skills. Existing knowledge concerning vocational factors and their impact on the ability to perform jobs in the national economy raises challenging questions about the continuing validity of the approach taken by SSA’s existing grid rules. It suggested a critical need for a program of research designed to validate or reform the use of vocational factors in SSA’s disability decision process.
SSA recognizes that it may have to make significant revisions to the rules it uses to determine disability, especially in light of the changes that the Department of Labor is making in its occupational data. SSA’s current rules, especially the grid rules, are based in part on both the organizational structure and the data content of the DOT. Without it, those rules will probably have to be revised in a fundamental way. SSA also recognizes that such a revision might also necessitate a change in the way it incorporates evaluation of age, education, and work experience in its disability decision-making process.3
Variations in Disability Allowances. As shown in the previous chapters, over the past two decades the number of disability beneficiaries as a share of the civilian labor force has risen steadily. Although applications for benefits have increased only moderately, the number of new beneficiaries has nearly doubled. Disability allowance rates (awards as a percentage of applications) have varied over time from 31.4 percent in 1980 to nearly 47 percent in 2000 (see Table 2-1). Variations in allowance rates occur for several reasons. For example, SSA’s standards for judging claims differ over time. Dramatic reductions in approval rates occurred when standards were abruptly tightened in 1980 and then subsequently made liberal. Significant differences are observed in approval rates across states, between the state Disability Determination Service (DDS) decision mak-
ers, and between DDSs and administrative law judges (ALJs). The approval rate is also influenced by legislative changes as well as court decisions. The adequacy of resources to process and review cases also affects the disability allowance rates.
Increased research is needed to explain the variation in the rates at which applications for disability benefits are approved, why these changes take place, and whether they are predictable. To what degree are the growth and changes in disability allowances related to societal factors, and to what degree have they been influenced by changes in the program rules and operations? Such research should involve examination of the disability decision-making processes and the standards applied by SSA, the differences among states, and the differences among DDSs and the ALJs and among states. For instance, in the short run, changes in the prevalence of impairments are not as likely as changes in the way SSA evaluates the various impairments. Untangling the effects of the demand side (the growth in the number of SSDI and SSI applicants) and the supply side (the SSA disability decision processes) and prescribing remedies is difficult, but careful research in these areas will help shed light on this comparatively neglected area.
Recommendation 6-2: The committee recommends that the Social Security Administration conduct research on
improving the ability to identify and measure job requirements for the purpose of determining work disability;
investigating the role and effects of vocational factors in the disability decision process; and
understanding reasons for variations in allowance rates among states and over time.
ALTERNATIVE APPROACHES TO THE CURRENT DECISION PROCESS
The objectives of the current disability decision process appear to be an attempt to make accurate decisions about the capacity to engage in substantial gainful employment consistent with the statutory definition of disability as consistently, expeditiously, and cost-effectively as possible within a system that is hierarchically accountable and makes determinations at a relatively low cost. Mashaw (1983) refers to it as “bureaucratic rationality.” Although interpretive judgment is clearly necessary in adjudication, the process has become an increasingly rule-bound system that strives, with greater or less success, to decide similar cases in a consistent
manner, in accordance with the statutory definition of disability, and in a timely and efficient manner.
Bureaucratic rationality, however, is not the only model of an adjudicatory process that might be applied to disability benefits determination. Some parts of the process, in particular decision making at the ALJ level, are more like an adversary adjudicatory process. Moreover, one could imagine the process as one that looks like the Internal Revenue Service system in which “claims auditors” might have the capacity to grant, deny, or even “settle” claims and would then defend those decisions at subsequent levels of review. Workers’ compensation and unemployment compensation systems provide other examples of adversary models of benefits adjudication.
Another, radically different approach would conceive of disability benefits designed to assist claimants in receiving appropriate medical attention and vocational rehabilitation as well as appropriate income supports. In this model the basic goal of the program would be to move claimants back toward productive work and to use benefits both as a means to facilitate the return to work process and as an ultimate fallback for those claimants whose impairments make continued work impossible. This is the approach used by many private disability insurers who manage employment-based disability plans in the United States, and it is the dominant model in certain foreign systems, such as those in Sweden and Germany.
Concern also is expressed that environmental factors, including environmental barriers to work are not taken into consideration in defining work disability.
Recent legislation makes clear that Congress is increasingly interested in the “return-to-work” model and is prepared to have SSA experiment with some alternative strategies that might facilitate the pursuit of work rather than benefits. The Ticket to Work and Work Incentives Improvement Act of 1999 (P.L. 106-170) was signed into law on December 17, 1999. One major provision of the law establishes the Ticket to Work and Self Sufficiency Program, or Ticket Program. This provision provides that eligible SSDI and SSI beneficiaries with disabilities will receive a ticket (or voucher) they can use to obtain employment services, vocational rehabilitation services, or other support services from an approved provider of their choice. The law also expands Medicaid and Medicare coverage to more people with disabilities who work.
Recommendation 6-3: The committee recommends that the Social Security Administration initiate a research program for testing decision process models that emphasizes rehabilitation and return to work.
In conclusion, although SSA has deferred a major redesign of the disability decision process, the committee believes that it is paramount that the determination of disability not only be timely, understandable, straightforward, and feasible, but also provide accurate and consistent decisions that are fair to the claimant and to the government. To this end the committee believes that SSA should undertake a systematic, long-term program of research—intramural and extramural—that provides baseline information on all key aspects of the current disability decision process and subsequent evaluative data on all future change aimed at improving the effectiveness and efficiency of the work disability determination process currently in use in the United States.