Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 15
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report 4 SSA's Research Plan for the Redesigned Disability Decision Process The previous chapter presented a general framework for the disability decision process research. This chapter reviews the Social Security Administration's (SSA) research plan in the context of this framework and identifies gaps in the research plan as the committee sees them at this early stage of its study. The committee hopes that its comments will serve to guide SSA as it proceeds with its research to develop, and to determine the feasibility of the various elements of the proposed new disability decision process. The goal of the redesign research should drive the research plan. According to SSA, “the goal of this research will be to devise a more efficient and more accurate method for making timely determinations of disability for Social Security claimants” (SSA, 1996). The new disability decision process, as envisioned by SSA, will assess a person's functional ability once, relying on objective, standardized, functional assessment instruments. SSA believes that focusing decisions on the functional consequences of a person's medical impairments will permit physicians and others who provide medical evidence, and decisionmakers to use a consistent frame of reference for determining disability, regardless of the diagnosis. SSA assumes that such a focus will reduce the need for developing voluminous medical records (SSA, 1994a). As described earlier in the report, under the proposed disability decision process, decisionmakers will consider if a person has a medically determinable impairment(s), but will no longer impose a threshold “severity” requirement. They will rely instead on a new “Index of Disabling Impairments,” which has not yet been developed. The index will replace the current listings of medical impairments. Presumably, “…it will only consist of descriptions of specific impairments and medical findings that are used to substantiate the existence and severity of the particular disease entity” (SSA, 1994a, p. 23). The index will describe quickly and easily impairments that are so severely debilitating that, when appropriately documented, they will be equal to functional inability to perform substantial gainful activity (SGA) regardless of the person's age, education, and work experience. SSA, therefore, believes that it will no longer need the concept of “medical equivalence” that is in the current decision process, thus eliminating one decision step in the current sequential evaluation process in relation to the index. If the claimant is not considered eligible for benefits based on the index, then the statutory requirements of age,
OCR for page 16
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report education, and work experience will be considered when deciding the claimant's ability or inability to engage in substantial gainful activity. SSA hopes to develop standardized functional assessment instruments that accurately measure a person's functional abilities and are universally accepted by all concerned parties. These instruments would measure an individual's abilities to perform a baseline of occupational demands. The agency further plans to develop a new standard to describe basic physical and mental demands of a baseline of work (i.e., work-related functions) that represent SGA and that exist in significant numbers in the national economy. SSA'S RESEARCH PLAN The research plan developed in 1995 by SSA has three components.2 Information gathering (comprehensive review and analysis of existing information) on: functional assessment instruments, occupational classification systems, disability determination processes used in other disability programs in the United States and other countries, and the effects of age, education, and work experience (vocational factors) of the applicant. Integration, synthesis, and development of a prototype for a new disability decision process: analysis and evaluation of the literature reviews undertaken in the first component, and development of prototype(s). Testing, analyzing, and refining the prototype: laboratory research and other small-scale testing; and the Disability Evaluation Study: to estimate the size and characteristics of the population eligible for disability benefits, determine factors permitting them to work, and assess future changes in the prevalence of disability; and also to serve as one of the evaluation mechanisms for the decision process prototype(s). Work on the first two components of the research plan is underway. Some information-gathering activities in the first component have been completed through specific contracted research. The remainder of the activity in the first component also is being conducted through contracts. Most of these latter contracts were negotiated at the end of fiscal year 1997. 2 SSA's research plan, along with a timeline for actions and completion dates, was published in the Federal Register in August 1996 and an update of the plan was published in November 1997 (SSA, 1996, 1997b).
OCR for page 17
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report SSA has contracted a series of projects to review and analyze research and literature done to date in the following subject areas: Functional assessment instruments (completed in 1996-contract with Virginia Commonwealth University [VCU]), Occupational classification systems (completed in 1996-contract with American Institutes for Research [AIR]), Disability determination processes used in other disability programs (contract awarded to Westat in September 1997 for completion in five months), Effects of vocational factors (Interagency Agreement with Library of Congress negotiated in September 1997 for completion in one year). Work on the second and parts of the third component has begun with the award of a task-order-type contract to AIR at the end of fiscal year 1997. The purpose of this contract is to synthesize and integrate the results of the literature review contracts and the Disability Evaluation Study (DES); to develop, test, evaluate, and refine alternative prototypes for a redesigned disability decision process; and to undertake additional research as yet unspecified. Additional work in the research and development of the decision process also may be undertaken in subsequent task orders under this umbrella contract. The number of task orders to be performed by the contractor and the duration of each will be determined by SSA. The committee reviewed the preliminary design of the DES and issued a report in July 1997. The DES is part of the third component of the research plan. Many of the recommendations in the committee's report have been incorporated in the request for proposals issued by SSA on January 15, 1998 (SSA, 1998a). SSA anticipates that a contract for the DES will be awarded in the summer of 1998. PROGRESS TOWARD THE FRAMEWORK FOR THE DECISION PROCESS RESEARCH The previous chapter outlined an initial conceptual framework of issues, research steps and methods for a research plan or design to develop and assess the proposed decision process as a workable solution to current problems. What follows is a preliminary review of SSA's research plan and the individual projects within that plan with reference to each of the research steps identified in the framework outlined in Table 3-1. SSA's research plan comprises projects that are oriented to developing and testing the functional assessment instruments in the disability decision process, exploring the effect of vocational factors on decisions, and developing a prototype for a revised disability decision process. Ultimately, however, the questions that need to be answered include: Will the projects together answer the questions that need to be answered? Is the approach adequate? Is the time frame for the completion of the research agenda sufficient? Are there critical gaps in the research that need to be filled? What problems will be faced if the sequencing of the various projects is not synchronized? Finally, will the research answer the basic concern expressed by the Commissioner in 1994—can the proposed revisions of the disability decision process be implemented?
OCR for page 18
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report Needs Assessment Research As discussed in Chapter 3, needs assessment research takes place early in any discussion of a major change to improve an ongoing program, such as the disability decision process, in order to understand the nature and extent of problems with the current process. It is an important early step for planning a research strategy. Assembling and analyzing existing information using administrative and statistical data, internal agency data and other documents, and results from special small studies and internal and external surveys to examine the relative roles of the factors associated with the problems and their concentrations, helps to understand the problems associated with the current disability decision process. It should also help define the goals to be achieved. Supplementing the knowledge gained from these activities with focus group discussions involving claimants and internal and external stakeholders, as well as satisfaction surveys is sometimes useful. Early in the reengineering effort, SSA held a series of focus groups throughout the country to obtain input from the general public and the claimant population. The agency staff also conducted fact-finding visits and interviews with members of the disability community. Similar meetings and surveys should be conducted with regard to the redesign of the decision process as it is being developed and after a prototype has been developed, but prior to implementation. If the purposes of redesigning the disability decision process are as stated earlier in the report, the research design calls for comparative judgments. It presumes analysis of baseline information from the current decision process to assess the effectiveness, or lack thereof, of the current decision process and compare it with similar analysis of a new decision process. As described in Chapter 2, during the past several years, SSA has conducted some baseline studies with respect to the current claims process. The committee has reviewed the available analysis of the amount of time and staff investment in processing disability claims, the problems associated with processing claims in a uniform and timely manner, as well as the nature and extent of the inconsistencies of decisions between the disability determination service (DDS) and the administrative law judges (ALJ). Early in the reengineering effort SSA also developed measurable criteria for evaluating if the reengineered claims process will achieve its stated objectives of being user-friendly, prompt, accurate, efficient, and satisfying for employees (SSA, 1994b). However, based on the information provided by SSA, the committee assumes that the agency has not conducted such baseline analysis with predetermined criteria for evaluating the components of the sequential disability decision process leading to the decision to its redesign. Moreover, such analysis does not appear to have a place in the current research plan. SSA's current research approach focuses mostly on the new decision process, therefore fails to build in tests that may be critical to answering the comparative questions and, ultimately, to the decision whether or not to adopt a new decision process. Evaluative Criteria. In order to assess if the decision process, current or proposed, is effective and efficient, objective measurable criteria need to be established against which the current disability decision process can be assessed. These criteria should reflect the purposes or objectives of the decision
OCR for page 19
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report process. Studies should be conducted on the existing process and data analyzed in the context of the established criteria in order to identify the nature of the gaps between what the program is supposed to achieve and what its actual performance is. Without these research steps and analysis, there is no objective way to conclude if the revised process is more effective and more efficient than the existing one. For example, SSA plans to replace the current listings of medical impairments with an index of disabling impairments. SSA has a conceptual notion of what this index should be, but no list has been developed and tested to date. Admittedly the current listings have expanded beyond their original purpose. Originally the listings included a short list of readily identifiable disabling impairments for the purpose of being a simple screener. Since then they have become lengthy and complex, requiring specialized medical evidence. In the mid-1980s, mental impairments were added to the listings, many of which take into consideration functional consequences of an impairment. The committee understands and supports the need to revise and update the listings to restore them closer to their original purpose. However, the committee is not aware of any attempt to evaluate the currency and consistency of the listings, or at least those groups of conditions that account for a significant proportion of the disability rolls. SSA appears to have made the decision to replace the current listings with an index without any attempt to first evaluate them and use the findings to update the listings or to guide in developing a new index. SSA should specify the desired levels of specificity and sensitivity and evaluate the current listings against those standards to serve as a baseline for creating the new index. The index is supposed to be simple enough so that lay persons will be able to understand what is required to demonstrate a disabling impairment in the index. To the extent possible, the medical findings included in the index will be in nontechnical language. Such a major change for determining a severely disabling medical impairment has yet to be developed, tested, and refined. Although conceptually appealing, the committee questions if such an index can be developed, tested, and implemented with confidence on a national level within the timeframe set by SSA. The committee supports the conclusion of the Disability Policy Panel of the National Academy of Social Insurance (Mashaw and Reno, 1996) to give high priority to research related to the medical listings as well as to evaluate the consistency of the presumptions underlying the medical listings for different body systems. SSA should conduct the necessary research, prior to implementing the new index, to determine: (1) whether or not the current listings satisfy the agency's goals for specificity and sensitivity, (2) whether or not these goals are satisfied consistently across the listings for the different body groups or conditions, and (3) evaluate the options to correct the problems detected by these evaluations, as it develops any new list of medical impairments. Throughout the documents reviewed by the committee relating to the redesign research, including the scope of work for the research contracts and in presentations before the committee, SSA has recognized the need to test the new disability decision process by applying standards of validity, reliability, sensitivity, specificity, credibility, and flexibility. In addition, the stated objectives of the redesign also include requirements such as simplicity in administration, consistency, accuracy, timeliness, equity of decisions at all levels, and fairness. However, to the committee's knowledge no measurement criteria have been established to test the current and the redesigned process along any of these lines. Measurement is the process of linking abstract concepts to empirical indicators (Carmines and Zeller, 1979). Various terms are commonly used to describe measurement. For instance, to determine the extent to which a particular empirical indicator(s) represents a given concept, one can
OCR for page 20
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report examine the reliability of the concept, that is, the reproducibility of a decision for each case within and/or across decisionmakers. But a process or indicator needs to be more than reliable if it is to provide accurate results. It must also be valid. Although the terms reliability and validity are often used together, they are not synonymous. A decision may be reliable but it may not be valid. Both reliability and validity reflect matters of degree. Validity represents a set of criteria by which the credibility of research may be judged. For example, it measures the degree of agreement between the disability decision and actual fact of disability. Moreover, validity has several meanings. These include construct validity; content validity; criterion validity, which includes concurrent validity and predictive validity; and study validity, which includes internal and external validity (Last, 1983). The definitions among the disciplines of logic, epidemiology, social science, and statistics do not always correspond. A measure of the sensitivity of a test is the proportion of truly “disabled” persons that are ruled “disabled” by the test, and a measure of specificity is the proportion of truly “nondisabled” that are ruled “nondisabled” by that test. If it is assumed that operations can be defined to measure empirically the variables of sensitivity and specificity, these variables can be considered jointly in the context of the relative operating characteristic (ROC). Now used in several diagnostic contexts, the ROC plots sensitivity against specificity to show the trade-offs of these two variables as decision thresholds (and cutpoints on component scores) vary (Swets, 1988, 1992). Effectiveness measures the extent to which the decision process is achieving its goals and purposes (Berk and Rossi, 1990), and the concept of effectiveness must always address the issue of, “compared to what?” regardless of whether it is marginal effectiveness, relative effectiveness, or cost-effectiveness. Finally, efficiency measures the results of a process in terms of resources expended and time. The committee notes that both the fields of science and law pay unusual attention to the definition of terms. When the two fields intersect, it is particularly important to be specific about the terms used. For example, although the word “disability” is used by SSA, the actual phenomenon focuses on an attribute more narrowly defined than the inability to perform the usual activities of daily living. It refers to the inability to engage in substantial gainful activity (SGA) because of a medically determinable physical or mental impairment leading to death or expected to last for at least 12 continuous months. Such distinctions are key to SSA's use of terms such as “validity,” and here the legal use of the term is important to guide the scientific criteria. The appropriate construct to use in judging whether the current or the new decision process achieves its goals must take into account the legal language. The committee, however, recognizes that the actual fact of disability may be unobservable. For example, the number of persons with disabilities under the Social Security definition will vary as a result of judicial interpretations. Each aspect of the law is subject to differing interpretations and judgments. Problems arise at each step of the sequential decision process, whether one is determining if a person has an impairment that will result in death or that is expected to last at least 12 months or, more crucially, whether by reason of that impairment a person cannot engage in SGA. The difficulty of making such a decision, however, is not the issue. Rather there is no single true answer to the question of whether a person with that impairment should be expected to engage in SGA. Simply put, a “gold standard” does not exist. Therefore, it is necessary to substitute some criterion or target and assess how well actual determinations are meeting this target. Whatever SSA chooses as a criterion or target also must be disseminated to stakeholders and decisionmakers as soon as possible, along with a plan for
OCR for page 21
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report validity assessment. Only with such openness will the validity assessments be accepted when they become available. The brief discussion of measurement terms and issues clearly demonstrates the need for SSA to specify early in the redesign effort what it means by the terms validity, reliability, sensitivity, specificity, credibility, flexibility, and all the other related terms that it uses; and how it plans to measure them, that is, what measurable criteria will be used to assess these standards vis à vis the disability decision process research. The same criteria should then be used to evaluate the quality of both the current process of disability determination and any prototype to be tested. Recommendation 4-1: The committee recommends that early in the redesign effort, the Social Security Administration should specify how it will define, measure, and assess the criteria it will use to evaluate the current disability determination process, as well as any alternative processes being developed. In any scientific process, standards of acceptance or rejection are declared before, and not after, data are analyzed. Similarly in an evaluation research process, evaluative criteria and validation plans should be determined by the agency early in the research process, and not as currently planned after the prototype decision process is developed. The committee cannot emphasize enough that the agency has responsibility for setting the standards or evaluative criteria, and for designing approaches to testing and experimentation to be used by a contractor. Identify Alternative Options In order to answer the question of what can be done to alleviate the problems in the current decision process, alternative solutions have to be identified and developed. The research methods that should be employed include: (1) review and analysis of existing literature relevant to the problem, and (2) initiation of additional specially targeted research that could lead to the development of solutions. Preliminary review of SSA's research plan suggests that the agency is employing these methods. As described earlier in the chapter, component one and part of component three of SSA's research plan fall in this category. What follows is a review of the individual projects undertaken. Review of Functional Assessment Measures In 1995, SSA contracted with the Virginia Commonwealth University (VCU) to review systems, methods, and instruments that measure a person's functional capacity to perform activities and tasks, to develop a matrix of categories to classify these instruments, and to evaluate them to determine their potential application in the disability decision process. VCU completed its research and submitted its findings and recommendations to SSA in 1996. The original literature search yielded nearly 700 different types of functional assessment instruments. Half of these instruments were discarded because they lacked the basic standardization criteria or were not relevant for SSA. The remaining 300 instruments were classified by further criteria, which narrowed the selection down to 46 instruments. Selection criteria focused on two of the more important issues: (1) primary purpose and description, and
OCR for page 22
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report (2) validation methods and samples. In addition, VCU also evaluated the availability of instruments in the public domain, global measures that can be applied to all types of disabilities, instruments not requiring administration by a physician or other highly trained specialist, cost of administration, and language. Finally, VCU looked at the extent to which the instruments had been used on the working age population and can be generalized to the SSA claimant population. VCU's main conclusion in its report was that no government or private entity is currently using functional assessment instruments specifically for determining work disability benefits, and a global measure of functional assessment does not exist that would be a valid indicator of disability for all populations currently served by SSA. Such an instrument will likely have to be developed and tested. After reviewing the advantages and disadvantages of both functional and clinical assessment measurements for SSA's needs, VCU concluded that objective functional assessment can and should be a component of the redesigned process. It recommended that SSA should use a global functional assessment measure to screen out people who do not have severe disabilities. A combination of clinical and functional assessment measures can be used in the next step to decide if the claimant meets SSA's disability definition. The global screening instrument to assess functional capacity could be standardized and validated on the SSA claimant population. VCU, however, stopped short of constructing the global measure of functional capacity; instead, it recommended several steps SSA should take in moving toward the development of a global tool. The committee has not been informed of SSA's progress in this activity. VCU further stressed that in order for SSA to use this type of process, it needs to have (1) normative and predictive data for initial screening and determination of severity of disability, and (2) an occupational classification system that can be linked to residual functional capacities. The committee is concerned that neither VCU nor SSA has made clear what conceptual or theoretical basis exists for believing that such a standardized, universally accepted global instrument applicable to persons with physical and mental impairments can be constructed. SSA needs to describe the basis for this belief, such as results from some developmental work in progress or literature that shows such instruments are in use elsewhere, before committing to such an approach in the new disability decision process. Recommendation 4-2: The committee recommends that the Social Security Administration develop an alternative plan for use of functional assessment measures in the disability decision process in the event that the proposed global, standardized, functional assessment instrument is not developed and tested in time for implementation. The committee is convening a workshop in June 1998 to discuss measures of functional capacity and work requirements as they relate to SSA's disability decision process research. The committee urges SSA to take into consideration the results of the workshop as it proceeds in the area of functional assessment. Identification of Occupational Classification Systems In 1996, SSA contracted with the American Institutes for Research to conduct a comprehensive review of the literature pertaining to systems and methods of classifying
OCR for page 23
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report occupations in terms of the physical and mental capacities required, to develop a taxonomy of occupational classification systems, and to assess the applicability of systems for SSA's redesigned disability determination process. This review relates directly to one of the key elements in the proposed redesigned disability decision process, namely, assessing baseline work. The purpose of the review is to determine if a standard exists, and if not, whether it is feasible to develop one to describe basic physical and mental demands of a baseline of work. AIR's initial search yielded 33 different occupational classification systems. Based on the needs of SSA's disability determination process, AIR further screened these 33 systems and narrowed the search to 14 occupational classification systems. Each of these systems was rated on three criteria—content, technical quality, and usability. AIR concluded that while none of the occupational classification systems exactly or ideally matched SSA's needs, the Occupational Information Network (O*NET) under development was the closest match to SSA's needs. O*NET is an occupational classification system being developed by the Department of Labor (DOL) under contract with AIR to replace the Dictionary of Occupational Titles. It consists of two databases. The first database is the incumbents' judgments about the skills and abilities needed for their jobs and the work activities they perform. There are over 300 variables per occupation for this database. Currently, O*NET only has data available for 30 of the largest occupations out of the estimated 1,200. DOL is scheduled to complete this database in 3 to 5 years. The second database contains job analysts' ratings of the skills and abilities that they believe are required for different jobs. This database covers all 1,200 occupations, however, it includes only a subset of the 300 variables per occupation. A prototype of the analyst-derived database was released by DOL to interested software developers in the fall of 1997. One of the reasons AIR recommended O*NET over the other systems is because it uses level scales to measure the amount of skill needed to perform certain jobs. Incumbents choose a numeric rating based on their reading of the behavioral anchors. Cognitive and mental descriptors are also included in O*NET, but the physical ability scales that O*NET uses may not be specific enough to help SSA. For instance, they do not address single arm movements. Measurement of the physical ability scales is more like gross body movements as opposed to, for example, one arm. Because the incumbents' database will not be released for some time, SSA will probably have to use the analysts' database that includes only a subset of the variables that are in the incumbents' database. Although most of the variables SSA is interested in are in this subset, SSA needs to test the analysts' database to see if it contains all the information it needs. AIR recommended that SSA establish a working relationship with DOL in the development of O*NET, construct a matrix of occupational classification and a functional assessment taxonomy, and conduct an analysis of the O*NET database, before deciding on O*NET as SSA's occupational classification system. The committee is not aware of the extent to which SSA has followed through on these recommendations. Given the alternatives, the committee believes that O*NET may be the best option for SSA. AIR's suggestion that SSA construct a matrix between O*NET and a functional assessment taxonomy is especially useful, given the unique needs of SSA's disability decision process. Such a matrix would allow SSA to observe, and possibly supplement, any potential gaps that exist within the O*NET database. Other issues need to be resolved before SSA uses O*NET in its disability determination process. First, there is no evidence that a simple definition of workplace functionality exists. In fact, those working in the field of occupational classification find it difficult to agree on a definition of an occupation that is precise enough to be applied across all workers and in all
OCR for page 24
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report business establishments. Too much depends on the worker, his or her ability to adjust to change, and on the employer's flexibility. O*NET comprises a classification of the abilities and strengths required for people to work at certain jobs. SSA's disability determination is primarily based on people's inability to engage in any SGA. It may be difficult for SSA to use O*NET to match people's level of ability to work with specific potential occupations. Another problem with O*NET is that it provides the average level of performance rating for each occupation. SSA's plans call for a baseline or minimum level of performance, since its goal is to see if people can do any work that is readily available in society, taking into consideration their age, education, and work experience. It is not clear how SSA plans to overcome this problem. Finally, O*NET's physical ability scales may be inappropriate for persons with disability. However, according to AIR, they can be linked to levels of functioning. For example, O*NET lacks job descriptors related to the ability to sustain work over an 8-hour day for extended periods; psychosocial characteristics; and executive functions, such as planning work activities and schedules; or finishing one task and deciding to start another. These are important reasons why people with disabilities do not have jobs. The committee is concerned that O*NET may not include these characteristics. The committee questions how O*NET will be used. SSA's current research design does not appear to be oriented to address this question. How does SSA plan to supplement O*NET with respect to contextual or other factors that are not well covered. There are no indications in the current research plan that the gaps in O*NET will be carefully considered and specific research identified to fill those gaps. The committee also is concerned about the synchronization of timing for completion of O*NET and SSA's target completion of the research for development and implementation of the disability decision process. Recommendation 4-3: The committee recommends that the Social Security Administration develop an interim plan for an occupational classification system in the event that the Occupational Information Network (O*NET) database is either not completed or insufficient to meet the needs of a new disability decision process. The committee believes that SSA should explore entering into some interagency arrangement with the Department of Labor to initiate a version of O*NET that would collect information on minimum as well as average job requirements to better serve SSA's needs to assess ability to engage in substantial gainful activity. Other Disability Decision Programs In September 1997, SSA awarded a contract to Westat to review other disability programs, public and private in the United States and in other countries, to determine if such programs have criteria, instruments, or any other features that may be appropriate for use in developing SSA's new disability decision process (SSA, 1997a). Undoubtedly, one can learn from the experience of other disability benefit programs that face the same general problems as the Social Security disability programs in the United States.
OCR for page 25
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report At the same time, it is important to recognize that no other country in the world has a program as large as the U.S. program. When the United States is compared to other developed countries with disability benefit systems, it is the only country that does not have either a short-term or a partial disability program at the federal level (SSA, 1997e). In the United States, an applicant must meet the test of being permanently and totally disabled, or have the benefit application denied. The task is considerable in light of the short duration of the contract—156 days. There is a paucity of evaluative materials in written documented form about the operations of these other programs, and comparing the published records can be problematic for a number of reasons: As the observer of other programs tries to relate specific provisions of the foreign or other domestic programs' laws to the number or types of decisions, it is difficult to take into account for the more elusive factors such as the adjudicatory climate. Public opinion or the public perception of how the program is operating affects any program's operations. If the perception is that too many disability claims are being allowed, stories begin to circulate about the person who is on disability and actively engaging in sports, or the rumors circulate about children being coached to feign disability symptoms. Once the law is changed to make access to benefits more difficult, the nature of the public's perception changes, and the stories circulate about how worthy recipients are being stripped from the rolls. In either case, the administrators react to prevailing winds of public opinion, as does the public itself. It is commonplace to note that applications tend to decline and decision criteria tend to change in advance of any actual changes in the legal provisions governing these criteria (Yelin, 1989). Another influence on decision outcomes is the quality of the administrative personnel. The competence of the decisionmakers and the extent of their freedom to operate free from political pressures are most important. For example, the tenure of some administrators of state workers' compensation programs is notoriously short. Another is what goes by the catch-all term, “cultural factors.” Differences exist among nations in the propensity of its citizens to sue one another or the administering authority, if a decision is not to their liking. There are vast differences in what people tolerate in the way programs are administered. In some nations, the administering authority can summon persons in for examination at the convenience of the authority, whereas in other countries, contact with the applicant would not be usual without extensive notice and appropriate safeguards. Nevertheless, there are important similarities among programs across countries. All disability programs must first screen applicants to determine if an individual falls within the covered or insured group. All programs must then determine if the applicant has a physical or mental impairment. Learning about the various processes by which this determination is made should prove useful to SSA. In some programs, benefit awards are made simply on the basis of the extent of the physical or mental impairment. In other jurisdictions, the decisionmaker is required to inquire into the extent of work incapacity, taking into account the applicant's education, age, and work experience, and possibly even the condition of the labor market.
OCR for page 26
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report Information from other programs about how such decisions are made, taking into account the adjudicatory climate, the type of administrators, and cultural factors, should prove valuable to SSA. It should prevent designers of the new process from making elementary mistakes. What should become clear after examining these other programs is that no obvious solution exists. Under particular conditions or at particular times, certain decision criteria seem to work, at least for a time. However, very sophisticated systems can lead to unsatisfactory results, and some simple systems can work to the satisfaction of the stakeholders in the system. What may emerge from examining other programs is that substituting measurements of functional capacity for assessment of physical impairments, or vice versa, is not necessarily a solution to a program's problems. The problems are complicated and deep-seated, and decision criteria cannot always be separated from the total administration of a program. They may stem from the adjudicatory climate, orientation to the welfare state, or the method of assessing functional capacity for work. In order for the examination of other programs to be most useful, SSA should view its results with attention to differences in level and type of benefits paid, differences in administrative methods and resources, and especially differences in decision criteria used to measure physical and mental impairment and work incapacity. Role of Vocational Factors in Determination of Disability Based on a request for proposals issued by SSA, a 9-month reimbursable interagency agreement was negotiated with the Federal Research Division of the Library of Congress in September 1997 to review and evaluate published literature and any other research pertaining to the effects of age, education, and work experience on the ability to work or adapt to work in the presence of functional impairment (SSA, 1997c). The purpose of this interagency agreement with the Library of Congress, as stated in the request for proposal (RFP), is to assist SSA in deciding an appropriate way to incorporate into the redesigned disability decision process the specific statutory requirement to consider an individual's age, education, and work experience in determining ability to work. The committee has concerns about several parts of the scope of work described in the request for proposals. One of the key objectives of the project is to determine the relationship of vocational factors and the interaction between them, to the person's ability to work. This is an important issue and the answer could be a major contribution to knowledge. The committee, however, wonders if this project, as stated in the scope of work and in the workplan developed by the Library of Congress, will be able to differentiate between those who cannot work, and those who are not working but could conceivably do so, and who have vocational deficits that suggest that vocational factors may be the cause. The scope of work for this project includes: a comprehensive review and evaluation of published and unpublished literature, both national and international, and of any other research pertaining to the effects of vocational factors on an individual's ability to work, or adapt to work, in the presence of a functional impairment; and an analysis of the development of, and basis for, the current statutory requirement for consideration of vocational factors. It further includes an assessment of how these vocational factors should be considered in evaluating a person's ability to work despite a severe, medically determinable, physical and/or mental impairment; and evaluation and recommendation of methods or techniques that may be appropriate for use by SSA.
OCR for page 27
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report SSA has listed in its RFP several related or subsidiary issues likely to arise in the course of conducting this research. SSA expects the Library of Congress to research and answer these questions in its evaluation and recommendations. Some of these issues are: the extent to which the actual process of aging, as distinct from medical impairment, affects an individual's ability to work; if measurement of functional capacity will address adequately the impact of age on ability to work and if not, what additional measures will be needed; how to define “nearing full retirement age;” can a baseline of work be defined in terms of clusters of jobs in order to account for the impact of vocational factors; and what are the effects of changes in the workplace. The final report of this interagency agreement will provide an in-depth analysis of the role of vocational factors in a new disability decision process. This process will include a detailed assessment of each factor's contribution to the evaluation process; documentation of the separate effects of each vocational factor, as well as their interaction with one another and with an individual's functional abilities; assessment of the ability of an individual to adjust to work other than that which has been previously performed; and recommendations for any change in the role of vocational factors in a new disability decision process. As a result of this review and analysis, SSA states that it hopes to determine if the present process; the proposed decision process; or some other, as yet unspecified, process provides the optimal mechanism(s) for considering vocational factors. Does that mean that the Library of Congress first will assess the role of vocational factors in the current decision process (or that such baseline analysis has been done by SSA) and then undertake similar assessments on the proposed decision process(es). In this context, baseline studies are needed to obtain answers to questions, such as how important to decision-making is the current evaluation of vocational factors; how are the factors specified operationally; and what effects are found from their application, with respect to the population of claimants and beneficiaries as a whole, and also with respect to different groups of beneficiaries. The committee is not aware if such a baseline study was or is being done. The committee was informed, however, that such an analysis might be part of one of the last tasks under the synthesis and integration contract awarded to AIR, but the Library of Congress project will be completed by then. Moreover, SSA is asking the Library of Congress to focus on how the effect of vocational factors would fit into the proposed sequential disability decision-making process, but if the alternative is “as yet unspecified,” it may be difficult to assess the effects of vocational factors. SSA currently uses the vocational rules to guide its decisions only for people who are classified by adjudicators as having physical impairments. In this study, SSA asks the Library of Congress to go beyond the current methods and analyze how these vocational rules can be used for people with mental and cognitive impairments. SSA only briefly mentions this in the scope of work, but the implication is that the results of this research will lead to vocational factors that can be applied to individuals with both physical and mental and cognitive impairments. In conclusion, the scope of work in the RFP for this study is formidable especially in light of the short timeframe for completion of the work (259 days). At the same time the workplan for this research is general and brief. The committee, therefore, is unclear to what extent and depth the issues laid out by SSA will be addressed. The tasks stated in the RFP, if performed adequately and in a comprehensive manner, are considerable.
OCR for page 28
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report Recommendation 4-4: The committee recommends that the Social Security Administration conduct baseline studies on the role of the evaluation of vocational factors in the current decision-making process and the effects of these factors on the populations of claimants and beneficiaries. Specially Targeted Research. In addition to review and analysis of existing literature, specially targeted research is usually needed to identify and develop alternative solutions to existing problems. SSA's task-order-type contract with AIR negotiated at the end of September 1997 is in this category. To the committee's knowledge, to date no other research has been planned or undertaken that is specifically targeted for this purpose. The task order mechanism of this contract permits SSA to rapidly award additional contracts to AIR as separate tasks. Synthesis, Integration, and Prototype Development As described earlier in the chapter, SSA has awarded the first task order to AIR. The Boston University, VCU, Washington University's Program in Occupational Therapy, Westat, and numerous consultants are also involved in this task order. The purpose of the first task order under the umbrella contract is to synthesize and integrate the research conducted under the first component of the research plan; the work of SSA staff and designated consultants; and the work of the contractor for the DES; and to identify and conduct additional research needed. In addition, the contractor will develop and assess one or more prototypes for the new decision-making process and its individual components, and recommend testing strategies for the prototypes. These strategies will include evaluating and testing for validity, reliability, specificity, sensitivity, and other applicable measures of the prototype of a new decision process and its individual components (SSA, 1997d). This first task order includes 13 tasks with numerous subtasks to be completed in two years. The committee is unable to comment on the specific empirical research components because the workplan in response to the RFP is general and appears to rely mostly on opinions of consultants and interest groups, expert reviews, and focus groups for obtaining objective scientific information. These methods have a definite place, but they should supplement not substitute for, essential well-designed studies to obtain and analyze empirical data. The RFP issued by SSA suggests a loose, open-ended research strategy, lacking in specifics and directives to a contractor. Also some of the tasks appear to duplicate similar tasks in the other related contracts, and the timeframe for completion of tasks appears to be too short for adequate investigation of the issues. To illustrate the point, task two of this task order contract requires the contractor: (1) to review critically and analyze the findings of the four literature review projects (Functional Assessment Instruments conducted by VCU, Occupational Classification Systems conducted by AIR, Other Disability Programs awarded to Westat, and the Effects of Vocational Factors awarded to the Library of Congress as an Interagency Agreement); (2) to review the work of SSA's consultants, staff's research activities and deliberations, and other sources to assess the adequacy of the knowledge base underlying the development of a prototype of a new disability
OCR for page 29
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report decision-making process; and (3) identify additional research needs and make recommendations on the conduct of this research. All of these activities are to be completed and the final report on this task submitted to SSA in 5 months after the date of the agreement. However, the final report of the vocational factors study will not be available for AIR to review, analyze, and integrate with the other research projects until months later. The RFP for the DES contract was issued in January 1998, and a contract is expected to be awarded sometime in the summer of 1998. A month later, the contractor is required to complete task four, which is an in-depth review and analysis of the proposed decision process, including a preliminary assessment of the feasibility of developing such a process. The assessment will address potential development costs and timeframes, potential administrative and programmatic effects of such a new process, and make recommendations for development of alternative decision methods. Some of the other tasks to be conducted in this task order include: investigating issues related to the definition of a single baseline of work demands applicable to all circumstances in terms of common and fundamental functional requirements and use of O*NET to develop such a baseline, identifying the dimensions of functioning to be measured in the functional assessment instruments, developing standardized functional assessment protocols, and analyzing and assessing the feasibility of using these protocols. The contractor is required to develop several recommendations for testing the prototype(s) of the new decision process in various settings and to evaluate its reliability, validity, sensitivity, and specificity; and to assess its effects on decision outcomes, as well as on a number of administrative and program variables of importance to SSA. The committee is concerned that SSA is attempting to get answers to complex research questions in a very short timeframe. The committee supports the objectives of redesign research, but questions whether these objectives can be achieved within the time limits specified for completion of the task order and the individual tasks within it, all the individual projects, and of the total research plan. Proper sequencing of the projects is as important as the conduct of the discrete projects. Results from earlier projects should be available for development and conduct of later projects that will be influenced by these results. Research, development, and testing relating to such fundamental redesign as that proposed by SSA should be satisfactorily completed before the recommended changes are implemented. Recommendation 4–5: The committee recommends that the Social Security Administration reconsider the timeframe for completion of the redesign research so that the necessary questions can be answered in an appropriately sequenced and coordinated manner. Any changes in the disability decision process will be of critical importance to people with disabilities, their families, and advocates. When the redesigned process is presented to these groups, they will want to know what alternatives were considered in developing the new decision process and what evidence SSA has that the new process will be better than the current one. Small-Scale Testing and Field Evaluation Can a hypothetical concept developed in a research environment be converted into a workable program that can be implemented in the real world? In order to answer that question, iterative testing is required on small samples in a laboratory environment and then in the field.
OCR for page 30
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report Based on the results of this testing, prototypes can be developed and pilot tested, followed by field evaluations in the form of national surveys to further refine and revise the prototype. As the committee emphasized in its first interim report, SSA should use data from its national surveys and special studies, as well as surveys conducted by other agencies, to assist in evaluating the prototype and developing workable alternatives. Laboratory Research SSA's research plan includes testing in a controlled setting the proposed components of a new decision process, such as functional assessment tools and proposed screening mechanisms. SSA is hopeful that data from this testing will narrow the choices among possible policy options by developing empirical data on which to base decisions. In addition, SSA envisions “laboratories within which other potential components of a new disability decision process may be tested in a controlled setting. These laboratories may provide controlled setting and representative samples within which data may be gathered and a new disability decision process may be tested.”3 According to SSA, the setting and testing procedures will vary depending on the issue under investigation. SSA uses the term laboratory rather loosely to refer to any controlled setting in which it could test or evaluate ideas relevant to the project. In response to the committee's questions, SSA explained that a “laboratory” for purposes of its research plan could be anything from a small number of disability decisionmakers reviewing case files to determine whether they contain sufficient occupational information to a formal evaluation of the validity and reliability of a specific functional assessment protocol. To date, SSA has not developed any specific requirements regarding key questions and subject pools to be included. Clearly, SSA does not envision a laboratory similar to those established in some other federal agencies and in some organizations in the private sector. With the changing composition of the population, advances in medicine and technology, major shifts in the labor market and the changing perception of disability in society, the current major effort SSA has undertaken will not be a one time effort, but will need periodic modifications and changes in the years ahead. Recommendation 4-6. The committee recommends that the Social Security Administration establish a cognitive laboratory for the Disability Evaluation Study, disability decision process research, and for other purposes of the agency. For example, the National Center for Health Statistics (NCHS) cognitive laboratory was established to promote and advance interdisciplinary research on the cognitive aspects of survey methodology among federal statistical agencies, universities, and research centers. Identifying the underlying cognitive difficulties that respondents experience in dealing with difficult or impossible tasks implicit in some survey questions or procedures facilitates the process of revising a questionnaire or procedure. Intensive interviewing techniques in a laboratory setting detect design flaws in instruments by observing the cognitive stage of the response process. Since the establishment of the NCHS laboratory, other laboratories have been established in 3 Letter from the Social Security Administration, dated August 11, 1997.
OCR for page 31
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report other federal agencies including at the Bureau of the Census, the Department of Labor, Statistics Canada, and the private sector (Sirken, 1991). The committee urges SSA to explore with agencies, such as the Bureau of the Census, DOL, NCHS, and the private sector, experiences that can guide SSA in implementing this recommendation. Pretesting, Pilot Testing, National Survey The Disability Evaluation Study planned by SSA is one source for testing and field evaluation. The committee addressed the need for such testing in its first report dealing with the design of the DES. It recommended that a formal field experiment should be performed during the research and development and testing phase of the survey to determine the validity, reproducibility, and cost-effectiveness of the approaches developed. The report further stated that studies are needed to address measurement issues, such as assessing functional status and quality of medical evidence of record. These tests call for repeated measurements and will need to be iterative to be able to test the validity and reliability of measures. Process Engineering Input Determining eligibility to receive Social Security disability benefits under current eligibility rules is a highly complex process involving thousands of SSA staff trained to apply a multitude of specific instructions generally described in the Social Security Handbook and in greater depth in thousands of pages of process instructions and amendments found in documentary sources like, the Program Operations Manual System, Modern System Operations Manual, and Social Security Rulings. Essentially, the process of administering disability benefits is thus a large set of interconnected instructional steps, such as preparing a medical file for the applicant, completing an applicant interview, and completing the questionnaire in considering a beneficiary for continuing disability benefits. Completing these steps involves one of more of the following participants: applicants, beneficiaries, and various staff, such as claims representatives and vocational disability examiners. Viewed another way, each participant in the disability benefits process has a set of instructional steps that he or she is expected to complete, thus defining the process role for each type of participant. Two ingredients are essential for the success of any administrative process. First, instructional steps must effectively meet the administrative goals of the process, that is, correctly establish an applicant's eligibility for SSA disability benefits. Second, instructional assignments must be made to the right person(s), such as participants in the disability determination process. Evaluating the effectiveness of the first ingredient can be accomplished by considering the adequacy of various measures of process outcome, but evaluating the second ingredient—efficiency of task assignment and participant roles—requires input from those who study processes. Whereas, many of the committee's recommendations have addressed the need to assess the utility of the disability decision process (i.e., the first ingredient), none of its recommendations have specifically addressed the need for research into the process engineering of SSA's disability claims process (i.e., the second ingredient). While SSA has obtained some guidance early in
OCR for page 32
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report planning the reengineering of the disability claims process from process engineering experts, the committee believes that SSA would benefit from additional guidance in this area. Recommendation 4–7: The committee recommends that the Social Security Administration actively engage process engineering experts (such as industrial engineers or operations researchers) to evaluate and improve the Social Security Administration's disability benefits administrative process to assure that task assignments and participant roles achieve a maximum level of effectiveness and efficiency. This recommendation implies that SSA form a group of suitably oriented faculty from some of the leading academic process engineering programs (e.g., industrial engineering, operations research, and operations management in business) to assist SSA with planning its research agenda for the disability claims process. For instance, appropriate experts in industrial engineering might be drawn from programs at Georgia Institute of Technology, North Carolina State University, Purdue University, Stanford University, and Virginia Polytechnic Institute and State University. This process research advisory group might help SSA define specific issues that would become part of a research agenda aimed at improving the administration efficiency of the disability claims process. For instance, one item on the advisory committee's process research agenda might be to decide if the initial applicant interview is best administered by the claims representative over the telephone or self-administered and mailed in by the applicant, or some combination thereof. The advisory group might also assist SSA in soliciting and selecting an academic center(s) or private firm(s) to provide SSA with ongoing process research support of the kind implied by this recommendation. Program Evaluation and Transition to Implementation. Once it is decided that a proposed prototype for a new decision process is feasible, that is, has a realistic probability of success when implemented, the next step is a transition from a research mode to implementation of the process. The question that must be answered is: will the new process be more effective and efficient than the existing one. Research methods that are appropriate for such assessments include simulation and modeling using data from the national surveys, analysis of data using the same evaluative criteria developed at the initiation of the research and applied to the existing process, clinical trials, and applicant and claimant satisfaction surveys. The process also must be tested in selected sites and refined to assure its smooth implementation. Simulation, Modeling, and Analysis of Information Using Established Evaluative Criteria Over the years SSA has developed models to simulate the application process, assess the factors affecting disability decisions, and estimate the size of the population potentially eligible for benefits (Hu, et al., 1997; Lahiri, et al., 1995). To date the principal limitation is that these models depend mainly on household survey data which are supplemented with program data
OCR for page 33
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report only on applicants. These models, therefore, rely on limited self-reported measures of health and severity of limitations and incorporate the implicit assumption that self-reported data for applicants are consistent with self-reports for nonapplicants. However, perceptions of health and limitations may be different among applicants and nonapplicants, which in turn could result in differences in self-reports. Such differences in response may result in significant biases in estimates of the size of the population of eligible nonparticipants in the program. The DES for the first time will provide objective health related measures that can be used to evaluate the appropriateness and reliability of self-reporting measures for both applicants and nonapplicants. Such data can be used to both corroborate the validity and consistency of self-reported data collected in surveys and to improve models that may use data from current and future data sources. The committee hopes that data generated from this effort will be analyzed for modeling alternative disability decision processes, using the evaluative criteria established early in the redesign research. Moreover, the effect of redesigning the decision process on the number and characteristics of future beneficiaries also needs further study and development, and this will require specific and detailed information on health, impairment, as well as functional assessments. However, until the DES is more fully developed, the committee is unable to comment further at this time on SSA's plans for simulation and modeling using data from the DES and other data sources. The committee believes that SSA needs to enhance its research capacity in this area of research, including developing the capacity and expertise to undertake these activities in-house. Adequate resources should be made available for this important research area. Recommendation 4-8: The committee recommends that the Social Security Administration develop plans for simulation and modeling of alternative disability decision processes and other policy options, and devote adequate resources for this activity. Clinical Trials Clinical trials are mainstay tools of medical, therapeutic, and technology assessment. The most definitive type of clinical trials is the randomized controlled clinical trial (RCT). In an RCT, patients are randomly assigned to experimental and control groups. Randomization reduces bias that might otherwise be introduced by prognostic and other selection factors not accounted for in the design of the trial (Institute of Medicine, 1985). The committee believes that when alternative disability decision processes are developed, they can be formally tested in a direct comparison with the current process by the conduct of clinical trials. An example of a clinical trial that can be conducted is to subject existing claims files, for which a decision already has been made, to the alternative process and the results compared in terms of outcomes and process variables. Alternatively, a prospective randomized design could be implemented for comparing alternative disability decision processes.
OCR for page 34
The Social Security Administration's Disability Decision Process: A Framework for Research, Second Interim Report Testing in Selected Sites As with many applied research programs, the assessment of SSA's decision process involves a phenomenon that is affected by changes in the external environment. For example, as the universe of jobs changes, the set of functional requirements one needs to be qualified for any job would also change. As the economy changes or as physical and mental impairments recognized as debilitating evolve, the set of functions to measure might also change. As researchers invent medical and social measurement techniques, the set of functions that are feasible to measure changes. For all these reasons, SSA should not view the current research activities as discovering permanent solutions to the problems it faces. Research to evaluate decision processes should be an ongoing program. This could be integrated into a system designed to monitor the size of the pool of persons with disabilities. In addition, SSA's current plans generate the need to consider a longitudinal program of evaluation. The functional measurement to be used in DES will be based on measures newly developed by the integration contractor. Analysis of the DES may show that the set of functional assessments is not adequate for use in a decision process. Alternatively, other SSA research may demonstrate that new measures would offer potential improvements over the set used in the DES. Missing from all the contract activities completed and underway is any effort to stimulate exploration of alternatives to the traditional approaches. SSA should devote more attention to testing and experimentation. It should consider awarding a series of small grants or contracts to bring together experts and consumers to brainstorm innovative approaches. SSA will need tools to evaluate the performance of the new candidate measures relative to those used in DES. The committee believes that SSA should identify ways to test new ideas in future studies, including randomized controlled experiments of alternative procedures, which would facilitate ongoing improvements in SSA's disability decision process.
Representative terms from entire chapter: