3
Committee Reactions

The previous chapter gave a brief overview of the design and procedures for the Disability Evaluation Survey (DES) as proposed by Westat, the survey contractor to the Social Security Administration (SSA). This chapter highlights some of the committee's reactions to the design and its recommendations for improvements.

The committee has reviewed the reports prepared by Westat on the survey procedures, the design of the DES (including that of the pilot study), and the strategies to maximize response rates. The committee commends SSA and Westat for the tremendous amount of work they have done and the progress they have made in designing this complex study. Both SSA and Westat have heeded many of the recommendations made in the committee's first interim report. Their plans are an improvement over the preliminary design for the study reviewed by the committee in 1997 (IOM, 1997). The reports they have prepared are thoughtful, and contain sound ideas; in addition, they have addressed the correct issues for a survey design.

The committee, however, believes that the DES cannot adequately address all of the stated objectives using only a personal or proxy report of the factors related to the person's work environment. The complex interaction between environmental factors external to the individual (such as social, geographic, and economic influences) and the perceptions of the individual also should be studied.

In its first interim report, the committee reviewed and commented on the preliminary design developed by SSA for the DES. In that report, the committee strongly endorsed the conduct by SSA of a well-designed, carefully pretested, and statistically sound DES. The committee has not changed its position; in fact, it reemphasizes that position.

The committee, however, continues to be concerned that some of the key problem areas, and the recommendations for resolving them that were identified in 1997, are not addressed adequately in the reports prepared by Westat. In its 1997 report on the DES design, the committee had recommended that the period prior to and including the pilot study should be expanded and extended into a research, development, and testing phase for the survey, with applications to samples of the type that are more traditionally used in methods testing. The national survey should not be launched until the functional assessment instruments, survey operations, and other issues are fully developed, tested and resolved.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report 3 Committee Reactions The previous chapter gave a brief overview of the design and procedures for the Disability Evaluation Survey (DES) as proposed by Westat, the survey contractor to the Social Security Administration (SSA). This chapter highlights some of the committee's reactions to the design and its recommendations for improvements. The committee has reviewed the reports prepared by Westat on the survey procedures, the design of the DES (including that of the pilot study), and the strategies to maximize response rates. The committee commends SSA and Westat for the tremendous amount of work they have done and the progress they have made in designing this complex study. Both SSA and Westat have heeded many of the recommendations made in the committee's first interim report. Their plans are an improvement over the preliminary design for the study reviewed by the committee in 1997 (IOM, 1997). The reports they have prepared are thoughtful, and contain sound ideas; in addition, they have addressed the correct issues for a survey design. The committee, however, believes that the DES cannot adequately address all of the stated objectives using only a personal or proxy report of the factors related to the person's work environment. The complex interaction between environmental factors external to the individual (such as social, geographic, and economic influences) and the perceptions of the individual also should be studied. In its first interim report, the committee reviewed and commented on the preliminary design developed by SSA for the DES. In that report, the committee strongly endorsed the conduct by SSA of a well-designed, carefully pretested, and statistically sound DES. The committee has not changed its position; in fact, it reemphasizes that position. The committee, however, continues to be concerned that some of the key problem areas, and the recommendations for resolving them that were identified in 1997, are not addressed adequately in the reports prepared by Westat. In its 1997 report on the DES design, the committee had recommended that the period prior to and including the pilot study should be expanded and extended into a research, development, and testing phase for the survey, with applications to samples of the type that are more traditionally used in methods testing. The national survey should not be launched until the functional assessment instruments, survey operations, and other issues are fully developed, tested and resolved.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report The single most important issue surrounding the DES as currently developed and the biggest deficiency in its design is the truncation of the research and development phase that the committee had strongly recommended. The current schedule provides no flexibility in terms of the available time to correct problems and to test alternative solutions if some questions or procedures do not work in the pilot study (Table 3.1). The schedule developed for the conduct of the various phases of the survey and the time allotted to absorb and retool for the main survey will not permit deliberate and rigorous decisions about revisions of the design, procedures, or questionnaire content. Decisions will have to be made throughout the process, and there is insufficient time to resolve issues and test alternatives. SSA and Westat are aware of these uncertainties. The committee believes that unless these issues are resolved before the national study, SSA may be taking enormous risks that could compromise the scientific integrity of the national survey and its findings. PILOT STUDY Westat has developed extensive plans for testing and experiments to be conducted before and during the pilot study, but the short period of time allowed for research, development, and testing in the rush to launch the national survey will cause serious logistical inflexibility during the various phases of the survey. Before starting a national survey, sufficient time should be allowed (a) to conduct and analyze the results of the various pretests, focus groups, and cognitive tests; (b) conduct a comprehensive pilot study with the planned and other built-in experiments; and (c) analyze and test alternative solutions in areas that need resolution as a result of the pilot study. Under the current time frame, the pilot study may not yield much evaluative information and not enough time is allowed to make modifications in the national study after alternatives are tested. As far as the committee can gather from the documents it reviewed, the pilot study may only show that "the design works" or "the design does not work." Westat will not know if the alternative selected will work in the national study without further tests and evaluation of the alternatives. Unfortunately, the present schedule for the pilot study allows just about 2-3 months from the end of the pilot study (November 2000) to the start of data collection for the main study (January 2001). Although some decisions on instrumentation can be made prior to the end of the pilot study, sample sizes and results will not be sufficient to allow a thorough analysis of issues until close to the end of data collection phase in the pilot study. Even if analysis of some tests and experiments could begin earlier in the analysis phase of the DES pilot study, additional time will be needed to examine the implications and plausibility of several different "adjustments" in the problem areas. Westat needs to explain to SSA exactly what it plans to do when problems arise in the pilot study. There should be some advance plans about how they will identify the obvious problems, assess the risks they pose, and then what they propose to do about these problems. Moreover, given the complexity of the DES, once all the issues are resolved, Westat should consider conducting a dress rehearsal before starting the national study. The documents reviewed do not explicitly state that Westat plans to conduct a dress rehearsal and no time is allocated for it in the current timetable.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report TABLE 3-1. Timeline for Survey Activities Proposed by Westat   Time Schedule   Survey Activity No. of Months After OMB Approval Calendar Month After OMB Approval OMB Approval Received   October 1 (est.) Pilot Study     Print survey instruments 1 October Conduct listing activities and administer initial screener 2-11 Nov 99-Aug 00 Conduct interviewing and medical exam activities, including targeted referrals 2-12 Nov 99-Sept 00 Obtain medical records 3-13 Dec 99-Oct 00 Ongoing data preparation and processing 3-14 Dec 99-Nov 00 Data analysis and preparation of simulated case folders 5-14 Feb 00-Nov 00 Methodological Report 1: Evaluation of Pilot Study Methods 14-15 Nov 00-Dec 00 Submit protocol revisions to OMB 15 December 2000 Methodological Report 2: Instruments and Procedures for the DES Main Study 15-16 Dec 00-Jan 01 Main Study     Print revised survey instruments 16 January 2001 Conduct listing activities and administer initial screener 16-26 Jan 01-Nov 01 Conduct interviewing and medical exam activities, including targeted referrals 16-27 Jan 01-Dec 01 Obtain medical records 17-28 Feb 01-Jan 02 Ongoing data preparation and processing 17-29 Feb 01-Feb 02 Data analysis and preparation of simulated case folders 18-29 Mar 01-Feb 02 Descriptive Report 1: Disability Profile of Working-Age Americans: Preliminary Data from the DES 30 March 2002 Descriptive Report 2: Employment Profile of Working-Age Americans: Preliminary Data from the DES 32 May 2002 Descriptive Report 3+: other reports as specified 28-31 Jan 02-Apr 02 Analytic Report 1: Estimates of Trends in SSDI and SSI 30 March 2002 Analytic Report 2: Relationships Between Disability and Work 30 March 2002 Analytic Report 3: Evaluation of Disability Data Collection Questions and Procedures 31 April 2002 Analytic Report 4: The Effects of Disability on Retirement 32 May 2002 Analytic Report 5: Disability Decision Process 32 May 2002 Final project report 33 June 2002 NOTE: OMB = Office of Management and Budget. SOURCE: data provided by Michele Adler, SSA, June, 1999.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report PERFORMANCE OF THE INITIAL SCREENER AND ACHIEVEMENT OF TARGET SAMPLE SIZES An example of the inadequate time for developmental work is Westat's plan to conduct initial screening to classify people into four disability-status categories. As indicated in the previous chapter, a key element of Westat's plan for the DES is its intent to meet respondent sample size targets for four disability-status categories (with sample size targets for a completed examination indicated in parentheses): non-beneficiaries likely to have severe disabilities (3,090); non-beneficiaries who have less severe disabilities (1,545); non-beneficiaries with no or slight disabilities (515); and  people currently receiving disability benefits (515). The capacity of the screener to work, and for Westat to achieve the response rates needed, will have serious implications in terms of achieving the targets set for the sample sizes for each of the categories. In addition to levels of survey nonresponse at the various steps of the DES design and the extent of differential attrition among key population segments, the ability to hit these targets is heavily dependent on the capacity of the initial screener to correctly classify people screened by telephone or in person. In the current design, Westat does not know if the screener will be sensitive enough to classify the people in the sample in one of the four categories or if the categories themselves will be feasible. Also clouding this issue is the operational interpretation of being ''nonimpaired," as well as the qualifiers "severely," "less severely," and "slightly." The committee is concerned that Westat may not have an adequate opportunity within the present timeframe of the DES contract to adequately deal with category misclassification that is likely to occur with the initial screener. Westat's plan for DES instruments and procedures indicates that a "major objective of the pilot will be to analyze the proposed cut-points (to delineate the four groups) and make adjustments for the main study" (Westat, Inc., 1999a, pp. 2-7). Unfortunately, as indicated above, the present timeline for the pilot study allows just a short period from the end of data collection in the pilot study to the start of data collection for the main study. Even if analysis of the initial screener could begin earlier in the analysis phase of the DES pilot study, additional time will be needed to examine the implications and plausibility of several different "adjustments" that might need to be made as a result of the analysis; to examine the specific effects of attrition on the screening phase of the DES; and to seek the best available national data on the distribution of the four groups that is needed to establish post screening rates of disproportionate subsampling. In the committee's judgment, the addition of several months to the period between the end of data collection in the pilot study and the start of data collection for the main study will allow the screening and attrition findings of the pilot study to be more carefully considered and used to ensure that sample size targets for the four groups are met. Without this extension, the adequacy of the job done by the initial screener may well become an important issue, in that an adverse outcome could poses a serious risk to the success of the DES. The committee, therefore, urges Westat to allow enough time for it to carefully examine (a) the implications and plausibility of several different "adjustments "; (b) specific effects of attrition on the screening phase of the DES, and (c) the best available national figures on the distribution of the four groups.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report Targeted Sample Sizes The committee continues to have several questions and concerns about the rationale for the ability to achieve, as well as for the adequacy of the total sample size, and especially of the allocation of people among the four subgroups. It is not clear how SSA and Westat arrived at this particular disproportionate sample design. The committee has never seen the statistical rationale for setting the sample size targets. What precision targets were used to arrive at these sample sizes? Westat assumes that at response rates of 90 percent for each component of the DES, it should get the planned sample sizes. The committee believes that the rate is overly optimistic, especially for a population with disabilities. Even if Westat can achieve these planned sample sizes, the cells very likely will be much too small, especially if SSA stratifies for analytical purposes on more than one disabling condition, and or demographic and socioeconomic characteristics such as age, gender, minority status. The committee raised these issues in its first interim report (IOM, 1997); it reemphasizes the problems that could arise as a result of sample selection, size, and allocation if adequate advance planning and testing are not undertaken. SSA should produce a clear explanation of how and why it came up with the sample size targets, particularly for the four study groups. Such an explanation should include: (a) an explicit statement of SSA's plans for the analysis of DES data (with priorities indicated regarding information goals and population subgroups for generating domain estimates); (b) statistical precision and or power requirements for these analyses (particularly those of highest priority); and (c) a clear statement of how they arrived at their target sample sizes based on (a) and (b). Further SSA should require Westat to devote sufficient time after the pilot and based on the pilot data to conduct power calculations to assess in more detail whether the main survey is adequate in size. Based on the results, Westat should re-evaluate and revisit the numbers it originally set as targets, and if necessary, to assess Westat's ability to meet any new targets. MEDICAL EXAMINATION SCHEDULE The scheduling of medical examinations in the Mobile Examination Centers (MEC) is another example of logistical inflexibility resulting from the tight timeframe for the survey. The timing of the exams is so tight that there would be little flexibility if several respondents required more time to complete the examination. In addition, Westat is using the National Health and Nutrition Examination Survey (NHANES) model for the MEC part of the study, but it is not clear from the documents the committee reviewed if any tests were undertaken to assess whether the NHANES model will work for people with disabilities. As noted in the previous chapter, Westat plans to expend significant effort in testing and experimenting with different options related to the efficiency of processes and methods before and during the pilot study. The committee commends Westat's plans but does not think Westat can appropriately conduct these needed tests, analyze the results, test alternatives, and make the necessary modifications within the current timeframe.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report PROXY RESPONDENTS Westat has addressed many of the concerns about the use of proxies in surveys and has identified useful mechanisms and procedures for interviewing and examining the survey participants; in addition, it plans to use proxies only when unavoidable for the comprehensive interview and the follow-up interview. The committee continues to be concerned about the effect of the use of proxies on the validity of responses. Westat recognizes that the use of proxies is a critical issue for any household survey. It has heightened importance for the DES because the target populations are likely to have difficulty responding due to their impairments, resulting in the potential for higher levels of proxy responses and proxy involvement throughout the administration of the survey (Westat, Inc. 1999c). In addition, Westat must be able to deal with the divergence of responses from proxies and from the sample persons on questions that may lead to misclassification of participants in one of the four categories for the subsequent phases of the survey. For example, proxy responses in the initial screener interview may lead Westat to classify the person as severely impaired, whereas the respondent may be found to be mildly or moderately impaired during the follow-up screener and medical examination. To reduce respondent burden during the initial screener, a "household reporter" will respond for all working-age adults in the household. Westat is concerned about the validity of data, especially those collected from a household reporter, and particularly about the ability of this reporter to respond to questions about the mental and cognitive health of the other individuals living in the household. It intends to experiment with alternative approaches in the pilot study. The experiments identified include (a) attempting to conduct mental and cognitive interviews with all working-age adults in the household; (b) collecting information from the reporter and then interview each individual; and (c) using the interviews for mental and cognitive ability (e.g., MMSE and CIDI) with the household reporters, asking them to answer these questions for the other adults in the household, and then selecting those who test positive on the interviews, based on the reporters' responses for the follow-up screener and the comprehensive interview. Adequate time is required to conduct these experiments, analyze the data and make changes based on the findings. Moreover the training needed for the CIDI-SF and the SCID is time consuming, as is the documentation and evaluation of quality control. Sufficient time will be needed after the pilot study to compare the two systems and to develop and monitor the assessment. Proxies have been shown to be valid reporters for much of the information that is sought in the initial screening instrument (e.g., Epstein et al, 1989; Bassett, et al., 1990; Long, et al., 1998; Sneeuw et al, 1997, 1998). The screener has been designed so that the various mental and cognitive exams will only be administered to the respondent because they cannot be accurately or reliably completed by proxies. However, in addition to difficulties of detecting mental disability through proxies, research also has shown differences in proxy versus respondent reports in other areas, including a tendency for proxies to overreport or underreport the severity of disability (Epstein, et al, 1989; Hays, et. al., 1995; Grootendorst, et al., 1997; Magaziner et. al, 1997). QUESTIONNAIRE AND PROCEDURES ISSUES Questionnaire Development Much of the work on questionnaire development and procedures for data collection could be tested with small samples using different approaches, including the laboratory research. In

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report creasingly, survey operations, especially for large-scale surveys, are being preceded by small-scale, carefully structured, field testing coupled with laboratory research, so that definitions, questionnaire wording and the order effects of the questions, response burden on the respondents, and concepts critical to the results of the survey can be better understood. Questionnaire pretesting can involve a number of different methods. One qualitative method often used in questionnaire pretesting is the cognitive interview in which a respondent is interviewed one-on-one by a trained questionnaire design specialist. The respondent is asked to think aloud as he develops his answers. This technique can be useful in detecting problems with questionnaire items such as retrieval difficulties (Sudman, et al., 1996). Another approach is to ask more directed probes after the respondent has answered the question. These probes can help assess the respondent's comprehension of some term or concept (Forsyth and Lessler, 1991). These techniques also can be helpful in developing successful probes to be used with open-ended questions. While understanding how respondents arrive at their answers will be important, the answers themselves will also provide invaluable information in the development of response options for some questions. The responses given by cognitive interview participants should provide some idea of what responses to expect for these questions. The committee is pleased to note that Westat's plans include field testing and cognitive testing, followed later by a larger pretest and then a large pilot test. Westat's detailed plans for testing and experimentation also should include the applicability of the NHANES mode of medical examination for people with severe disabilities and functional assessment instruments. Moreover, in the course of the workshop titled "Measuring Functional Capacity and Work Requirements" held in 1998, participants generally agreed that there is no one instrument available to assess the functional capacity to work that could be incorporated into the DES or into the disability decision process. Many participants were of the opinion that more research and experimentation are needed in this area (IOM, 1999). The committee notes that Westat has incorporated many measures and procedures to include people with mental or cognitive impairments. However, the committee is especially concerned about the applicability of the questions and procedures to this group of people. For instance, some people with chronic mental illness and traumatic brain injury are able to get jobs but cannot retain the jobs for more than a few days for various reasons related to their condition. These individuals are likely to be eligible or borderline eligible for disability benefits under SSDI and or SSI. The work status questions in the various sections of the DES generally do not address this situation; they need to be reviewed to ensure that these questions are appropriately worded and tested using cognitive processes and then again in the pilot study. Another example is the use of the term "work." People understand the meaning of this wordterm work differently, and people with disabilities are often encouraged to think of sheltered work and activity programs as "work." For purposes of the DES, it is important to differentiate these different situations. Although this distinction is made in many places in the current DES questionnaires, all sections of the DES questionnaires should be reviewed to ensure that every time respondents are asked about work, it would be clear whether their answer refers to regular work, sheltered work, or an activity program.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report Respondent Burden Each of the DES survey instruments is lengthy and complex, thus creating a risk that respondents will be unwilling or unable to provide useful data to SSA (see Table 3.2). For example, SSA has noted that that the Comprehensive Survey Interview will impose a burden on some respondents who have a complicated medical history, considerable income or assets, and a complex work history. The committee agrees and expects that other DES components will also impose a significant burden on those and other respondents. Another concern is the initial screener, because its results will be used to sort individuals into the four categories. For this screener, one household member will be asked to respond to numerous questions, including questions about mental and emotional problems, for all household members 18-69 years of age. If the respondent does not answer these questions correctly for all household members, individuals who have conditions that should result in their selection for the follow-up screener may be missed. Because of its length and complexity, SSA and the committee agree that the instrument will have to be pared down between the end of the pilot study and the start of the national study. SSA first must decide which questionnaire items are to be eliminated, and then the shortened version must be evaluated and field-tested to ensure its viability as an instrument that is able to meet the study's goals. These steps will take several weeks or months to be done well. The committee believes that the current DES contract does not allow sufficient time for these steps. SUMMARY The pilot design generally looks reasonable, but the committee seriously doubts that enough time is allotted to determine what changes are needed to implement those changes before the conduct of the national survey. The type of research and experimentation described in the previous sections that is planned to be conducted before and during the pilot study would facilitate final resolution of the various issues identified, and appropriate changes could then be made in the design and procedures before the start of the main survey. In order to resolve the problem areas in a satisfactory manner more time will be needed especially between the completion of the pilot study and the start of the national study. The current plan allows 2-3 months to assess the findings of the pilot study prior to the start of the main study. Given the potential for adverse outcomes in the current design of the pilot study, this amount of time for reflection and response is likely to be wholly inadequate. Recommendation: The committee strongly recommends that SSA revise the project schedule to allow significantly more time to plan and analyze the pilot study and test alternative solutions for problem areas before starting the national study. While the committee is unable to prescribe specific time needs, a number of plausible scenarios may provide some guidance to SSA on the amount of additional time that may be required for the DES timeline. One possibility is an inability to meet response rate targets set for the screening, interview, and examination phases of data gathering under each of the data collection options and incentive levels proposed by Westat. This outcome could easily require 3-4 additional months to develop and field test alternatives to achieve these targets. In another scenario,

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report development of the initial and follow-up screeners may require one or more additional iterations to better achieve sample size targets for disability status groups. These iterations could add several months to the completion of the pilot study phase of the survey. Finally, planned quantitative assessments of examination instruments may reveal unacceptably low levels of reliability and validity. This situation would demand 4-5 additional months for another round of instrument redesign and field testing before SSA could comfortably implement this phase of data gathering in the national study. Simultaneous occurrence of all of these scenarios could force a delay of a year or more, during which time the entire design of the DES is revisited and new survey strategies are devised and tested. The committee recognizes that increasing the time and level of research between the pilot study and the national survey may have cost implications.

OCR for page 12
Review of Disability Evaluation Study Design: Third Interim Report TABLE 3-2. Components of Survey Instrumentation for the Disability Evaluation Study Initial Screener: To be conducted primarily through telephone interviews with one household member who will be asked to answer questions for all household members age 18-69. • Household enumeration • Functional limitations • Health status • ADLs and IADLs • Current recipients of Social Security disability benefits • Work status Follow-Up Screener: To be conducted in person with each subject selected into the sample. • Household enumeration • Functional limitation scales • ADLs and IADLs • Mental status examination • Composite International Diagnostic Interview 12-month short screening scales • Performance measures (actual tasks to be completed by the subject) • Perceived disability Comprehensive Survey Interview: To be conducted in person with each subject in the sample. • Health behaviors • Medical history • Access to health, educational, and vocational services • Social and community living • Work status, history, and environment • Program knowledge • Economic resources Medical Examination: To be conducted in person with each subject in the sample. • Core medical examination • Cardiovascular system module • Endocrine system module • Gastrointestinal system module • Genitourinary system module • Hemic and lymphatic system module • Immune system module • Musculoskeletal system module • Neoplastic disease module • Neurological disease module • Respiratory system module • Dermatology system module • Performance measures, including special senses, speech, and actual tasks to be completed by the subject. Procedures for Collecting the Medical Evidence of Record Disability Decision Process NOTE: ADL = activities of daily living; IADL = instrumental activities of daily living. SOURCE: Westat, 1999a.