Review of the Disability Evaluation Study Design: Third Interim Report (1999)

This chapter describes briefly the general features of the Disability Evaluation Study (DES) as planned by the Social Security Administration (SSA) and identifies the key survey design and sampling plan, data collection plans, and operational decisions made to date regarding the national survey and its pilot study¹.

The DES is a complex, national sample survey to estimate the number and characteristics of a broad range of people with disabilities that affect their ability to work and carry out activities of daily living. In addition to personal interviews, medical and physical examinations will be conducted on a nationally representative sample of the noninstitutional population 18-69 years of age.

The primary objectives of the DES are to:

1. Estimate the total number and characteristics of people who are severely enough impaired such that, but for work or other reasons ², would meet SSA's statutory definition of disability. (This group would represent the universe of potentially eligible nonbeneficiaries who could apply and meet the current criteria, but who are not now receiving benefits.)

2. Identify the factors that enable persons with disabilities who could qualify for benefits to remain in the workforce.

3. Identify the variables needed to monitor and assess, in a cost-effective manner, future changes in the prevalence of disability.

4.  Identify the number and types of people who could be affected by a change in the disability decision process.

The sample design for the DES is driven by the following four core objectives (Westat, 1999b, p. 5): The design should:

1. yield a sample of the various subgroups of working-age people with severe enough disabilities to be likely to be eligible for disability benefits for SSA purposes if they applied;

2. yield a sample of the ''borderline" group of people with disabilities sufficient to permit estimates of the number and characteristics of those who might become eligible, or cease to be eligible, if the current SSA disability decision criteria are altered;

3. yield a sample of people with only mild or no disabilities sufficient to permit comparisons with the population with disabilities on measures of physical and functional performance and medical conditions in the population; and

4.  yield a sample of people receiving disability benefits under SSDI and or SSI.

The sample for the DES is a dual-frame, multistage, stratified probability sample design. The first stage will be a stratified sample of primary sampling units (PSU) selected with probability proportional to size. Within the PSUs, households with persons 18-69 years of age will be subsampled at rates designed to yield a nationally representative sample. Households with telephones will be selected for telephone interviews by list-assisted random digit dialing (RDD) sampling. Households without telephones will be selected for in-person interviews using standard area probability sampling methods (i.e., segments within PSUs, listing of dwelling units, sampling of households). In addition, a small subsample of households with telephones in the area sample will be selected for in-person administration of the initial screening interview. Group quarters will be chosen mainly from lists for the sample PSUs and will be sampled for in-person interviews.

The first-stage sample will consist of 80 PSUs³ selected from more than 3,000 counties and independent cities in the United States using probability proportional to size (PPS) sampling. To limit the travel distance between the respondents' homes and the mobile examination centers (MEC), Westat has decided to use individual counties as PSUs for this study instead of the typical metropolitan statistical area or groups of contiguous nonmetropolitan counties.

The PSUs selected for the survey will be used for both the area sample and the RDD sample. Westat plans to construct the area sampling frame from the Census Bureau P.L. 94 data tape⁴ that contains housing unit counts and geographic information for every block in the United States. Blocks and block groups are uniquely associated with counties in this data tape. In the telephone sampling frame, telephone numbers within the same area code and exchange can sometimes straddle two or more counties. To solve this problem Westat plans to adopt an "unbiased" rule in which an exchange is associated with the county only if a plurality, or a specified percentage, of households in the exchange is contained within the county. Under this approach each exchange will be associated with only one county. However, the definition of the county for sampling households in the telephone frame will not match the exact geographical definition of the county. Westat expects little loss in efficiency in employing different PSU definitions for the area and RDD samples.

In sampling units of unequal population sizes (counties, in this study), it is standard practice to sample them with PPS or some other measure of size. In the DES, selection of the measure of size is driven by the need to equalize the workloads at the MECs. Using an estimate of the number of persons aged 18-69 for a county as its measure of size would lead to an equal-probability sample with approximately equal sample sizes per county for the screening interview. Westat plans to use, for that purpose, the 1997 county-level population estimates⁵ available from the U.S. Census Bureau. To equalize the workloads at the MEC, these estimates will be adjusted by county-level disability statistics.

The SSA has set a target to identify and complete all data collection for a total sample of 5,665 persons made up of the following four study groups or strata:

1. a "core" group of nonbeneficiaries with severe disabilities (the likely eligible group);

2. people with significant but lesser impairments (the "borderline" cases);

3. people with only mild or no disabilities; and

4.  current SSDI and or SSI disability beneficiaries, who will be included primarily for the purpose of benchmarking the distinctive characteristics of the core group.

Westat assumes an initial sample size of 98,095 persons, about 89 percent of whom will be sampled using the RDD component. The initial sample sizes for the RDD component of the survey are:

• 80 PSUs;

• 110,238 telephone numbers have residential/nonresidential status determined;

• 54,016 telephone numbers determined to be residential;

• 92,636 persons 18-69 years of age sampled in identified households; and 86,955 eligible persons for whom the initial screening interview is completed.

The initial sample sizes for the area frame component can be summarized as follows:

• 80 PSUs;

• 2,000 total area segments, 25 per PSU;

• 64,124 total occupied dwelling units canvassed, 25 per segment;

• 4,168 nontelephone households canvassed, all sampled;

• 59,956 telephone households canvassed, 3,504 sampled;

• 12,378 total persons 18-69 years old sampled, 6,189 each in nontelephone and telephone households; and

• 11,444 persons for whom in-person screening interview is completed.

As stated previously, noninstitutionalized persons residing in group quarters are included in the DES. After the selection of PSUs, Westat plans to develop PSU-wide lists of group quarters. Group quarters will be sampled from these lists within each PSU, with sampling rates designed so that people living in group quarters will have the same selection probabilities as people living in households. Group quarters for which lists are not available and those that are missed during list construction will be identified during the area sample listing process.

To ensure geographical spread and demographic diversity of the sampled PSUs, the PSU frame will be stratified by geographic region, metropolitan status, income level, minority status, prevalence of work disability using 1990 census data, and population size. In addition, because research on work disabilities has shown marked differences by geographic region, Westat is considering stratifying by prevalence of work disability to ensure appropriate representation of areas with very high and very low levels of disability.

Once the sampling strata have been specified, the largest counties will be included in the sample with certainty. Westat expects five certainty PSUs. The remaining counties then will be assigned to strata of approximately equal size. Counties too small to be sampled as individual units will be grouped to meet minimum measure-of-size requirements. Finally, two PSUs will be selected from each stratum with probability proportional to size.

The second-stage sampling unit for the RDD sample is the telephone number. The sample of telephone numbers will be selected from the exchanges linked to sampled counties. The sam

ple will be unclustered within the PSUs; the number of households with telephones to be screened per PSU will be roughly equal across the noncertainty PSUs.

As indicated earlier in this chapter, the DES sample design includes an area sample of households without telephones to complement the RDD sample. To obtain a self-weighting sample of households of sufficient size for the initial screening process, about 64,124 dwelling units in the area sample will be listed and canvassed to include 4,168 nontelephone households in the sample. In the area sample, each PSU will be divided into sampling units or segments, which are defined to be census blocks. Each segment will include 25 dwelling units. Within each PSU, a sample of segments will be selected with probability proportional to the number of dwelling units. A sample of households will be sampled from within each selected segment. Assuming that nontelephone households are sampled at the same rate as telephone households, a total sample of 64,124 households will include about 4,168 nontelephone and 59,956 telephone households. Group quarters will be sampled as part of the area frame. Group quarters will be selected using probability proportional to size. The measure of size will be the best estimate of the number of residents 18-69 years of age living in the group quarter.

The main purpose of the DES screening process is to collect information needed for sampling purposes and to classify people aged 18-69 into the four target study groups noted previously. A two-stage screening process will be used. The purpose of the initial screener is to collect information needed for sampling purposes as well as information needed to classify the estimated 98,095 screener respondents into the four study groups. Following the initial screener, a subsample of respondents will be selected for the second stage screening, which will be an in-person interview. The initial screener will be mostly a telephone interview, followed by a face-to-face interview. The initial screener will include 57,712 households, and will result in interviews with about 98,095 adults 18-69 years old. It will involve three types of interviews. The majority of people will be interviewed by telephone, however a face-to-face interview using the same screening questions will be conducted with two groups. One group will include a subsample of approximately 3,609 telephone households, the other will be a subsample of approximately 4,168 nontelephone households. Westat assumes a 10 percent attrition rate for further study participation.

Of the total number of people participating in the initial screener, a sample of about 11,444 will be selected to participate in the follow-up personal interviews. The purpose of this follow-up screener is to confirm the preliminary classification of each person into one of the four categories and to more precisely classify people into severe and borderline-impaired study groups. It will also confirm suspected mental illness and cognitive deficit. Westat expects approximately 10,300 persons to complete the follow-up interview. All of these interviews will be done in person. Upon completion of the follow-up screening, sampled persons will be assigned once again to one of the four study groups, with an expected breakdown of 515 nonimpaired nonbeneficiaries; 1,545 borderline-impaired nonbeneficiaries: 3,090 severely impaired nonbeneficiaries; and 515 beneficiaries. They will then be subsampled to obtain the target sample sizes for each of the study groups.

The total target sample size is 5,665 persons, and the target sample sizes for the four study groups completing the medical examination are as follows:

1. nonbeneficiaries likely to be severely disabled (3,090);

2. nonbeneficiaries who are less disabled (1,545);

3. nonimpaired or slightly impaired nonbeneficiaries (515); and

4.  persons receiving disability benefits (515).

This sample of 5,665 will then receive a medical examination.

Westat's assumptions about the sample size that needs to be screened to obtain the required 5,665 persons distributed disproportionately in the four strata for the various components are based on achieving the following response rates:

• 90 percent for the initial screening interview;

• 90 percent for the subsequent in-person interview and medical examination; and

• an overall response rate of 80 percent for the combined interview and medical examination components.

Assuming that these high response rates can be achieved, Westat estimates that a sample of about 98,095 persons in about 57,712 households will be sufficient to yield 5,665 persons for the DES study group.

The issue of the use of proxies arises in this survey because a large number of people in the sample will have disabilities or some kind of functional limitation. Westat plans to avoid proxies whenever possible. However, it may be necessary to collect information from proxies to ensure the highest possible response rate and to obtain as much information as possible from people who have difficulty responding on their own.

Westat's current plans call for a household reporter to answer questions in the initial screener about all working-age adults in the household. Westat is concerned, however, that such reporters may not be able to answer accurately and honestly questions about the mental and cognitive health of other members of the household. Westat is also concerned about the risk of very low response rates if it attempts to interview each person in the household about his or her mental and cognitive health. During the follow-up screener and the comprehensive survey interview, Westat plans to use medical exam proxy assistants in interpreting for and assisting the sample person with medical needs or language problems.

Proxy interviews have varying levels of accuracy depending on the topic of the interview and the relationship of the subject to the proxy. Westat believes that the use of proxies in the initial screening process will make it over-sensitive, for purposes of the initial screener, however, that would be acceptable. Beyond the initial screener, Westat plans to avoid using proxy reporters but does expect to have proxy-assisted interviews. The decision to use or not use a proxy respondent

will be made when the sample person is initially contacted. If the respondent is available and able to complete the interview, the interviewers will be discouraged from accepting a proxy.

In response to a recommendation in the committee's first interim report, the DES now includes a large comprehensive pilot study prior to the conduct of the national study. The stated purposes of the pilot study are to experiment with several data collection methods and procedures, and to ensure that the questionnaires are clear and concise, and that all procedures run smoothly and efficiently, with the burden and discomfort placed on the respondent kept to a minimum. The pilot study will also test the effectiveness of the screening instruments and the accuracy of the screening algorithm; determine the best procedures for maximizing the response rates, both total and item; and develop estimates of prevalence rates to determine the final sample sizes for the main study. Finally, the pilot study will test the operational procedures for medical examinations, including the reliability of physician and nurse practitioner examinations; medical examinations performed in the home and in mobile MECs; and the logistics, reliability, and validity of the simulated disability decision process.

Current plans call for the pilot study to be conducted in eight PSUs, with a total sample of 1,000 persons in the final data collection step. A sample of approximately 13,202 households will be contacted in the initial screener. Of the 13,202 households, about 11,882 households, or about 20,316 persons, are expected to complete the initial screener. A single respondent per household will be administered the screener for all persons in the household 18-69 years of age. The screener will result in the assignment of respondents into the categories of interest for the DES. From this group, a sample of 2,000 persons will be selected for follow-up. Assuming a 10 percent attrition rate, Westat expects 1,800 persons to complete the follow-up screener and the comprehensive survey instrument. From these 1,800 persons, 1,000 will be selected for the medical examinations. Included in this 1,000 will be 200 current beneficiaries on the disability rolls, about half of them will be selected from SSA files

Before the pilot study starts, Westat plans to ensure that the instruments are clear, understandable, and concise and to test all explanatory information and procedures. These developmental activities will include focus groups, cognitive laboratory tests, and pretests. These activities will continue into the pilot study to allow further improvements.

Focus Groups will be conducted to learn about four areas:

1. Four focus groups will be held to simulate the disability determination process-two will be held early in the design effort, and two will be held after half of the pilot study simulations are completed.

2. Two focus groups will be held with individuals with disabilities who are not current beneficiaries to review and discuss advance materials and materials provided after the sample person is selected.

3. Two focus groups will be held with current beneficiaries to discuss issues surrounding participation in the study.

4.  Three focus groups will be held with respondents on the pretest to discuss reactions to the interviews and examinations.

Each focus group session is expected to last two or three hours.

The objectives of the Cognitive Laboratory Tests are to reduce response errors and to ensure that each instrument serves its purpose. Westat is planning to conduct cognitive tests early in the design process with nine individuals for each test to help identify problems and issues related to item flow, item content assessment, and length of the selected sections of the instruments. Each cognitive test session is expected to last from 45 to 90 minutes. About 50 interviews are planned and they will cover various sections of the screeners and the comprehensive survey instrument. These tests will use several techniques to capture respondents' views about the questionnaire items, including postinterview debriefing, think-aloud protocols, and paraphrasing.

Westat plans to conduct Pretests before and after approval is received from the Office of Management and Budget (OMB). Westat plans to conduct three sets of nine interviews, for a total of 27 interviews, before OMB approval. One of the stated purposes of these pretests is to provide an initial examination of how well the questions work when they are used as a complete protocol. The pretests also will allow an assessment of overall flow, length of interview, and interviewer-respondent interaction.

Another purpose of the pretests will be to determine which instruments provide the desired results in the follow-up screener, particularly for confirming mental illness and cognitive deficit. As Westat correctly points out, the capacity of the screener to identify and classify people with mental illness is a persistent concern with the DES. Westat plans to test the effectiveness of two options each for confirming mental illness and cognitive deficit. The Composite International Diagnostic Interview, Short Form (CIDI-SF), which is brief and can be administered by a lay interviewer, will be compared with the Structured Clinical Interview for DSM-IV (SCID) which is a highly sensitive and specific instrument that must be administered by a psychiatrist or a psychiatric social worker, to determine if the CIDI-SF can identify the type and level of severity of psychiatric illness as well as the SCID. The Mini-Mental State Examination (MMSE) will be compared with the Mental Status Examination (MSE) to determine which instrument more accurately identifies and classifies people with cognitive impairments. Participants for these tests will be obtained from SSA rolls of people with known psychiatric illness and known cognitive impairments.

After approval for the survey is obtained from the OMB, Westat plans to conduct a pretest of 50 persons representing a cross-section of the population of interest for the DES.

Westat plans to conduct several experiments during the pilot study. Response rates will be a critical factor in determining the final sample sizes for the main study. To increase response rates, Westat plans to conduct four experiments with different data collection methods, four with refusal conversion incentive strategies, and one with medical providers. Another area of experimentation regarding improved response rate relates to current beneficiaries. Three alternative consent forms will be tested in the pilot study for this purpose.

Westat plans to test several process and methodological issues related to medical examinations. Some of these issues concern the efficiency of the MEC examinations, coordination and communication of the administrative and interviewing office and examination center, participant satisfaction, layout of the MEC, functioning of the equipment, data tracking, reporting, and documentation, efficiency of the process of referrals to outside medical providers, and efficiency of the home examination. The pilot study also offers an opportunity to test the comparability of medical examinations administered by physicians and by nurse practitioners, the comparability of examinations given by nurse practitioners at the respondents' homes and in the MECs, and the determination of when home examinations should be offered. A sample of 30 persons will be asked to be examined three times: twice in the MEC-once by a nurse practitioner, once by a physician-and then again in their homes by a nurse practitioner.

The pilot study is also designed to test instrument designs for the DES and more thoroughly test the screens and questionnaires. The tests concern the screener methods used to allocate the general population into the four study groups and will attempt to answer these questions: Does the screening process misclassify individuals? Is the screener sensitive to the full range of people with mental illness? Is the screener sensitive to people with cognitive deficits? The resulting classification analysis will address the sensitivity and specificity of the screening process. Pretests and laboratory testing of the instruments prior to the pilot study, help concentrate on issues of item wording, response options, item sensitivity, sequencing, and flow. Westat plans to fully test the questionnaires in the pilot. Westat believes that the large sample in the pilot study affords the opportunity to conduct item analyses, which could provide clues to items in the questionnaires that respondents routinely do not answer or that offer little or no variability in response. Such information could not be obtained in small pretests. Alternative formats of the content and length of the various sections of the screener and questionnaires also will be tested in the pilot study. The work status section is a case in point.

Westat's proposed plans for the DES include a comprehensive series of tests and experiments covering all aspects of the survey operations, design, response rates, and the effectiveness of the questionnaires before the start pilot and during the pilot study.