4
Data Collection Plan

Data collection for the DES involves:

  • a screening interview;
  • a personal interview and physical performance tests;
  • an extensive medical, and if needed, psychological examination; and
  • a series of core and special medical tests.

In addition, SSA plans to obtain all medical evidence of record identified by the respondent and by third party reports on all persons in the sample to supplement information from the interviews and medical examinations in order to determine if the person meets SSA's current definition of disability. SSA indicates that the medical evidence of record can provide more information about impairments and other health problems than can be obtained from a medical examination at a single point in time. Survey designs that seek the medical records of persons in the sample face the high cost of conducting record searches as well as the risks of missing data. SSA and the contractor should carefully develop and evaluate protocols for record acquisition during the early months of the DES contract and evaluate subsampling designs, possibly disproportionately sampling important target groups. Any subsampling scheme could reduce the cost of DES relative to a design that obtains medical records for all persons in the sample.

Screening Interview

As stated in the previous section, the survey design includes, as a first phase, a large-scale screening survey to identify a sample of working age persons to permit accurate identification of (a) current SSA disability beneficiaries; (b) nonbeneficiaries with severe disabilities who appear likely to meet SSA's medical/functional disability criteria; (c) "borderline" nonbeneficiaries who have some degree of impairment, and are likely to be candidates for disability benefits at some later time; (d) people who, on the basis of the screening, do not appear to be disabled. At the time of the screening interview, the interviewer is supposed to schedule an in-person main survey interview and medical examination.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 18
4 Data Collection Plan Data collection for the DES involves: a screening interview; a personal interview and physical performance tests; an extensive medical, and if needed, psychological examination; and a series of core and special medical tests. In addition, SSA plans to obtain all medical evidence of record identified by the respondent and by third party reports on all persons in the sample to supplement information from the interviews and medical examinations in order to determine if the person meets SSA's current definition of disability. SSA indicates that the medical evidence of record can provide more information about impairments and other health problems than can be obtained from a medical examination at a single point in time. Survey designs that seek the medical records of persons in the sample face the high cost of conducting record searches as well as the risks of missing data. SSA and the contractor should carefully develop and evaluate protocols for record acquisition during the early months of the DES contract and evaluate subsampling designs, possibly disproportionately sampling important target groups. Any subsampling scheme could reduce the cost of DES relative to a design that obtains medical records for all persons in the sample. Screening Interview As stated in the previous section, the survey design includes, as a first phase, a large-scale screening survey to identify a sample of working age persons to permit accurate identification of (a) current SSA disability beneficiaries; (b) nonbeneficiaries with severe disabilities who appear likely to meet SSA's medical/functional disability criteria; (c) "borderline" nonbeneficiaries who have some degree of impairment, and are likely to be candidates for disability benefits at some later time; (d) people who, on the basis of the screening, do not appear to be disabled. At the time of the screening interview, the interviewer is supposed to schedule an in-person main survey interview and medical examination.

OCR for page 18
The first issue in choosing sampling frames concerns the target population. In general, most national household population surveys routinely exclude people who are institutionalized and those homeless people who cannot be found in households or other living quarters visited during household surveys. SSA has decided to exclude from the DES the institutionalized population and the segment of the homeless population who cannot be found in households or other quarters at the time of the interview. However, the question of including or excluding homeless people from the DES is not as straightforward as for the other household surveys. The committee recognizes the likelihood of relatively high rates of disability among homeless and institutionalized populations, and the resulting negative bias resulting from their exclusion. At the same time it has serious questions about the operational and methods issues. Can reliable information be obtained, feasibly and economically, from homeless and institutionalized populations? Techniques have been developed to locate, sample, and obtain data about each of these populations. Yet locating and screening respondents for eligibility require special efforts involving careful, and long-term planning, large amount of staff resources, considerable time, and high levels of funding. Homeless people present problems in scheduling, interviewing, and administering performance tests and medical examinations. Maintaining contact with them and getting them to participate in adequate numbers in the medical examination is also problematic. Likewise, obtaining permission from family members for the participation of people in long-term care institutions who are not able to grant permission themselves may be difficult. The committee discussed these various issues and also reviewed a draft discussion paper prepared by Westat for the SSA (Westat, Inc., 1995a). The committee concurs with SSA that adding homeless and institutionalized populations to the sampling frame at this time would not be cost effective. Much research and testing are required to develop the necessary protocols and procedures for conducting the DES among homeless people and those living in different types of institutions. The costs of sampling and interviewing in the various types of institutions would be prohibitive. Thus, limiting the target population to the household population seems appropriate. The committee, however, urges SSA to undertake research as part of its long-term research plan leading to the inclusion of these populations in subsequent studies or a separate supplement to the DES in the near future. The current proposal specifies the use of telephone number frames for DES. This decision by SSA appears to be primarily driven by cost considerations. The choice of sampling frame determines the nature of noncoverage error in any survey. Common choices in surveys in the United States are area frames, offering theoretically complete coverage of households and institutions; dual frame designs, combining telephone and area frames; dual frame designs combining area and institutional list frames; and telephone number frames. If SSA uses a telephone frame as the screening tool, the DES will miss households without telephones. Approximately 5 percent of households in the United States are without telephones. Persons in households without telephones have a higher rate of disability than those in households with telephones. This rate is 17 percent for those without telephones compared with 15 percent among persons with telephones (Thornberry and Massey, 1988; LaPlante and Carlson, 1996). The availability of telephones also is negatively correlated with income. This noncoverage of persons in households with no telephones should be of particular concern with regard to the representativeness of the sample of persons with disabilities.

OCR for page 18
The nontelephone population can be covered through a dual frame approach, whereby nontelephone households are obtained through an area frame and telephone households through both an area frame and a telephone number frame. (Such a design is currently being used in the New Federalism Survey conducted by the Urban Institute and Westat.) Alternatively, a full area probability design can be implemented to provide such coverage. SSA's current decision omits the nontelephone population from coverage in the DES.7 Because DES will be the principal instrument by which the United States will obtain prevalence data on disability as defined by law, the committee concludes that a national survey of disability that does not offer statistical estimates of the nontelephone population is statistically inadequate. Further, telephone sampling and screening is likely to offer lower response rates than face-to-face screening (Groves, 89; Lessler and Kalsbeek, 1992). As a consequence the screening sample will need to be increased to compensate for the losses from the sample because of nonresponse; the higher nonresponse rates are likely to increase the risk of bias in the estimates. Thus, although telephone screening may be less expensive, some aspects of the quality of the data collected are more suspect. Careful study of mechanisms to increase the screener response rate is required. These mechanisms might include incentives, refusal conversion efforts, switches to alternative modes of data collection, and so on. In addition, there is no indication about how SSA will deal with people with hearing loss, communication disorders, mental and cognitive impairments, and emotional disturbances. SSA also has the problem of response burden for the total household if more than one person in the household has a disability and proxy reporting is not encouraged. Similar problems will have to be faced in the main interview and in administering medical examinations and performance tests to persons with severe disabilities. The effect on response rates and bias could be significant. SSA should test several options dealing with these problems in pretests prior to the start of the national survey. It is clear that in terms of coverage of the adult working-age population, survey response rates, and some features of the screening measurement, the preferred design is an area probability, face-to-face survey. It is also clear that the cost of such a design is higher than the alternative proposed by SSA. The additional costs for a survey of this importance and complexity should be considered in the context of the size of the program itself (SSDI and SSI)—which, if current trends continue, could cost $100 billion annually—and the implications of poor or imprecise information. The committee, therefore, urges a careful review of the costs of a full area probability survey, in light of the cost savings proposed in later recommendations. 7   SSA is considering the possibility of supplementing the telephone screening with face-to-face interviews on a small scale for the purpose of supporting some estimates of the bias introduced. No decision about that has been made.

OCR for page 18
RECOMMENDATION 4-5. The committee recommends that the Disability Evaluation Study be based on a design offering full coverage of the U.S. household population of adults. If resources are lacking to mount an area probability sample using face-to-face interviews, the Social Security Administration should use a multiple-frame design of a statistically optimum mix of the general population followed by face-to-face interviews of the eligible population. Moreover, the cost effectiveness of various mixes of telephone and area probability frames for the screening phase should be investigated as part of the research, development, and testing phase of the study. For example, one possible dual frame design would use a frame of listed telephone numbers, and cluster the listed telephone sample geographically (to reduce the cost of face-to-face interviews). This design would supplement the telephone frame sample geographically with a clustered area probability sample to cover the unlisted telephone and the nontelephone households, as well as the listed telephone households. Thus, listed telephone households enter the sample in two ways; others enter only through the area frame sample. Appropriate statistical adjustments exist for combining the samples to obtain national estimates (Hartley, 1962; Lund, 1968; Lepkowski and Groves, 1986). Medical Examination A general medical examination has been designated as a key component of the DES. According to the draft scope of work, the results of a general medical examination, combined with relevant "medical evidence of record" from health care providers, will be obtained for DES respondents. These two data sources will constitute the body of "medical, functional, and other pertinent data" from which "to make the most accurate possible prediction of a disability eligibility decision based on current program criteria'' (SSA, 1996b, p.13). The committee assumes that the main purpose of the examination is to provide clinical input to establish a respondent's level of disability when determining program eligibility. The examination, however, may also serve as a validation standard for the data in the medical evidence of record collected from health care providers. SSA has considered two options for such an examination: (1) employ local physicians, and (2) use a traveling team consisting of a physician, nurse, and mental health social worker with mobile examination centers of the type used in NHANES. SSA has determined that the mobile examination centers approach is the best way to obtain acceptable response rates and acceptable levels of quality and standardization. SSA further states that locating the mobile units in relatively close proximity to those people with severe impairments who will be examined, may yield an adequate response rate. The amount of detail associated with the planned medical examination phase of the DES is not fully apparent from the draft scope of work, and the examination cannot be fully developed until procedures for the disability assessment in the DES are set. If the DES medical examination is to include an in-depth medical interview and performance test of all body systems, as well as a set of relevant laboratory and other diagnostic tests, the proposed examination would need to be administered by trained medical staff. This model for data collection would be similar to that of

OCR for page 18
NHANES. That survey uses traveling medical teams consisting of a group of physicians, nurses, and medical technicians who gather examination data in ''stands" at mobile centers that move among sampled areas of the country (NCHS, 1994). If, on the other hand, the extent of medical information needed from the examination phase is far less (e.g., limited to administering simple clinical and performance tests), the NHANES model may not be cost effective. Current SSA thinking, therefore, leaves unresolved some of the key issues tied to this important and potentially very costly portion (in terms of dollars, time, and statistical precision) of DES. Three fundamental questions arise in reaching closure on design aspects of the examination phase of DES: (1) what tests and diagnostic procedures will be included in the examination, with particular attention to the relationship between a person's claims of functional limitation and functional performance testing?, (2) what types of health care professionals will conduct the examinations?, and (3) in what setting will the examination take place? Because the answers to these questions depend on examination content (which is yet to be resolved and is also a later agenda item for the committee), the committee defers comment on the first question altogether, sets out some options for addressing the other two questions, and outlines a general strategy for reaching a final resolution for all three questions. One aspect of the examiner issue is the mixture of professional skills required by the team as a whole. The mix of health care professionals needed to staff the examination teams would depend on the content of the examination. For instance, the more medically specialized the tests and procedures, the higher the level of professional training and experience required for the examination team. If, on the other hand, the examinations were to include a variety of procedures with varying ranges of professional competencies needed for their successful administration, the team might consist of persons with varying specialties (e.g., medical doctor, nurse, laboratory technician, and others). An aspect related to the examiner issue is whether the examinations will be conducted by a few traveling teams or by separate teams of local health professionals for each sample area. The current DES plan adopts the NHANES model of traveling teams with mobile examination centers, the main argument being the potential for achieving improved data quality and examination response rates by having a group of professionals specifically trained for the DES examination. The SSA prefers the traveling team option over having local health professionals administer examinations at their own facilities. If there is a need for less comprehensive and specialized examination data than is required for NHANES, however, the plausibility of using local professionals for DES may be greater, though clearly the content of the examination must be determined before such decisions can be justified. Several options have been raised regarding the setting for the examination. However, before making a final decision, each option needs to be carefully considered. True to the NHANES model currently adopted by SSA for the DES, the examination would be conducted in a mobile examination center which, like NHANES, would be a modified semitrailers or a recreational vehicle equipped to provide a suitable venue and facilities for all procedures and tests. At each sample location, examinations of area residents participating in the DES would be completed before the mobile center is moved to the next location. An alternative to this option (tied to the use of local professionals) would have the examinations conducted at the locations where the professionals practice, thus decentralizing the examination process around the sample area. This alternative, with the accompanying need to

OCR for page 18
schedule visits by respondents at several locations to complete the examination, was rejected by SSA because of its presumed greater response and measurement problems. A third option to consider is to have either a traveling or locally trained team set up an examination center in space that is rented or leased specifically for the DES. The advantage of this option is that like the mobile centers, the examination becomes a locally centralized process, thus affording greater quality monitoring and simplifying examination scheduling for study participants. Because of the current state of uncertainty concerning the approach to determining the disability status of study participants, the committee strongly urges SSA to undertake a series of steps before final resolution of relevant design issues tied to the medical examination. Specific components of the examination must be identified and details of their content developed based on the information needs tied to the medical and functional criteria needed to establish eligibility for disability benefits under (a) the existing decision process and (b) a redesigned process. Examination components may be limited and their contents general if medical information needs for determining disability status are limited, conversely, the number of components may be larger and their content broadly detailed if the medical information needs are great. Once the examination content is set, the next step is to identify the appropriate knowledge, background, and skills needed by examiners to collect the required information. If performance tests and other medical aspects of the examination are relatively simple, nurses, physical therapists, or both (rather than physicians) could constitute the examination team. Where there is some uncertainty as to team composition, alternative team compositions might be set for later testing. Teams with appropriate knowledge, background and multidisciplinary skills including but not limited to occupational therapists, psychologists, social workers, and rehabilitation workers could be considered. Lastly, field procedures should be developed that are sufficient for formal field testing, laboratory testing, or both of the various data collection options. Illustrated below is an example of one possible menu of options for the DES examination: Option Team Composition Team Origin Examination setting A Physician + nurse + technician + psychiatric social worker Traveling Mobile center B Physician + nurse + technician + psychiatric social worker Local Team member's office C Physician + nurse + technician + psychiatric social worker Traveling Leased/rented space D Nurse + physical therapist + psychiatric social worker Traveling Leased/rented space

OCR for page 18
RECOMMENDATION 4-6. The committee recommends that once the options for using different combinations of team composition and origin, examination setting, and other dimensions are sufficiently set for assessments, a formal field experiment should be performed during the research, development, and testing phase of the survey to determine the validity and reproducibility of these options as well as the most cost- effective approach to meeting the objectives of the survey. This methods-testing activity should include a comparison of both the relative operational efficiency by measuring productivity and unit costs (including the costs and time involved in building and equipping these units, moving them, renting space for them, bringing in electricity and sewer at each location, dismantling and putting them back together, and calibrating the equipment at each site), as well as the statistical merit of all options (e.g., measures like refusal rates, inter-rarer reliability, response validity, the cost impact on the number of sample PSUs, the impact on response rates resulting from the short period of time the centers can remain in each PSU). Also, studies are needed to address measurement issues, such as assessing functional status, quality of medical evidence of record, quality and validity of examinations by the various teams, and other similar issues. These tests call for repeated measurements and will need to be iterative to be able to test the validity and reproducibility of measurements. Taken as a whole, these validity studies as well as operational and statistical comparison measures would provide a more sound basis for setting the final approach to completing this important part of DES. Miscellaneous Data Quality and Procedural Issues The committee notes the following important procedural issues that have been considered by SSA for the DES but not fully resolved: (a) the use of incentives to encourage study participation, (b) offering a home examination to those who are unable to travel to an examination site, and (c) assessing the quality of medical evidence of record data to support a person's claim of functional limitation. Various potentially significant sources of error in DES estimates are affected by these procedures. For example, nonresponse rates and bias are likely to be reduced by using monetary incentives (Lessler and Kalsbeek, 1992, p. 168) and offering a home examination. However, measurement error may be adversely affected by offering the home examination or relying on provider data for the medical evidence of record (Cohen and Carlson, 1994). Although SSA appears to have given considerable thought to each of these procedural matters, none was sufficiently resolved in the draft scope of work to provide a useful response to the RFP. SSA asks the bidders to the RFP for a discussion of options for dealing with these issues and a statement of the rationale for the choice that is put forth, but it provides no solid basis for the contractor to come to a final resolution. Without explicit directives to the bidders concerning the means and method, the best possible proposals will not be forthcoming. To ensure a degree of rigor to the resolution of these three issues,

OCR for page 18
RECOMMENDATION 4-7. The committee recommends that the Social Security Administration require in the scope of work a rigorously designed experiment in the field testing and development phase of the survey to identify mechanisms for enhancing participation in the Disability Evaluation Study, to guide decisions on the use of a home examination for those unable to travel to an examination site, to establish the validity of the measures obtained, and to assess the quality of the medical evidence of record. SSA should explicitly spell out the methods or procedures it wishes to have considered. Decisions on these design questions then can be applied to the plan for the full national survey to be conducted after the development and testing phase is completed. The committee urges SSA to incorporate testing for the three sample and data quality issues in the above recommendation and the examination issues in Recommendation 4-6 into the same field test. Recommendation 4-7 implies the need for the field tests to be of sufficient size to allow testing along several dimensions of study. An illustrative example of options that might be considered for each of the incentive and home examination quality issues listed above follows: Option Home Examination Offered? Incentive Offered? 1 No No 2 No Yes 3 Yes No 4 Yes Yes Furthermore, suppose that an assessment of the quality of the medical evidence of record data were to be done through an adjudication process involving medical information from the medical examination, the respondent's medical history, and all health care providers visited. The DES examination options (A to D on page 23) combined with the options for incentives and home examination would imply the need for 16 (4 x 4) cells in a full factorial design. Each cell requiring say, 50 households would imply a sample of 800 (16 x 50) for the field test. The committee recognizes that the rigorous field testing it proposes will significantly add to the cost of the research, testing, and development phase of DES, however, it is convinced that the benefit to the planning of the critical national survey will be substantial and ultimately may even reduce total costs by using the most cost-efficient and valid methods in the large national survey. Without this testing, the unique context of the DES leaves SSA with few solid sources in the methods literature for sound decisions on these and other important survey design questions. The committee assumes that SSA has plans for advance arrangements and public awareness campaigns in the local areas to promote support and increase response rates prior to interviewing and administering medical examinations. This type of activity is commonly done for large surveys and is often vital in increasing response rates and local goodwill. Because of the complexity and sensitive nature of this survey, SSA should pay special attention to advance arrangements. The plan of work should clearly delineate responsibilities of SSA and the contractor. Advance activities over a span of several months could include notifying

OCR for page 18
local officials, selecting field offices, sites for medical examinations, and gaining support from media and other contacts. Introductory letters and fact sheets are usually sent to local officials including local health directors and key staff, county executives, police, representatives of local advocacy organizations, and provider organizations. Meetings can be held with local officials and advocate groups to explain the purpose and procedures. Written endorsements are usually helpful in allaying fears and increasing participation. In fact, the presence of an advocate for persons with disabilities with the examination team may aid in increasing participation in medical examinations. Some examples of such campaigns prior to a survey are NHANES III conducted by NCHS and the decennial census conducted by the Bureau of the Census. Data Linkage with SSA Files and Confidentiality Issues Matching data files on individuals is of increasing value to analysts of survey data because it permits inexpensive expansion of the data record. In recent years concerns about the ethics of matching data on human subjects have become prominent in research circles (Duncan, et al., 1993). Several principles appear to be emerging from these concerns: (a) persons whose identifiable data records are matched should be informed about the matching as well as the content of the medical examination and several other items in the survey instruments; (b) the purposes of the matching must be described to the respondents; (c) permission from the respondents must be obtained for the matching; (d) nonstatistical uses of the data must not be threatened by the match; (e) and participation must be voluntary. Although such procedures are common for health surveys, in a survey such as DES there are added complications of obtaining informed consent from persons with mental and cognitive impairments who may not be competent to give their informed consent. In addition, beneficiaries and potential beneficiaries may not want to respond to certain questions or participate in certain tests unless confidentiality is guaranteed because of a perceived threat to their receipt of benefits. Assurance of confidentiality would require the contractors to strip identifiable information from the files as soon as it is no longer needed for survey purposes. However, such a procedure does not allow SSA to link the data with other SSA files. Laws that control access to administrative records, such as reports of earnings covered by Social Security, restrict their use for statistical purposes. In seeking to match individual cases in DES through a Social Security Number to the longitudinal earnings and other administrative files and to third parties (e.g., health insurance providers, physicians, employers, or pension plans), investigators can track longitudinally the work earnings of the sample. This would be a great benefit to the analytical value of the data. SSA views such a linkage as important to the success of the current DES and possible future extensions to the DES sample for measuring changes over time. The draft scope of work states that the contractor will turn over identifiable individual records to SSA, and that SSA will carry out the matching. It further requires the contractor to develop appropriate language for a waiver of confidentiality, reflecting the specific commitments to be made at the time of the interview. Although SSA recognizes these confidentiality issues, the committee believes that additional steps must be taken by SSA to obtain adequate informed consent from the participants, to ensure sufficient protection of confidentiality for study subjects, and to deal with these issues in a way that will not compromise the scientific integrity of the DES design. Current beneficiaries and the group potentially eligible for benefits particularly may have

OCR for page 18
concerns about participating. The impact of such waivers on the response rates and the resultant bias introduced by refusals also should be taken into consideration and evaluated during the research and testing phase of the survey. The presence in a regulatory environment of statistical databases linked to earnings and other administrative files is precarious. It is not clear whether individuals in the sample would have legal protections from regulatory uses of their survey data when they are matched to the longitudinal earnings file. But they deserve such protection. There are several accepted models of matching that remove from the regulatory agency direct access to the information that would allow actions to be taken against individuals. SSA is urged to review the procedures used in the Health and Retirement Survey sponsored by the National Institute on Aging and other relevant surveys to protect the respondents' confidentiality and yet permit linkage with other data files. For example, the matching could be performed by the contractor, and a file could be returned to SSA stripped of all identifiers. The contractor also could update the match over the years. Other arrangements to guarantee that matched files cannot be abused deserve attention. RECOMMENDATION 4-8. The committee recommends that the Social Security Administration enhance its safeguards for matched data according to accepted practices by employing procedures used in recent federal surveys, and that it take into consideration the effect of such procedures on response rates.