. "3 Sample Surveys: Overview, Examples, and Usefulness in Studying Aviation Safety." An Assessment of NASA's National Aviation Operations Monitoring Service. Washington, DC: The National Academies Press, 2009.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
An Assessment of NASA’s National Aviation Operations Monitoring Service
There are many ways of selecting probability samples, but the simplest method is simple random sampling. Under this method, every possible subset of a fixed size from the population has equal probability of being selected. For practical and statistical reasons, it may not always be desirable or feasible to use simple random sampling. A variety of other probability sampling techniques, such as stratified sampling, cluster sampling, and multistage sampling, as well as corresponding estimation methods, have been developed in the literature.1
In surveys of human populations, the data are generally collected using a questionnaire as the survey instrument—the participants are asked to respond to a set of questions. The design of the questionnaire is critical to ensuring that the data collected are of good quality and can provide information that is generalizable to the target population. There is a vast literature on questionnaire design.2 There are also many ways of conducting surveys of human populations—for example, by mail, through telephone interviews, or in on-line surveys.
Since a sample survey collects data from only a subset of the population, the estimates have sampling error. The use of probability sampling methods allows one to characterize and estimate this error. The sampling error depends on the probability sampling method used; methods for estimating the sampling error are discussed extensively in the literature.3
There are also several other types of errors (often called nonsampling errors) that occur commonly in surveys (including censuses). For example, coverage bias can occur when the sampling frame (a list of identifiable units from which to draw the actual sample, such as identification numbers, geographical coordinates, household addresses, or telephone numbers) is incomplete. The sampling frame may systematically miss some classes of population members entirely. The sampling frame may also include units that are not members of the target population. Also, failures to contact sampled subjects or nonresponse by those who are contacted can lead to additional biases. Measurement error can result when the survey instrument is poorly designed or if problems arise in the field implementation of the survey.
The nature and the magnitude of added uncertainty because of nonsampling errors cannot be ascertained from the sample itself, regardless of whether it is a probability or nonprobability sample, or even if it is a census. Thus, an important part of the survey planning and implementation process is to determine ways to make these errors as small as possible.4
There are several steps in planning and implementing a good survey. For the purpose of the discussion here, the relevant steps include the following:
Identify the population of interest (the set of units from which the survey would ideally collect data in the absence of concerns over cost or respondent burden) and the characteristic(s) to be studied.
Determine the method(s) for conducting the survey (such as mail or telephone interviews) and implementing the survey.
Develop a sampling frame that will be used to select the sample.
Determine a sampling design, the probability sampling method and the sample size, and the number of elements to be selected (the latter depends on the sampling design and the desired precision as well as on available resources).
Design the data-collection instrument (for example, the questionnaire for a human population).
Examine possible sources of error, ways to reduce them, and ways to estimate them.
Analyze the data and report the results.
Despite the presence of sampling errors, sample surveys have several advantages over censuses:
Samples are less costly than censuses. For example, for populations with large, hard-to-find, or highly
William G. Cochran, Sampling Techniques, Wiley, New York, 1977; and Sharon Lohr, Sampling: Design and Analysis, Duxbury Press, Pacific Grove, Calif., 1999, pp. 4-8.
Norman M. Bradburn, Seymour Sudman, and Brian Wansink, Asking Questions: The Definitive Guide to Questionnaire Design, Jossey-Bass, San Francisco, Calif., 2004.
Cochran, Sampling Techniques, 1977; and Lohr, Sampling: Design and Analysis, 1999.
Judith Lessler and William Kalsbeek, Nonsampling Error in Surveys, Wiley, New York, 1992.