Technical Appendix to NCHRP Report 571: Standardized Procedures for Personal Travel Surveys (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

119 CHAPTER 6 6. Pilot Surveys and Pretests 6.1 P-2: REQUIREMENTS FOR PRETESTS OR PILOT SURVEYS 6.1.1 Definition Pretests and pilot surveys are the process of testing various aspects of the survey design, protocol, instruments, analysis, etc. on a small sample of the population, prior to fielding the main survey. The intention of pretests and pilot surveys is to determine whether or not everything in the intended survey will work and produce the expected results. In some instances, pretests or pilot surveys may be conducted to compare two or more methods for some element of the survey process, and to determine which to choose. In other cases, there is no comparison test involved, although it may be anticipated that some refinements to elements of the survey process will result. 6.1.2 Review and Discussion Various reviews of travel surveys showed that carrying out pilot surveys or pretests is by no means a universal practice. Yet, evidence shows that when pretests or pilot surveys are carried out, they usually lead to changes and improvements in the survey instrument or procedures. It appears that many agencies commissioning surveys are unaware of the importance of pretests and pilot surveys, and that neither time nor budget is usually provided for these activities. As a consequence, there are a number of cases in which data have been collected at considerable cost, only to be found to be inadequate for the intended task. Given the expense normally associated with a household travel survey, this is a serious problem, representing a substantial waste of public resources, and usually making it impossible to collect additional data in that region for that purpose for some years into the future. Although it is unlikely that thirty or forty years of mislabeling can be corrected by the results of this project, it is worthwhile to distinguish between a pilot survey and a pretest. The two terms are used interchangeably in the transportation profession. However, the survey literature distinguishes between them in that a pretest is a test of one or a few elements of the survey, usually without testing other elements, while a pilot survey is a complete run through of the survey (i.e., a dress rehearsal), including analysis of the results. In general, a pretest is necessary when any element of a survey has been changed from an earlier version that has been applied to essentially the same population. A pilot survey should usually be done each time a new survey is designed, or a survey performed on one population is to be performed on another population. It is most important to note that, when the pilot survey reveals the need for significant changes to one or more elements of the survey, the changed elements should be pretested again before full fielding of the survey. The fact that pretests and pilot surveys are not routinely included in the study design, schedule, and costs, and the fact that such failures often lead to some level of failure of the main survey, demonstrates that a standard is required that would specify that pretests or pilot surveys are normally to be done, and that may specify under what circumstances this step might be skipped. Yates (1965) states, in reference to the questions that should be answered at the planning stage of censuses and surveys:

120 “If prior knowledge in these matters is not available a pilot or exploratory survey will be necessary. Even if there is adequate knowledge of the statistical properties of the material, pilot surveys are frequently advisable in large-scale surveys in order to test and improve field procedure and schedules, and to train field workers.” (Yates, 1965, pp.48-49) Cochran (1963) similarly states: “It has been found useful to try out the questionnaire and field methods on a small scale. This nearly always results in improvements in the questionnaire and may reveal other troubles that will be serious on a large scale, for example, that the cost will be much greater than expected.” (Cochran, 1963, p.8). Kish (1965) also states: “To design efficiently a large sample in an unknown field, a pilot study may be conducted prior to the survey, to gain information for designing the survey.” (Italics in original, p. 51) Yates (1965, p.99) goes on to describe some of the roles of pilot surveys, and specifies these as: • Providing information on the various components of variability within the subject population; • Development of fieldwork procedures; • Testing questionnaires; • Training interviewers; • Provision of data for estimating survey costs; and • Determining the most effective type and size of sampling unit. While these authors of basic texts in Survey Design do not specify that pilot surveys must be undertaken, taken together, these statements clearly indicate that pilot surveys should be considered to be essential unless there is considerable prior survey research experience with the subject population. They also indicate that large-scale surveys need pilot surveys. While large scale is never defined in these texts, the type of survey usually undertaken in a metropolitan region, where the sample is several thousand households, probably meets the implied definition of a large-scale survey. Dillman (2000) states “Pilot studies frequently result in substantial revisions being made in the survey design, from adding additional contacts or an incentive to improve response rates, to eliminating or adding survey questions.” (pp. 146-147). The AAPOR Quality Guidelines as quoted by Biemer and Lyberg (2003) have, as their sixth point, the following: “6. Pretest questionnaires and procedures to identify problems prior to the survey. It is always better to identify problems ahead of time rather than in the midst of the survey process.” (p.364) Biemer and Lyberg (2003) also introduce some differences in their definition of the terms relating to pilot studies, pretests, etc. They define pretests as “…small studies using informal qualitative techniques…” that are used to acquire information that helps in the design of the survey. They define pilot surveys as surveys “…to obtain information that can improve the main survey.” They then define Dress Rehearsal as “…a miniature of the main survey, conducted close to the main survey to reveal weaknesses in the survey design…” and generally to perform those functions described previously in this section for a pilot survey.

121 A further type of preliminary test is a rolling pilot survey (Pratt, 2003). This is defined as using the first two or three days of surveying to ensure that the survey is proceeding as intended. Minor adjustments can often be made at this stage, whether to survey instruments, procedures, or other aspects of the survey. This is particularly useful when both time and money are limited, and a full pilot survey cannot be undertaken. Depending on the extent of changes that are made at this point, the surveys from these first few days may still be used in the main survey, or may be discarded, with a subsequent adjustment of the final sample size. There are limited circumstances under which a pilot survey or pretest could be considered unnecessary. A full pilot survey is unnecessary only when the survey being conducted is essentially unchanged from one that has been conducted successfully in the past, so that instruments, sampling procedures, protocols, analyses, and reporting from the survey are essentially the same as another survey and that the population on which the survey is to be conducted is similar in most respects to the population on which the survey has been conducted previously. Under these circumstances, it can be assumed that the correct questions, phrased in appropriate ways, are already included in the survey, that it is known that the analysis of results will work with this design, and where there are no new difficulties in drawing the sample, recruiting respondents, etc. Under these circumstances, a pretest is also not necessary. Yates (1965, p.99) also states that pilot surveys will not normally be required for surveys of populations “…on which there is considerable previous survey experience.” While his focus was largely on agricultural surveys, where the population may remain unchanged for many years, it is necessary to keep in mind that human populations may undergo substantial change in relatively short periods of time, so that a survey at a location ten years ago does not provide the “considerable previous survey experience” noted by Yates. It is, of course, of value to point out that, in the rare event that a pilot survey or pretest leads to no substantive change in design, the data collected could become part of the main survey data, provided only that the sampling for the pilot survey or pretest has been done in a way that is consistent with the main survey and will not add bias to the overall sample. Recommendations on standardized procedures for Pilot Surveys and Pretests are provided in section 2.3.1 of the Final Report. 6.2 P-3: SAMPLE SIZES FOR PRETESTS AND PILOT SURVEYS 6.2.1 Description As defined in the preceding section, pretests are tests of one or more individual components of the survey process, while pilot surveys are a complete run through or dress rehearsal of the survey. Because it was recommended that pretests and/or pilot surveys be conducted in all future travel surveys, it is appropriate to establish the required sample size of these initial tests or surveys. 6.2.2 Analysis In the Atlanta Household Travel Survey, one of the questions asked of the panel of experts that was overseeing the project, was what sample size should be adequate for the pilot survey. The decision made was to use a sample of 50 households. There was no unanimity of the transportation experts on the panel on the sample size. In contrast to this decision, the NPTS in 1995 used a pilot survey sample of over 2,000 households for what was eventually a 42,000 household survey. Similarly, the 2001 NHTS used a pilot survey sample of 2,740 households for the eventual 69,817 households survey. These recent experiences highlight the need for guidance on the sample sizes required and the rationale behind them. There is, in fact, little guidance in the literature on this. Biemer and Lyberg (2003) note that “The design

122 and use of pilot studies are sadly neglected in the survey literature.” Further, they state “The same casual treatment that pilot survey design has received in the literature is also seen in the surveys themselves.” However, having said this, and because Biemer and Lyberg are not writing about survey design, per se, they do not suggest what might be appropriate sample sizes. It seems likely that the sample sizes needed for true pretests – that is, the testing of a single element of a survey, such as a redesign of certain questions – would be able to be done effectively with a very small sample of households, such as 25 to 50 households. However, before adopting such a sample size range, it is necessary to see if there is anything in the general survey literature that addresses the issue, or provides any guidance on the size of pretest samples. If not, then a rationale needs to be found for specifying the sample size. For pilot surveys, it would seem likely that a larger sample should normally be necessary. Again, however, a rationale for this is needed. It appears that present practice in choosing such sample sizes is no more than a “thumb in the air.” This topic addresses not only the size of the samples required for pretests and pilot surveys, but also the composition of the sample and how the sample should be drawn. There are no clear statistical procedures for determining the sizes of samples for pretests and pilot surveys. Clearly, the first issue must be one of what is desired from conducting the pretest or pilot survey. Since this will vary from survey to survey, it is possible that no standard can be set, but only guidance offered. However, some fundamentals can be considered here. Kish (1965) notes that “If the pilot study is too small, its results are useless, because they are less dependable than the expert guesses we can obtain without it.” (p.51). Dillman (2000) suggests that a pilot survey should have a sample size of 100 to 200 respondents in general, and notes that the size may be larger than this, if resources allow. He also states that “…entering data from 100-150 respondents allows one to make reasonably precise estimates as to whether respondents are clustering into certain categories of questions.” (p.147). Another important area to consider here is how the samples are to be drawn for pilot surveys and pretests. It is clear that we do not wish to survey the same households in the main survey as were surveyed in the pretests or pilot survey. To do so would generally produce an unacceptably low response rate and would also be likely to cause significant adverse publicity for the survey. Therefore, those households that are used in the pilot survey and/or pretests should be excluded from the main survey. If, however, these samples are drawn at the outset of the study, and are then excluded for the drawing of the main sample, a bias has been introduced. Random sampling which is essential for representativeness of the sample, requires that all households have an equal probability of being sampled. If households used in the pilot survey or pretest are excluded, then representativeness is compromised, even if only slightly. To avoid any possibility of compromising the main survey, the main sample should be drawn first. Then the pretest samples and pilot survey sample can be drawn from those households not included in the main survey. There is a problem, of course, in this if one of the purposes of the pretest or pilot survey is to gauge nonresponse levels and determine the size of the needed recruitment sample. In this case, the potential to bias the sample is probably unavoidable. In all other cases, however, the pretest and pilot samples should be drawn after the main survey sample. In those cases where this cannot be done, great care should be taken to draw the pretest or pilot survey samples in a completely random process and to exclude all attempted households, irrespective of outcome from further consideration in the main sample. For example, a pretest is often desired to find out what the response rate will be. One might ask if a sample of 50 attempts is sufficient to determine this. Let us suppose that, using 50 telephone numbers, an attempt is made to recruit households to undertake the survey. Suppose that 20 households agree to be recruited, representing a recruitment rate of 40%. We can ask what the confidence is that the actual recruitment rate will be 40%. The sampling error on this figure will be ±7%. This means that, with 95 percent confidence, the recruitment rate will lie between 26 and 54 percent. This is probably not very adequate. Furthermore, if 8 households actually complete the survey, representing a response rate of 40 percent of the recruited households, this would lead to the statistical conclusion that the response rate in the main survey from the recruited households would range, with 95 percent confidence, from 17 to 63 percent. Thus, with a final total response rate from attempted households of 16 percent (40% times 40%),

123 the pretest on 50 households would indicate that, with 95 percent confidence, the overall response rate would appear to lie between 4 percent and 34 percent. Supposing that the survey firm has performed the pretest in order to determine how many samples to draw for recruitment, and that a final sample of 3,500 households is required, then this result would define that the number of households that must be drawn in the sample would range between 10,300 and 87,500. This seems unlikely to be of sufficient precision to benefit the survey design. In this same example, suppose that the number of households sampled for recruitment was increased to 250, with the same 40% response rate, followed by a further 40% response rate for completion of the survey. This would result in 40 households completing the survey. Now the 95% confidence figures on the response rates change to a recruitment rate of between 34 and 46 percent, and a completion rate of between 30 and 50 percent. The overall response rate would now appear to lie between 10 and 23 percent. With the same overall sample aim, the number of households to draw for the recruitment would range between 15,250 and 35,000. In this case, the survey firm may opt for the most conservative figure and purchase a sample of 35,000 households. This result is clearly much more useful to the survey firm than in the prior case, where the conservative figure would be to purchase a sample of 87,500 households. We perform similar computations to determine the minimum sample sizes that are likely to be required in order to answer certain typical questions that are usually asked as part of a pretest or pilot survey. This includes estimates of the average household trip rate and its sampling error, in order to confirm the sample size, determining the expected response rate and hence the total recruitment sample size needed, as well as providing information on how well various questions are answered, how well procedures work, and whether or not analysis can be completed as desired. For example, suppose that one of the items being tested is a question on household income, and it is decided to use the item non- response rate as the measure. Similar computations to those reported in the preceding paragraphs could be done using whether or not a response was obtained to the income question for computing sampling error and a 95 percent confidence on the outcome. The minimum sample sizes required for different possible outcomes from a pilot survey or pretest are shown in Table 69. These sample sizes are all based on the assumption that the relevant statistic of concern to the pretest or pilot survey is to be known with the specified level of accuracy at a 95 percent confidence level. If the confidence level is lowered to 90 percent, the sample sizes reduce, while they increase if the confidence level is raised to 99 percent or higher. Table 69: Sample Sizes Required for Specified Levels of Accuracy Measure Assumed Value Desired Accuracy Sample Size Measure Assumed Value Desired Accuracy Assumed Variance Sample Size 50% ±5% 384 10 ±1 100 384 50% ±10% 96 10 ±2 100 96 50% ±15% 43 10 ±3 100 43 50% ±20% 24 10 ±4 100 24 60% or 40% ±5% 369 10 ±1 50 192 60% or 40% ±10% 92 10 ±2 50 48 60% or 40% ±15% 41 10 ±3 50 21 60% or 40% ±20% 23 10 ±4 50 12 75% or 25% ±5% 288 7 ±0.5 70 1076 75% or 25% ±10% 72 7 ±1 70 269 75% or 25% ±15% 32 7 ±1.5 70 120 Response Rate 75% or 25% ±20% 18 7 ±2 70 67 10% ±3% 384 7 ±0.5 50 768 10% ±5% 138 7 ±1 50 192 10% ±8% 54 7 ±1.5 50 85 Nonresponse to a Question 10% ±10% 35 Household or Person Trip Rate 7 ±2 50 48

124 20% ±3% 683 4 ±0.4 40 960 20% ±5% 246 4 ±0.8 40 240 20% ±8% 96 4 ±1 40 154 20% ±10% 61 4 ±1.5 40 68 30% ±3% 896 4 ±0.4 16 384 30% ±5% 323 4 ±0.8 16 96 30% ±8% 126 4 ±1 16 61 30% ±10% 81 4 ±1.5 16 27 To use Table 69, the following example is provided. Suppose a pilot survey is to be done in which it is desired to determine the response rate to within ±10 percent accuracy, where it is expected to be 40 percent, to determine the non-response rate to the income question to ±5 percent, when it is assumed that the level will be 20 percent, and to estimate the household trip rate, expected to be around 10 with a variance of 100, to within ±2 trips per household per day. Entering the table first for the response rate, this shows the need for a pilot survey sample of 92 completed households. Entering the table for the income nonresponse yields a sample size of 246 households, and for the trip rate, a sample size of 96. The critical element proves to be the non-response to income, which requires a sample size of 246 households. If we now suppose that, based on this, and the scarcity of resources, it is decided instead to reduce the desired accuracy on the non-response to income to ±8 percent, then the sample size for this is seen to be 96, which is the same as that for the trip rate, and only slightly larger than that required for the response rate. Based on this, the decision would be to obtain a completed sample of 100 households, which, assuming the response rate to be 40%, would require contacting and attempting to recruit a total of 250 households. Recommended approaches to sample size estimation for pilot surveys and prestests are to be found in section 2.3.2 of the Final Report.