Technical Appendix to NCHRP Report 571: Standardized Procedures for Personal Travel Surveys (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

185 CHAPTER 10 10. Assessment of Quality 10.1 Q-1: COMPUTING RESPONSE RATES 10.1.1 Background Proper calculation of response rates is important because response rates are used by analysts to assess survey quality. Higher response rates are usually desired to reduce the likely incidence of non- response bias. For example, in household travel surveys, it has been found that non-respondents have different travel and demographic characteristics to those of respondents. Hence, the resulting data set is biased – not representative of the general population. This has been widely documented (DeHeer and Moritz, 1997; Kam and Morris, 1999; Richardson, 2000). However, in transportation surveys, no standard has been established and many surveys compute quite different rates. 10.1.2 Methods of Computing Response Rates Until recently, the Council of American Survey Research Organizations, CASRO, was the only organization with its own method for calculating response rates. However, some years after the development of the CASRO method, the American Association of Public Opinion Research (AAPOR) developed another method for calculating response rates. Both the CASRO and AAPOR formulas are commonly used by survey practitioners. For example, the Advertising Research Council (ARC), Council of Marketing Opinion Research (CMOR) and Marketing Research Association (MRA) use a modified version of the AAPOR method for calculating response rates (CMOR, 1999). The World Association of Opinion and Marketing Research Professionals (ESOMAR) does not have its own method for calculating response rates. Estimating Response Rates The response rate is simply defined as the ratio of the number of completed interviews divided by the number of eligible sample units, where eligible sample units are the sample units that have met certain eligibility criteria (CASRO, 1982; CMOR, 1999; Ezzati-Rice et al., 1999; Richardson and Meyburg, 2003; AAPOR, 2004). The main difference between the CASRO and AAPOR methods lies in the estimation of the eligibility rate for sample units of unknown eligibility. In addition, despite the fact that the response rate formulas are rather simplistic, a complex issue arises when trying to determine the number of eligible sample units from the eligibility unknown sample units, especially when using the AAPOR method, given that the CASRO method assumes that the eligibility rate of the unknown sample units is equal to the eligibility rate of the known sample units. Furthermore, the number of non-contacts (eligibility unknown sample units) is increasing in sample surveys and this accentuates the need to appropriately estimate the eligibility rate for the sample units of unknown eligibility. Before describing the formulae used to calculate response rates in more detail, broad classifications regarding eligibility status are discussed. This provides a better understanding of the problems encountered during the analyses of call history files and subsequently, the calculation of response rates.

186 In the literature on response rate calculations, a sample is divided first into two groups. The first group is called the “eligibility known” group, and the second group is called the “eligibility unknown” group. The eligibility known group divides into two further subgroups: the eligible and ineligible. In the first group and subgroup, there is a further sub-grouping into respondents and non-respondents. This is shown diagrammatically in Figure 25. The second group, of eligibility unknown, comprises all sample units whose eligibility for the travel survey is never established. Figure 25: Sample Grouping By Eligibility In many cases, in transportation surveys, the response rate is presented as the respondents divided by the eligible sample units (i.e., R/E). This is actually the cooperation rate, defined by the AAPOR (2004) and is similar to the response rate formula (RR5), also devised by the AAPOR (2004), except that the RR5 formula includes non-contacts in the denominator. The removal of these would in fact give the cooperation rate (COOP1). By definition, the COOP1 rate ignores the portion of the sample that have not been contacted successfully, and within which there is presumably a number of eligible sample units. However, this is also the case for RR5. Thus, the response rate (RR5) formula is not useful in relation to travel surveys and other surveys of the general population, because it assumes that the eligibility rate of the unknown cases is actually zero. In addition, the response rate formula (RR5) is likely to overestimate the response rate of surveys of the general population. Other possible definitions of response rate might include the number of respondents divided by the total sample units (R/S), which would provide a response rate that is generally considered too low. Many of the eligibility unknown units may prove to be ineligible, so that including them as though they are eligible produces an incorrect estimate of response rate. Another, also generally erroneous calculation would be the respondents divided by the eligibility known units (R/K). In one paper consulted, this formula for the calculation of response rates was used (Singer et al., 2000). The result was an under estimation of response rates because all known ineligible sample units were included in the calculation (denominator). The problem is accentuated if many of the attempted contacts are ineligible sample units. Response rates are calculated by analysts to observe the overall quality of the completed survey (Beerten et al., 2000; Lynn et al., 2001). However, the response rate to a survey is only one survey quality indicator, therefore, one cannot assume that a high response rate relates to good quality data. Although Sample Units (S) Eligibility Unknown (U) Eligibility Known (K) Ineligible (I) Eligible (E) Nonrespondents (N) Respondents (R) Non-contacts Contacts

187 response rates are not the only indicators of survey quality, they are important indicators that are readily quoted by survey practitioners, reinforcing the need for this item to be standardized. Response rates have become more of an issue because response rates have been falling over recent years (Ezzati-Rice et al., 1999; Dillman and Carley-Baxter, 2000; Dillman et al., 2001; Kalfs and van Evert, 2003). In relation to travel surveys, it has also been widely documented that the differences in terms of key statistics, between respondents and non-respondents is significant (DeHeer and Moritz, 1997; Kam and Morris, 1999; Richardson, 2000). This highlights the desire by most travel survey practitioners to obtain higher response rates to travel surveys. However, due to the inconsistency of the definition of response rates often quoted in travel surveys, it is difficult to state explicitly that declining response rates are the result of less people willing to participate in surveys or are attributable to the calculation of response rates. It is most likely to be a combination of the two. This then leads to the problem of incomparability: hence, the need for a standard for the calculation of response rates. The widely used CASRO method is: where: RR = response rate SR = complete interviews E = eligible sample units eC = CASRO eligibility rate (eligible units divided by the sum of the eligible and ineligible units) U = unknown sample units refers to the sample units with unknown eligibility (unresolved). The CASRO formula assumes that the proportion of eligible units amongst the eligibility unknown sample units is equal to the proportion of eligible units amongst the eligibility known sample units. For example, if a Random-Digit-Dialing survey was conducted and 20,000 telephone numbers are called, there may only be 4,800 people successfully recruited to participate in the survey, of which only 1,579 complete the survey. The rest of the sample is characterized by refusals (1,200), ineligible respondents (2,400) and 11,600 cases where eligibility is unknown. The eligibility rate for this survey is: (4,800+1,200)/(4,800+1,200+2,400) = 71 percent. Applying the CASRO formula for response rates, the result is 11.1 percent, a very low response rate for the entire survey procedure, because CASRO requires that 71 percent of the unknown eligibility cases are assumed actually to be eligible. The formula for response rates (RR3) devised by the AAPOR, is shown below: ………(2) where: SR = complete interview/ questionnaire PI = partial interview/questionnaire RB = refusal and break-off NC = non-contact O = other UH = unknown if household occupied UO = unknown other UeE SRRR C *+ = )UOUH(e)ONCRB()PISR( SR 3RR A ++++++ =

188 eA = estimated proportion of cases of unknown eligibility that are eligible (AAPOR eligibility rate: the same formula for calculating the eligibility rate is used). Sample units labeled as non-contacts, according to the AAPOR formula, are allocated an eligibility known status. The AAPOR reasoning for this is that prior knowledge of the household has determined the household as an eligible sample unit. Interestingly, another paper also categorized non-contacted sample units as eligible sample units (Lynn et al., 2001). A diagram shown on page 7 of Lynn et al. labels a sampling unit as eligible before contact takes place: this does not seem correct. Furthermore, if eligibility criteria have to be met, than this is certainly incorrect and this does not seem relevant to surveys of the general population. However, it may be relevant to panel surveys, in relation to subsequent waves. With this in mind, the above response rate is re-written as: ………(3) where the symbols have the same meanings as in equation 2. Apart from the different labeling in relation to the non-contacts, the AAPOR formula (RR3) is only slightly different from the CASRO formula, and this difference is in relation to the specification of eA. The two methods are similar because the sum of SR, PI, RB, and O is simply the total of eligible units in the sample (E), and the sum of the UH, UO and NC is the total of the unknown eligibility units (U). Despite the modification of the AAPOR RR3 formula in this analysis (referred to as RR3A), the AAPOR breakdown of disposition codes enables the research agency to understand better the possible contact outcomes and therefore label correctly the disposition codes, in terms of eligibility status. In addition, the AAPOR formula more or less requires the agency to distinguish between the responses that are complete and those that are partial. Even though this should be determined by the agency before fieldwork commences, the AAPOR formula reinforces the distinction and hence, does not allow for the over estimation of response rates. The real question, in relation to the calculation of response rates, is the determination of the eligibility rate for the unknown sample units (Ezzati-Rice et al., 1999; Brick et al., 2002; AAPOR, 2004). The AAPOR definition of response rates (RR3) states that the estimation of the eligibility rate is left to the discretion of the organization(s) and individual(s) undertaking the research, that the estimate for eligibility from unknown cases should be based on the best available scientific information, and that the basis of the estimate must be explicitly stated and explained. A relatively recent study used the AAPOR (RR3) formula to calculate response rates (Keeter et al., 2000). In this study the eligibility rate for the unknown sample units was estimated to be around 20 percent due to investigations that indicated that around 20 percent of eligible units were among the unknown sample units. Two or More Stage Surveys There is a further complication in a survey that involves two or more steps. For example, most household travel surveys involve an initial recruitment contact, followed by a data retrieval procedure that may take place some days later, as shown in Figure 26. This process often leads to incorrect estimates of response rates. Some surveys ignore the response rate from the recruitment, and report only the response rate of the retrieval process (SR/R). Others may calculate the response rate from the recruitment incorrectly using one of the methods discussed above, and then correctly multiply the resulting response rate from the retrieval. Agencies calculating response rates for two or more stage surveys should not )NCUOUH(e)ORB()PISR( SR A3RR A ++++++ =

189 encounter difficulties as long as disposition codes are correctly labeled in terms of known and unknown eligibility. This would allow for the overall response rate to be calculated directly as demonstrated in equation 4: ………(4) where: RR= response rate, SR =successful retrievals, RH= recruited households (respondents in the recruitment phase), E= eligible sample units, e= eligibility rate, and U= unknown sample units. Actually, this equation is very similar to standard 1-3-3 developed by the National Center for Education Statistics (NCES, 2002). The first part of the formula gives the recruitment response rate and the second part calculates the retrieval response rate. Equation 4 reduces to equation 5, the formula for response rates (CASRO, 1982; Groves and Couper, 1998; AAPOR, 2004). ………(5) However, calculating the response rate for each stage of the survey may be useful for agencies to identify problematic areas encountered during any phase of the survey process. For example, the recruitment response rate is calculated by using equation 6. ………(6) )(*) * ( RH SR UeE RHRR += U*eE SR RR += U*eE RH RR +=

190 Figure 26: Two-Stage Survey Process: Recruitment and Retrieval Equation 6 may make agencies aware that recruiting methods and materials used were not suitable, if the response rate calculated for this stage of the survey is poor. This exercise is even more beneficial to agencies wishing to undertake follow-up studies to surveys that yielded very poor overall response rates. Through analyses of two call history files for the recruitment phase for two recent household travel surveys, an attempt to propose standards or guidelines for the estimation of the eligibility rate across sample units of unknown eligibility was made. This is described in the following section. Estimating the Eligibility Rate Given that many agencies use either the AAPOR or CASRO methods for calculating response rates, we examined call history files to determine the eligibility status of the unknown sample units after ten call attempts. We selected ten, because this was the number of calls made to the same sample unit to try to resolve the sample unit in relation to its eligibility status (eligible or ineligible), although not every sample unit that had not been resolved was called ten times, because time may have run out before some units could be called that many times. The status of some sample units will never be known, because either time did not permit ten attempts to be made, or because they were still never contacted after ten attempts. These are the sample units that remain as units of unknown eligibility after the ten call attempts have been made, in this analysis. In addition, an analysis of five call attempts was used to show the difference in the response rate. By looking at each call attempt, the rates at which previously unknown sample units become resolved are determined for each call attempt. This is important because the rates at which the unknown units become resolved are not fixed across the ten call attempts, and this information is vital when trying to establish a suitable eligibility rate to use in the AAPOR method for calculating response rates. In addition, this is important when comparing this method to the CASRO method for calculating response rates. Sample Units (S) Eligibility Unknown (U) Eligibility Known (K) Ineligible (I) Eligible (E) Non-respondents (N) Respondents (R) Recruitment Phase Unsuccessful Retrievals (UR) Successful Retrievals (SR) Non-contacts Contacts

191 CALL HISTORY FILES. The prime purpose for undertaking the analyses of call history files is to determine eligibility rates of the eligibility unknown sample units. However, call history files are not commonly referred to and, therefore, it is useful to provide a definition of such a file. A call history file is the file that houses disposition codes (labels) for each call attempt for each sample unit, during the recruitment phase of the survey process. It therefore contains temporary and final disposition codes for each call attempt for each sample unit (AAPOR, 2004). It also contains other information such as the type of recruitment, (for example whether a cold call is made or the intercept recruitment method is adopted), records the time, day, and date when each call was made, and importantly, the telephone number. Eligibility status is not explicitly shown in a call history file. However, if the number is re-called, this does not necessarily mean eligibility status of the number has not been determined. This depends on how the survey agency decides to categorize certain disposition codes. For example, some call history files categorize call backs as calls of known eligibility whereas other call history files categorize these as calls of unknown eligibility. This is so because a screener interview, if conducted, may have been able to establish the eligibility of the number called, in relation to the bounds of the study undertaken. Thus, it is important to examine the call history file, in terms of the disposition codes used, and any relevant documentation before undertaking any analysis. Disposition codes for the two files are shown in Table 84 and Table 85. Table 84: Disposition Codes, Call History File 1 In Table 84 and Error! Reference source not found., a few differences should be noted in terms of the disposition codes categorized as eligible sample units. The first call history file categorized requests for call backs as units of unknown eligibility whereas the second call history file categorized these as units of known eligibility. This was because, for the second call history file, a screener question determined the eligibility status of the household before a request for call back was made. For the first household travel survey (relating to the first call history file), no attempt was made to convert households that refused to participate, and contacted households in which respondents did not speak English were not called back (this was a function of the bounds of the study, as well as budget). The different temporary and final disposition codes, used in these two call history files, demonstrate the complexity of this analysis as well as highlighting the need for agencies to use the AAPOR standards for temporary and final call disposition codes. Table 85: Disposition Codes, File 2 Disposition (Labels) Code Eligibility Status Disposition (Labels) Code Eligibility Status Complete 1 E Over quota cell 59 I Disposition (Labels) Code Eligibility Status No answer 2 U Busy 3 U Disconnected/changed 4 I Answering machine 5 U Wrong number/ business number 6 I Language barrier/deaf 7 I Party not available 8 E Party terminated (refused) 10 E Scheduled for call-back 11 U Terminated by quota 13 I Party terminated mid-survey 16 E New number 17 I Completed interview 20 E

192 Hard refusals 2 E Over quota county 60 I Second refusals 3 E No answer 101 U Disconnected number 4 I Busy 102 U System default 6 I 1/2 Busy 103 U Business number 8 I Call back specific 104 E Second language barrier not Spanish 13 I Call back non-specific 105 E Second fax machine/ modem 14 I System default (live number) 110 U Terminated interview/ Q BR 18 E First fax machine/modem 127 U Terminated Q1 50 E All other reasons 128 U Terminate out of area 51 I First refusals 140 E Bad zip code 52 I Answering machine 141 U Terminate Q20 53 E First language barrier not Spanish 143 U Terminate Q21 – household count 54 E Wrong number but second attempt chain – live 144 U Refused to participate at invite 55 E Language barrier Spanish 191 U Refused address component(s) 56 E Eligible 212 E Unable/Refuse to reassign date 58 E Ineligible 213 I The second call history file had a more detailed breakdown of call dispositions. The research agency was able to provide Spanish speaking interviewers; hence, “language barrier Spanish” was not given an ineligibility status, but rather a status of unknown eligibility after first contact. These households were re-called by Spanish speaking interviewers to determine whether the households were eligible or ineligible. In the report by the AAPOR (2004), it is indicated that language barriers can be allocated an unknown eligibility status if the survey can account for non-English speaking respondents. Also in the second call history file, a distinction was made between hard and soft refusals: 1. Hard refusals refer to respondents who made it clear that they did not want to participate in the survey and who may have also specifically stated they should not be called back; and 2. Soft refusals (first refusals) were called again. If respondents refused a second time, the disposition was labeled as a second refusal and the household was not called again. These sample units were eligible, which is why they were referred to as “eligible households.” Hard refusals were not re-called; hence, the call disposition is the final call disposition. Fax machines were allocated a separate disposition code. This should be adopted in call history files given that many households may have more than one phone line; however, first contact should be allocated a status of unknown eligibility. If the second call attempt confirmed that the line is dedicated to a fax machine or modem, then the number is given a status of ineligibility because telephone contact with an individual will never take place. The last two disposition codes listed in Table 85 were created to allow for the analysis of the call history file. Once eligibility is established, subsequent call dispositions cannot be categorized as unknown. This too has been suggested in the report by the AAPOR (2004). Therefore, the disposition codes for these households have to be recoded to temporary disposition codes that still represent eligibility. For example, if the request for call back is made after eligibility has been established, the call should be allocated a different disposition code to signify that the household has requested to be called back and that the eligibility status was known and determined as eligible. This clearly demonstrates the need to look across the disposition codes for all call attempts made for each specific number. In addition, it would not make any sense to call back a household determined as ineligible, because such a number has been resolved. After consulting the documentation and examining the call history files, it was obvious that some disposition codes were incorrectly categorized in the second call history file, in terms of eligibility status.

193 For example, in the documentation for the second call history file, call dispositions “All other reasons”, “Wrong number but second attempt chain – live”, and “System default (live number)” were considered as ineligible sample units, which in turn, indicated that these numbers should not have been called again; the numbers were resolved. However, these numbers were called again meaning that these disposition codes should be grouped with the units of unknown eligibility. Correctly re-classifying these numbers was vital for the call history file analyses to yield meaningful results. ANALYSIS. An important step required was to devise a program that corrected for cases where eligibility was established but on later calls was labeled as unknown (as described above). A temporary or intermediate call disposition code was created. These are shown in Table 85. To create this program, first the data were examined and the disposition codes for calls one to ten were either categorized as eligibility known (eligible and ineligible) or eligibility unknown. For example, the disposition code labeled “refused” was categorized as eligible, the disposition code labeled “over quota” was categorized as ineligible, and the disposition code labeled “machine answering device” was categorized as unknown. These three categories need to be determined to calculate the eligibility rates, where the eligibility rate is defined as the number of eligible units divided by the sum of the eligible units and the ineligible units, which reduces to the number of eligible units divided by the total number of eligibility known units. Second, for cases when a call back has been determined as an eligible sample unit and is given the disposition of no answer, busy, answering machine, or any other disposition code of unknown eligibility after subsequent call attempts, the program recoded all cases coded “unknown” to eligible. A number cannot be labeled as a known unit and on later calls be given a status of unknown eligibility. For cases that were initially coded as call backs and later determined as ineligible sample units, the program also recoded these cases to ineligible and created a new variable. In addition, running a frequency count, in terms of call disposition codes for call one, enabled the calculation of the eligibility rate for the known units after call one. Third, another new variable was created to group the eligibility known units (eligible and ineligible). The eligibility known units were allocated the code “0”, and the code “1” was allocated to the eligibility unknown units. Finally, a cross tabulation was performed: call one from step three was cross tabulated against call two in step two. By looking at the eligibility unknown column for the variable created in step 3 (coded as 1) and looking at the disposition codes for the variable created in step 2, the eligibility rates for the unknown units (call 2 to call 10,) in the variables created in step 3, were determined by applying the eligibility rate formula. RESULTS. The results from the analyses of the two call history files are displayed graphically in Figure 27 to Figure 31. In call history file one, no units of known eligibility were called on subsequent calls. Thus, the eligibility rate of the known units is the eligibility rate of the known units determined after the first call; units of unknown eligibility after call one are the sample units called in call two. Given this, the eligibility rate of the unknown units in call one can be determined from call two onwards. For example, the eligibility rate of the unknown units in call one equals the eligibility rate of the known units in call two. This pattern repeats itself for the remainder of the call attempts.

194 0 5 10 15 20 25 30 35 40 45 50 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call 10 w eighted av. Call Number El ig ib ili ty R at e (P er ce nt ) e.r. know n e.r. unknow n Figure 27: Eligibility Rates for Known and Unknown Sample Units, File 1 Figure 27 shows that there is a substantial difference between the eligibility rate of the known units and the eligibility rate of the unknown units (weighted average) for call history file 1; the eligibility rate of the unknown units is higher than the eligibility rate for the known units. This is surprising and disputes what the CASRO formula states; the eligibility rate of the known units equals the eligibility rate of the unknown units. Despite the eligibility rate of the unknown units in call 1 equaling the eligibility rate of the known units in call 2, the weighted average should be used for the eligibility rate of the entire recruitment process, and not just the eligibility rate of an individual call attempt. If one was assessing the eligibility rate for every call attempt, then the CASRO definition of the eligibility rate would be correct (eligibility rate of the unknown units in call 1 equals the eligibility rate of the known units in call 2). 0 10 20 30 40 50 60 70 80 90 100 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call10 Call Number Pe rc en t %unknow n Figure 28: Percentage of Calls that Remain Unresolved After Each Call, File 1 In addition, the percentage of calls for which eligibility status could not be determined (the total number of units of unknown eligibility divided by the total number of calls made on each call attempt), increased as the number of call attempts increased. This is because a high number of the units of unknown eligibility were non-contacts.

195 0 10 20 30 40 50 60 70 80 90 100 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call10 Call number Pe rc en t Figure 29: Percent of the Total Calls Made on Each Call Attempt that Were Non-Contacts For the first call history file examined, the percentage of calls that remain unknown (unresolved in this case because no calls determined as having an eligible status were called back) increased across the ten call attempts. Called numbers whereby contact with an individual did not arise (non-contacts), hence eligibility status is unknown, include call dispositions busy, no answer, and answering machine. Looking at Figure 29, the number of unresolved numbers for call history file one consisted mainly of non- contacted sample units. 0 10 20 30 40 50 60 70 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call 10 w eighted av. Call Number El ig ib ili ty R at e (P er ce nt ) e.r. know n e.r. unknow n Figure 30: Eligibility Rates for Known and Unknown Sample Units, File 2

196 0 5 10 15 20 25 30 35 40 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call10 Call Number Pe rc en t %unknow n Figure 31: Percentage of Calls that Remain Unresolved after Each Call, File 2 Table 86 shows the response rate for the first household travel survey, using the CASRO and AAPOR formulas. In this case, the CASRO formula yielded a higher response rate. This was expected given that the eligibility rate for the known units was lower than that for the unknown units. Table 86: AAPOR and CASRO Response Rates, File 1 Statistic CASRO Statistic AAPOR SR 15064 SR 15064 E 117291 E 117291 E e.r. unknown = e.r. of known units = 22.6% E e.r. unknown = average weighted for ten calls = 41.1 % U total unknowns= 174979 U total unknowns= 174979 RR 15064/ 117291+(0.226*174979) = 9.6 % RR3A 15064/ 117291+(0.411*174979) = 7.9% *RR 9.3% (-0.3%) *RR3A 7.6% (-0.3%) * Response rate if five call limit set The known and unknown eligibility rates determined for the second call history file are shown in Figure 30. There is only a slight difference between the eligibility rate of the known units and the eligibility rate of the unknown units (weighted average); 55.1 percent and 58.5 percent respectively. It is also important to note that these eligibility rates are in fact weighted averages for nine call attempts because it was not possible to determine the eligibility rate of the unknown units for the tenth call attempt. As mentioned earlier in this section, this call history file involved calling numbers with an eligible status, on subsequent call attempts. When a cross tabulation was performed, the eligible cases depicted in the known column were the units where eligibility was pre-determined. Hence, performing the cross tabulation enabled the avoidance of double counting of eligible cases. This was not an issue for the first call history file because cases determined as eligible were not called on subsequent call attempts. Comparing the eligibility rates of the two call history files, the eligibility rates for the second file are much higher than for the first. According to Ellis (2000), the national estimate of residential working numbers is around 41.8 percent. Given that both call history files involve the recruitment phase of the household travel survey, where the eligible unit is a household, the eligibility rates calculated conform to the national estimate.

197 The eligibility rates for the second file examined are higher. This may be the result of the survey being able to interview households that speak Spanish only; therefore, the eligibility status of these households could be determined. In addition, these sample units were not all pooled with the ineligible sample units, which is part of the denominator in the eligibility rate formula. Importantly, the eligibility criteria will affect the eligibility rates observed, and this will vary across surveys. Figure 31 shows that the number of calls resolved increased across the ten call attempts. This is also very different to the situation in the first call history file. Figure 32 is very similar to Figure 31, because the number of call-backs and first refusals called in subsequent calls diminished as the number of call attempts increased. These two call dispositions are not non-contacts and are, therefore, not included in Figure 32, hence the similarity between Figure 31 and Figure 32. 0 5 10 15 20 25 30 35 40 call 1 call 2 call 3 call 4 call 5 call 6 call 7 call 8 call 9 call10 Call number Pe rc en t Figure 32: Percentage of Calls Made on Each Call Attempt that Were Non-Contacts Table 87 shows that eligibility rates estimated using the CASRO and AAPOR methods gave almost identical response rates. This occurred because the difference between the eligibility rate for the known cases and unknown cases was very small. Table 87: AAPOR and CASRO Response Rates, File 2 Statistic CASRO Statistic AAPOR SR 3996 SR 3996 E 19197 E 19197 E e.r. of unknown = e.r. of known = 55.1% E e.r. of unknown = average weighted for nine calls = 58.5 % U total unknowns= 13029 U total unknowns= 13029 RR 3996/ 19197+(0.551*13029) = 15.2% RR3A 3996/ 19197+(0.585*13029) = 14.9% *RR 14.5% (-0.7%) *RR3A 14.3% (-0/6%) * Response rate if five call limit set Another important issue is how to set an appropriate call limit and how this may affect the overall response rate. For example, it has been proposed that non-contact and refusal conversions (that may also involve the temporary dispositions codes non-contacts and requests for call backs), as well as call back requests, should incur a five call limit. After this, the number will remain unresolved. From this research, the change in the conversion of non-contacts, call backs and first refusals, to complete household interviews, as a result of a five call limit instead of a ten call limit, was either non-existent or negligible. Given these results, the effect on the overall response rates is shown in the last rows in Table 86 and Table 87.

198 The changes in overall response rates, as a result of a five call limit, range from a 0.3 percent reduction to a 0.7 percent reduction. It appears as though the CASRO method is slightly more sensitive to the five call limit than the AAPOR method; the reduction in the response rate for the CASRO method after a five call limit is greater than the reduction in response rate after a five call limit for the AAPOR method. Obviously, for file 2, the decrease in the response rate due to a five call limit is more pronounced than for file 1. This is because many of the unknown units in file 2 were actually resolved by the tenth call. Therefore, setting a call limit to five will decrease the response rate because many of these units are still of unknown eligibility after the fifth call (denominator in response rate calculation). Section 2.7.1 of the Final Report provides recommendations on the method to use to calculate response rates, while section 2.2.1 provides recommendations on number of contacts. 10.1.3 Standardizing Disposition Codes Disposition Codes The analysis in the preceding section also demonstrated that there need to be consistent disposition codes adopted, so that response rates can be calculated correctly. AAPOR has recommended disposition codes for each of the three main types of survey – telephone, face-to-face, and mail. The AAPOR recommended codes are shown in Table 88 to Table 90, respectively. Specifically for transportation surveys, a set of recommended consistent disposition codes have been proposed and are provided in section 2.7.1 of the Final Report. Table 88: Final Disposition Codes for RDD Telephone Surveys Primary Disposition Code Secondary Disposition Code Complete 1.1 Interview 1.0 Partial 1.2 Refusal and break off 2.10 Household level refusal 2.111 Refusal 2.11 Known respondent refusal 2.112 Break off 2.12 Non-contact 2.20 Respondent never available 2.21 No message left 2.221 Telephone answering device (message confirms residential household) 2.22 Message left 2.222 Other 2.30 Dead 2.31 Physically or mentally unable/incompetent 2.32 Household-level language problem 2.331 Respondent language problem 2.332 Language 2.33 No interviewer available for needed language 2.333 Eligible, Non- Interview 2.0 Miscellaneous 2.35 Unknown if housing unit 3.10 Not attempted or worked 3.11 Always busy 3.12 No answer 3.13 Telephone answering device (don’t know if housing unit) 3.14 Telecommunication technological barriers, e.g., call-blocking 3.15 Technical phone problems 3.16 Housing unit, unknown if eligible respondent 3.20 No screener completed 3.21 Unknown Eligibility, Non Interview 3.0 Other 3.90

199 Primary Disposition Code Secondary Disposition Code Out of sample 4.10 Fax/data line 4.20 Non-working/disconnected number 4.30 Non-working number 4.31 Disconnected number 4.32 Temporarily out of service 4.33 Special technological circumstances 4.40 Number changed 4.41 Cell phone 4.42 Residence to residence 4.431 Call forwarding 4.43 Nonresidence to residence 4.432 Pagers 4.44 Nonresidence 4.50 Business, government office, other organization 4.51 Institution 4.52 Group quarters 4.53 Not Eligible 4.0 No eligible respondent 4.70 Table 89: Final Disposition Codes for In-Person, Household Surveys Primary Disposition Code Secondary and Tertiary Disposition Code Complete 1.1 Interview 1.0 Partial 1.2 Refusal and break off 2.10 Household level refusal 2.111 Refusal 2.11 Known respondent refusal 2.112 Break off 2.12 Non-contact 2.20 Unable to enter building/reach housing unit 2.23 No one at residence 2.24 Respondent away/unavailable 2.25 Other 2.30 Dead 2.31 Physically or mentally unable/incompetent 2.32 Household-level language problem 2.331 Respondent language problem 2.332 Language 2.33 No interviewer available for needed language 2.333 Eligible, Non- Interview 2.0 Miscellaneous 2.35 Unknown if housing unit 3.10 Not attempted or worked 3.11 Unable to reach/unsafe area 3.17 Unable to locate address 3.18 Housing unit, unknown if eligible respondent 3.20 No screener completed 3.21 Unknown Eligibility, Non Interview 3.0 Other 3.90 Out of sample 4.10 Not a housing unit 4.50 Business, government office, other organization 4.51 Institution 4.52 Group quarters 4.53 Vacant housing unit 4.60 Regular, vacant residences 4.61 Seasonal/Vacation/Temporary residence 4.62 Not Eligible 4.0 Other 4.63

200 Primary Disposition Code Secondary and Tertiary Disposition Code No eligible respondent 4.70 Quota filled 4.80 Table 90: Final Disposition Codes for Mail Surveys of Specifically Named Person Primary Disposition Code Secondary Disposition Code Complete 1.1 Returned Questionnaire 1.0 Partial 1.2 Refusal and break off 2.10 Other person refusal 2.111 Refusal 2.11 Known respondent refusal 2.112 Blank questionnaire mailed back, “implicit refusal” 2.113 Break off questionnaire, too incomplete to process 2.12 Non-contact 2.20 Other notification that respondent was unavailable during field period 2.26 Completed questionnaire but not returned during field period 2.27 Other 2.30 Death (including USPS category: deceased) 2.31 Physically or mentally unable/incompetent 2.32 Respondent language problem 2.332 Language 2.33 Wrong language questionnaire sent for needed language 2.333 Literacy problems 2.34 Eligible, “Non- Interview” 2.0 Miscellaneous 2.35 Nothing known about respondent or address 3.10 Not mailed 3.11 Nothing ever returned 3.19 Unknown if eligible respondent in unit 3.20 No screener completed 3.21 Refused to accept 3.231 USPS category: refused by addresses 3.23 Refused to pay postage 3.232 USPS category: returned to sender due to various USPS violations by addressee 3.24 USPS category: illegible address 3.251 USPS category: insufficient address on mail from one Post Office to another Post Office 3.252 USPS category: no mail receptacle 3.253 USPS category: cannot be delivered 3.25 USPS category: delivery suspended to commercial mailing agency 3.254 Unknown whereabouts, mailing returned undelivered 3.30 USPS category: attempted – addressee not known 3.311 USPS category: postal box closed 3.312 USPS category: no such number 3.3131 USPS category: no such office in state 3.3132 USPS category: no such street 3.3133 No such address 3.313 USPS category: vacant 3.3134 USPS category: unable to forward 3.3141 USPS category: outside delivery limits 3.3142 Not delivered as addressed 3.314 USPS category: returned for better address 3.3143 Not delivered as addressed 3.314 USPS category: unable to forward 3.3141 USPS category: outside delivery limits 3.3142 Cannot be delivered as addressed 3.31 USPS category: returned for better address 3.3143 USPS category: moved, left no address 3.32 Unknown Eligibility, “Non Interview” 3.0 USPS category: returned for postage 3.33

201 Primary Disposition Code Secondary Disposition Code USPS category: temporarily away, holding period expired, unclaimed 3.34 USPS category: unclaimed – failure to call for held mail 3.35 USPS category: no one signed 3.36 Returned with forwarding information 3.40 Returned unopened – address correction provided 3.41 Returned opened – address correction provided 3.42 Other 3.90 Not Eligible 4.0 Selected respondent screened out of sample 4.10 10.2 Q-2: TRANSPORTATION MEASURES OF QUALITY 10.2.1 Definition A variety of data quality measures have been proposed in this study but, in this section, we consider variables that have not been used elsewhere. The type of variables considered are specific to personal travel surveys and are those that are temporally and spatially stable and, therefore, should acquire similar values among surveys. Special circumstances may cause values to deviate from the norm but, generally, deviations from standard values are an indication of a breach in the quality of the data. 10.2.2 Potential Measures and Their Attributes For the variables considered in this section, it is necessary to agree on which variables should feature as transportation measures of data quality, what their expected values are, and what deviation from these values should be considered tolerable. It is common practice to compare values from new surveys with those from surveys that are considered reliable. Data sets that are generally considered to produce reliable results include the national census, national household surveys such as the Nationwide Personal Transportation Survey (NPTS) and the National Household Travel Survey (NHTS), or carefully designed and executed local household travel surveys. The Institute of Transportation Engineers published average values of socio-economic, travel, vehicle usage, time-of-day behavior, and network characteristics from 12 urban areas in the U.S. specifically for the purpose of providing such a reference for new surveys (ITE, 1995). Average values from numerous past surveys have also been published in NCHRP Synthesis 236 (Stopher and Metcalf, 1996) and NCHRP Report 365 (Martin and McGuckin, 1998). It is intuitively expected that variables that relate to the characteristics of a traveler rather than the environment in which travel occurs, are more likely to be stable among surveys. For example, it is known that trip lengths are affected by metropolitan size, and mode choice is affected by the level of transit service and road congestion existing in an area. On the other hand, the number of trips made by an individual are primarily determined by the characteristics of the individual. With this in mind, candidate variables investigated for stability in this study were those that characterize the traveler. Variables investigated included: • The proportion of non-mobile households; • The proportion of non-mobile persons; • The average activity rate per household; • The average activity rate per person; • The average trip rate (overall) per household;

202 • The average trip rate (overall) per person; and • The average trip rates per household and per person for specific trip purposes. The number of households or persons making no trips during a travel survey are seldom reported in survey documentation. However, the statistic can easily be calculated from the raw data. Values of non-mobile rates from several past studies are shown in Table 91. The values in the table are the percentage of persons or households who reported no travel activity during an observation period of one day. Table 91: Percentage of Non-Mobiles Observed in Past Travel Surveys Percentage Non-Mobile Data Persons Households NPTS, 1990 21 - San Francisco, 1981&1990 18 - Sydney, 1981 22 - Adelaide, 1977 13 - Salt Lake City, 1993 18 0.9 Ohio, Kentucky, Indiana Survey, 1990 17 1.6 Dallas Fort Worth, 1996 - 0.8 Southeast Florida Regional Characteristics Study, 2000 - 1.3 The use of non-mobility as a measure of data quality has been suggested in the past (Kitamura, 1995). The premise is that beyond the actual immobility of some respondents, failure to report trip- making reflects a shortcoming in the survey. The reason for respondents failing to report trips actually made are varied. Some do not want to go to the time and effort of reporting them. Others may believe that the travel they made was too insignificant to be of interest to those conducting the survey. Some merely forget the travel they did make or forget to record it. However, in all cases the incidence can be reduced by good survey design and execution. The portion of recorded immobility that is true inactivity is difficult to estimate because at least some immobility on any given day is elective. For example, older people in particular may often choose to stay home all day. However, statistics are not available on elective immobility as a whole. On the other hand, there are those that are permanently or temporarily incapacitated and unable to travel, and some statistics are available for these cases. In the U.S., approximately 12 percent of the U.S. population is characterized as “severely disabled” and approximately one-third of these people require “assistance with activities of daily living” (U.S. Bureau of the Census, 1997). Individuals are classified as severely disabled if they use a wheelchair, cane, crutches, or a walker, if they have mental or emotional conditions that seriously interfere with everyday activities, if they receive federal benefits based on an inability to work, have Alzheimer’s disease, mental retardation, or another developmental disability, or are unable to work or perform every-day activities such as walk, speak, hear, grasp objects, etc. Those needing assistance with “activities of daily living” are individuals requiring assistance in moving inside or outside the home, getting in or out of bed, bathing, dressing, eating, taking medicine responsibly, using the telephone, preparing meals, etc. Thus, while some of the severely disabled persons may indeed make a trip on any given day, virtually none of those requiring assistance with activities of daily living are expected to make a trip. Therefore, it appears that between four and ten percent of the population is either unable or unwilling to travel due to a disability. Illness that prevents an individual from traveling is another possible reason why individuals may not travel on any given day. Statistics from the U.S. Bureau of Labor Statistics and from Canadian Statistics suggest that, on average, the number of days lost per worker due to injury or illness is seven days per annum (Bureau of Labor Statistics, 2003, Statistics Canada, 2002a). Thus, on any given day a worker would have approximately a two percent (7/365) chance of missing work due to injury or illness.

203 What proportion of these workers would make no trips is not known but the statistic does show that the source of immobility due to illness is small relative to that due to disability. Activity rates, like non-mobility rates, are statistics that are seldom reported. Because activity levels are intuitively expected to be more a function of the characteristics of an individual or household rather than their location, activity levels could be expected to remain relatively stable among surveys. The activity rates of a few activity-based surveys are shown in Table 92. Table 92: Activity Rates from Selected Travel Surveys Number of Activities Per Day Data Per Household Per Person Salt Lake City, 1993 13.3 4.2 Ohio, Kentucky, Indiana Survey, 1990 13.5 5.3 Dallas-Fort Worth, 1996 9.1 - The activity rates between the Salt Lake City and Ohio, Kentucky, Indiana Survey are relatively similar, but different to the household activity rate in Dallas-Fort Worth. One of the major obstacles in obtaining similar activity rates is the difference in the classification of activities among surveys. The issue of standardized time use activities has been the subject of several endeavors in the past decade. The United Nations Statistical Office has developed a Trial International Classification of Activities for Time Use Statistics (ICATUS) that is “… an international classification of activities for time use statistics that is sensitive to the differences between women and men in remunerated and unremunerated work” (UNSD, 1997a and UNSD, 1997b). In Europe, an alternative time use classification scheme was developed for the Harmonized Time-Use Study Project (Eurostat, 1996). However, there has not been widespread acceptance of these proposed standards and alternative classification schemes have been developed by several agencies in the United States, Canada, and Australia (Hoffmann and Mata, 1997; UNSD, 1997c; UNSD, 1998; Harvey, 2001). Activity classification schemes require specification of both what is done and the context in which it is conducted (Hoffmann and Mata, 1997). This is because an activity is qualified by its setting. For example, cooking (as an activity) for one’s own family is quite different to cooking as a commercial activity, and caring for a family member is different to providing care to a stranger in a hospice. The classification schemes that are currently under development take these factors into account. However, they are different from each other, and until a single, standardized activity classification system for transportation is established, it will not be possible to identify standard activity rates. Reviewing past experience on the stability of trip rates among surveys suggests that there is indeed a degree of stability among the values. A review of more than 50 recent urban travel surveys in NCHRP Synthesis 236 ( Stopher and Metcalf, 1996) show that the number of trips per person per day can be expected to range between 3.5 and 4.5, and trips per household per day between 8 and 11 (Stopher and Metcalf, 1996). This is also supported by the research which led to publication of NCHRP 365 – the update of standard trip-making characteristics first established in NCHRP 187 in 1978 – that household trip rates vary between 8.5 and 9.2 trips per household per day (Martin and McGuckin, 1998). Household trip rates from a number of studies, including those from NCHRP 187 (Sosslau et. al, 1978), 236, and 365, are shown in Table 93. The data in Table 93 appear to support the contention that the average household trip rate falls within the range of 8-11 person trips per day. Table 93: Average All-Purpose Household Trip Rate from Recent Travel Surveys Data Survey Date Source Person trips/hh/day San Francisco 1981 ITE, 1995 8.71 Albany, NY (Capital District) 1983 ITE, 1995 8.25 Houston-Galveston 1984 ITE, 1995 9.32 Denver, CO 1988 ITE, 1995 7.89 Philadelphia, PA – Southern N.J. 1989 ITE, 1995 7.81

204 51urban travel surveys 1990-1995 NCHRP 236 8.91 Home interview surveys 1956-1976 NCHRP 187 7.6-14.1 12 urban travel surveys & NPTS 90 1985-1990 NCHRP 365 8.5-9.2 Ohio, Kentucky, Indiana Survey 1990 - 10.03 Salt Lake City 1993 - 13.8 NPTS 95 1995 NPTS 95 9.73 Baton Rouge Personal Tr. Survey 1997 LTRC/LSU 9.69 Dallas/Fort Worth 1996 NCTCOG 9.47 Nashville, Memphis, Knoxville, TN 1998-2003 Everett, 2003 8.04 - 8.44 South East Florida 2000 - 7.19 Florida - Schiffer, 2003 7.31 - 9.80 Twin Cities (urban) 2001 Filipi, 2003 10.3 Twin Cities (rural) 2001 Filipi, 2003 9.5 Oregon 1996 Ayash, 2003 7.8 Atlanta (SMARTRAQ) (day 1) 2001-2002 Rousseau, 2003 8.31 Atlanta (SMARTRAQ) (day 2) 2001-2002 Rousseau, 2003 7.95 As pointed out by Stopher and Metcalf (1996) in NCHRP Synthesis of Highway Practice 236, measuring trip rates is not without ambiguity. First, there is seldom a clear specification of whether the trip reported is a linked or unlinked trip. A single linked trip between an origin and destination consists of two or more unlinked trips (or, synonymously, segmented trips) if the traveler changes mode, or if the trip is interrupted to drop off or pick up a passenger (Stopher and Metcalf, 1996; RTI, 1997). In transportation planning, linked trips are typically used, and unlinked trips are combined to form linked trips before analysis begins. Reported trip rates are typically linked trip rates but care must be taken to ensure that this is the case since unlinked trip rates will inevitably be higher. Second, the definition of a trip has not been standardized and this can affect the observed rates. Specifically, the inclusion of all non-motorized travel and the inclusion of very short trips can alter the number of trips recorded. Third, the issue of weighting, employed to adjust the sample for bias, can affect trip rates. Weighting is conducted in a variety of ways during the processing of travel survey data, and the procedure used can affect the weighted trip rate. More importantly though, is knowing whether the reported trip rate is of weighted or unweighted trips. Weighted and unweighted trip rates can be quite different, as demonstrated in the NPTS 95 data where the weighted household trip rate is 10.5 compared to 9.7 for the unweighted trips. In most studies, if not specified, unweighted trip rates are reported. Fourth, care must be taken to ensure that the trips reported are person trips and not vehicle trips, since both are often reported in travel survey results. Household trip rates by purpose are shown in Table 94. The values average 1.7, 4.7, and 2.8 person trips per day for home-based work, home-based other, and non home-based trip purposes, respectively. This implies an average all-purpose household trip rate of 9.2 person trips per day, which is consistent with the rates shown in Table 93. Table 94: Average Household Trip Rate by Purpose from Recent Travel Surveys Person trips/hh/day Data Survey date Source HBW HBO NHB San Francisco 1981 ITE, 1995 1.89 - - Houston-Galveston 1984 ITE, 1995 1.72 4.65 2.95 Philadelphia, PA – Southern N.J. 1989 ITE, 1995 2.14 4.03 1.64 Ohio, Kentucky, Indiana Survey 1990 1.72 - - Salt Lake City 1993 1.66 4.93 - NPTS 95 1995 NPTS 95 1.56 4.99 3.03 Baton Rouge Personal Tr. Survey 1997 LTRC/LSU 1.57 4.94 3.18 Dallas/Fort Worth 1996 NCTCOG 1.63 4.68 3.16

205 A problem with measuring trip rates at the household level is the impact household size has on the results. The effect of household size can be eliminated by observing trip rates per person. However, this will not necessarily reduce the variation in trip rate values because of the different levels of aggregation at which the two trip rates are measured. The Coefficient of Variation (COV) of the trip rates per person in Table 95 is 0.20 while the COV for the household trip rates shown as single values in Table 93 is 0.17. The average all purpose trip rate in Table 95 is 3.38 trips per person per day. A review of the trip rates per person by purpose revealed considerable variation among the data sets considered in this study. Subsequently, we were unable to identify representative values that could function as useful reference values. Recommendations on mobility rate per person and per household, and on trip rates per household as transportation measures of quality are provided in section 2.7.2 of the Final Report, together with reference values for each measure. Table 95: Average All-Purpose Person Trip Rate from Recent Travel Surveys Data Survey Date Source Person trips/person/day San Francisco 1981 ITE, 1995 3.40 Albany, NY (Capital District) 1983 ITE, 1995 2.05 Houston-Galveston 1984 ITE, 1995 3.48 Denver, CO 1988 ITE, 1995 2.54 51urban travel surveys 1990-1995 NCHRP 236 3.50 Ohio, Kentucky, Indiana Survey 1990 - 3.87 Salt Lake City 1993 - 4.23 NPTS 95 1995 NPTS 95 3.76 Baton Rouge Personal Tr. Survey 1997 LTRC/LSU 3.70 South East Florida 2000 - 2.30 Atlanta (SMARTRAQ) (day 1) 2001-2002 Rousseau 3.90 Atlanta (SMARTRAQ) (day 2) 2001-2002 Rousseau 3.80 10.3 Q-3: COVERAGE ERROR 10.3.1 Definition Coverage error in surveys is the error incurred by having a sampling frame that deviates from the survey population. It has been described as the “failure to include some units, or entire sections, of the defined survey population in the actual operational sampling frame” (Kish, 1965), or the error that “results from every unit in the survey population not having a known, non-zero chance of being selected” (Dillman, 2000). However, in addition to the “under-coverage” which results from exclusion of valid units in the sampling frame, it is also the unintentional inclusion of units in the survey sample (including duplication of units) that do not belong there (Kish, 1965 p. 529; Statistics Canada, 1998, p. 16). This “over-coverage” can occur, for example, when telephone numbers are used as a sampling frame in a random digit dialing (RDD) sampling process, and households with multiple telephone lines are, subsequently, sampled at a higher rate than those with a single line. Similarly, “under-coverage” can occur in the same type of survey because some households do not own a telephone or have interrupted telephone service. Coverage error is distinct from non-response error although both result from not obtaining information from units in the survey population. Coverage error results from not having some units in the

206 sampling frame, or from having units in the sampling frame that do not belong there. Non-response is failing to obtain a response from units that are within the sampling frame. Coverage error does not include intentional deviation of the sampling frame from a complete and accurate listing of the population (Kish, 1965). In travel surveys, certain portions of the population are often intentionally excluded from the sample frame, either because they do not contribute in any meaningful way to travel in the area, or they are too difficult to survey. For instance, household travel surveys exclude those in hospital and in prison from the sampling frame, and usually exclude households living in group quarters such as military barracks or university residence halls. Even children under the age of 5 have typically been excluded from the sampling frame because they were considered to generate virtually no trips of their own. This may change as more children under the age of five are placed in day- care centers, nursery schools, and pre-Kindergarten classes and they begin to generate significant travel of their own. However, the intentional omission of these sections of the population from the sampling frame are considered a redefining of the survey population and not a contribution to coverage error. 10.3.2 Potential as a Measure of Survey Quality Coverage error is seldom estimated and rarely reported in travel surveys. Among the eleven data sources reviewed and analyzed for various purposes in this study (the Research Triangle survey is not included, because of lack of documentation), none were found to have estimated and reported coverage error. However, coverage error can be significant and, therefore, it is important that it be measured and reported as a means of assessing the quality of data. Establishing a standardized method of measuring coverage error, and recommending that it be estimated and reported in all future surveys will provide a useful additional measure of the quality of survey data in the future. Two alternative procedures are typically used to estimate coverage error (Kish, 1965). The first involves a second survey with improved procedures and good sampling frame where coverage error is supposedly absent or, at least, substantially diminished. With this method, comparison between the original survey and the results from the improved procedure provide an estimate of coverage error. The method is expensive and is generally not appropriate in all but large surveys with big budgets. The second alternative is to estimate the population of the study area using the sample, and compare it with an estimate of the population from an external source. The estimate of the population using the sample is obtained by multiplying the sample population by the inverse of the sampling rate to produce an estimate of the total population. External estimates of the population are obtained from sources with low coverage error, such as the decennial Census or the Current Population Survey (CPS) (Zimowski et al., 1997a). The CPS is a sample survey conducted monthly by the U.S. Bureau of the Census for the Bureau of Labor Statistics to produce a variety of statistics on employment and related items. The sample frame used in the CPS is an updated version of that used in the last census. Coverage error in the CPS is currently estimated at 7.4 percent (U.S. Bureau of the Census, 2000). Interestingly, coverage error in the CPS has increased in the last two or three decades, because it was 3.7 percent in the mid 1970s (U.S. Bureau of the Census, 2000). Coverage error is traditionally measured by the extent to which the population is accurately measured by the sample frame (U.S. Bureau of the Census, 2000). A statistic which achieves this is the following formulation which measures the percentage error in population estimation resulting from deviation of the sampling frame from the true population (Kish, 1965): 100)~1( X FCE x−= where: CE = coverage error in percent

207 Fx = sample population multiplied by the inverse of the sampling rate X~ = population from an external source. If the sample consists of a disproportionate stratified sample, the population estimated from the sample (Fx) is the sum of the individual products of the sample population and inverse of the sampling rate over the strata. The sampling rate used in this expression is the planned sampling rate and not the sampling rate ultimately obtained in the survey. That is, it is the sampling rate designed for the survey and, subsequently, is not affected by refusals, non-contacts, non-response, or incomplete responses. It should be noted that the measure of coverage error in this equation provides an estimate of the net effect of over-coverage and under-coverage. For example, in a telephone interview using RDD, the sampling frame would be all residential telephone numbers in a study area, and an accurate estimate of the population of telephone-owning households would be obtained if each household had one telephone and the sample size divided by the sampling rate were multiplied by the average household size. However, those households with multiple telephone lines have a higher chance of selection and, if the average household size is different among these households to that among the remainder of the telephone- owning households, an incorrect estimate of the population will be obtained. Similarly, if the average household size among those with interrupted telephone service is different to those with full service, an incorrect estimate of the population would be obtained. The presence of households without telephone service reduces the estimate of the population to an estimate of the population in telephone-owning households only. In the presence of all of these conditions, as is typically the case, only the net effect of these over-, under-, and zero-cover conditions would be observed because one condition plays off against the other. Generally, over-coverage in RDD surveys can be eliminated by weighting households with multiple lines so that they reflect the same chance of selection as a household with a single telephone line. This can be achieved because information on the number of voice lines used by the household can be gathered during the interview. If over-coverage due to multiple telephone lines is eliminated through weighting, the remaining coverage error in a RDD survey will be reduced to reflect under-coverage only. This would make measurement of coverage error more useful as a measure of data quality. Consequently, it is recommended that over-coverage caused by multiple telephone lines be eliminated by weighting whenever possible, to allow the coverage error measure to reflect more accurately the level of under- coverage present in the sample. The level of under-coverage can be quite high in certain types of surveys. For instance, the number of households without telephone service was estimated at 2.4 percent by the U.S. Bureau of Census in 1997 (U.S. Bureau of the Census, 2000). From the Nationwide Personal Transportation Survey (NPTS) of 1995, a further 2.2 percent of households in the U.S. are estimated to have interrupted telephone service. Those with interrupted or no telephone service are not a random selection from the population. Analysis of the Public-Use Microdata Sample (PUMS) data has shown that low phone ownership is more common among low income groups, persons below 25 years of age, and African Americans (Federal Highway Administration, 1998). Research conducted by Banks et al. (2001) compared the demographic and travel characteristics of people with a history of interrupted telephone service with those with no telephone service and found that there are significant differences between the two groups. People in households with interrupted telephone service were more likely to own their home, have more workers in the household, and have more vehicles available than people in households without telephone service. Thus, while both groups are poor, any weighting that may be applied to try to adjust for this under-coverage would need to take these differences into account. Coverage error can be reduced by weighting, if it is known how to weight respondents in the sample so that they represent the entire population. Sometimes, as in the case of multiple-line households, the appropriate weighting can be determined easily. However, when the characteristics of those omitted from the sample frame are not well understood, it is difficult to determine appropriate weights. In this case, it would probably be more appropriate not to attempt to effect any change to the sample and merely

208 estimate the coverage error on the sample as is. However, for coverage error that can be reliably corrected with weighting, these corrections should be made to the sample before the estimation of coverage error using the equation above. Interpretation of what constitutes acceptable levels of coverage error will remain subjective. One suggestion is that good surveys should produce CE values of less than ten percent (Kish, 1965). Considering the level of telephone ownership quoted above, surveys using the RDD sampling process may be able to achieve such values. However, if “do not call” lists are commonly perceived to include research surveys, if more households make exclusive use of cell phones, if cell phone numbers are not included in the sample frame, and if devices such as caller ID effectively eliminate certain numbers from the design sample, then much higher coverage errors will be produced. Recommendations on coverage error are provided in section 2.7.3 of the Final Report. 10.4 Q-5: PROXY REPORTING AS A QUALITY INDICATOR 10.4.1 Introduction As discussed in section 5.2 of this Technical Appendix, proxy reporting in a travel survey is the reporting of one person on behalf of another. Sometimes it is necessary to perform proxy reporting, because some persons in the household are too young to answer the questions themselves, individuals are temporarily incapacitated due to illness or injury, or they are permanently incapable of answering questions due to language difficulties or mental incapacity. However, beyond these cases, proxy reporting also occurs when participants feel little commitment to the survey or the survey is conducted in such a manner as to make individual participation less of a requirement than is desirable. This latter condition occurs, for example, when data are retrieved by telephone and the person answering the telephone is encouraged by other members of the household, or is forced by their absence or refusal to talk on the telephone, to provide the information required. Thus, while proxy reporting is unavoidable in some cases, it is also susceptible to survey design and the method of survey execution. Because proxy reporting affects the accuracy of the data, it is reasonable to suggest that more proxy reporting is likely to lead to less accuracy in the data. Accuracy is an important component of data quality and, therefore, it is suggested that the incidence of proxy reporting can be used as a measure of data quality of the data set. This section addresses that issue. 10.4.2 Proxy Reporting as a Quality Measure Proxy reporting is known to bias reported data (Richardson et al., 1995, p. 49; Greaves, 2000). Analyzing data from the 1995 Nationwide Personal Transportation Survey (NPTS), Greaves (2000) found that among persons over the age of 13 (children were not permitted to report their own travel in the survey), those that completed a diary and reported their own trips had, on average, trip rates that were 21 percent higher than those who completed a diary but had someone else report the data. Among those who did not complete a diary, self-reported trip rates were 63 percent higher than those using proxy reporting. However, of even greater significance was the fact that these differences were not consistent among the different trip purposes; in some cases proxy reporting produced higher trip rates than self reporting. For trip purposes involving regular trip activity such as work and school trips, proxy reporting tended to overestimate the trip rate while the more spontaneous or discretionary trips such as non-home-based trips were severely underestimated. Thus, while proxy reporting displays a clear impact at the aggregate level, its impact is even larger at the disaggregate (i.e., individual or household) level.

209 To use the incidence of proxy reporting as a measure of data quality, the definition, measurement, and interpretation of proxy reporting must be standardized. That is, a common understanding of what proxy reporting is, how it is measured, and how the results are interpreted, must be formulated so that a consistent expression of this measure can be generated in each data set. Each of these aspects is discussed more fully below. First, there is currently no definition or common agreement among survey practitioners of what constitutes proxy reporting. The general concept of proxy reporting is easily understood, but its application in practice is often more difficult. For example, if a person filled in a travel diary but does not personally report the information in a CATI retrieval, is that proxy reporting? That is, is proxy reporting linked to the reporting or recording activity? Similarly, how much information must be supplied by another person for a response to be qualified as a proxy response? For example, if a person recorded their own personal information but someone else furnished their travel information, would this qualify as a proxy response or not? The complexity of the possible forms of proxy reporting are demonstrated by an analysis of the 1995 Nationwide Personal Transportation Survey (NPTS) in which the questions “Is this a proxy interview?” and “Who completed the diary?” were both asked. Table 96 shows the joint answers to these questions from all persons over the age of 13. The 1995 NPTS survey involved telephone recruitment, a mail-out travel diary in which household members were to record their travel on the travel day, and a CATI retrieval of all the information after the travel day. Table 96 shows that while approximately 22 percent of total sample provided information by proxy (17,608/81,252), almost half of those (8,497/17,608) filled in their own diary. Another quarter (4,022/17,608) had someone else complete their diary, while most of the remainder had no completed diary at all. Thus, among the reports that were classified by the respondents themselves as proxy reporting, considerable variation in the level of involvement of both the proxy reporter and the subject is evident. Review of the self reported cases is equally interesting. Approximately two-thirds of the self reporters completed their own diaries (41,154/63,644), but some (3,831) reported on diaries completed by someone else, and most of the remainder (10,605+7,939), or 29 percent of the self reporters, reported their own travel without having completed a diary. Thus, there is again a clear difference among the cases in this category although it is perhaps not as large as among the group of self-professed proxy reporters because in this case the majority of cases involve the subjects reporting on their own behavior or personal characteristics. Table 96: Proxy Reporting of Persons over 13 years of Age in NPTS 95 Who completed the diary? Self Other No one No diary Missing Total Yes 8,497 4,022 3,178 1,873 38 17,608Proxy interview? No 41,154 3,831 10,605 7,938 116 63,644 Total 49,651 7,853 13,783 9,811 154 81,252 Second, there is a need to ensure that the information necessary to define whether a report is a proxy report or not is included in the data. Merely asking a respondent whether they are making a proxy report or not will, on its own, not be sufficient to distinguish among cases. Information must be provided to allow the analyst to determine whether the subject is someone who could report their own information, whether the subject recorded the information being reported, and whether the person reporting the information is also the subject. It is suggested that information of this nature is needed in both interview and self-administered surveys so that an estimate of proxy reporting can be obtained irrespective of the type of survey conducted. It has been suggested in the past that self-administered questionnaires do not afford a reliable means of determining proxy reporting but even interview-type surveys rely on the integrity of the respondent for some of the information needed to identify a proxy report. Because there is relatively little incentive for a respondent to falsify information on, for example a question on who

210 prepared the information being reported, it would seem advantageous to include questions that allow identification of proxy reporting in all types of surveys. Third, there is a need to evaluate the levels of proxy reporting produced. That is, how are levels of proxy reporting to be interpreted in terms of data quality? Beside the necessary proxy reporting for children and those unable to participate in the survey at the time, the tolerable level of proxy reporting among other household members needs to be specified. Looking at the analysis reported in Table 97, it is clear that even a relatively moderate level of proxy reporting (22%, based on Table 96) can induce large errors in certain trip purposes. At the same time, not all capable respondents are likely to participate. Diehard refusals are probably better handled using proxy information rather than spending an inordinate amount of effort to convert the individual or forego all information on the individual entirely. Using the example of the NPTS 95 data shown in Table 96, the number of proxy reports would be 4,022+3,178+1,873+3,831, or 12,904 out of the 81,098 cases for which the source of the information is known. Thus, using the above definition, the level of proxy reporting in the NPTS 95 data is 15.9 percent (12,904/81,098). To be able to estimate the level of proxy reporting as defined above, the necessary information must be included in the data. Thus, it is necessary to record in each data set information that would allow the suitability of the subject to be determined (e.g., age, language barrier, sickness) as well as information similar to that recorded in the NPTS 95 data which asked “Who completed the diary?” If no diary is involved, respondents are to be asked whether they are reporting on behalf of themselves or someone else. If no interview is involved, a question is to be included in the questionnaire or diary asking the respondent to state whether they are reporting their own data or that of someone else. These questions must be posed in the questionnaire for mail-back responses, and to the respondent for telephone retrieval and/or telephone surveys. The options for each of the above questions should be “self”, “someone else”, or “no one”. If the data which are reported have been prepared, or recorded, by the subject, then it is self- reported irrespective of whether that person actually reports the information or not and irrespective of whether the information was prepared in advance, involved writing it down, or was generated spontaneously at the time of data collection. When it is not known who prepared or recorded the data transmitted, the case is omitted from the calculation of the level of proxy reporting in the data. Table 97: Differences Between Proxy and Self Reporting in the 1995 NPTS (Greaves, 2000) Purpose Category Home-Work Home- School Home-Shop Home-Other Non-Home- Based Total Self, Diary 0.89 0.11 0.70 1.73 1.82 5.26 Proxy, Diary 0.99 0.22 0.49 1.35 1.28 4.33 Self, No Diary 0.89 0.14 0.45 1.18 1.19 3.75 Proxy, No Diary 0.67 0.22 0.22 0.70 0.50 2.30 Total 0.86 0.14 0.57 1.46 1.49 4.52 Other data sets that have information on proxy reporting, such as the North Central Texas Council of Governments (NCTCOG) data, have different questions identifying proxy reporting which makes the comparison difficult. For example, in the NCTCOG data the question was asked during the telephone retrieval “what is your relationship to the person who filled out the form?” However, the questionnaire that was sent out in advance of the travel day and which the respondents used to record their household, vehicle, and travel information, informed them that while telephone retrieval would be conducted they should mail back their completed questionnaires. Under these circumstances, individual respondents were less likely to provide their own information during the telephone retrieval. Of the 12,172 persons in the sample, 4,711 (39%) identified themselves as the person who filled out the form suggesting that the rate of proxy reporting was 61 percent. However, with the definition of proxy reporting suggested above, the true rate of proxy reporting could only be determined if information on who completed the questionnaires

211 relative to the subject of each questionnaire was provided. Recommendations on using proxy reporting as a data quality measure are provided in section 2.7.4 of the Final Report. 10.5 Q-6: VALIDATION STATISTICS 10.5.1 Definition Validation is the process of verifying the authenticity of collected data by recontacting a sample of households. It is used in interview-based surveys to determine whether the interviewer actually conducted the interview and whether the information obtained is accurate (TMIP, 1996, p. 6-171). It can also be used in self-administered questionnaires where the validation survey then usually involves a face- to-face or telephone interview to check the quality and completeness of data (Richardson et al., 1995, p. 241). Validation surveys typically involve a limited set of key questions only. These usually include identifying and trying to make contact with the person involved in the original survey, and verifying a few trips reported by the respondent. Validation surveys are conducted to ensure the authenticity and integrity of the data. 10.5.2 Design and Use of Validation Surveys Validation surveys have been relatively rarely performed in travel surveys in the past. From a review of nine recent studies (1991 California Statewide Survey, 1993 Wasatch Home Interview Travel Survey, 1995 Origin Destination Survey For Northwestern Indiana, 1996 Bay Area Travel Survey, 1996 Broward Travel Characteristics Survey, 1996-97 Corpus Christi Study Area Travel Survey, 1997-98 Regional (New York, North Jersey) Travel Household Interview Survey, 1998-99 Greenville Travel Study, and the 2000 Southeast Florida Regional Travel Characteristics Survey), only one reported conducting a validation survey. Validation surveys are not popular because of the time and effort involved and the need to explain to each interviewee why they are being contacted again. However, the mere fact that interviewers know that validation surveys will be conducted is often enough to discourage them from being lax in the execution of the survey, or, in extreme cases, of falsifying information (Richardson et al., 1995, p. 248). A second advantage is that validation surveys provide information that can be used to assess the quality of the survey data. To use statistics from validation surveys to assess the quality of a survey, variables that feature in the statistic must be identified, their combination in a statistic must be formulated, and the ability to interpret the values must be developed. Each interview in the validation survey must be conducted by someone different to the one who conducted the initial interview. Validation surveys must be conducted progressively throughout the travel survey so that problems can be identified and remedied, and interview standards are maintained throughout the study. Whenever possible, the validation survey must be conducted with the initial respondent. The questions included in the validation survey must not be verbatim quotes from the earlier survey but, rather, should express the same question in different terms. Also, the questions should be phrased as if further information is being sought, rather than that the purpose is to verify the integrity of the data gathered earlier. The questions should not ask for detail that the respondent has difficulty in recalling, while still asking something that would be difficult to guess. For example, in validating a trip, a feature that is relatively easy to remember but difficult to fabricate, is the approximate time spent at the destination of the trip, or the number of accompanying persons on the trip.

212 It is suggested that the following set of core questions be included in every validation survey conducted: 1. Did you complete the initial survey? (yes or no). If “yes”, go to question 3 below. If “no”, go to the second question below. 2. Did someone else in your household complete the survey? (yes or no). If “yes” go to question 3 below. If “no” terminate the validation survey. 3. Select a trip that the respondent is likely to remember from among the trips reported in the initial survey and note the time spent at the destination. Ask the respondent to recall the trip in question and to report the approximate time spent at the destination. If the answers to both of the first two questions are negative, then these two questions identify a household that apparently was never surveyed. This may be due to a lack of knowledge on the part of the person being interviewed, forgetfulness, or a case of genuine falsification of an entire interview. The interviewer conducting the validation survey must discretely determine which one of these possibilities is the most likely. For those who admit to being interviewed, the third question provides a brief check on the trips reported. Due to the difficulty of recall, only large differences should be considered evidence of possible falsification. The tolerable limits of falsified information are a matter that must be decided by each agency commissioning a travel survey. The main purpose of the validation survey is to identify and remedy problems within the survey company. An indirect purpose is to act as a disincentive to interviewers when they know that validation surveys are conducted. Falsification of data by interviewers is likely to be dealt with very severely in survey companies. However, anecdotal evidence suggests that it does exist and that, under pressure to reach certain goals, interviewers will develop very innovative ways in which to introduce such data. Statistical analysis of reported data is often used to detect the lack of randomness, and particularly the change in the relationship among variables that characterizes falsified data (Richardson et al., 1995, p. 248-249). Validation surveys may be directed to cases identified through such analysis. Recommendations on using validation surveys, and suggested acceptability levels are provided in section 2.7.5 of the Final Report. 10.6 Q-7: DATA CLEANING STATISTICS 10.6.1 Definition Data cleaning or data checking is an activity that is conducted almost routinely in travel surveys. It involves checking and, where possible, correcting data values that can be identified as being incorrect. It is usually performed as soon after the data are retrieved as possible. This is to enable queries to be made while the information is still fresh in the memories of the respondents. For errors that are caused or accentuated by the survey process, it also allows timely correction. 10.6.2 Data Cleaning and Its Use as a Quality Indicator A review of nine recent travel surveys showed that all of these studies conducted error checking with subsequent call-backs to respondents and correction of data where possible. Thus, it is common practice to perform data cleaning. What is not common is to use statistics of this operation in assessing the quality of the data. If the incidence of errors is assumed to be indicative of data quality then statistics of error incidence can serve in this role.

213 Current practice of detecting and correcting errors in travel survey data tends to vary from survey agency to survey agency. While it is common practice to call respondents back to retrieve missing data on critical data items, the feasibility and logic checks used by agencies, and the practice of calling respondents back on these items, varies from agency to agency. There is also no common definition of what the critical data items are with the result that counting how many “errors” or queries are identified in a survey is not a good measure of data quality. What is needed is a standardized list of data items that are “critical” for the purpose of counting missing values, and a standardized set of checks to detect out-of- range, inconsistent, or illogical responses in data. In addition, the “flagging” of such cases in the data and the statistics that are derived from these values must be a required feature of future travel surveys so that they can be easily detected. Ideally, error checking should be conducted at the time of data collection by the interviewer (Richardson et al., 1995, p. 264). CATI and CAPI surveys can help achieve this by incorporating range, logic, and consistency checks in the program, as well as procedures that detect missing information beyond merely missing information on a data item. For example, if the travel portion of the survey does not include travel on every person in the household, the interviewer should be prompted to verify that the person or persons who reported no trips were indeed immobile during the survey period. In self- administered surveys, illegible writing, misspelled street names, and illogical or inconsistent statements, must be addressed by the reviewer as soon after self-administered surveys are returned as possible. The number of variables differs from survey to survey. In addition, the potential to generate missing values or erroneous responses differs from variable to variable. For these reasons, it is recommended that the estimation of data quality from the data cleaning process be restricted to the data cleaning required among the set of core questions recommended for a travel survey (see “Minimum Question Specifications” in section 4.1). Using a fixed set of variables allows an equitable comparison among data sets. The following index provides a mechanism to measure the incidence of cleaned data items in a data set: questions (core)minimum of number I survey in srespondent of number N otherwise 0 cleaned was n respondent ofitem data i if 1 n respondent ofitem data i (DCS) Statistic Cleaning Data = = ⎭⎬ ⎫ ⎩⎨ ⎧= = ×= ∑∑ th n,i th n,i N n I i n,i )x(count x ,where IN )x(count The DCS statistic above measures the proportion of the core question data that underwent cleaning. It will vary from zero, when no cleaning occurred, to a maximum of 1 when all data on the core questions were cleaned. It is recommended in section 2.7.6 of the Final Report that this statistic be reported in all future travel surveys without specifying what are acceptable and what are not acceptable values of the index.

214 10.7 Q-8: NUMBER OF MISSING VALUES 10.7.1 Definition The number of missing values in a data set is a measure of how much information was not collected. If expressed as a proportion of the total number of data items in the data set, it serves as a measure of the relative information content of the data. Thus, it could be used as a measure of data quality. It is important to define what a missing data item is and what it is not. As described in section 8.3, recommended coding practice is to distinguish between non-responses that are refusals, those where a respondent does not know the answer to the question, and those in which a response would not be applicable. Among these categories, only responses where a respondent either refuses or does not know the answer, are truly missing values 10.7.2 Missing Values as a Quality Indicator The need for standardized procedures arises from the fact that no common practice exists with respect to the definition of missing values and how they may be measured to give an overall assessment of missing information, and hence quality, in a data set. Standardizing these aspects of missing data measurement will allow setting of minimum requirements that would be universally understood and would allow comparison among data sets using a common measure of assessment. Missing values can be defined as data items where respondents have: • Failed to provide a response because they refuse to divulge the information, or • Are unable to provide an answer to the question because they do not know the correct answer. Given this definition of missing values, a missing value index can be calculated that is the proportion of missing data items among all the data items in the data set. That is, the following missing data index can be calculated: set datain srespondent ofnumber variablesofnumber applicablenot is response a if 0 n respondent toapplicable is i variable toresponse a if 1 otherwise 0 missing isn respondent of i item data if 1 Index Value Missing , , * , 1 , 1 1 * , 1 = = ⎭⎬ ⎫ ⎩⎨ ⎧= ⎭⎬ ⎫ ⎩⎨ ⎧= = = ∑∑ ∑∑ == == N I x x MVI where x x MVI ni ni I i ni N n I i ni N n Recommendations on the use of this index as a quality indicator are provided in section 2.7.7 of the Final Report.

215 10.8 Q-9: ADHERENCE TO QUALITY STANDARDS AND GUIDELINES 10.8.1 Background One of the ways to improve the quality of data is to have a checklist of actions that must be performed or standards that must be met in each survey. Such a checklist is not currently accepted or used in reporting on household and personal travel surveys. 10.8.2 Checklist of Quality Indicators An example of a checklist of actions is the “Survey Design Checklist” listed in Appendix C of Richardson et al., (1995). However, a more encompassing set of requirements, which cover all aspects of the survey process from management, through quality control, to survey design, subcontracting, inspection and testing, and product delivery and storage have been suggested by Richardson and Pisarski (1997). Using principles promoted by the International Standards Organization (ISO) and applying them to travel surveys, they have developed a list of 55 aspects of a travel survey that collectively describe adherence to ISO standards (Richardson and Pisarski, 1997, pp. 27-28). A comprehensive checklist of activities or standards that each survey should perform or comply with, will help ensure that individual aspects of the survey are not overlooked or neglected. The degree of compliance with these requirements in each survey can serve as an indirect measure of data quality. If the checklist is standardized, the measure can also be compared among surveys. To be able to use the degree of adherence to quality guidelines as a measure of data quality, the quality guidelines must be defined. Further, to be able to use the measure of adherence from survey to survey, the items that make up the quality guidelines must be fixed. Thus, a need exists to standardize the items that make up the quality guidelines for all travel surveys so that a stable, comparative measure of data quality can be developed. This may prove difficult to do since it depends on the definition of standards on all included items and setting standards on some of these items may be beyond the scope of this project. The items identified by Richardson and Pisarski (1997) form the basis of the items included in this measure, but rather than including all items in that list, it is suggested that a subset of relatively easily-collected item values be used in the analysis. From the original 55 items identified by Richardson and Pisarski (1997), ten questions have been compiled to assess the quality of the survey process. These are listed in section 2.7.8 of the Final Report.