Technical Appendix to NCHRP Report 571: Standardized Procedures for Personal Travel Surveys (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

39 CHAPTER 3 3. Identification and Categorization of Potential Procedures and Assessment Measures 3.1 APPROACH This chapter discusses specifically which aspects of the survey procedure and which measures of assessment of the survey were considered to have potential for standardization. The items were identified using information gathered and synthesized in the preceding chapters of the report, and by considering the steps in the design and execution of a typical travel survey and assessing the potential that each activity in that process presented for standardization. The specific categories of design and execution used in this process were: • Design of survey instruments; • Design of data-collection procedures; • Sample design; • Pilot surveys or pretests; • Survey implementation; • Data coding including geocoding; and • Data analysis and expansion. The above structure was also used to categorize the items identified for standardization. While identifying opportunities for standardization, the research team was mindful of the need to not “over-standardize” so as to stifle future innovation and improvement. In setting forth these opportunities, therefore, care was taken to look for potential pitfalls that would be stifling to the further growth and development of the personal travel survey. It also became apparent, in examining all of the aspects of personal travel surveys, that some aspects were susceptible to defining and recommending standardized procedures, while others were suitable only for guidance or guidelines. Therefore, the subsequent sections of the report indicate which elements of the survey were recommended for standardized procedures, and which are suitable only for guidelines. It is also clear that the standardized procedures proposed in this project must be revisited from time to time. As more surveys are executed, social mores change, societal habits and values change, and what is considered good consistent practice today may become far below what is considered good practice in the future. In the remaining sections of this chapter, the elements are grouped into three categories – those that are ready for immediate implementation as standardized procedures or guidelines, those that required development into standardized procedures or guidelines within this project, and those that were beyond the scope of this project. These aspects are referenced within the table that summarizes all the potential areas of standardization, so that the context of each can be seen.

40 3.2 EVALUATION OF POTENTIAL AREAS FOR STANDARDIZATION In this section, the potential areas for standardization are summarized, categorized, and evaluated. The evaluation includes an assessment of the level of effort required to implement the standardized procedure, the potential benefits, the expected costs of implementation, and whether a field test was considered necessary or not. The ranking of importance of each potential area of standardization is also provided. The elements are categorized in terms of whether standardization could be accomplished immediately, whether it required further analysis, or whether it was beyond the scope of this project. The level of effort required to implement the procedures were subjectively assessed as being either low, medium, or high. The benefits of standardization were estimated in terms of the contribution standardization was expected to make to achievement of each of the following goals: • Improvement in survey quality; • Improvement in survey data reliability; • Improvement in survey data usefulness; • Improvement in cost effectiveness or value; • Improvement in comparability among surveys; • Improvement in ability to measure survey quality; • Improvement in clarity; and • Improvement in completeness. The benefits are listed in Table 19 (on page 79) in the order of significance for each item. Thus, if comparability is listed first, it is because this is seen as the greatest benefit from standardizing this item. If reliability is last, it is because reliability is seen as the least significant of the benefits that would arise from standardizing the item. The costs were estimated only in broad terms and are categorized as high, medium, low, none, and negative. High costs are those that were expected to lead to increases in the unit costs of a survey on the order of 25 percent or higher. Medium costs were those in the range of 5 to 25 percent, while low costs were those that result in cost increases of less than 5 percent. The category of “none” arises when implementation leads to no increase in cost of a survey, because it involves only a redefinition of a task already undertaken. Negative costs arise in the event that adoption of a standardized procedure or assessment measure is expected to lead to a decrease in unit costs of the survey. Table 19 (on page 66) provides the summary of the potential procedures and assessment measures for standardization. It is important to note that costs are assessed on the basis of increases to unit costs. Some standards may have no effect on unit costs, but may result in overall higher or lower survey costs, while others may affect unit costs but may affect overall costs in the opposite direction. An example of the first of these is the specification of sample sizes, which does not change unit costs, but may increase overall survey costs for those regions that have traditionally used inadequate samples. An example of the second is the number and type of contacts, which is likely to increase unit costs, while decreasing overall costs, as a result of decreases in the amount of sample needed and greater completion of the sample initially selected. To avoid confusion, we have footnoted these types of occurrences. For those tasks where existing data sets are considered adequate to address the issue, or where a fieldwork test is not appropriate, the fieldwork category is indicated as “No”. If a fieldwork test appears to be necessary in addition to working with existing data, the category is indicated as “Yes”; in the event that only a fieldwork test is useful to establish the potential of a standard, then the category is indicated as “Only”. If existing data were thought to be adequate for the task, but a fieldwork test could be beneficial, then the category was marked “Maybe”. Finally, each item was ranked in importance to help select those that could be completed in the project. Items were marked in importance using categories ranging from Very High to Low and from this the research program was derived based on the time and funding available.

41 3.2.1 Categories of Classification The aim of the second task in this project was not only to elaborate on the list of potential procedures and assessment measures for standardization, but also to categorize these into whether a standard was ready for immediate implementation, whether it could be researched sufficiently to adopt as a standard within the time and budget available, or that it was beyond the scope of this project. However, as the categories were considered it became apparent that there were issues and assessment measures where part of it was categorized at one level and part at another level. For example, the issue of survey ethics was assessed as falling partly into the “immediate” category and partly into the “in this project” category. Furthermore, it also became apparent that the meaning of “immediate” was generally not the same as instantaneous, or without additional work. Even those aspects considered to be possible for immediate implementation as standards, e.g., the time of day at which to begin and end the diary period, still would require writing up as a standard, and could involve at least some review by the team and others outside the project. 3.2.2 The Rating Procedure To undertake the rating of potential procedures and assessment measures for standardization, eight criteria were proposed that are related to the benefits of standardization. These criteria are the same for both procedures and assessment measures, but, because a procedure specifies how an activity is to be conducted while an assessment measure measures how well a survey has been executed, the manner in which procedures and measures are evaluated on the criteria is different. The evaluation of a procedure involves measuring the benefit of standardization on the eight criteria. An assessment measure is evaluated by its ability to assess the quality of the survey, including the quality of the data obtained and the effectiveness and efficiency of the process employed. The criteria to evaluate the merit of standardizing procedures are: • The ability of the procedure to promote the quality of the data as represented by the: • Accuracy of data collected; • Accessibility to the data; • Interpretability of the data (i.e., correctly understand the nature of the data); and • Coherence or Comparability of the data (i.e., its consistency in terms of terms, codes, concepts, and procedures). • The ease with which the procedure can be applied (i.e., low level of effort and lack of complexity); • The clarity of the procedure (i.e., nonambiguity or lack of uncertainty regarding the nature of the procedure); • The universality of the procedure (i.e., the applicability of the procedure to the majority of surveys); and • The criticality of the procedure (i.e., the urgency of applying the procedure). For assessment measures, the criteria reflect how well an assessment measure is able to measure the condition or quality of a survey. The criteria that may be used to evaluate an assessment measure are: • The ability of the assessment measure to assess survey accuracy; • The ability of the measure to assess the accessibility of the data; • The ability of the measure to assess how well data is documented so that the nature of the data can be correctly interpreted by a new user; • The ability of the measure to assess the coherence (i.e., comparability or consistency) of the data;

42 • The ease with which the assessment measure may be applied; • The clarity of the assessment measure; • The universality of the assessment measure; and • The criticality of the assessment measure. The procedures and assessment measures identified in this study were assessed on the above criteria using a numeric weighting-and-rating process to provide a single index of assessment. First, the criteria were given weights, and then each potential procedure or assessment measure was rated on each of the eight criteria on a scale from zero to three. Because both of these activities are subjective, all members of the team were asked to review the weights and the individual ratings, to determine a consensus on the aspects and assessment measures. The weights assigned were: • Improve accuracy of the data – 2.0; • Improve accessibility to the data – 1.0; • Improve interpretability of the data – 1.0; • Improve Coherence or Comparability of the data – 2.0; • Ease of use of the procedure – 1.0; • Clarity of the procedure – 0.5; • Commonality or universality of the procedure – 0.5; and • Criticality of the procedure– 0.5. As an example of the use of these, the first candidate procedure for standardization in Table 19 is minimum question specification. This received ratings of 1, 0, 1, 2, 2, 2, 3, 2 on the eight criteria. Applying the weights to these ratings and summing them produces an aggregate total of 12.5, or an average rating of 1.47. The maximum rating that could be achieved, if an item were scored as 3 on all eight criteria is an average score of 3.00. However, in application, no aspect received an aggregate score higher than 18.5, or an average of 2.18, so that this average was considered to indicate an item of high importance. In fact, in the final scorings, quartiles of the aggregate score were used to divide the items into four groups. Values in excess of 18.5 were considered to indicate items of very high importance, those between 13.5 and 18.5 high importance, between 12 and 13.5 medium importance, and those below 10.5 low importance. No aspect scored below 5.5. 3.2.3 Summary Table and Evaluation of Potential Areas for Standardization In subsequent sections of this Technical Appendix, the reference category shown in the leftmost column of the table is used to refer to the item. The initial letter indicates which part of the survey process the item belongs to, while the numeric value was simply assigned in the order in which the items were initially presented and developed in this project. Thus, S-3 indicates that this is the third item in the sample design part of the survey process. Table 19: Summary of Potential Procedures and Assessment Measures for Standardization Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance Design of Survey Instruments I-1 Minimum Question Specification In this project Low ƒ Comparability ƒ Completeness ƒ Usefulness ƒ Reliability None No High

43 Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance I-2 Standardization of Categories In this project Low ƒ Comparability ƒ Usefulness ƒ Reliability ƒ Quality ƒ Clarity None No Very High I-3 Collection of In- Home Activities In this project Medium ƒ Comparability ƒ Completeness ƒ Usefulness ƒ Quality Low Yes Low I-4 Ordering of Questions In this project Medium ƒ Quality ƒ Reliability None Yes Low I-5 Standard Question Wordings In this project Medium ƒ Quality ƒ Clarity ƒ Usefulness ƒ Reliability ƒ Comparability None Yes Very High I-6 Instrument Design In this project/ Beyond scope Medium to High ƒ Quality ƒ Clarity ƒ Usefulness ƒ Comparability Low Yes in future work High I-7 Multi-Tasking of Activities In this project/Beyond scope Medium to High ƒ Usefulness ƒ Comparability ƒ Completeness ƒ Quality ƒ Reliability Low Yes in future work Low I-8 SP Data Beyond Scope High ƒ Comparability ƒ Quality ƒ Clarity ƒ Usefulness ƒ Reliability Low to Medium Yes in future work Low Design of Data Collection Procedures D-1 Number and Type of Contacts In this project/Beyond scope Medium ƒ Cost Effectiveness ƒ Quality ƒ Reliability Low to Medium3 Yes Low D-2 Who Should be Surveyed In this project Medium ƒ Comparability ƒ Usefulness ƒ Quality Low No Medium D-3 Proxy Reporting In this project Medium ƒ Cost Effectiveness ƒ Quality ƒ Completeness ƒ Reliability ƒ Usefulness ƒ Comparability ƒ Measure of Quality Low to Medium Yes High D-4 Complete Household Definition In this project Medium ƒ Quality ƒ Cost Effectiveness ƒ Clarity ƒ Completeness ƒ Comparability ƒ Measure of Quality ƒ Usefulness Medium No Very High D-5 Classification of Contact Outcomes In this project Low ƒ Comparability ƒ Measure of Quality None No Medium D-6 Sample Replacement In this project Medium ƒ Cost Effectiveness ƒ Quality ƒ Measure of Quality Low No High 3 This is an instance where an increase in unit costs should lead to an overall decrease in survey costs.

44 Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance ƒ Comparability ƒ Reliability D-7 Item Nonresponse In this project Medium ƒ Quality ƒ Cost Effectiveness ƒ Comparability ƒ Reliability ƒ Measure of Quality Low Yes Low D-8 Unit Nonresponse In this project/Beyond scope Medium ƒ Quality ƒ Comparability ƒ Reliability ƒ Cost Effectiveness Low Yes Low D-9 Times of Day for Contacts In this project Medium ƒ Quality ƒ Comparability None to Low No Low D-10 Initial Contacts In this project Medium ƒ Quality ƒ Comparability ƒ Cost Effectiveness None to Low Maybe Medium D-11 GPS Surveys Beyond Scope Medium to High ƒ Comparability ƒ Quality ƒ Cost Effectiveness Medium Yes Low D-12 Internet Surveys Beyond Scope Medium to High ƒ Quality ƒ Comparability ƒ Cost Effectiveness Medium Yes Low D-13 Incentives In this project/Beyond scope Low ƒ Quality ƒ Cost Effectiveness ƒ Reliability Low to Medium Yes Low Sample Design S-1 Sample Size In this project Medium to High ƒ Cost Effectiveness ƒ Comparability ƒ Reliability ƒ Quality ƒ Usefulness None4 No Medium S-2 Sizes and Procedures for Augment Samples In this project Medium ƒ Comparability ƒ Quality ƒ Usefulness None5 No Medium S-3 Collecting Augment Samples In this project/Beyond scope Medium to High ƒ Quality ƒ Completeness ƒ Usefulness ƒ Comparability None6 No Low S-4 Stratification Options In this project Medium to High ƒ Cost Effectiveness ƒ Quality ƒ Reliability ƒ Comparability ƒ Usefulness None to Low No Low S-5 Specification of Sampling Error Requirements In this project Medium ƒ Quality ƒ Measure of Quality ƒ Comparability ƒ Reliability ƒ Usefulness None7 No Medium S-6 Default Variances In this project Low to Medium ƒ Quality ƒ Comparability None No Medium 4 Although unit costs will not change, overall survey costs will increase for those cases where the sample sizes are significantly larger than those used prior to establishing standards. 5 Where an augment sample has not been collected in the past, this would increase overall survey costs. Where an augment sample needs to be changed in nature, this could lead to either an increase or a decrease in overall survey costs, but is not likely to affect unit costs in most cases. 6 Will not increase unit costs, but may result in a significant increase in overall survey cost. 7 Will not impact unit costs, but has the potential to increase (or decrease) overall survey costs.

45 Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance ƒ Usefulness ƒ Reliability ƒ Measure of Quality Pilot Surveys and Pretests P-1 Focus Groups In this project/beyond scope Medium to High ƒ Usefulness ƒ Quality ƒ Reliability Low Maybe Low P-2 Requirements for Pretests or Pilots In this project Medium ƒ Quality ƒ Cost Effectiveness ƒ Usefulness ƒ Reliability None8 Maybe High P-3 Sample Sizes for Pretests and Pilots In this project Low to Medium ƒ Quality ƒ Cost Effectiveness ƒ Usefulness ƒ Reliability None7 No High P-4 Sample Sizes for Comparing Methodologies In this project Low to Medium ƒ Quality ƒ Cost Effectiveness ƒ Usefulness ƒ Reliability None7 No High P-5 Reporting of Pretests and Pilots In this project Low to Medium ƒ Usefulness ƒ Comparability None9 No Medium Survey Implementation E-1 Interviewer Training In this project/ Beyond scope Medium to High ƒ Quality ƒ Reliability ƒ Usefulness None to Low No Medium E-2 Ethics Immediate/In this project Low to Medium ƒ Quality ƒ Comparability None No Low E-3 Mailing Materials Immediate/In this project Low ƒ Cost Effectiveness ƒ Completeness ƒ Comparability Low Yes Low E-4 Respondent Questions Immediate/In this project Low ƒ Cost Effectiveness ƒ Quality Low Maybe Low E-5 Caller ID Immediate Low ƒ Quality ƒ Cost Effectiveness ƒ Completeness None Maybe High E-6 Retention of Data on Incomplete Households In this project Low ƒ Quality ƒ Measure of Quality ƒ Reliability ƒ Usefulness None Maybe Very High E-7 Cross-checks in Data Collection and Data Review In this project Low to Medium ƒ Cost Effectiveness ƒ Quality ƒ Reliability ƒ Usefulness ƒ Comparability None to Low Maybe High E-8 Days and Periods to Avoid for Data Collection In this project Low to Medium ƒ Comparability ƒ Cost Effectiveness ƒ Usefulness ƒ Quality None No Very High E-9 Answering Machines and Repeated Call-Back Requests In this project Low ƒ Cost Effectiveness ƒ Comparability ƒ Reliability Low No High E-10 Methods to Reduce Incorrect Reporting In this project?10 Low to Medium ƒ Quality ƒ Reliability Low Maybe Very High 8 None of these items will impact unit costs, but each one may add significantly to the time and cost requirements of the overall survey. 9 This will not impact unit costs, but documentation will add slightly to the overall costs of the survey.

46 Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance of Non-Mobiles ƒ Cost Effectiveness ƒ Usefulness E-11 Reporting Time of Day Immediate Low ƒ Comparability ƒ Reliability ƒ Cost Effectiveness None No Medium E-12 Time of Day to Begin and End Reporting Immediate Low ƒ Comparability ƒ Usefulness None No Low E-13 Creation of Identification Numbers Immediate/In this project Low ƒ Comparability ƒ Usefulness None No Very High Data Coding Including Geocoding C-1 Geocoding Standards In this project Medium ƒ Quality ƒ Completeness ƒ Comparability ƒ Usefulness Low to Medium No High C-2 Level of Geocoding to be Performed In this project Medium ƒ Quality ƒ Comparability ƒ Usefulness Low Maybe High C-3 Geocoding Out-of- Region Addresses In this project Medium ƒ Completeness ƒ Usefulness ƒ Cost Effectiveness Low No Medium C-4 Missing Values, Use of Zero, Etc. Immediate Medium ƒ Clarity ƒ Comparability ƒ Usefulness ƒ Cost Effectiveness None No Very High C-5 Coding Complex Variables Immediate Medium ƒ Clarity ƒ Comparability ƒ Usefulness None No Very High Data Analysis and Expansion A-1 Assessing Sample Biases Immediate/In this project Medium ƒ Measure of Quality ƒ Comparability ƒ Reliability Low No Medium A-2 Weighting and Expansion of Data Immediate Low ƒ Reliability ƒ Quality ƒ Usefulness Low No Medium A-3 Missing Data Imputation In this project/Beyond scope Medium to High ƒ Quality ƒ Comparability ƒ Reliability ƒ Usefulness Low No Low A-4 Data Archiving In this project/Beyond scope Medium to High ƒ Comparability ƒ Usefulness ƒ Cost Effectiveness Low No High A-5 Glossary of Terms In this project Low ƒ Comparability ƒ Clarity ƒ Usefulness None No Very High A-6 Documentation In this project Low ƒ Comparability ƒ Quality ƒ Completeness ƒ Reliability None to Low No Very High Assessment of Quality Q-1 Computing Response Rates In this project Medium ƒ Measure of Quality ƒ Comparability ƒ Reliability None to Low No Very High Q-2 Transportation Measures of Quality In this project Low to Medium ƒ Measure of Quality ƒ Comparability None to Low No High 10 Not a clear potential for standardization.

47 Ref Item Category Effort Req. Potential Benefits Expected Costs Fieldwork Required Import- ance ƒ Reliability Q-3 Coverage Error In this project Low to Medium ƒ Measure of Quality ƒ Reliability ƒ Comparability ƒ Quality None to Low No High Q-4 Sampling Error In this project Low ƒ Measure of Quality ƒ Quality ƒ Reliability ƒ Comparability None No High Q-5 Proxies Immediate Low ƒ Measure of Quality ƒ Quality ƒ Reliability ƒ Completeness ƒ Comparability Low to Medium No Very High Q-6 Validation Statistics In this project Low ƒ Measure of Quality ƒ Quality ƒ Comparability ƒ Reliability Low Maybe Low Q-7 Data Cleaning Statistics Immediate/In this project Low ƒ Measure of Quality ƒ Comparability ƒ Usefulness ƒ Completeness None to Low Maybe Medium Q-8 Number of Missing Values Immediate Low ƒ Measure of Quality ƒ Comparability ƒ Completeness Low No High Q-9 Adherence to Quality Guidelines Immediate/In this project Low ƒ Measure of Quality ƒ Quality ƒ Comparability None to Low No Medium