psychometricians, survey sampling statisticians, and methodologists. While the design and measurement processes are complex, the results must be reported in a manner that is useful to policy makers and researchers, yet comprehensible at some level to the general population of parents and teachers.
This complex system of multiple ongoing surveys, with its demand for high levels of both accuracy and timeliness, did not emerge full-blown on the first attempt. Over the four decades of its existence, almost every technical aspect of design and implementation has undergone revision while still, with some exceptions, maintaining trend measures of performance changes over time.
The NAEP provides several lessons for NAOMS. The nature of the concepts that the NAEP measures has changed over time. For example, the content of mathematics topics and skills in 1970 are vastly different from those in 2000. The availability of alternative performance measures based on other tests has also continued over this time, with the introduction of more state testing programs. The NAEP has evolved over time to accommodate these changes, while still regularly producing reliable data that allow trend estimation.
The Behavior Risk Factor Surveillance System (BRFSS), started in 1984, is managed by the Centers for Disease Control and Prevention, and is another example of a survey designed to identify trends over time. It is designed to collect information about health behaviors (known to be related to particular health risks) of the general population of the United States. Unlike the NAEP, which is entirely a centralized enterprise, the BRFSS is designed and managed by the individual states, with federal guidelines. This approach came about for historical and political reasons, as well as for technical and resource considerations.
The BRFSS is a telephone survey of the general population that is repeated annually (through the aggregation of quarterly data), using a core instrument that is supplemented with state-selected questions. From a sample design perspective, the BRFSS has to be concerned with issues of frame coverage. The sample design omits cellphone-only households and, by definition, nontelephone households. These frame shortcomings are not stationary “targets,” but change over time. The cellphone-only households are increasing and are known to have generally younger residents. The number of nontelephone households varies somewhat with the economy, and the group is known to be disproportionately likely to contain young children. The estimation procedures try to extend results, in a defensible manner, to the nontelephone household population.
The survey relies on respondent self-reports of behaviors within a specified reference period. Some of the behaviors (for example, substance abuse) are socially undesirable and thus potentially subject to under-reporting. All of the behaviors are also subject to errors of memory and frequency. These various challenges have necessitated a program of methodological research. This research has included consideration of alternative estimators for extending results to the nontelephone population and of questionnaire design to find out how best to encourage the reporting of socially undesirable behaviors.
The National Household Travel Survey (NHTS), conducted for the Bureau of Transportation Statistics (BTS), measures both the daily and the long-distance travel of the U.S. household population. The NHTS is an example of a survey designed to update and build on information from prior survey series. The earlier surveys focused on either daily travel (Nationwide Personal Transportation Survey) or long-distance trips (American Travel Survey). The NHTS was intended to cover both, as well as walking and biking trips.
The survey was conducted from March 2001 through May 2002. Households were recruited by telephone. The recruited households were each sent a survey form and asked to report all travel by household members on a randomly assigned “travel day.” Telephone interviewers conducted a follow-up interview that asked respondents about their travel on the travel day and the preceding 27 days.
This survey has a number of design issues relevant to NAOMS. It required self-reports of travel, some requiring very detailed information, over a previous reference period. The survey also illustrates one way to deal with a major public event that occurs during the data-collection period and is clearly related to the survey measure-