Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
â4â Matching Design Features to Desired Goals F ROM OUR REVIEW OF THE GOALS of the National Crime Victimiza- tion Survey (NCVS; Chapter 2) and the challenges it faces (Chap- ter 3), we ï¬nd seven fundamental goals to have particular salience and use them as the basis for evaluating various NCVS design options. As we elaborate below, we suggest this as a set of desirable goals; they are cer- tainly not the only possible goals, and others may place different weights on particular goal statements. Four of these goals are historical in nature, in that they reiterate or reï¬ect the various formal task statements recounted in Chapter 2. They are: â¢ Production of a national measure of crime independent of ofï¬cial re- ports to the police; â¢ Provision of information on the context, consequences, and etiology of victimization; â¢ Ability to measure aspects of crime beyond the production of basic, overall rates; and â¢ Utility for producing information on hard-to-measure crimes that are difï¬cult or impossible to detect in police reports. The remaining three goals for the NCVS that we consider to be particu- larly relevant come from current data uses and needs: 81
82 SURVEYING VICTIMS â¢ Capability to readily provide information on emerging crime problems; â¢ Capability to provide small-domain and subnational data of direct in- terest to states or localities; and â¢ Timeliness of resultant data (by which we mean that policy-relevant data can be collected and tabulated sufï¬ciently quickly as to assess emerging trends and inform policy responses). This chapter summarizes the relationships between the goals of the NCVS, the design of the survey, and the implications of various designs in terms of cost, error, and utility. We ï¬rst consider alterations to the current NCVS design that could be put in place quickly for the purpose of saving money, while more substantial changes in design are assessed and imple- mented (Section 4âA). We then consider long-term changes in the form of the NCVS (4âB), outlining a set of survey design packages, some represent- ing relatively minor changes to the current design and others overhauling the basic approach to measuring victimization. Section 4âC describes the trade-offs in cost, error, and utility associated with various design features and presents our general assessments. 4âA SHORT-TERM FIXES: COST REDUCTION STRATEGIES AND THEIR IMPLICATIONS Any thoroughgoing redesign of the NCVS will require research and de- velopment work. At the same time, the survey is in dire straits with respect to funding. The Bureau of Justice Statistics (BJS) has managed this problem by making short-term changes in the survey that will allow it to continue while the panel does its work and while some research and development work is done. These changes are designed to reduce cost with the most min- imal disruption to the survey. However, most have been implemented with no empirical tests of their likely impact, a very risky survey management strategy. In this section, we review a number of cost reduction strategies and assess their implications for the cost, error, and utility of the survey. Some of these strategies have been introduced into the survey and others have not. Table 4-1 describes some short-term changes that might be carried out in order to reduce NCVS costs (some of which are already planned for im- plementation in 2007 as part of the most recent cost-saving efforts by BJS). One of these changesâreduction in sample sizeâhas been the alternative of most frequent resort over the history of the NCVS. The table does not include a change in the reference period of the NCVS (i.e., from 6 to 12 months) even thoughâas we discuss in Section 4âC.1âwe favor it as an alternative to continuing to reduce the NCVS sample size.
MATCHING DESIGN FEATURES TO DESIRED GOALS 83 Two small-group expert meetings convened by Lauritsen (2006b:9) sug- gested reductions in the length of the NCVS questionnaire as a potential source of cost savings. Several of the alternatives we discuss in the next section include attempts to streamline the content of the survey; however, we do not include reductions in questionnaire length in the table of short- term alterations. As the summary of those meetings indicates, cost-saving implications from questionnaire length are not immediately obvious: Efforts to shorten interviewer screening time are likely to produce very small savings because the ï¬eld costs of conducting the screening inter- view are likely to be the same. Eliminating or vastly reducing incident report details for less serious crimes may result in greater savings, espe- cially if such experiences account for some of the ï¬eld costs associated with future efforts to improve retention and participation. In general, condensing the content of the incident report portion of the ques- tionnaire was seen as a better alternative because âit would reduce respon- dent burden and perhaps minimize errors such as survey fatigue and future participation.â However, we again note that large portions of the NCVS costs arise not in interviewing but in contacting and gaining the cooperation of sample households. We think that it is critical to emphasize that even small changes to the design of a survey can have signiï¬cant impacts on resulting estimates and the errors associated with them. Design changes made (or forced) in the name of ï¬scal expediency, without grounding in testing and evaluation, are highly inadvisable. Recommendation 4.1: BJS should carefully study changes in the NCVS survey design before implementing them. We use âstudyâ in wording this recommendation because the appropriate measures may vary based on the speciï¬c methodological changes being con- sidered. The comprehensive redesign of the NCVS included a sufï¬ciently large bundle of changes that implementation was phased in over the course of several years; during that time, data were available using both the old and new survey instruments, for comparison and evaluation purposes. Not all possible changes would require such a lengthy and costly phase-in process, but some would; these include changes in the stratiï¬cation of the sample of addresses or a shift in reference period from 6 to 12 months. A study of proposed changes may include smaller scale testing or the reanalysis and re- calculation using existing data; changes in this class might include decisions on how to count series victimizations in estimates (see Section 3âB.2). As noted in Table 4-1, one adjustment that BJS decided to implement in 2007 (for the production of 2006 NCVS estimates) was to include the ï¬rst
84 Table 4-1 Short-Term Changes to the NCVS to Achieve Cost Reductions Changes to Current Design Cost Error Utility Using the bounding interview in NCVS This would reduce costs if the effective Some increase in standard errors due to Availability of unbounded interviews ad- estimatesâ 1/6 increase in sample was offset by a adjustment. vantageous for methodological research. sample cut. The methodological work on the adjustment would involve costs. Keeping units in sample for an eighth Effectively increases sample size by 1/6, Some increase of time in sample bias. No None. interview which would reduce costs if it was offset adjustments currently made for known by a cut in sample. time in sample bias. Including series incidents in the estimates Reductions in standard errors due to Some reduction in standard errors due to High-end users have always had access to higher victimization rates would permit the increase in the rate of victimization, series incidents. However, including them sample size reductions or other cost especially violent victimization and the in the general estimates could beneï¬t other savings. Some methodological work inclusion of some suspect events that NCVS consumers, as the products would necessary to select maximum number of cannot be accurately allocated to reporting more accurately estimate the magnitude incidents to allow in a series. years or crime category. of such incidents as domestic assaults and assaults at school. Reduce or eliminate allocation to central- Unclear; use of centralized CATI was Pilot work on CATI showed large effect on The mode of interview would be some- ized CATI facilityâ intended as major cost saver, but BJS and reporting, but early implementation sug- what more uniform across the sample. Census Bureau experience suggests no gested that centralized CATI use yielded There might be some reduction in inter- gain over current mix of in-person and higher reported incidence rates (Hubble, view refusals due to caller ID screening. informal CATI interviews. 1999). This seems likely to be due to the automation of the instrument, generally; estimates may not vary much between interviewer-initiated calls using a CAPI instrument and calls from a centralized CATI center. SURVEYING VICTIMS
Sample reductions Cost savings in proportion to the size of Increases in standard errors of estimates Decreasing precision of the estimates for the reduction. commensurate to size of the reduction. violent crime; further reductions would seriously limit usefulness of estimates. Biannual estimates Current BJS reports already combine two Aggregating two years of data would Threatens the relevance of survey as a years of data to assess short-term change; increase the precision of the estimates source of annual information on crime. this would formalize this approach by somewhat. switching to a 2-year release cycle. This would reduce costs if the increases in precision were offset with sample cuts or other cost reductions. Subsampling prevalent incidents for re- This could reduce time in household but ceiving incident form since much of the cost is in contacting households this will not save much. Use police statistics to improve stratiï¬ca- To the extent that stratiï¬cation reduced If stratiï¬cation was used without sample This revision stops short of a wide-scale MATCHING DESIGN FEATURES TO DESIRED GOALS tion in the sample standard errors, commensurate sample cuts, then there would be gains in preci- revision of survey content but is not cuts could reduce costs. sion with changes in sample stratiï¬cation. really a short-term effort. In addition to planning, it would take several years to phase in and implement. * Planned for implementation in 2007. NOTE: CATI, computer-assisted telephone interviewing. CAPI, computer-assisted personal interviewing. 85
86 SURVEYING VICTIMS interview at a sample address in the estimates. The NCVS has always in- cluded some mix of bounded and unbounded interviews in its estimates due to moversâhouseholds that depart an address during that addressâs 3.5-year stay in the survey sample and are replaced with new households; the ï¬rst in- terview with the ânewâ household at an âoldâ address cannot be bounded by the previous interview. Catalano (2007:131â132) observes that the level of unbounded interviews included in the NCVS estimates âhas ï¬uctuated between 10.7 and 14.7 percentage points during the course of the surveyâ; Lauritsen (2005:Note 3) comments that the fraction âis roughly 6% and does not appear to have changed much over the past decade.â Bounding had been the focus of some studies prior to the 1992 NCVS redesign (see, e.g., Biderman and Cantor, 1984; Murphy, 1984; Woltman et al., 1984), but there was a paucity of research on combining bounded and unbounded data in postredesign NCVS estimates. Addington (2005) was able to assess the relative effects of bounding and residential mobility on reported victimiza- tions in what may be the only published, peer-reviewed article in the postre- design era. However, due to its reliance on public-use ï¬les, that analysis was limited to data from an administration of the School Crime Supplement for which incoming respondent (ï¬rst interview) data were included. The deci- sion to include unbounded, ï¬rst interviews in NCVS estimates was made as our panel was being established and assembled, and so we do not think it proper to second-guess it; we understand the ï¬scal constraints under which the decision was made. However, it serves as an example of a seemingly short-term ï¬x with major ramiï¬cations, and it would have beneï¬ted from further study prior to implementation.1 4âB ALTERNATIVES TO THE CURRENT DESIGN IN THE LONG TERM Tables 4-2 and 4-3 present a set of design packages that can be compared with the current NCVS design. The columns of Table 4-2 describe a set of speciï¬c design features on which the packages vary. Each design feature poses some trade-off decision for BJS, offering beneï¬ts of some type, but also potential costs or risks of not fulï¬lling the BJS mission with regard to the measurement of crime. We discuss ï¬rst the columns of Table 4-2, the speciï¬c design features, by pointing out the beneï¬ts and costs of each 1 On December 12, 2007, the ï¬rst 2006 estimates from the NCVS were released. The inclusion of the ï¬rst-time-in-sample interviews, combined with the effects of a new sample based on 2000 census information, led to âvariation in the amount and rate of crime [that] was too extreme to be attributed to actual year-to-year changesâ; BJS concluded that âthere was a break in series between 2006 and previous years that prevented annual comparison of criminal victimization at the national levelâ (Rand and Catalano, 2007:1).
MATCHING DESIGN FEATURES TO DESIRED GOALS 87 feature; assessments of advantages and disadvantages of each design package follow in Table 4-3. We emphasizeâand elaborate in greater detail in Section 4âB.2âthat the set of design packages we discuss in this chapter is not, and is not in- tended to be, exhaustive of all design possibilities for the NCVS. Instead, they are meant to suggest a range of choices and broad visions for the sur- vey. It should also be noted that we do not have the capacity to provide an estimated price for all of these various options; while some can clearly be assumed to be less costly than the current NCVS design, others may not. Our focus is on the mapping of possible design choices to desired goals, and the weight one puts on the end cost of the design (relative to other choices) is important in selecting one; we think that it is our charge to sug- gest possibilities but not to impose any particular weighting of goals and design characteristics. 4âB.1 Characteristics of Possible Design Packages Table 4-2 compares each of these alternative design packages across nine characteristics, ï¬ve related to the general survey design and four speciï¬c to the type of instrumentation used to implement the design. Single Survey Versus System of Surveys The wealth of information needed to help policy analysts and researchers alike is ever changing. One surveyâeven one that is omnibus in natureâ may not be able to handle changing needs. A âsystem of surveysâ is a set of independent but coordinated data collections; in the context of victim- ization, the individual surveys could vary by the type of crime studied, the setting of the crime, or the population victimized. A system of surveys may be better suited to focus on measuring a particular type of crime or a par- ticular social context; a system of surveys may also be amenable to a âquick surveyâ capability, being able to provide meaningful data in a compressed time period (a few weeks, rather than several months or years) in response to critical social events, such as a college or school shooting. A system- of-surveys approach could increase the richness of covariates and so may be more useful for the study of correlates of victimization. However, this greater ï¬exibility must be reconciled with an increased burden of coordina- tion and heightened dependence on the continued existence (and funding) of, and access to, multiple other surveys. By comparison, the single-survey approach allows for focusing of effort and consistency of measurement.
88 SURVEYING VICTIMS General Structure of Sample By âgeneral structure of sample,â we mean the combination of two con- cepts. One is the unit of analysis used in the survey: choices include house- holds, persons, addresses, or phone numbers. The second is the way the sample of those units is selected: for instance, an element survey makes draws directly from a frame or listing of eligible units, compared with clus- tered or multistage approaches. Choices for the sample design affect the efï¬ciency of the data collection or the degree to which respondents can be found at units chosen for the sample. Due to extensive clustering, a multi- stage sample is less efï¬cient than an element sample, as is the case in a pure random-digit dialing (RDD) survey. Oversampling (state, local area, or small population) improves the precision for those areas or populations in which sample is added but not the overall national picture. Under this heading, we also include potential alternatives to the NCVS current design that would serve to boost basic efï¬ciency. One such approach would be adding crime-related questions as supplements to existing surveys, such as the American Community Survey (ACS), National Health Interview Survey (NHIS), or Current Population Survey (CPS), all of which are con- ducted by Census Bureau. Cross-Section or Panel Panel surveys are generally useful to assess the change over time and per- mit longitudinal analysis of change in behavior. The NCVS is a 7-wave panel of addresses, not individuals; this permits analysis of longitudinal behavior of nonmoving households.2 A rotating panel design also permits bound- ing interviews with nonmoving households, using information collected in a prior interview to make sure that events are not duplicated. Rotating panels are less costly than the repeated cross-sectional surveys and produce lower cost per interview. However, the advantage of rotating panels on response rates is likely to be a function of how frequently sample households are vis- ited; households may drop off in their cooperation with a survey the longer they remain in the sample. By comparison, repeated cross-sectional surveys may have a lower response rate but can be more ï¬exible in terms of content and design modiï¬cations. 2 It is important to note that the NCVS panel of addresses is different from the panel of persons used in some other surveys, such as the Survey of Income and Program Participation and the National Longitudinal Survey of Youth. Although a panel-of-persons structure would be ideal for capturing within-household experiences of victimization (and reactions to them), a panel of addresses is a substantially less costly compromise approach.
MATCHING DESIGN FEATURES TO DESIRED GOALS 89 Reference Period The NCVS currently asks respondents to report victimizations occurring in the six months prior to the interview (âa 6 month reference periodâ). The current practice came about as part of the 1992 NCVS redesign and was intended as a simpliï¬cation for respondents. Prior to the redesign, the survey questionnaire emphasized boundaries on the reference period, ask- ing respondents âto report for the 6-month period ending at the end of the previous month and beginning 6 months priorâ (e.g., JanuaryâJuly for an interview conducted in August). The new, and current, questionnaire im- plicitly asks respondents to report victimizations up to and including the day of the interview by referencing only the past 6 months (Cantor and Lynch, 2000:112). By comparison, most other ongoing victimization surveys (e.g., the British Crime Survey and the International Crime Victimization Survey; see Appendix E) use a 12-month reference period. Between 1978 and 1980, the Census Bureau conducted a reference pe- riod research experiment using parts of the National Crime Survey sample; see Bushery (1981a,b); additional early work on reference period under the initial design is described by Singh (1982) and Woltman and Bushery (1984); see also the summary by Cantor and Lynch (2000). Mode Telephone interviewing reduces the proportion of time that interviewers are engaged in noninterviewing activities (i.e., travel). Thus, the mode tends to be cheaper per interview than face-to-face interviewing. However, the telephone model appears to be more sensitive to longer interviews, exhibit- ing higher break-off rates and partial interviews. Furthermore, the telephone prohibits use of visual aids in measurement, such as the ï¬ashcards currently used in the NCVS to clarify responses to some questions (e.g., race, His- panic origin, employment, education) and provide respondents with infor- mation about survey privacy (U.S. Census Bureau, 2003:A7-6âA7-9). For some types of populations (e.g., those in multiunit structures and walled subdivisions), the telephone might yield higher contact rates, but for oth- ers there appears to be no mode difference. The face-to-face mode, while costly, generally is viewed as yielding the highest cooperation rates. The self-administered mode (e.g, paper questionnaires, web administration) is gaining increasing attraction. Self-administration often has very low per- unit costs. Self-administration is found to yield more honest reporting on sensitive topics (e.g., domestic violence, rape).
90 SURVEYING VICTIMS Screener and Incident Form The current NCVS separates the functions of counting and listing vic- timization events from more detailed classiï¬cation of those crime events through its separate screening questionnaire and incident report. This sep- aration of screening from classiï¬cation is an effective way of promoting re- call and ï¬ltering out ineligible events; moreover, the additional information collected in the more detailed incident form permits a fuller description of crime events and their context. It is possible for surveys to take a different, more omnibus approach, integrating screening and classiï¬cation in one pass. Surveys that use responses to screening questions to classify events may be less burdensome, but they may yield data of lower quality. (Further discus- sion of the nature of the screener portion of the questionnaire follows under âEvent History.â) Core and Supplement Some surveys have a set of questions that are consistently asked of all re- spondents, sometimes labeled the âcore.â The full survey questionnaire con- tains core questions and a rotating set of supplement questions. Scheduled supplements allow topical reports from the survey, enriching the breadth of reports. These supplements might change over time, to reï¬ect the changing nature of crime. Subsampling Frequent Events In the current NCVS design, respondents are asked to complete an in- cident report for every victimization incident counted during the screening portion of the questionnaire. A possible approach to reduce overall inter- viewing time (and hence lower costs somewhat) is to subsample the most frequently occurring victimization types (e.g., theft, simple assault) and to collect the detailed incident report data for only some of these incidents. The subsampling rate could depend on the frequency with which a particu- lar crime is reported as well as complex structures involving multiple crimes. For example, a matrix sampling framework could be used to subsample inci- dent reports to preserve certain associations or joint occurrences. Probability sampling of such events permits estimation of standard errors of estimates. Subsampling could be implemented with slight modiï¬cation of the current computer-assisted instrument used by the NCVS. Event History We include âevent historyâ as a category as one relatively new technique in survey methodology that may be useful to consider in promoting accu-
MATCHING DESIGN FEATURES TO DESIRED GOALS 91 racy in the screening portion of the NCVS interview, in addition to contin- ued emphasis on completeness of recall. As described in Section 3âB.1, the problem of respondent recallâthat survey respondents often have difï¬culty recalling the number or exact timing of events and that telescoping may occur as a resultâis a long-standing one in survey research. Accordingly, a major objective of the 1992 NCVS redesign was to improve the screen- ing portion in order to promote completeness of reporting. As Cantor and Lynch (2005:296) observe, the structure and wording of the screener ques- tions made the new NCVS the ï¬rst federal statistical survey to be designed based on psychological models of the survey response process, or what have come to be known as cognitive aspects of survey measurement (see, e.g., National Research Council, 1984; Sudman et al., 1996; Sirken et al., 1999; Tourangeau and McNeeley, 2003). Speciï¬cally, the new NCVS screener em- phasized the cognitive steps of comprehension and retrieval, adding a large number of âshort cuesâ and structuring the interview into multiple frames of reference (Cantor and Lynch, 2005:297â298). Martin et al. (1986) re- view screening procedures considered in the redesign. (In addition to the revisions to the screening questions, the NCVS retained its high-level ap- proach to dealing with recall problems by withholding the ï¬rst interview at an address and using it only to âboundâ responses given in the next in- terview, until that practice was changed for the production of 2006 NCVS estimates.) In recent years, survey research has suggested methods for structur- ing and designing questionnaires that can improve the recall and dating of events. These event history methods typically involve the use of calendar- type structures in questionnaires, along with special cues, that permit re- spondents to make best use of thematic groupings and landmarks in the way memories are coded in the human mind. See, for example, Belli (1998) for an overview of event history methodology and Belli et al. (2001) for a direct comparison between event history and standard âquestion listâ methods. The implementation of event history methods in the NCVS would require major restructuring in the existing screener portion of the survey. However, they could become particularly important if respondents were asked to recall victimization incidents over a longer time frame, as would be the case with a switch from a 6-month to a 12-month reference period. Although they tend to improve recall of events, event history methods generally require more interviewing time. They are also generally easier implement in face-to-face interviewsâa setting in which a ï¬ashcard or a questionnaire section struc- tured in the form of a calendar gridâthan in telephone interviewing, which is the mode most commonly used in NCVS interviews. However, event his- tory methods have been developed for use in telephone surveys; Belli et al. (2007) compare the implementation of an event history approach for tele- phone interviews in the Panel Survey of Income Dynamics with standard
92 SURVEYING VICTIMS telephone interviewing methods and ï¬nd beneï¬ts in improved retrospective reporting. The implementation of event history techniques in the NCVS may improve the accuracy of the data but may also increase the cost, and so would need to weighed and dealt with in the context of other modiï¬cations adopted by the NCVS. 4âB.2 Survey Design Packages for Comparison There are many possible combinations of speciï¬c survey design features that satisfy the different characteristics that form the columns of Table 4-2. We focus on 10 survey design bundles that, in the panelâs view, contain de- sign features that are compatible and form a cohesive design option. Several of them are modeled after data collection efforts already in place, whether in other countriesâ victimization surveys or in other social surveys of the American public. The rows of Table 4-2 describe a set of alternative structures for the NCVS, beginning (with item (0)) with the current design. Core-Supplement Models (1)â(2) Core-supplement models would retain the rotating panel design of the current NCVS (contacting the same households for several interviews) but differ from the current NCVS in the emphasis placed on topical supple- ments. They are a partial implementation of the British Crime Survey (BCS) model in that they would make topical supplements a regular, built-in part of the survey structure (paired with a streamlined core NCVS questionnaire) instead of being conducted on an irregular and as-available basis. We dif- ferentiate between (1) a core-supplement model that would maintain the 6-month reference period of the existing NCVS for maximum continuity of estimates and (2) a model that would shift to a 1-year reference period (and fewer interviews with the same household). British Crime Survey Type (3) The British Crime Survey type model (3) is built around a core- supplement design but would represent a major shift from the current NCVS as it would abandon the rotating panel design for an annual cross-section sample. It would also potentially include the addition of event history meth- ods to improve recall of victimization incidents. This option is described as âBCS Typeâ rather than a pure replication of the BCS design, because some key parts of the design are left unspeciï¬ed (and subject to change). Specif- ically, decisions under this model type would have to be made on whether the survey is administered by a government agency or by a private ï¬rm (the BCS is contracted out) and whether interviews are collected continuously
MATCHING DESIGN FEATURES TO DESIRED GOALS 93 throughout the year or targeted at one speciï¬c time interval (the BCS, like the current NCVS, uses continuous interviewing). Local-Area Boost Models (4)â(6) Local-area boost models would maintain the existing NCVS structure but would oversample some areas or populations on a rolling basis so that subnational estimates could be produced. Given the sample sizes involved, these subnational estimates would most likely be multiyear averages of the sort produced by the Census Bureauâs American Community Survey; the ACS combines data for 36 and 60 months of data collection in order to pro- duce estimates for geographic areas as small as census tracts. In the table, we differentiate between (5) a local-area boost that would focus on states as the natural unit of interest and (6) one that would permit more ï¬exible alloca- tion of rolling sample, perhaps to generate estimates for large metropolitan statistical areas. International Crime Victimization Survey Type (7) While packages (0)â(6) generally maintain the current structure of the NCVS, packages (7)â(10) are substantially more aggressive redesigns that would effectively end the NCVS as it is currently known. The International Crime Victimization Survey (ICVS) is an ongoing semi- annual survey administered by the United Nations Interregional Crime and Justice Research Institute. Its methodology is described in more detail in Sec- tion Eâ2 in Appendix E. Relative to the current NCVS design, an ICVS-type model (4) involves at least two major compromises: it would abandon both the rotating panel design and the potential face-to-face interviewing capa- bility of the current NCVS, focusing instead on a cross-sectional telephone- only survey.3 Replication of the ICVS one-stage screening process would also probably substantially reduce the quality of reporting in the survey. However, the instrument for such a design could fairly readily accommo- date topical supplements. The ICVS-type model we describe here is not a pure replication of the ICVS, as that would abandon annual victimization measures (the ICVS is conducted on a periodic basis only, about every 4â5 years). Partnership Model (8) The option that we call the âpartnership modelâ would diffuse vari- ous topics into the current NCVS into different data collection vehicles. 3 As described in Section Eâ2, personal interviewing is carried out on a small scale in a few ICVS participant countries, and then only in the countryâs capital city.
94 Table 4-2 Current and Possible Alternative Designs for the NCVS Survey Design Instrumentation Sub- Single Cross- Screener Sampling Survey vs Section Reference Incident Core and Frequent Event Description System Sample or Panel Period Mode Form Supplement Events History (0) Current NCVS Design Single Household multistage cluster Rotating 6 months CATI/ Yes No No No panel CAPI Core-Supplement Modelsâstreamlined core questionnaire with structured topical supplements (1) Simple Single Household multistage cluster Rotating 6 months CATI/ Yes Yes Yes No panel CAPI (2) Longer Reference Single Household multistage cluster Rotating 12 months CATI/ Yes Yes Yes No panel CAPI (3) BCS TypeâCross-sectional design Single Household multistage cluster Repeated 12 months CAPI Yes Yes Yes Yes with a 12-month reference period, event cross- history methods, routine supplements section Local-Area Boost ModelsâCross-sectional design with a 12-month reference period, event history methods, and routine sup- plements, plus rolling sample to support subnational estimates (4) State BoostâPeriodic increase of Single Household multistage cluster Cross- 12 months CAPI Yes No Yes Yes sample to tiers of states sample clustered by states with section states rotating in and out (5) Other-Area BoostâPeriodic sample Single Household multistage cluster Cross- 12 months CAPI Yes No Yes Yes increases in some other subnational sample; additional sample allo- section groups (e.g., SMSAs or agglomera- cated to subnational areas rotat- tions like ârural statesâ) ing in and out (6) Boost with SupplementsâCross- Single Household multistage cluster Cross- 12 months CAPI Yes Yes Yes Yes sectional design with a 12-month sample clustered by states with section reference period, event history states rotating in and out methods, core and supplement format, a quick survey capability and a sample clustered to provide state-level estimates for rotating group of states SURVEYING VICTIMS
(7) ICVS TypeâPure ICVS replication Single Households; RDD Repeated 12 months/ 5 years CATI No Yes No No would be conducted every 4â5 years and cross- use both 5-year and 1-year reference section periods (8) Partnership ModelâSystem of victim- System 1) add current screener-type ques- Either Varied Mixed Yes NA NA ization surveys with a supplement to an tions for crimes well-reported to ongoing household survey and a series of police (e.g., robbery, burglary, interrelated surveys focused on particular motor vehicle theft) to other na- crimes or populations tional survey vehicle, like ACS or CPS; 2) routine surveys of domestic violence, workplace, and schools, possibly including general âharmâ survey to include assault; 3) episodic special topic surveys on looming issues (9) Surveillance ModelâDistributed sys- System 1) Individual state surveys would Either â Mixed Yes â No â tem of surveys, administered by central follow core content, with annual core but conducted through BJA-type appropriations 2) NCVS program grants to state/university consortia would administer content. Ide- ally, there would be a continuing, small-level national sample for calibration; a pure version of this MATCHING DESIGN FEATURES TO DESIRED GOALS model would make the âNCVSâ strictly the compilation of the state-level surveys (10) Periodic Survey without incident Single Phone (likely RDD) Cross- â Mixed No No No No reports: Cross-sectional design with section changing annual content based on policy interests; not expressly limited to victim- ization but could include questions on perception of risk NOTES: ACS, American Community Survey. BCS, British Crime Survey. BJA, Bureau of Justice Assistance. CAPI, computer-assisted personal interviewing. CATI, computer-assisted telephone interviewing. CPS, Current Population Survey. ICVS, International Crime Victimization Survey. RDD, random-digit dialing. SMSA, standard metropolitan statistical area. 95
96 SURVEYING VICTIMS A basic premise of this model is that it would still be useful to have some measureâindependent of the policeâof certain crime types that are gen- erally well reported to police, such as robbery, burglary, and motor vehi- cle theft, but that it might not be necessary to gather speciï¬c incident-level data on these crimes. Hence, the partnership model would shift the basic screener questions for these crime types to some other national surveyâ possibly the American Community Survey (ACS) or the Current Population Survey (CPS), both conducted by the Census Bureauâand waive the collec- tion of detailed incident data for them. Having shifted measures of some crimes to other surveys, a series of smaller scale, interrelated surveys could be targeted to speciï¬c crimes or populations. This would have the beneï¬t of training resources on crime types for which a sample survey is best suited (relative to reliance on admin- istrative or facility records). For example, some continuity in the measure of assault could be preserved though a smaller scale harm survey, perhaps combining information from a smaller scale survey with emergency room data. Although the choice to focus efforts on individual surveys for speciï¬c crime types would ideally be driven by the desire to increase the quality of information, it could also come about as the result of cost cutting. Hence, the number and frequency of these smaller surveysâand the ï¬scal resources dedicated to themâwould be a major consideration. Surveillance Model (9) While the partnership model differs by spreading the NCVS topic area content over a number of alternative sources, the surveillance model (9) maintains fairly tight control over topic area content but disperses the data collection task. This model is based mainly on the Centers for Disease Control and Preventionâs (CDC) Behavioral Risk Factor Surveillance Sys- tem (BRFSS) (described more fully in Box 4-1), in which CDC provides funding to state health departments to collect data. It is also based in part on the Federal-State Cooperative Program for Population Estimates (FSCPE) between the Census Bureau and state demographic units, which has become an important partnership agreement for improvements in various population estimate and general census processes. Most signiï¬cantly, though, we consider this model because of BJSâs unique placement relative to other federal statistical agencies: inside the Of- ï¬ce of Justice Programs, whose core mission is providing assistance to state and local law enforcement agencies. Under the BRFSS, CDC provides core funding and central coordination to state health departments to administer a telephone survey. Likewise, under this model, BJS and the Bureau of Justice Assistance would provide funding and central support to states (or consortia of states) to collect victimization data, either in-house or through subcon-
MATCHING DESIGN FEATURES TO DESIRED GOALS 97 tract with private or university survey research organizations. In return for administering a core set of questions and, optionally, topic supplements, states could add their own questions to the survey. As a state-based system, the states (or groups of states) would also have direct estimates at their level of geography. If this network of afï¬liates were to provide full coverage of the nation, the state-level data could be pooled to provide a national-level estimate. Crime Poll (10) A ï¬nal option follows the example of the Texas Crime Poll, a semiannual data collection that is directly intended to inform speciï¬c legislative issues at the state level; the poll is described in more detail in Box 4-2. Alternately, it can be viewed as promoting a set of questions that have historically been handled through supplements to the NCVSâquestions on popular attitudes on the extent of crime and perceptions of public safetyâto be the exclusive contact of the victimization survey. This option is the starkest contrast with the current NCVS design, as it would involve giving up the goal of producing annual rates of criminal victimization and the collection of detailed incident information. However, it is alsoâalmost certainlyâthe lowest cost alternative considered in this table. Depending on its implementation and how close it follows the Texas example, it could also arguably be the most responsive to speciï¬c legislative directives, the least burdensome on individual respondents, and would be readily amenable to questions on attitudes on crime and justice issues other than victimization. The speciï¬c implementation envisioned by the model would be a representative cross-sectional survey, conducted by telephone. Other Design Possibilities We do not intend these 10 design packages to be exhaustive of all alter- native design possibilities; rather, these 10 are chosen to illustrate a range of design choices. In our judgment, they constitute packages that merit ï¬rst review, either because they force attention to their impact on alternative goals of the NCVS (and thus sharpen thinking on goals) or focus attention on innovations that could reduce the costs or improve the measurement ef- ï¬ciency of the NCVS (and thus focus attention on process). We do not wish to exclude other combinations of design features; indeed, detailed consider- ation of the 10 options may identify a combination of design features that is preferable to any of the 10 packages identiï¬ed here. Thus, the 10 op- tions are a ï¬rst step in a decision process to choose which of the goals of the NCVS should be emphasized and which should be deemphasized in the future.
98 SURVEYING VICTIMS Box 4-1 Behavioral Risk Factor Surveillance System The Behavioral Risk Factor Surveillance System (BRFSS) is a collaborative data collection project in which data are collected by state health departments through monthly telephone interviews, with funding and technical and methodological coordination from the Behavioral Surveillance Branch of the Centers for Disease Control and Prevention (CDC). Initiated in 1984 with 15 participating states, the BRFSS had participation by all 50 states, the District of Columbia, Puerto Rico, Guam, and the U.S. Virgin Islands as of 2001. The system is intended to measure risk behaviors and preventive practices in the adult population (18 years of age or older). The BRFSS has a set of core questions that are agreed to by the states and CDC and that are administered in all states without modiï¬cation. Since a 1993 redesign, some of these core questions are ï¬xed and asked every year while a ârotating coreâ set of questions is asked every other year. In addition, some slots on the core questionnaire are reserved for âemerging issues questions.â The individual states may also elect to add optional topical modules that are developed by CDC; optional modules were ï¬rst ï¬elded in 1988. Other branches within CDC sponsor questions in the BRFSS, as do external agencies such as the National Institutes of Health and the Department of Veterans Affairs. States are also permitted (and encouraged) to add their own questions to the instrument without direct CDC editing; the state health departments may obtain funding from other state agencies to place questions on the BRFSS questionnaire. BRFSS samples are drawn by CDC from a commercial telephone number database, or they may be constructed by the states on their own as long as they comply with general standards. CDCâs objective is to support at least 4,000 interviews per state, although effective sample sizes vary; in 1995, Iachan et al. (2001:Table 1) report that state sample sizes ranged from 1,193 (Montana) to 5,107 (Maryland). Most state health departments let contracts to commercial or university survey research units to conduct BRFSS interviews; in 2006, only 14 state health departments conducted the interviews in-house. The BRFSS is intended to generate state-level estimates, and over time the BRFSS has sought to provide both higher and lower level estimates. As an amalgam of state samples, the BRFSS is not intended to provide direct national estimates; ânevertheless,â Iachan et al. (2001:221) observe, âthere is much interest among the research community in using the BRFSS for such estimates because all states use the same core instrument, sample size is relatively large, and the annual data are available within 6 months after collection.â Iachan et al. (2001) discuss different methods for pooling the state samples and ï¬nd consistent results between their national BRFSS estimates for some items with corresponding items from the National Health Interview Survey. In 1997, pressures for estimates below the state level resulted in the Selected Cities Project and, since 2002, the Seletcted Metropolitan/Micropolitan Area Risk Trends (SMART) program. Through the project, estimates have been generated for counties and metropolitan/micropolitan statistical areas in which at least 500 BRFSS interviews are completed in a year; estimates were available for 99 cities in 2000, and SMART estimates were derived for 145 statistical areas and 245 counties in 2006.
MATCHING DESIGN FEATURES TO DESIRED GOALS 99 Box 4-1 (continued) The BRFSS has been used to collect data on some incidents of violence. A module of questions on intimate partner violence and injuries was implemented in the state of Washington in 1998 (Bensley et al., 2000). In 2005 and 2006, CDC offered both an 8-question sexual violence module and a 7-question intimate partner violence module; 29 states or territories opted to use one or both of these modules during 2005â2006. Individual states have periodically added their own questions on violence-related matters, including generic questions on injury and on access to ï¬rearms. SOURCES: Centers for Disease Control and Prevention (2003, 2005, 2006a,b, 2007). Other design features could be explored or adjusted to forge additional design options: â¢ Variations in the number of repeated interviews conducted at a sample address (and, correspondingly, the length of time an address remains in sample) can be made within the framework of a rotating panel design as is currently used in the NCVS. Although the nature of the current NCVS as a longitudinal sample of addresses is one of its key attributes and has provided the basis for bounding interviews, that longitudinal structure is an underutilized feature. NCVS data ï¬les have not com- monly been constructed in linked, multiyear longitudinal segments; most recently, this was done for 1995â1999 data (Bureau of Justice Statistics, 2006b). Some of the designs we consider in Table 4-2 would switch from a rotating panel to a cross-sectional design, but it is also possible to conceive designs that would emphasize longitudinal struc- ture (e.g., conductingâin part or in wholeâa longitudinal sample of people or households, making efforts to contact the same respondents regardless of whether they move.) â¢ Implicit in what we call the partnership model (8) is the notion of shift- ing NCVS screener-type questions to another survey, like the Ameri- can Community Survey (ACS) or Current Population Survey (CPS). This is one way in which an alternative mode of survey collectionâ via the mailâcould be used in victimization framework, but not the only one. Particularly if the NCVS continues to be collected by the Census Bureau, one approach could be to use the ACS or the CPS as a prescreener: construction of the NCVS sample could be targeted based on âyesâ or ânoâ responses to victimization questions on the other survey. (Obviously, the total burden on respondents who would then be included in multiple federal surveys would have to be taken into account.) More generally, it is possible to consider detaching the NCVS screening questionnaire by mail and then performing follow-up by phone or personal interview (although this would hurt the cuing
100 SURVEYING VICTIMS Box 4-2 Texas Crime Poll First conducted in 1977, the Texas Crime Poll is a semiannual public opinion survey administered by the Survey Research Center, Criminal Justice Center, Sam Houston State University. The poll is authorized by the 1965 legislation that created the Criminal Justice Center, which required the center to conduct âsurveys of pertinent problems in the ï¬eld of crime, delinquency and correctionsâ (Teske and Lowell, 1977:1). By design, âthe format [of the poll] remains the same each time and many of the questions are replicated on a regular basis in order to allow for measurement of changes in public opinion. Other items are topical and are included only onceâ (Teske and Lowell, 1977:1). Basic attitudinal questions (e.g., 1977âs âOver the past three years, do you feel the crime problem in your community is: â¢ Getting better â¢ About the same â¢ Getting worse?â) are among the questions that are repeated regularly. Topics covered by the poll questions have ranged from the appropriateness of capital punishment to perceptions of obscenity to trust in the stateâs use of forensic science in prosecutions. Because the poll is meant to help inform state legislative issues, question text can become detailed and complex. For instance, question 1.7 on the 2007 poll read: Exempting criminal justice professionals from tuition and fees would cost some universities as much as $500,000 per year. If you responded âYesâ to either items 1.5 or 1.6 above, which of the following best represents your thoughts on how these losses should be addressed? (Select only ONE of the following.) â¢ Additional legislation should be passed to compensate the universities for these losses. â¢ Tuition and fees for other students should be increased to compensate the universities for these losses. â¢ The universities should consider these âlossesâ as their contributions to the development of criminal justice professionals and adjust their budgets to accommodate them. â¢ Other options: Please specify: Area for free response The Texas Crime Poll was originally conducted as a systematic random sample from the frame of persons with valid Texas driverâs licenses, collected by mail; in 1977, the poll included 642 respondents (a 67 percent response rate). As of 2007, the poll remains a mail survey but its frame is based on âwhite pagesâ telephone listings. From these listings, the sample is drawn systematically, stratiï¬ed by county. In its two most recent iterations (2004 and 2007), the sample design was altered in an attempt to increase the representation of minorities in the sample. In 2007, the poll included two sample groups of 1,500 people: one intended to be generally representative of the whole population and the other drawn speciï¬cally from those area code listings corresponding to areas with high-density Hispanic concentrations in the 2000 census. Respondents also were encouragedâin a follow-up postcardâto complete an Internet version of the questionnaire. However, the 2007 poll received only 332 total responses out of 2,874 valid questionnaires (11.6 percent), a decline in response rate from 2004âs 22.8 percent. Documentation and data sets for the Texas Crime Poll are archived at http://www.cjcenter. org/cjcenter/research/srp/txpi.html.
MATCHING DESIGN FEATURES TO DESIRED GOALS 101 and recall structures built into the current structure, doing both the screener and incident form in the same administration). The panel thinks that deliberate and rigorous evaluation of the design packages listed above will generate other ideas for packages that balance costs of the NCVS and the quality of key NCVS estimates. Such evaluation requires leadership within BJS and the statistical system to assemble the sup- port for professional review of the key data series. Quick decisions in the absence of evaluation carry with them great risks of misleading the country with error-ï¬lled estimates. A careful examination is required and consistent with the principles of a federal statistical agency. 4âC ASSESSMENTS OF DESIGN FEATURES AND PACKAGES Table 4-3 summarizes our basic assessment of how well the design pack- ages described by Table 4-2 satisfy seven desirable goals, as articulated at the outset of this chapter. Table 4-3 also lists basic advantages and disadvantages of each design. A basic observation from Table 4-3 is that our assessments in the table suggest an underlying difï¬culty concerning the goal of timeliness of the re- sultant data. An ideal design with respect to timeliness is one that produces estimates that can shed light on new and emerging crime trends (and types) and inform policy strategies; the design should be nimble, with the capacity to add new questions in a relatively short amount of time in order to pro- vide âquick surveyâ empirical data on emerging issues. Yet we judge even the more streamlined core-supplement models as being ill-suited to this ideal timeliness goal; the only design that meets the goal adequately is the crime poll (10) design in which âquick surveyâ capability is the entire measurement goal. Furthermore, the table suggests two other, related conclusions that de- serve emphasis because they shape the discussion that follows. The ï¬rst is that there is no especial magic to any of the proï¬led designs: none of the listed design packages is optimal across all of the desirable goals, and each represents certain trade-offs and compromises. As we note below, there are speciï¬c design elements that we think are worthy of consideration in the NCVS, but we do not think that any of the full-blown design packagesâas a wholeâis uniquely superior to the others. The superiority of packages de- pends on oneâs choices of goals and the weight that is placed on them; hence, scoring the alternative models and picking a âbestâ one is not as simple as counting the numbers of shaded boxes in the table. We do not presume that our set of listed goals is uniquely correct, or that other stakeholders would provide exactly the same assessments as we do, and so do not intend to try to justify a single package as better than all of its competitors.
102 Table 4-3 Goals of the NCVS and Alternatives to the Current Designs â¢ indicates that the panel believes that the goal is well served by the design package; â¢ that the goal is adequately served; Ã that the goal is poorly served. Natâl Vict. Info Design Measure Context Emerging Detail for Beyond Hard-to- and Indep. of and Crime States and Crime Measure Products Police Etiology Problems Localities Rates Timely Crimes Pros Cons Current NCVS â¢ Annual estimates â¢ â¢ Ã Ã â¢ Ã â¢ â¢ High-quality national estimates of â¢ Limited detail on crime events rates for âstreet crimesâ â¢ Inï¬exible information content of level and â¢ Long, stable time series â¢ Not particularly timely change â¢ DoJ controls survey content â¢ Scope of basic BJS reports restricted â¢ Special reports â¢ Omnibus vehicle reserves resources to rate estimation, though data have on social context for victimization data been used extensively in other ways of crime and on â¢ Limited coordination with other â¢ Expensive speciï¬c types of agencies or entities required â¢ Little information of direct use to crime and states and localities subpopulations of victims Core-Supplement (simple) â¢ â¢ â¢ Ã â¢ Ã â¢ â¢ DoJ controls survey content â¢ Development and ï¬elding of â¢ Omnibus vehicle reserves resources supplements will add to survey cost â¢ Annual estimates for victimization data â¢ Some increases in coordination with of level and â¢ High-quality national estimates of other agencies change rates for street crimes â¢ Additional stafï¬ng for BJS â¢ Regular series of â¢ Long, stable time series for core â¢ Little information of direct use to topical reports items states and localities based on â¢ More information on context, scheduled etiology, and consequences supplements â¢ More information on emerging crime problems â¢ More information on issues beyond rates â¢ Reduced cost from trimming core SURVEYING VICTIMS
Table 4-3 (continued) BCS Type â¢ â¢ â¢ Ã â¢ Ã â¢ â¢ Save money with 12-month reference â¢ Reduction in quality of national period estimates â¢ Greater ï¬exibility in information â¢ Lower response rates in cross-section content â¢ Increase in recall bias with longer â¢ DoJ controls survey content reference period â¢ Omnibus vehicle reserves resources â¢ Telescoping without bounding for victimization data interview â¢ Limited coordination with other â¢ Complicates introduction of agencies or entities required self-administered technology â¢ Loss of information on all household members, due to one respondent per household Local-Area Boost â¢ â¢ â¢ â¢ â¢ Ã â¢ â¢ High-quality national estimates of â¢ Lower response rates in cross-section rates for street crimes â¢ Greater recall loss from longer â¢ More information on context, reference period etiology, and consequences, but not â¢ Modest increased telescoping with as much as core-supplement event history â¢ Development and ï¬elding of MATCHING DESIGN FEATURES TO DESIRED GOALS â¢ More information on emerging crime problems supplements will add survey cost â¢ More information on issues beyond â¢ Additional stafï¬ng for BJS rates â¢ Greater ability to provide state and local estimates and other subnational data â¢ DoJ controls survey content â¢ Omnibus vehicle reserves resources for victimization data â¢ Limited coordination with other agencies or entities required 103
104 Table 4-3 (continued) Natâl Vict. Info Design Measure Context Emerging Detail for Beyond Hard-to- and Indep. of and Crime States and Crime Measure Products Police Etiology Problems Localities Rates Timely Crimes Pros Cons Boost with Supple- ments â¢ â¢ â¢ â¢ â¢ Ã â¢ â¢ High-quality national estimates for â¢ Some reductions in quality of rates of street crimes information â¢ More information on context, â¢ Lower response rates in cross-section etiology, and consequences but not as â¢ Greater recall loss from longer much as core-supplement reference period â¢ More information on emerging â¢ Modest increase in telescoping with crime problems cross-section and event history â¢ More information on issues beyond â¢ Development and ï¬elding of rates supplements will add survey cost â¢ Greater ability to provide state and â¢ Additional stafï¬ng for BJS local estimates and other subnational data â¢ DoJ controls survey content â¢ Omnibus vehicle reserves resources for victimization data â¢ Limited coordination with other agencies or entities required ICVS Type Ã Ã â¢ Ã â¢ Ã Ã â¢ Reduced costs of sampling and â¢ Large reductions in the quality of the administration with all-telephone data RDD design â¢ Much lower response rates â¢ Changing information content is â¢ Increased recall bias due to longer simple reference period â¢ DoJ controls survey content â¢ Underreporting due to abandonment â¢ Limited coordination with other of screener/incident logic agencies or entities required â¢ Increased telescoping â¢ Increase in out-of-scope events due to lack of incident form â¢ Loss of all geographical information â¢ Loss of detail incident information â¢ Reduced resource set devoted to victim statistics SURVEYING VICTIMS
Table 4-3 (continued) Partnership Model Ã â¢ â¢ Ã â¢ Ã â¢ â¢ Better reporting on hard-to-ï¬nd â¢ Large sample sizes would be needed crimes in specialized surveys in narrower-scope component â¢ Increases information on context, surveys, due to relative rareness of etiology, and consequences of these some crime types crimes â¢ Loss of omnibus vehicle on crime â¢ Divides labor for crime statistics may lead to reduction in resources between UCR, survey and health dedicated to victimization statistics â¢ Increased coordination required to ï¬eld surveys and to maintain coverage of important crimes â¢ Loss of control by DoJ â¢ Time series disrupted â¢ Large reductions in information on context, etiology, and consequences of street crime Surveillance Model Ã Ã â¢ â¢ â¢ Ã Ã â¢ Ability to provide extensive â¢ Loss of omnibus vehicle on crime information on victimization in states may lead to reduction in resources and localities dedicated to victimization â¢ Consistent with law enforcement â¢ Increased coordination with states assistance focus of OJP and localities required to ï¬eld surveys and to maintain coverage of important crimes â¢ Loss of control by DoJ â¢ Time series disrupted MATCHING DESIGN FEATURES TO DESIRED GOALS â¢ Large reductions in information on context, etiology, and consequences of street crime â¢ Funding uncertainty due to reliance on BJA funding Crime Poll Ã Ã â¢ Ã â¢ â¢ Ã â¢ Reduced cost from abandoning goal â¢ No estimates of victimization of estimating victimization rates â¢ Break in time series of victimization â¢ Smaller samples rates â¢ Attitudes more prevalent than â¢ No information on context, etiology, victimization and consequences of victimization â¢ More extensive information on â¢ Time series disrupted attitudes toward justice issues â¢ Much lower response rates NOTES: â¢ indicates that the panel believes that the goal is well served by the design package; â¢ that the goal is adequately served; Ã that the goal is poorly served. BJS, Bureau of Justice Statistics. DoJ, U.S. Department of Justice. RDD, random-digit dialing. 105
106 SURVEYING VICTIMS The second basic conclusion is that there is nothing inherently wrong or lacking in the current NCVS design. As noted elsewhere, the NCVS has beneï¬ted from years of experience and methodological probing, particularly the intensive redesign effort that culminated in 1992. The NCVS design is a model that has been adopted by international victimization surveys as well as subnational surveys in the United States, and it is a good and useful ex- emplar. The principal fault of the current NCVS design is not a design ï¬aw or methodological deï¬ciency, or even that the design inherently costs too much to sustain, but ratherâsimplyâthat it costs more than has been ten- able under current federal budgetary priorities. We argue in Chapter 3 that collection of victimization data is substantially undervalued in the United States. There are design packages suggested in our tables that would ultimately be cheaper than the current NCVS, but they can involve considerable com- promises in the quality and detail of knowledge that the present NCVS has been able to provide about crime. For instance, the partnership model (7) that would dissolve the current NCVS and allocate topics to other survey vehicles would be likely to be less expensive in the long run than the cur- rent NCVS (depending on the choice of topics for smaller, focused surveys). However, this model would involve âpiggybackingâ some questions (for gen- eration of basic victimization rates) onto some omnibus survey vehicle like the American Community Survey or Current Population Survey. This piggy- backing raises a number of challenges: â¢ By design, the NCVS screening interview is long, in order to more completely elicit victimization incidents from respondents. For consis- tency with other ACS/CPS questionsâand to keep respondent burden under controlâany victimization questions added to another survey would almost certainly have to be radically simpliï¬ed, with a corre- sponding lack of accuracy. â¢ NCVS screener questions are asked of all persons in the household over age 12, in turn, whereas other surveys like CPS have only one respondent per household. That single respondent may then provide information about other household members or about characteristics of the household as a whole. Separate interviews with multiple house- hold members would be a major departure from usual procedure for the ACS and the CPS, while reliance on proxy information from a sin- gle household respondent will lead to underreporting of victimization incidents. Likewise, the crime poll (10) option is arguably the least expensive of the options in the table, but the sacriï¬ces in information would be profound. Although important information about attitudes toward crime and public
MATCHING DESIGN FEATURES TO DESIRED GOALS 107 safety would ï¬ow readily from this type of model, there would be no capac- ity to obtain details about the characteristics of speciï¬c incidents. For continuity of measurement, the best-case situation would be the pro- vision of budgetary resources sufï¬cient to keep the current NCVS in opera- tion at a stable size. For long-term viability of the NCVS, the ideal would be an increase in budgetary resourcesâand sample sizeâso that at least some subnational (e.g., large metropolitan statistical areas) can be made available on some regular basis. Barring those very optimistic outcomes, we also rec- ognize thatâin terms of its sample sizeâthe current NCVS is at a critical point: its sample size has slipped sufï¬ciently that terminating the survey would effectively be preferable to sustaining additional across-the-board cuts in sample size. 4âC.1 Length of Reference Period In light of these arguments, we suggest switching to a 12-month refer- ence period (thus achieving savings by reducing the number of contacts with sample households) in the event that resources to continue the NCVS using the current 6-month reference period are not available. Recommendation 4.2: Changing from a 6-month reference pe- riod to a 12-month reference period has the potential for im- proving the precision per-unit cost in the NCVS framework, but the extent of loss of measurement quality is not clear from exist- ing research based on the post-1992-redesign NCVS instrument. BJS should sponsor additional researchâinvolving both exper- imentation as well as analysis of the timing of events in extant dataâto inform this trade-off. If this recommendation is followed, a decision will have to be made about what number of waves (interviews conducted at a particular address) would best balance cost, measurement error, and nonresponse error. Relative to other countriesâ victimization surveys, the NCVS 6-month reference period is something of a luxury, with a 12-month reference period being the norm. It bears repeating, though, that the switch would decrease the cost of the survey but would not be without consequences. The choice of a best-length reference period for the NCVS (e.g., 3, 6, or 12 months) has been a source of discussion and research since the surveyâs inception; tests varying reference periods were recommended by National Research Council (1976b:68), and Cantor and Lynch (2000:110â112) sum- marize reference period experiments conducted following that recommen- dation. The effects of length of reference period have also been treated in general survey methodological research. The results of these studies, NCVS-
108 SURVEYING VICTIMS speciï¬c and otherwise, are generally quite consistent. Cognitively, an opera- tive phenomenon is telescoping: distant events tend to be moved forward in time relative to when they occurred, while recent events tend to be shifted to an earlier date. Traumatic distant events (like victimizations) may be more easily recalled and subject to more severe forward telescoping (Bradburn et al., 1994). Accordinglyâuntil adjustments in 2007 led to ï¬rst-time-in- sample interviews being included in estimationâthe ï¬rst interview with an NCVS household has historically been withheld and used as a boundâa check that events are not projected forward into the 6-month reporting time window. A switch to a 12-month window means that forward telescoping might be less problematic but that complementary problems would become prominent: backward telescoping victimization incidents so that they are outside of the 12-month recall period or, for some less traumatic incident types, simply forgetting that they occurred within the past year. The extent of underreporting may differ by crime; studies like those summarized by Cantor and Lynch (2000) suggest about a 30 percent general reduction in reporting by doubling the length of the reference period to 12 months. Consequently, it is important that a move toward a 12-month reference period be paired with research on developing event history techniques and other methods for improving accurate recall over a lengthy time window. This could involve major redesign so that events are placed on calendars or that the list of recalled victimization incidents is anchored to important dates and events (such as birthdays or anniversaries). At the same time that the screener-type questions are revised to promote more accurate recall, attention would also have to be paid to protocols for the detailed incident form. Even allowing for the possibility that some less recent (and less traumatic) incidents may be forgotten, the net number of incidents experienced by a sample interviewee can be expected to be larger for a 12-month window than a 6-month window. Hence, respondent bur- den would correspondingly increase if each and every incident was subject to the detailed incident form questioning; break-offs in the interview and general nonresponse could be expected to increase as a result. Accordingly, protocols for sampling which events elicited in an interview are subject to the full incident form questionnaire would have to be tested and evaluated. The change in reference period will also have an impact on the vari- ances of some statistics that involve multiple incidences because of increased intraperson correlation. Due to these logistical and technical complications, our recommendation concerning a change in reference period is deliberately nuanced. It is not a change that should be rushed into, in the name of ï¬scal savings, but rather one that needs grounding in pilot work and testing. Changing to a 12-month reference period while maintaining continuous interviewing in the ï¬eld (as the Census Bureau currently does) throughout
MATCHING DESIGN FEATURES TO DESIRED GOALS 109 the year would also warrant reconsideration of the production cycle for an- nual NCVS estimates. Table C-2 in Appendix C illustrates the current time coverage of events using the NCVS 6-month reference period, illustrating the distinction between collection year and data year estimates. Seeking âto publish more timely estimates from the survey,â BJS switched to collection- year estimates in 1996 (the actual data release cycles have since been syn- chronized so that both NCVS and Uniform Crime Reports estimates of crime are made public each September). The change to collection year estimates rather than data year estimates was discussed in Bureau of Justice Statistics (2000:163), acknowledging that these âestimates for any given year will in- clude some incidents that actually took place in the previous calendar year and will exclude some incidents that would have been reported in interviews conducted in the following calendar year.â In support of that decision, col- lection year and data year estimates for 1995 were compared, disaggregating by type of crime, and no statistically signiï¬cant differences were detected (Bureau of Justice Statistics, 2000:Appendix Table 1). Extending Table C-2 to reï¬ect a 12-month reference period, a straight collection year estimate for year t would more thoroughly overlap events that actually occurred in calendar t â 1. Labeling these as âyear t estimatesâ would thus be inher- ently misleading, conceptually; presenting them as estimates âfor the past 12 months of collectionâ (and avoiding explicit mention) could be confusing to the media, the public, and other users of the data. However, reverting to a data year estimate for calendar year t would require waiting until interviews were completed in December t + 1 (and then allowing time for processing), which would have serious implications for the timeliness of the data. The BCS-type model (3) described in our tables would combine a switch to a 12-month reference period with another major change: converting from a rotating panel, multiple-interview-per-household design to a cross- sectional single-interview-per-household structure. (The model also assumes that some of the cost savings would be redirected into the ï¬elding of regular, systematic supplements, providing additional ï¬exibility in measurement.) To be clear, we do not recommend a change to a cross-sectional design. How- ever, it is worth noting that a 1-year reference period and cross-sectional design could be combined with a third major change to interesting effect. Speciï¬cally, continuous interviewing throughout the year could be replaced with focused, intensive data collection in the ï¬rst few months of year t + 1. Doing so, respondents would have a very natural common sense reference period to work with (calendar year t ), and the resulting estimates would likewise be easy to interpret as âyear t â estimates. Logistically, the drawback to this scenario is the strain on ï¬eld operations that would result. Prior to its switch to annual operations, the BCS was collected on this basis and the workload required the use of multiple private survey groups to carry out the interviewing workload. Such a large once-a-year effort would be a major
110 SURVEYING VICTIMS change from current NCVS operations by the Census Bureau and would be unique among the Census Bureauâs demographic survey programs. As we discuss further in the next chapter, exactly how other survey research orga- nizations might be able to carry out the work as the NCVS data collection agent is an open question. It is also likely that the cost of moving away from continuous data collection would be prohibitive since the advantage of hav- ing a survey infrastructure in place would be lost (and such infrastructure would have to be stopped and rebuilt annually). In addition, this change in cycle would raise the problem of seasonality: the volume and type of crime varies by the month of the year. Hence, seasonality and respondent bias toward reporting more recent events could combine to distort the overall picture of crime. 4âC.2 Role of Supplements Table 4-3 gives generally high marks to models that emphasize the role of topic supplements. Core-supplement models would streamline the body of the NCVS questionnaire to a minimal âcoreâ and structure the remainder of the survey around regular topic supplements. Likewise, although we argue against the BCS-type model in the previous section due to a reluctance to switch away from a rotating panel design, we do ï¬nd much to admire in the BCS concentration on supplements as a regular and structural part of the broader survey. Recommendation 4.3: BJS should make supplements a regular feature of the NCVS. Procedures should be developed for solic- iting ideas for supplements from outside BJS and for evaluating these supplements for inclusion in the survey. The role of supplements in the NCVSâas is part of the current design of the British Crime Survey (see Appendix E)âis to enhance the ï¬exibility of the survey content. It is meant to prevent the overall content of the survey from becoming stale; in some cases, a topical supplement could serve as a âmethods panelâ for testing new questions that might, in time, be added to the core NCVS content. We think that the implementation of such NCVS topic supplements as the Police-Public Contact Survey and the School Crime Survey have broadened the scope of the victimization survey and that further branching out into topic supplements will serve to ï¬rm up the constituencies for the NCVS and other BJS products. Although a focus on a systematic set of topic supplements would, on its own, fall short of correcting a long- standing shortfall of the NCVSâthe lack of geographic detail in estimatesâ it could be the vehicle for much richer substantive detail.
MATCHING DESIGN FEATURES TO DESIRED GOALS 111 Recommendation 4.4: BJS should maintain the core set of screening questions in the NCVS but should consider streamlin- ing the incident form (either by eliminating items or by changing their periodicity). To be clear, we do not suggest trimming the screening questions on the NCVS questionnaire. As noted in Section 4âA, reducing the screener portion of the interview could slightly decrease interview length and yield very small cost savings, but the quality of resulting data would suffer. That said, we have not comprehensively reviewed the incident form to suggest items to cut, either. The point of this recommendation is that if reductions in interview length are deemed necessary (particularly in order to facilitate a more regular set of topical supplements), then some means of condensing incident form content should be considered. It follows from this recommendation that we think that BJS and the De- partment of Justice should dedicate staff to ï¬nd ways to effectively market the sample for periodic supplements while insisting that supplement costs are fully covered by the sponsors. Conceptually, an advantage of a core-supplement design (or other design that includes a strong role for supplements) is that it allows the NCVS to play to its strengths as a survey. 4âC.3 Supporting Subnational Estimates Many federal government national surveys measure key social phenomenaâtransportation and travel patterns or health risks, for exampleâthat have importance to the country as a whole as well as to local areas and small subpopulations. Crime is certainly a phenomenon that has policy relevance for local, state, and federal governments. Thus, it is not surprising that a long-term tension affecting support for the NCVS has been how useful it is for local interests versus national interests. NCVSâs role as a major national social indicator (and a point of com- parison with international victimization rates) notwithstanding, we think that the long-term viability of the NCVS will depend critically on its ability to provide small-domain, subnational information. âSmall-domainâ means subpopulations that may or may not be spatially proximate but offer impor- tant policy issues; these may include such groups as new immigrants, persons over 70 years of age living alone, or persons in areas with different growth rates (e.g., fast-growing counties, relative to stable or declining-population counties). âSubnationalâ refers to levels of geography smaller than the na- tion as a whole, such as regions, states, metropolitan statistical areas, and large cities and counties.
112 SURVEYING VICTIMS As Pepper and Petrie (2003:5) summarize: Public attention to crime and victimization often focuses on particular subgroups in which deviant behavior may be most troublesome. For ex- ample, hate crimes, crimes committed by youth, and crimes committed against vulnerable subpopulations including children, the elderly, and people with disabilities have all been the subject of recent investigations and legislation. The current NCVS is capable of generating estimates by these basic demo- graphic criteria. However, for very small subpopulations, sample sizes from each year of the NCVS may be too small to yield stable estimates. As crime mapping has become an increasingly useful tool in assessing trends and plan- ning police activities, and as geographic information system (GIS) software and Internet mapping tools have become more widespread, markets have developed for richer spatial data on social variables like crime and victim- ization. Recommendation 4.5: BJS should investigate the use of model- ing NCVS data to construct and disseminate subnational esti- mates of major crime and victimization rates. As described in Section 3âD.2, state and local agencies do still ï¬nd value in having a national-level measure from the NCVS as a benchmark, in many cases because NCVS data are the only available source for some analyses. Hence, it is an overstatement to say that the NCVS would be relevant or important to state and local users only if subnational reporting was provided. It is also the case thatâperhaps with some intuitive kinds of adjustments for differences between the national and local levelsâsome of the existing demographic splits available in the NCVS can be applied to good effect. Small states, for instance, may be able to make effective use of estimates derived only from the NCVS sample for rural areas. When the national or other larger area estimate is used as a proxy for a local estimate, there arises an additional component of error because the proxy is imperfect. A better alternative than using a proxy often can be found using meth- ods for small-area estimation (Pfeffermann, 2002; Rao, 2003). Using such methods, an estimate for each area of interest can be reported along with an estimate of its mean squared error (MSE). The MSE estimates reï¬ect both sampling error and area-to-area variability but not biases from response er- rors or nonresponse errors. It is also possible to pool data over time (if short-run change analysis is not the focus) and thereby increase the sample size for local areas. Small-area estimation techniques have greatly advanced since the NCVS redesign efforts yielded the redesigned questionnaire in 1992. The NCVS
MATCHING DESIGN FEATURES TO DESIRED GOALS 113 would seem to be a good candidate for evaluating small-area models of vic- timization estimates both for spatial domains and for demographic domains. The attraction includes the possibility of borrowing estimation strength from the police-report data of the UCR and other demographic data (perhaps en- riched by the new American Community Survey data), as well as the rich covariates in NCVS, to model the victimization rates of small domains. Fur- thermore, the fact that the NCVS is a longitudinal study of addresses would allow the small-area models to take advantage of the covariances across time in repeated measurement areas. Since the redesign efforts, the United States has now become accustomed to small-area poverty estimates for program administration (National Research Council, 2000b,a, 1997, 1998, 1999); in terms of geography, these estimates extend to the county and school district levels and have been expanded to include estimates of health insurance pro- grams. The Bureau of Labor Statistics Local Area Unemployment Statistics program combines data from the monthly Current Population Survey and state unemployment insurance records, along with other sources, to gener- ate estimates for states and selected cities and counties (National Research Council, 2007:256). Folsom et al. (1999) describe methodology for produc- ing small-area estimates of drug use, and Raghunathan et al. (2007) discuss the utilization of two survey sources to estimate cancer risk factors at the county level. In addition to small-domain modeling using NCVS data, it may also be useful to explore ways to strengthen victimization surveys conducted by states and localities. The surveillance model (9) we describe in our table has drawbacks in its full form, as it would shift the burden for collecting victimization data on states (or groups of states) and make the ânationalâ survey a concatenation of the state measures. Although we lean against its implementation in full, we think that the basic idea underlying the model is a sound one. Currently, BJS operates a program under which it develops victimization survey software and provides it to interested local agencies; however, those agencies must supply all the resources (funds and manpower) to conduct a survey. An approach to strengthen this program would be to make use of BJSâs organizational position within the U.S. Department of Justice. The bu- reau is housed in the Ofï¬ce of Justice Programs, the core mission of which is to provide assistance to state and local law enforcement agencies; it does so through the technical research of the National Institute of Justice and the grant programs of the Bureau of Justice Assistance (BJA), among others. We suggest that OJP consider ways of dedicating fundsâlike BJA grants, but separate from BJS appropriationsâfor helping states and localities bol- ster their crime information infrastructures through the establishment and regular conduct of state or regional victimization surveys.
114 SURVEYING VICTIMS Recommendation 4.6: BJS should develop, promote, and coor- dinate subnational victimization surveys through formula grants funded from state-local assistance resources. Such surveys would most likely involve cooperative arrangements with research organizations or local universities and make use of the existing BJS statistical analysis center infrastructure. 4âC.4 Efï¬ciency in Sample Design The sample design of the NCVS uses data for stratiï¬cation of primary and secondary units that are available from the decennial census. Stratiï¬ca- tion and the size of the sample are the two features that are tools to reduce the standard errors of NCVS estimates. When variables are available on the frame to preidentify groups that will vary on their victimization experience, the standard errors of the estimates might be reduced. One possible approach for using external information would be to stratify areas based on other crime-related data; UCR data might be considered for this purpose. Although the sample for the NCVS was originally designed so that each household had the same probability of being selected into the sample, such a design is not optimal for the uses of the NCVS data. Since many of the uses are at the local level, reallocating the sample to strengthen the estimates for the smaller areas will be advantageous even though the sampling variance for the largest areas (and national level) will increase, provided that those increases are not too severe. Optimization for multipurpose samples is dis- cussed in Cochran (1977:119â123) and Kish (1976), for example. Review- ing the optimality of the numbers of primary sampling units versus number of housing units within primary sampling unit may also be beneï¬cial. Efï¬ciency gains can also occur if information can be exploited to pre- dict which housing units tend to have higher victimization rates; in that case, one can then sample the blocks containing such housing units at higher rates. Properly done, such âoversamplingâ can improve precision, some- times markedly. Several approaches to predicting housing unit victimization rates may be considered: (i) Merge block-cluster covariates into the NCVS data ï¬le and use the covariates to develop a predictive model based on past NCVS data. Prior research using the area-identiï¬ed NCVS has shown that tracts with the highest levels of socioeconomic disadvantage (e.g., percentage living below poverty) have respondents with the highest victimization rates (Lauritsen, 2001). (ii) Ask a question in the ACS about victimization and use the results to improve on the predictions in (i).
MATCHING DESIGN FEATURES TO DESIRED GOALS 115 (iii) Use the ï¬rst wave results for housing units to modify retention proba- bilities for the second wave, so that housing units that report victimiza- tion in the ï¬rst wave are more likely to be retained. Recommendation 4.7: BJS should investigate changing the sam- ple design to increase efï¬ciency, thus allowing more precision for a given cost. Changes to investigate include: (i) changing the number or nature of the ï¬rst-stage sampling units; (ii) changing the stratiï¬cation of the primary sampling units; (iii) changing the stratiï¬cation of housing units; (iv) selecting housing units with unequal probabilities, so that probabilities are higher where victimization rates are higher; and (v) alternative person-level sampling schemes (sampling or subsampling persons within housing units). 4âC.5 Other Improvements In early redesign efforts in the early 1990s, a consistent ï¬nding that CATI interviews yielded higher reports (Hubble, 1999) led to the belief that CATI interviewing could both save money and obtain higher quality data. Now that the dispersed interviewing of the NCVS uses computer-assisted per- sonal interviewing, the role of CATI in the overall cost-error structure of the NCVS is worth reconsidering. Both CATI and CAPI can use the same soft- ware for automatic routing and editing of responses, and the only distinction between the two approaches is connected with (a) different interviewers and (b) centralization of CATI that might affect impacts of supervision. There are time and administrative costs from shifting cases to and from CATI facilities to ï¬eld interviewers in cases in which the mode initially as- signed is not desired by the respondent. These tend to reduce the response rates of the NCVS. It seems likely that the cost-quality attributes of the NCVS might be improved by shifting all telephone calling to interviewersâ homes. Recommendation 4.8: BJS should investigate the introduction of mixed mode data collection designs (including self-administered modes) into the NCVS. As with most federal household surveys, the person-level response rates (known as Type Z rates in the NCVS) are declining over time. Whether these declines have produced estimates with higher nonresponse bias is not clear from the data at hand. Increasing evidence from survey methodology
116 SURVEYING VICTIMS has shown that the relationship between nonresponse rates and nonresponse bias is more complicated than previously believed to be true (Curtin et al., 2000; Keeter et al., 2000; Groves, 2006). It is quite likely that response rates will continue to fall in the NCVS. If BJS attempts to keep response rates constant, it is likely that costs of the NCVS will increase. Thus, great importance should be given to determining whether low propensities to respond to the NCVS are related to different likelihoods of different types of victimizations. There are diverse methods of such nonresponse bias studies: studying the movement of victimization rates by increasing effort to measure the cases, follow-ups of samples of nonrespondents, examining changes in estimates from alternative postsur- vey adjustments, and so on. These methods and others have been noted and promoted in recent U.S. Ofï¬ce of Management and Budget (2006b,a) guidelines for federal surveys. Recommendation 4.9: The falling response rates of NCVS are likely to continue, with attendant increasing ï¬eld costs to avoid their decline. BJS should sponsor nonresponse bias studies, fol- lowing current OMB guidelines, to guide trade-off decisions among costs, response rates, and nonresponse error.