Data Priorities for Population and Health in Developing Countries: Summary of a Workshop (1996)

The questions addressed in this session included: What are the major advantages and limitations of relying on a household survey based methodology, and how might household surveys be modified to provide better answers to key policy questions? Are there other data collection techniques that could provide quick and relatively inexpensive estimates of key demographic variables with reasonable confidence?

In separate presentations, Julie DaVanzo and Leo Morris discussed the relative merits of household surveys for providing population and health information for monitoring, evaluation, and planning purposes. In the absence of reliable vital registration data, household surveys have been used extensively to provide information for calculating key population and health indicators such as fertility, infant and child mortality, and contraceptive prevalence rates on a representative sample of the population. Increasingly, household surveys have provided rich detail on maternal and child health practices, suitable for use in monitoring the outcomes of health programs. By taking advantage of probability sampling techniques, household surveys can be designed to provide statistics on the general population from a very small percentage of the population of interest. Hence, one of the principal advantages of household surveys is that they can provide reasonably accurate information quickly and relatively cheaply on both users and non-users of services.

Of the four strategic objectives identified by USAID above, household surveys have been most useful for estimating the prevalence of unintended pregnancies and prevailing levels of infant and child mortality. Historically, household surveys have tended to be much less important for producing strategic or management information related to USAID's other two strategic objectives, namely, reducing maternal mortality and reducing the prevalence and incidence of sexually transmitted diseases (STD) including human immunodeficiency virus (HIV).

Maternal mortality is an increasingly important indicator as reproductive health programs are implemented, and several participants discussed the difficulty in getting accurate estimates. DaVanzo noted that the DHS in some countries is in the process of evaluating the worthiness of the sisterhood method (in which women and sometimes men are asked about the number of their sisters who have died in adulthood) to produce indirect estimates of maternal mortality rates. These can be very useful for baseline measurement, but they are not designed to measure short-term changes due to interventions. For direct esti-

mates of maternal mortality rates, DaVanzo noted, large sample sizes and longitudinal or panel designs are needed.

Increased program emphasis is also needed for the development of credible population-based indicators on STDs. In DaVanzo's view, ideal sample household surveys would include both men and women, in a broad age range, rather than only women of reproductive age, who form the sample for most surveys primarily concerned with contraception and fertility. Morris, in his presentation, added that efficient measurement of the impact of programs aimed at reducing the transmission of STDs/HIV will require oversampling of special population groups. When men are needed as respondents for these topics, it may not be useful to select only the husbands of the women in the DHS sample; an independent sampling of men would be needed to get a representative sample of sexually active men for data on topics like condom use. The Young Adult Reproductive Health Surveys (YARHS), for example, have been implemented by CDC to collect data on sexual activity from young people ages 15–24 (Morris, 1993).

Besides self-reports, the collection of biological markers from population-based samples was considered as a possible source of data on STD prevalence. Marge Koblinsky reported that John Snow, Inc., is developing a filter paper test to identify both syphilis and anemia in postpartum women that could be added to a DHS survey. In Ethiopia, DHS staff are providing assistance for a special study of STD prevalence using blood and urine from respondents. Kate Stewart noted that this experimental survey is designed only for baseline measurement and not for monitoring change over time. With biological markers, the participants agreed that practical, logistical, and ethical issues need to be explored, including the costly and complicated logistics of collecting blood and urine samples, the responsibility to treat those identified with an illness, and considerations for protection of privacy.

In her presentation, DaVanzo considered some of the uses of survey data beyond estimation of rates, for example, for studying effects of programs on behavior and health outcomes. One of the chief problems with using cross-sectional survey data to evaluate interventions is that it is often difficult to attribute observed changes in the indicators of interest to particular interventions. Familial, cultural, and environmental factors influence both individual behaviors and program participation and must be accounted for when collecting and analyzing survey data. For example, non-coresident kin (e.g., migrant husbands, grandparents, brothers, etc.) may make important contributions to household income and have a direct impact on fertility and health decisions, which would be overlooked if one focused exclusively on co-resident household members. Thus, to assess the effects of those influences, surveys must collect information on behaviors and characteristics of women and their families and communities, as well as relevant program dimensions. The DHS and similar surveys typically collect only a limited amount of information on family members and socioeconomic characteristics.

Several multipurpose surveys designed by RAND have attempted to provide a better understanding of the myriad of factors that influence demographic and health decisions. DaVanzo suggested that some of the features of the RAND surveys, such as the Malaysian and Indonesian Family Life Surveys (FLS), be considered for other situations. For example, the FLS in Malaysia collected information on:

• health and mortality of all children;
• community and family characteristics;
• household members' earnings and other income;
• migration histories, including dates and locations;
• housing histories, including dates, locations, and water supply and sanitation information about each previous dwelling; and,
• community-level data on family planning and health care facilities.

Collaborating with the Institute for Nutrition in Central America and Panama (INCAP), RAND developed a survey in Guatemala that piggy-backed onto another data collection effort. The RAND survey collected demographic and socio-economic data from households that were being surveyed over a period of time as part of a longitudinal health and nutrition survey. DaVanzo pointed out that the use of existing surveys can provide enriched, high-quality data that covers an extended period of time. Opportunities to piggyback DHS surveys onto other nationally representative surveys such as national labor force surveys, warrant consideration.

In his presentation, Morris discussed several issues that need to be considered when devising population-based household surveys, including determining the appropriate sample size and length of interview. The demand for ever larger surveys has become an increasing problem as policy makers and planners request more refined stratification of statistics on population subgroups and as the goal of successive surveys changes from estimating baseline rates to detecting whether any change has occurred in the baseline over the recent past. Morris gave estimates of the approximate sample sizes needed to measure certain population and health indicators with reasonable accuracy. For most purposes, a sample size of 1,000 is needed, in his view, to measure contraceptive prevalence rates. To reliably measure change in the rate, the sample size would have to be increased by 500 in every year. Estimates of infant or maternal mortality rates requires much larger sample sizes, on the order of 5,000–9,000 women.

The length of interviews was perceived by most participants as an increasing problem. There are always great pressures to include additional topics in demographic and health surveys. For evaluation purposes, analysts want data not only on outcome measures and program participation, but also on ever more numerous independent variables, as well as family and personal characteristics. Morris argued that there is a cutoff point of 45 minutes, beyond which data quality

suffers because of the fatigue of both respondent and interviewer; however, some participants noted that there are examples of well-constructed and well-administered questionnaires that are considerably longer than an hour.

Stan Becker discussed the usefulness of a number of refinements and alternatives to nationally representative cross-sectional surveys for collecting information on key population and health indicators including using reinterviews after household surveys as a way to estimate the reliability of estimates of key parameters. Alternatives to cross-sectional surveys include vital registration data, health facility data, panel studies, and quick cluster surveys.

Amy Tsui suggested that follow-up modules could be used more, to allow intensive interviews with subsamples of respondents selected on the basis of characteristics identified in the first interview. These follow-up interviews could collect data on topics particularly relevant to that subsample or on sensitive issues requiring a long sequence of questions. This design would permit case-control studies (Schlesselman, 1982). For example, a study of practices that pose high risk for HIV transmission could select those reporting in the earlier survey that they had multiple sex partners in the past month for an intensive follow-up on attitudes toward condoms and condom use.

The DHS, Martin Vaessen pointed out, has already started the model of two surveys in some countries, one a standard DHS, the second an intensive study of a special topic. In Egypt, for example, an intensive survey on unmet need for family planning is being implemented in two governorates for which estimates of unmet need were unusually high in analyses of the 1992 national DHS. The design of the National Safe Motherhood Survey in the Philippines provides another model. The sample consisted of DHS respondents who had ever had a pregnancy (almost two-thirds of the original DHS sample, which had included never-married women). The Safe Motherhood Survey, fielded three months after the main DHS, focused on a wide variety of reproductive health topics, in greater detail than would have been possible simply with a module added to the DHS. The implications of this design for cost, as well as for data quality, need to be assessed. Vaessen maintained that a large part of the cost of field work consists of finding and getting to the respondent. Once there, the incremental cost of additional items in the survey is fairly small.

John Casterline and others also supported the idea of modules, asked of selected subsets of a full DHS sample. This approach would represent a compromise between the idea of adding to the length of an interview for everyone and the idea of going back with virtually separate survey rounds. Savings could come

at the expense of complexity for the interviewers (who would have to select which respondent get which modules) and for data processing. However, if there is to be any linkage of the data between the first survey and subsequent modules, the gap between data collection efforts should not be too long.

An alternative to a single, cross-sectional survey is the follow-up survey, in which some or all of the original respondents are reinterviewed some time after the original survey (for reasons other than quality control or studies of validity of the original data). Two of the main reasons for such designs were discussed at the workshop: producing efficient estimates of change in key variables and of the impact of interventions occurring between the two surveys and splitting up what would otherwise be excessively long interviews. Reinterviews are costly (primarily due to the costs of additional fieldwork) and time consuming, but they allow a more comprehensive check on data quality than do studies of internal consistency of data from a single round of interviews. The DHS has undertaken reinterview surveys in Pakistan and Nigeria. In the Pakistan reinterviews, only about one-third of women reported the same age (or only 1 year older) and only three-quarters of the women reported the same number of children ever born (Curtis and Arnold, 1994). The report on the Nigeria reinterview survey was never made public. Another reinterview survey in Liberia found that approximately 17 percent of the child deaths were unreported in the initial survey (Becker et al., 1993). Robert Black and others argued that studies of data quality and of the validity and reliability of indicators have received too little attention in recent years.

A variant of the household panel design is to reinterview in the same primary sampling units as in an earlier survey. This variant saves some of the cost of drawing a new sample, listing, and mapping. If nationally representative estimates are needed, however, a supplemental sample would be needed if the original sampling frame had become outdated. It also allows efficient estimation of effects of interventions implemented at the community level, for example, opening of new clinics or information, education, and communication campaigns. This design is being used for a DHS survey in Morocco, and it has been used for evaluation of program impact in Tanzania.

Panel studies, in which members of the same households are respondents in a second survey fielded some time after the original one, are an efficient design for measuring changes in outcomes over time. Locating people who have migrated out of an area or matching originally interviewed people can be time consuming and costly but the value of matched data often makes this design

worthwhile. The Second Malaysian Family Life Survey included a panel of women who had first been interviewed 12 years earlier (Haaga et al., 1994), and DaVanzo pointed out that such surveys produce information on changes over time that cannot be generated by cross-sectional surveys.

Ian Diamond argued that longitudinal or panel studies are the most effective means for evaluating programs. A longitudinal study can collect population and health data together with individual-, family-, and community-level data on socioeconomic status, attitudinal information, local-level social and cultural norms, program characteristics, etc. Considering the high costs of such designs, Diamond proposed that pilot studies be explored in order to examine the concept more fully. If successful and feasible, a multilevel longitudinal framework could be used routinely for evaluations in all countries to monitor national or subnational programs as well as social and economic changes.

Further discussion at the workshop dealt with the potential value of panel studies for obtaining data on sensitive topics, such as sexual practices and income. Such topics are often placed near the end of questionnaires in comprehensive surveys, so that some rapport would have built up between interviewers and respondents (and so the damage done by a termination would be minimized). But a second intensive survey, or more qualitative methods, could yield more accurate results. Respect for cultural sensitivities, the adequacy of interviewer training and motivation, and the self-selecting bias of an interviewer must also be considered.

In countries in which vital registration coverage levels are reasonably high, a relatively low investment from an outside source could make the national data complete enough for many important uses. In particular, these data could provide useful detailed information for analysis by time period and for local areas. Michael Vlassoff reported that the United Nations Population Fund (UNFPA) has devoted much emphasis to supporting national censuses, vital and civil registrations systems that improve the national statistical knowledge base of developing countries.

For quick, relatively inexpensive estimates of infant mortality, Becker suggested that the preceding birth technique could be an effective measurement tool. This technique involves asking women who are at a health facility for a delivery or to have their young children immunized whether or not the child from their preceding birth is still alive. There are sample selection biases associated with this technique, however, because interviewees are already preselected by being at

a health facility. To circumvent this problem, an adjustment has been proposed by some researchers (Hill and Aguirre, 1990).

Finally, for determining contraceptive prevalence and fertility, the cluster survey method developed by the World Health Organization for estimating immunization coverage might be applicable (Henderson and Sundaresan, 1982). These surveys have low costs (since they rely on samples drawn without complete listing of households and enumeration of inhabitants) and provide results that are comparable to DHS results (Boerma et al., 1990). Cluster surveys could also be used to estimate infant mortality, but that would require a larger sample size than is needed to estimate more prevalent events. Becker noted that by using both the preceding birth technique and the quick cluster design, infant mortality, fertility and contraceptive estimates could be obtained without a full-scale DHS-type survey.

Qualitative approaches to the collection of population and health information can be useful in identifying and understanding the behaviors and beliefs of a population. Such techniques could be used to increase program managers' ability to design programs and researchers ability to evaluate them.

There was much discussion at the workshop about the potential for using qualitative studies linked to household surveys. John Knodel argued that qualitative studies can complement surveys in three ways: by confirming survey findings; by explaining answers elicited by standardized questionnaire items; and by providing a more complete understanding of issues not addressed by the standardized surveys. Knodel's comments focused on the relative merits of focus group discussions and intensive one-on-one interviews. Each approach has its uses: focus group discussions can elicit rich information on attitudes, perceptions, and motivations, while in an in-depth interview the respondent's own experiences relating to a topic can be explored. Questions on sensitive behavior, such as sexual practices and issues related to HIV risk, could be covered more easily with an individual in-depth interview than in a group setting.

Kate Stewart reported that the Philippines National Safe Motherhood Survey successfully experimented with combining qualitative and quantitative methods for development and implementation on the basis of insights of women's perceptions about complications during pregnancy. She said that additional research is needed to determine how best to ask questions and how to determine the most effective method for gathering qualitative information.

There has been a growing willingness, particularly in USAID missions, to use both qualitative and quantitative data for program design and program plan-

ning. David Oot discussed an in-depth study of sexual practices in Nepal that was instrumental in the design of an HIV communication program (see Cox, 1993). Oot noted the need to determine which of various data collection methods are most effective and useful. National surveys could be complemented by qualitative studies and allow program managers to gain a better understanding of how to improve programs and services. Although much qualitative work is being done on STD/HIV issues, Susan Hassig indicated that there remains a need to develop a methodology that will establish the degree to which qualitative insights and perspectives can be generalized. Hassig noted that some qualitative studies have structured guidelines for moderators or interviewers so that relatively structured data can be examined.

One problem discussed by participants is that qualitative data are not systematically maintained. Knodel suggested that USAID consider greater efforts to coordinate research on qualitative techniques and begin archiving studies. Workshop participants supported the idea that USAID support efforts to establish more systematic methods for the collection and coordination of qualitative studies and also conduct further research to determine best methods, practices, and potentials for linking qualitative and quantitative methods of data collection.