Click for next page ( 228


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 227
8 Management and Oversight Many elements combine to make a cost-effective and efficient survey pro- gram that meets the needs of its users for relevant, high-quality, timely, and accessible information on the topics within its mandate. Some of these elements are well-designed questionnaires; motivated respondents; capable interviewers; talented systems staff; innovative experiments and evaluation studies; knowledgeable analysts; and state-of-the art data collection, pro- cessing, and dissemination technology. Not usually considered in the re- view of a survey program is the management of the survey, including the means for obtaining feedback and advice from others. However, an effec- tive management structure underpins all of the above elements and is a key component to a survey's success. In this chapter we review the current management structure for SIPP at the Census Bureau, including the channels through which feedback from users and others outside the SIPP program is sought. Such feedback is essential to keep a survey program oriented to the concerns and needs of its users and also up to date with the latest improvements in survey design and methods. We believe strongly that a different and more effective management structure is needed if SIPP is to achieve its full potential in the future. In our view, the Census Bureau should not treat SIPP as "just like any other Census Bureau survey" from a management perspective. SIPP is far more complex than the Bureau's other household surveys. More important, the Census Bureau has leadership, analysis, and dissemination responsibilities 227

OCR for page 227
228 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION for SIPP above and beyond its responsibilities for collecting and processing the data. In this regard, the Census Bureau's role for SIPP resembles its role for the decennial census and differs from its role for other household surveys: the Bureau both sponsors and operates the decennial census but typically conducts household surveys on behalf of an outside sponsor agency. The Census Bureau must manage SIPP in a manner that is commensurate with its responsibilities as the survey's sponsor and as the lead federal agency for analysis and publication of statistics on family and individual income. (The latter role also encompasses management of the March Cur- rent Population Survey (CPS) income supplement.) MANAGEMENT STRUCTURE Description The responsibility for household surveys that are conducted by the Census Bureau (including SIPP, CPS, the Consumer Expenditure Survey, and oth- ers) is lodged with Demographic Programs~ne of six major organiza- tional units within the Bureau, each of which is headed by an associate director. (See Figure 8-1; subsequently, for clarity, we refer to Demo- graphic Programs as the "Demographic Directorate.") The Demographic Directorate in turn is organized into six divisions, two divisions that deal with international programs and four that are involved with U.S. household surveys: Demographic Surveys Division (DSD), Housing and Household Economic Statistics (HHES), Population (POP), and Demographic Statisti- cal Methods Division (DSMD). DSD performs many functions for each survey, including general man- agement and coordination, questionnaire design, and data processing. Staff of other divisions both inside and outside the Demographic Directorate- also play important roles. Within the Demographic Directorate, DSMD staff are responsible for sample design and selection and most methodologi- cal research and evaluation, while HHES and POP staff are responsible for publications and liaison with users. Outside the Demographic Directorate, the Field Division manages the interviewing staff, while the Center for Survey Methods Research and Statistical Research Division contribute to methodological research. Data User Services Division (DUSD) fills orders for public-use microdata products and documentation (primarily for SIPP and the March CPS, among the Bureau's household surveys). DUSD will also develop user products, such as users' guides, but only by request and with funding from another division. 1HHES and POP perform these functions for the March CPS as well as SIPP, but not generally for surveys that are conducted for other federal agencies.

OCR for page 227
229 - 4' E E0 C) C ~ .m ~ I O ~ ~ ~' l U) .o ~ .m dO E 2 o W "C _ I _ _ I I ~-F~m ~ ;] ' - ~ a, .O as ~ C `i' o -5 _ as CO o _ :> ~ss C) ~ E Cd c5 , ~. . 1 1 _ . ~ o ~ o-E o ~ Q e 1 _ giF_ 97 o -C-~ (D O. ~o ._ 0 .a, o _ 0328 ~ 1 3-S ~ e 1: l I D 1 'L _ J~ I cO , 1 0 o !~0-m 1 1 ~ 1 ~ ~ I o c" 1 E~C 1 ~ ' I lS 0 ~ 1 o c 1 o, 1 0 1 ~ 1 - s~ Ct v o . - C~ . o :' C~ C) ~s m C~ C~ v ~0 8 a5 C ~d ~C: .l

OCR for page 227
230 THE SURVEY OF INCOME AND PROGMM PARTICIPATION The responsibility for coordinating the work and generally for "making the trains run" for each household survey is typically lodged with a branch chief in DSD. One of the branch chief's functions is to obtain budget requests from all divisions with involvement in the survey and funnel them to the head of DSD and, ultimately, the head of the Demographic Director- ate, for review and adjudication. Because SIPP involves staff in so many divisions in the Census Bu- reau both inside and outside the Demographic Directorate and because of its relative newness and complexity, several cross-cutting committee structures have been tried to improve communication and coordination of-the work Initially, the income surveys (SIPP) branch chief in DSD chaired a commit- tee of representatives from each division that met frequently to keep every- one up to date. Also, the chief of POP (HHES did not yet exist) was delegated wide authority to monitor and guide the survey. Although not in charge of the survey's budget, he was instrumental in ensuring that a por- tion of the funding was available for methodological research and experi- mentation. He also played a deciding role in several key changes to the questionnaire (e.g., the decision to combine several scattered topical mod- ules into a single personal history module administered early in each panel). As problems developed (principally with data production and user liai- son), it became clear that a stronger coordinating mechanism was needed. Staff committees were set up to address specific areas, such as user needs and methodological research. An executive committee, consisting of the chiefs of divisions with a major role in SIPP, was charged to review propos- als from the staff committees and make major decisions for the survey. (The executive committee was chaired at times by the head of the Demo- graphic Directorate and at times by a special assistant to the head.) Some of the staff involved with SIPP work solely on the SIPP pro- gram-for example, the staff in the DSD income surveys branch and the DSMD SIPP branch.2 However, SIPP analysts in HHES and POP typically have other responsibilities as well-for example, analyzing data from the decennial census and supplements to the CPS. Initially, POP included sev- eral staff with a strong focus on SIPP who also served as contact points for questions and problems from outside users and who initiated research and development projects for SIPP. Subsequently, some of these staff were attached to the associate director's office. Plans were made to put these and other staff in an expanded Center for Demographic Studies within the De- mographic Directorate that would conduct analyses of longitudinal data from SIPP and other surveys, but those plans were dropped, and the Center was disbanded. At about the same time, several staff with a long association 2The DSD income surveys branch, despite its title, deals exclusively with SIPP and not also with the March CPS income supplement.

OCR for page 227
MANAGEMENT AND OVERSIGHT 231 with SIPP left the Census Bureau or the SIPP program, and some of them were not replaced. Recently, a SIPP user liaison position was established in HHES, which is the division that analyzes the core income and program participation data from SIPP. However, no additional positions for user liaison or SIPP analysis have yet been allocated. Assessment The management structure for SIPP, as for other household surveys, follows well-established practice at the Census Bureau. Given the Bureau's long record of solid performance in many aspects of survey operations, we do not lightly offer criticisms. However, we believe that the structure does not serve SIPP well. The basic decentralized staff organization for SIPP and other Census Bureau surveys follows a widely accepted model. Many agencies and firms that conduct surveys organize their staffs by function for example, with the field staff in one division, the data processing staff in another, the survey and sample design people in another, and so on. This type of struc- ture facilitates such important functions as staff training, mentonng, and career development. However, problems arise in taking the staffs from different divisions who are to participate in a particular survey project and turning them into a cohesive, efficient, well-focused team. Moreover, the work of the team must be driven first and foremost by the survey's goals, which are ultimately determined by the analysis and research needs of the users.3 Most survey organizations address this challenge by naming a project director or principal investigator for the survey who is a senior person with relevant substantive background and survey experience. (A variant struc- ture is to have a subject-matter-oriented principal investigator paired with an operations-oriented survey director.) This individual is assigned overall budget and management authority for the project and is designated the leader of the team. Most other staff members on the team retain a "home base" in their own division but are responsible to the principal investigator for their work on the survey. Often, members of one team will also participate in teams for other survey projects. However, if a particular survey is big and complicated enough, there will typically be a core staff who work full time (or close to full time) on it. This core staff will include task leaders (e.g., a leader for data processing or survey design). It will also almost always include analysis staff who play a major role throughout the project-in 3See Scott (1987:Ch. 9) for a discussion of the challenges that organizations face in devel- oping an appropriate division of labor for particular kinds of technical work and at the same time coordinating the efforts of multiple staff across divisions in order to carry out a project.

OCR for page 227
232 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION specifying questionnaire content and edits and imputations in addition to analyzing and reporting the survey results. In contrast, the Census Bureau assigns general management responsi- bilities for its surveys to DSD staff at the level of a branch chief, which is several steps down in the hierarchy. As needed, committees are set up to facilitate communication and coordination of work across divisions. How- ever, there is no equivalent of a principal investigator leading a designated team, and staff in different divisions often work at arm's length from each other (particularly analysts and data processing and other survey staff). This mode of operation can work well for the household surveys that the Census Bureau conducts on behalf of an outside sponsor agency, since the sponsor agency provides the substantive focus and an impetus for coor- dination. Typically, that sponsor agency whether it is the Bureau of Labor Statistics (BLS) for the Consumer Expenditure Survey and the main labor force portion of the CPS, the National Center for Health Statistics (NCHS) for the Health Interview Survey, or the National Center for Education Sta- tistics (NCES) for the Schools and Staffing Survey sets the survey goals and content, exercises authority over the budget, has substantial input on design issues, analyzes and evaluates the data, prepares publications and microdata products, and provides support services and liaison to users. In the case of SIPP, the Census Bureau itself is the sponsor agency, with full budget authority and responsibility not only for survey operations, but also for the content, analysis, and user service functions. We believe that these added areas of responsibility necessitate strengthening and focusing the management of the program. Evidence for our assessment comes from examining the start-up problems that SIPP experienced, which were most severe precisely in those areas of responsibility: for example, the lag in developing a publication program for the core data on income and programs and the setbacks experienced in developing adequate microdata products, documentation, and other services for users. We recognize that many fac- tors hindered SIPP at the outset, not least the externally imposed significant budget cuts. Nonetheless, we believe that a stronger, more focused man- agement structure would have made it easier for the Bureau to operate so complex a survey and to respond more quickly and effectively to problems. Looking to the future, the Census Bureau faces a major management challenge in planning for and implementing the redesign of SIPP in a timely and cost-effective manner. The redesign will affect all aspects of the sur- vey from content and design to collection and processing to analysis and dissemination. Moreover, the complexity and scope of the survey will con- tinue to make it challenging to manage even after the redesign is in place. Finally, it is critical for the survey to restore its capacity for flexible re- sponse to changing social welfare policy and research data needs, but a decentralized structure makes it difficult to accommodate flexibility and

OCR for page 227
MANAGEMENT AND OVERSIGHT 233 change and at the same time achieve operational efficiency. For all these reasons, we believe that it is imperative for the Census Bureau to change its management approach to SIPP. As noted above, the Census Bureau has experimented with more struc- tured coordination and decision-making arrangements for SIPP, indicating that it has not been completely comfortable with decentralized management of the survey. Currently, there is a SIPP executive committee comprised of relevant division chiefs and chaired by the head of the Demographic Direc- torate, with staff subcommittees on particular topics. An assistant division chief in DSD provides staff support to the executive committee. - However, this structure is not the same as a dedicated SIPP team. Moreover, it is awkward to have a Bureau associate director who is acting, in effect, as the principal investigator or project director for SIPP, given the general man- agement responsibilities of this position for all of the Census Bureau's de- mographic surveys and programs.4 Recommendation: A Different Approach The panel considered alternative management arrangements for SIPP. One model would be to lodge the sponsorship with another agency and so malice SIPP like the other household surveys conducted by the Census Bureau. The original Income Survey Development Program (ISDP) was a joint ven- ture of the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the Department of Health and Human Services (DHHS) and the Census Bureau, and the original plan for SIPP was to have the Social Secu- rity Administration (SSA) settee as the sponsor agency and the Census Bu- reau as the operating agency. However, it was the Census Bureau that came to the rescue of SIPP in the early 1980s, and it was then assigned full responsibility and budget authority for the program. To its credit, the Bureau successfully launched the survey, which, despite its problems, has provided invaluable data for critically important policy and research studies. Furthermore, the Census Bureau has a long history as the lead federal agency for analysis and publi- cation of statistics on family and individual income-one of the two princi- pal topic areas of SIPP. These are excellent reasons to keep SIPP at the Census Bureau. Moreover, there is no other obvious candidate to serve as the sponsor agency. Responsibility for federal assistance programs-the other main topic area of SIPP is divided among several operating and policy analysis 4Recently, the associate director delegated some planning and decision authority for the redesign to a committee comprising the division chiefs of DSD, HHES, POP, and DSMD. This group is meeting weekly to consider redesign issues and milestones.

OCR for page 227
234 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION agencies (ASPE, SSA, and the Administration for Children and Families in DHHS and the Food and Nutrition Service in the Department of Agncul- ture). None of these agencies is as well positioned as the Census Bureau to take a broad, objective view of the survey or to resist pressures for politi- cizing the program, and none of them is strong in survey methodology. Other statistical agencies, such as BLS, have limited experience in the topic areas covered by SIPP. Given the strong arguments to continue the Census Bureau as both sponsor and operator of SIPP, the question is how to create an internal group that cart provide the needed substantive focus and weld the survey staff into a cohesive, efficient, and committed team. There are undoubtedly several organizational arrangements that would serve this end, and the Cen- sus Bureau will have to weigh many factors in developing an organization scheme and operating plan to achieve the goal of a stronger and more focused management for SIPP.5 However, there He several points that we believe are key to the success of the overall concept. Most important is the need for a project director with full management and budgetary authority for the Bureau's income surveys (SIPP and the March CPS income supple- ment). We include the March CPS because it has been the mainstay of the Census Bureau's income statistics program in the past and will continue to make an important contribution for some years to come.6 To be in the best position to serve as an effective sponsor for SIPP, the project director should be at a very senior level: most likely, within the Census Bureau's current organizational structure, an associate director or another position that reports directly to the deputy director. The project director should be someone who combines considerable substantive policy analysis or research experience on such topics as the distribution of income, poverty, and the dynamics of program participation with a strong back- ground in management of complex projects involving original data collec- tion. Finally, the project director needs to have sufficient resources for an analysis staff that is large enough to fulfill the Census Bureau's leadership 50ne possibility would be for the Bureau to set up the equivalent of a "National Center for Income Statistics" within the Census Bureau, headed by a senior person reporting to the deputy director. The center could include the income and program analysis staff, as well as team leaders for the various staff groups (e.g., demographic surveys, field, statistical methods, etc). 6The March CPS income supplement (along with some demographic supplements to the CPS) is the only other household survey for which the Census Bureau serves as the sponsor as well as operating agency. However, the arrangement is not quite the same because the March supplement piggybacks on the main CPS, which is sponsored by BLS. Also, the March supplement is a much simpler data set than SIPP. Nonetheless, we believe that combining the overall responsibility for the March supplement and SIPP under the some project director will benefit both surveys and, more generally, the Bureau's income statistics program.

OCR for page 227
MANAGEMENT AND OVERSIGHT 235 and dissemination responsibilities for income and program statistics from SIPP (and the March CPS). We see important advantages from having a focused management struc- ture for SIPP, headed by a project director who is oriented to income and program statistics issues and who has the necessary budget and management authority for the Bureau's income surveys. Such a structure would provide a strong source of substantive guidance and direction for SIPP and the March CPS. It would also facilitate full involvement of the analysis staff with the survey staff in many aspects of survey design and operations (e.g., questionnaire content and editing decisions). It would better position the staff both analysts and survey people to communicate on an equal foot- ing with analysts in other agencies and with academic researchers about income and program concepts, data needs, measurement methods, and analysis techniques. Also, it would foster clear and effective channels for users to communicate problems, quenes, and suggestions and obtain timely feed- back. Finally, such a focused structure would facilitate the development of in-service training and other means for the staff to keep abreast of the latest conceptual and methodological developments related to income and pro- gram statistics-whether new methods to exploit the richness of SIPP long~- tudinal data for spell analysis or new techniques for improved measurement of asset income sources. In all of these ways, the new structure should enable SIPP to operate in a cost-effective manner and at the same time adapt in a timely manner to changing policy concerns and data needs. We note that there are precedents within the Census Bureau for the type of dedicated management structure that we are proposing for SIPP. In the mid-1980s, a new associate director position was created to manage the 1990 decennial census, and a number of divisions were moved into this office. This step made sense, given that the census is not only the Bureau's largest operation, but also one for which it serves as the sponsor as well as the operating agency. Recommendation 8-1: To be as effective as possible in carrying out its responsibilities to produce timely, comprehensive, rel- evant, high-quality, and analytically appropriate statistics on income and program participation, the Census Bureau should establish a senior-level position of project director for the Bureau's income surveys, SIPP and the March CPS income supplement. That position should include full management and budgetary authority for the income statistics program and sufficient re- sources to obtain the level of analysis staff that is needed to provide substantive guidance to the program, prepare reports, conduct analyses, and evaluate analytical concepts and meth- ods. The person who fills this position should have recognized

OCR for page 227
236 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION substantive expertise in topics related to income, poverty, and assistance programs, combined with strong survey management skills. OVERSIGHT Throughout SIPP's history, the Census Bureau has shown commendable initiative in seeking input from data users and survey experts on various aspects of the SIPP program. An array of advisory groups and input mecha- nisms has existed at one or another time, and these groups have made valuable contributions to the survey. Indeed, there is much in the SIPP advisory process that can serve as a model for other surveys. There are also ways in which the advisory mechanisms for SIPP can and should be im proved. We briefly describe past and present advisory mechanisms for SIPP and suggest ways to improve the oversight function in the future. We cannot stress enough the importance of obtaining and malting the best use of out- side advice and feedback to keep a complex operation such as SIPP on target and able to meet challenges and opportunities. An advantage of the focused management approach that we recommend is that it should facili- tate the development of an improved advisory function for the Census Bureau's income statistics program. Outside Input: Past Experience The Census Bureau has sought advice on SIPP from federal agency users, academic researchers, survey methodologists, and experts in data access and use of microdata products.7 Federal Agency Users The Census Bureau has used several mechanisms to obtain input on the content and other aspects of SIPP from the perspective of federal agencies that use SIPP data for policy analysis and research. A principal vehicle for input has been an interagency committee, chaired by the Statistical Policy Office of the U.S. Office of Management and Budget (OMB). The Census Bureau has also obtained agency input through requests to agency staff for written comments and discussions with the D.C. Users' Group, an informal The panel's description and assessment of advisory mechanisms for SIPP, particularly the interagency committee chaired by the Office of Management and Budget, draws on the interim report on SIPP of the Committee on National Statistics (1989). In addition, panel members and staff have participated in meetings and other activities of the various SIPP advisory groups.

OCR for page 227
MANAGEMENT AND OVERSIGHT 237 group of SIPP users from executive and legislative branch agencies and contract research organizations in the Washington area who meet monthly to exchange views and hear reports of research using SIPP data. The OMB interagency SIPP committee was established in 1983 and assisted in developing and revising the SIPP questionnaire, including the core and topical modules. Among other aspects of SIPP reviewed by that committee (or a series of its subcommittees) were file structures, longitudi- nal household definitions, self- versus proxy response, nonmetropolitan iden- tification, plans for reducing the sample size, and comparability with the CPS. At present, the OMB SIPP committee has no regular meeting schedule. It has experienced long periods of inactivity for example, the committee did not meet from July 1986 until June 1989. This situation was due in part to constrained resources and other problems affecting the statistical coordi- nation function within OMB and in part to a narrow perspective on the part of the Census Bureau of the committee's role. The Bureau has viewed the OMB committee as having a primary role in specifying the content of the variable topical modules in SIPP that are designed to respond to agency needs for data on emerging policy concerns, but as having a more advisory role with regard to the content of the core and fixed topical modules (e.g., on assets) or other features of SIPP. Academic Researchers The Social Science Research Council (SSRC) provided funding from 1982 through 1988 for a committee on SIPP that included members of the aca- demic research community with an interest in longitudinal data for social welfare policy analysis and research. The SSRC committee was initially set up following a conference in fall 1982 on the technical, conceptual, and administrative lessons of the ISDP, sponsored by the SSRC Center for Co- ordination of Research on Social Indicators (David, 1983~. The SSRC committee hosted several symposia (e.g., see David, 1985b) in which contributors addressed ways in which SIPP could be enhanced to better serve academic research needs in a variety of topic areas, including family structure, income distribution, labor markets, education, women and children, minorities. The committee later obtained Census Bureau funding for a conference on individuals and families in transition, which featured papers using SIPP and other panel surveys for longitudinal analysis (Bureau of the Census, 1988a). The committee met two or three times a year with Census Bureau staff in 1983-1988 to discuss various aspects of the SIPP program. The SSRC committee clearly played a very important role in making the research community aware of SIPP and encouraging use of the data (see

OCR for page 227
238 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION Appendix A). The committee and its symposia and conferences were also fertile sources of ideas for improvements to the SIPP design and content. However, the committee's relationship with the Census Bureau was not always smooth. The Bureau necessarily had an operational perspective on SIPP and a primary concern with running the survey and meeting deadlines. Suggestions from academics about content or design changes were often viewed as impractical, needlessly complicating an already complex opera- tion, or outside the survey's original mandate. The Census Bureau's deci- sion in late 1987 to freeze the core questionnaire brought into question the contribution that the SSRC committee could make. Having achieved many useful goals, the SSRC disbanded the committee in 1988. Since then, there have been no formalized channels for input to SIPP from the academic research community. Survey Methodologists In the first years of SIPP, the survey was often featured at the semiannual sessions of the Census Bureau advisory committees of the American Statis- tical Association (ASA) and American Economic Association (AEA). How- ever, the committee members found it difficult to provide useful commen- tary on complex methodological issues for SIPP because their time was so limited and their charge covered so many other Census Bureau programs. In spring 1986 the ASA committee recommended that the Census Bureau sponsor an advisory group of methodologists solely for SIPP. (The SSRC SIPP committee concurred and supported this development.) The Survey Research Methods (SRM) Section of ASA subsequently established the ASA/ SRM Working Group on Technical Aspects of SIPP, which held its first meeting in fall 1986. The ASA/SRM working group continues to meet about twice a year to discuss such issues as maximum telephone versus personal interviewing, longitudinal weighting concepts and techniques, and the results of record- check studies and other evaluations of the quality of SIPP data. As noted in Chapter 7, the group was a major instigator of the SIPP Quality Profile. The group has developed good working relationships with Census Bureau staff and has helped guide priorities for the SIPP methodological research and evaluation program. Experts in Data Access and Use of Microdata Products Problems with data files and documentation surfaced early on in the SIPP program. For several years, the Census Bureau obtained user input on improvements to the microdata products through informal means, for ex- ample, consulting with the D.C. Users' Group and listening to individual

OCR for page 227
MANAGEMENT AND OVERSIGHT 239 SIPP users who had complaints and suggestions. The Association of Public Data Users (APDU) became aware of SIPP users' problems and proposed a joint user-Census committee to address issues involving the design and content of SIPP microdata products, documentation, access mechanisms (such as an on-line extract capability), and means to improve communication be- tween users and Census Bureau staff. The APDU committee, which includes data librarians, microdata users, and systems analysts, along with key Census Bureau data processing and user services staff, first met in January 1989 and has since met once or twice a year. The APDU effort also involves publication of a supplement to the APDU newsletter, mailed to everyone on the Census Bureau's list of inquirers about SIPP, that provides minutes of the APDU committee meet- ings, notices of documentation and data file changes, and other articles of interest to SIPP users. The APDU committee helped specify a more accessible person-month format for the SIPP data files, which was recently implemented for wave files from the 1990 and later panels. The committee has also made numer- ous suggestions related to documentation, conventions for coding missing data, and similar matters. Only limited progress, however, has been made in responding to many of these suggestions, such as providing descriptions of edits and imputations in the documentation. And in spite of the committee's recommendation to increase the resources devoted to user liaison activities, there are fewer staff addressing this area now than 2 years ago, when sev- eral staff members who worked most closely with users left the SIPP pro- gram.8 Other Sources of Advice Formal advisory groups have not been (and should not be) the only source of input to SIPP. Other channels for Census Bureau staff to interact with others, encourage use of the data, and obtain feedback and ideas (see Chap- ter 6) have included: sessions at professional association meetings that featured SIPP-based research or methodological work; the ASA/Census re- search fellowship program, which attracted a number of researchers who used SIPP data on-site at the Census Bureau and provided their views on the program; and presentations and training sessions about SIPP given by Census Bureau staff (e.g., as part of the summer program of the Inter- university Consortium for Political and Social Research). Publication se 8A SIPP Data Products Process Action Team, consisting of staff from several Census Bu- reau divisions, was recently established to develop priorities for improvements in data prod- ucts, documentation, and other user services that could be funded with the expected savings from the recent switch to maximum telephone interviewing in SIPP.

OCR for page 227
240 THE SURVEY OF INCOME AND PROGRAM PARTICIPATIONr ries, including volumes of SIPP-based papers presented at ASA meetings and the SIPP Working Paper series, have also served to build the SIPP user community and encourage interchange among users and Census Bureau staff. Indeed, since SIPP was initiated, the Bureau staff have shown exceptional initiative in developing these kinds of communication channels. However, users have expressed concern that there are not sufficient resources at the Census Bureau to further these activities now that there is only a single SIPP liaison position in HHES. Observations and Directions for the Future We have a number of general observations about the experience with out- side advice for SIPP. First, it appears that the advisory process that has worked best is the ASAISRM working group. The Census Bureau has a long tradition of seeking input on technical issues related to survey research and evaluation, and the staff are comfortable in working with outside ex- perts in such areas as questionnaire design, sampling, weighting, and impu- tation. The ASA/SRM group is largely made up of survey methodologists who are not themselves data users and hence have no particular interest in steering SIPP in any particular direction The group's members and the Census Bureau methodological staff have been able to develop a collegial working relationship built around a common goal of finding ways to im- prove the quality of the SIPP data. Other SIPP advisory groups have had somewhat less success in devel- oping strong working relationships with the Census Bureau. Different per- spectives-for example, the viewpoints of outside researchers or policy analysts and Census Bureau operations people have been a problem in some in- stances. Limited progress in such areas as documentation (whether due to resource constraints or other factors) has been another source of tensions. Finally, there are some gaps in the advisory structure for SIPP. Cur- rently, there are no formal channels for input from academic researchers, who are important users of SIPP and important resources for keeping SIPP abreast of the state of the art in many areas. Also, there has never been an advisory group on technical issues relating to the analysis of SIPP cross- sectional and longitudinal data-for example, such issues as the definition and measurement of spells of poverty or program participation or appropri- ate ways to broaden the definition of income. (The ASAJSRM working group has considered weighting and imputation strategies but not other esti- mation issues.) We commend the Census Bureau's initiatives in seeking input on SIPP from a broad range of perspectives, but we conclude that the Bureau could usefully strengthen and clarify the advisory process in a number of respects. We believe that success in this regard depends, first, on adopting a focused

OCR for page 227
MANAGEMENT AND OVERSIGHT 241 management structure for the program as a whole. Such a structure would foster improved internal communications and cohesion among the various staff groups that work on SIPP, which, in turn, would make it easier to provide timely feedback to users. Also, strengthening the analytical aspects of the Census Bureau's income statistics program would put the staff in a better position to interact productively with outside researchers about con- tent issues and technical matters relating to data analysis and presentation. With regard to specific improvements to the SIPP advisory function, we emphasize that there is no single best way to organize the input process and that informal as well as formal mechanisms have a role. In particular, we are concerned that the Census Bureau not assume that the only way to fill gaps is to create additional expert review groups. We urge the Bureau to think creatively about the advice mechanisms and structures that are likely to be most cost-effective, particularly in the context of a new management approach for SIPP. Broadly speaking, the Census Bureau needs to obtain two kinds of input for SIPP on a continuing basis: advice from policy analysts, research- ers, and other users on issues of goals, direction, overall design, and con- tent; and advice from technical experts on matters related to survey method- ology, analysis, and data products and dissemination. In addition, as noted in other chapters, the Bureau needs mechanisms to ensure that all of the staff analysts, survey methodologists, and operations and data processing people have opportunities for regular in-service training to update their skills and knowledge as well as to learn from outside peers. Finally, as we note in Chapter 6, the Bureau needs to continue to encourage the use of SIPP data through a variety of means and to improve its communications with individual users about their problems and successes in working with the data products. Advice on Goals, Content, and Basic Design Federal agencies concerned with social welfare policy are obviously key constituents of the SIPP program, and the Census Bureau must regularly consult their views. Hence, it is important that the OMB interagency com- mittee continue to function. The Census Bureau should work actively with the OMB staff to put the operations of the committee on a sounder basis by planning its agenda for a 1- or 2-year period and scheduling meetings at least twice a year (more often, as needed). The Census Bureau should consult the committee, if this has not already been done, about key aspects of the proposed survey redesign. The Census Bureau should also consult regularly with academic re- searchers and other users outside the federal government about SIPP be- cause SIPP is important to the development of new knowledge in the social

OCR for page 227
242 THE SURVEY OF INCOME AND PROGRAM PARTICIPATION sciences (which is ultimately the basis for advances in policy analysis and information for decision-making) and because federal agencies frequently commission researchers to carry out SIPP-based policy studies on their be- half. It could be very useful to organize an advisory group of nonfederal agency users, on the model of the SSRC committee, that could coordinate input from researchers about priorities for topical modules and provide other guidance about the content and direction of SIPP. It could also be useful, as an added or alternative means of obtaining input, to organize periodic con- ferences to receive advice and feedback; such conferences could also fea- ture research uses of SIPP.9 Whatever the means chosen, we urge the Census Bureau to be very clear about the type and extent of advice that is being sought (and to pro- vide sufficient time for thoughtful input): for example, advice on content changes for a major redesign can be more expansive than advice on content changes over the next couple of years. Also, we urge the Bureau to main- tain a process to inform the people who provide input about the fate of their suggestions. Such steps can help bridge the different perspectives and de- velop more productive working relationships between researchers and Bu- reau staff. Advice on Technical Issues The ASA/SRM working group has functioned effectively to date in provid- ing outside review of the Census Bureau's research and evaluation program related to survey methods and data quality in SIPP. It seems very useful for the group to continue. Building on this model, we recommend that a parallel working group be established to advise the SIPP analysis staff. For this area, we believe that an additional group is needed because of the difficult issues that the Census Bureau must wrestle with in developing appropriate income statistics from the complex SIPP longitudinal data. We see no reason that an analytically focused group would not function in a similarly collegial manner as the survey-methods-focused ASAISRM group. Such an analytical group might be sponsored jointly by the AEA and the ASA and include economists, statisticians, and survey analysts with conceptual and analytical expertise related to income statistics. As part of its activities, the group could peri- odically review the analytical content of the Census Bureau's income statis- tics publications, including those in the research series. Finally, we believe it is important to continue to haste some type of mechanism, such as the APDU SIPP committee, for obtaining regular ad 9The Conference on the Future of SIPP sponsored by our study panel might serve as a useful model; see Chapter 2.

OCR for page 227
MANAGEMENT AND OVERSIGHT 243 vice and feedback on the Census Bureau's microdata products and data access systems for SIPP. We urge the Census Bureau to recognize, how- ever, that more resources and coordinated staff attention will be needed to respond to the issues that such an advisory group will raise. We urge the Census Bureau, in rethinking its management structure for SIPP, to place more emphasis on this important area. Recommendation Recommendation 8-2: We support the Census Bureau's efforts to obtain outside advice about the SIPP program and encourage the Bureau to further strengthen its advisory mechanisms. The Bureau should regularly seek advice about the content, overall design, and goals of SIPP from federal agency users and from other users, including academic researchers. The Bureau should also regularly seek advice about technical matters from experts in the field. Working groups should be formed or continued in three main areas: (1) survey methods and evaluation of ways to improve data quality; (2) conceptual and analytical issues in the development of appropriate income and program statistics from complex longitudinal data; and (3) microdata products, docu- mentation, and means of data access. As a concluding note, we have been impressed throughout our evalua- tion with the careful thought and attention that everyone we consulted has given to the question of the future of SIPP. Clearly, the many policy ana- lysts, researchers, and survey methodologists who have been involved with SIPP, as well as the SIPP staff at the Census Bureau, support the program and are anxious to see it improve. We urge the Census Bureau to change its approach to the survey in ways that will take full advantage of the interest and commitment of the SIPP community and enable SIPP to fully realize its promise to improve the nation's statistics on income and program participa- tion.

OCR for page 227