Part II
Commentary

This section comments on most of the topics in the principles and practices; the comments are offered to explain, illustrate, or further define the statement of principle in Part I.

DEFINITION OF A FEDERAL STATISTICAL AGENCY

A federal statistical agency is a unit of the federal government whose principal function is the compilation and analysis of data and the dissemination of information for statistical purposes.

A statistical agency may be labeled a bureau, center, division, or office or similar title, so long as it is recognized as a distinct entity. Statistical agencies have been established for several reasons: (1) to develop new information for an area of public concern (e.g., the Bureau of Labor Statistics, the National Center for Health Statistics); (2) to conduct large statistical collection and dissemination operations specified by law (e.g., the U.S. Census Bureau); (3) to compile and analyze statistics from sets of administrative records for policy purposes and public use (e.g., the Statistics of Income Division in the Internal Revenue Service); and (4) to develop broad and consistent estimates from a variety of statistical and administrative sources in accordance with a prespecified conceptual framework (e.g., the Bureau of Economic Analysis in the U.S. Department of Commerce). Once



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition Part II Commentary This section comments on most of the topics in the principles and practices; the comments are offered to explain, illustrate, or further define the statement of principle in Part I. DEFINITION OF A FEDERAL STATISTICAL AGENCY A federal statistical agency is a unit of the federal government whose principal function is the compilation and analysis of data and the dissemination of information for statistical purposes. A statistical agency may be labeled a bureau, center, division, or office or similar title, so long as it is recognized as a distinct entity. Statistical agencies have been established for several reasons: (1) to develop new information for an area of public concern (e.g., the Bureau of Labor Statistics, the National Center for Health Statistics); (2) to conduct large statistical collection and dissemination operations specified by law (e.g., the U.S. Census Bureau); (3) to compile and analyze statistics from sets of administrative records for policy purposes and public use (e.g., the Statistics of Income Division in the Internal Revenue Service); and (4) to develop broad and consistent estimates from a variety of statistical and administrative sources in accordance with a prespecified conceptual framework (e.g., the Bureau of Economic Analysis in the U.S. Department of Commerce). Once

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition established, many statistical agencies engage in all of these functions to varying degrees. This definition of a federal statistical agency does not include many statistical activities of the federal government because they are not performed by distinct units, or because they do not result in the dissemination of statistics to others—for example, statistics compiled by the U.S. Postal Service to set rates or by the U.S. Department of Defense to test weapons (see National Research Council, 1998b, 2002b, 2003b, on statistics and testing for defense acquisition). Nor does it include agencies whose primary functions are the conduct or support of problem-oriented research, although their research may be based on information gathered by statistical means, and they may also sponsor important surveys, as do, for example, the National Institutes of Health, the Agency for Healthcare Research and Quality, and other agencies in the U.S. Department of Health and Human Services. Finally, this definition of a statistical agency does not usually include agencies whose primary function is policy analysis and planning (e.g., the Office of Tax Analysis in the U.S. Department of the Treasury, the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services). Such agencies may collect and analyze statistical information, and statistical agencies, in turn, may perform some policy-related analysis (e.g., produce reports on trends in after-tax income or child care arrangements of families). However, to maintain credibility as an objective source of accurate, useful information, statistical agencies must be separate from units that are involved in developing policy and assessing policy alternatives. The work of federal statistical agencies is coordinated through the Interagency Council on Statistical Policy (ICSP), created by the U.S. Office of Management and Budget (OMB) in the 1980s and authorized in statute in the 1995 reauthorization of the Paperwork Reduction Act. The ICSP is chaired by OMB and currently includes representation from 10 principal statistical agencies and from the statistical units in the Environmental Protection Agency, the Social Security Administration, the National Science Foundation, and the Internal Revenue Service (see Box 1). Throughout the federal government, OMB recognizes more than 70 units and agencies that are not statistical agencies but that have annual budgets of $500,000 or more for statistical activities (U.S. Office of Management and Budget, 2004c:4-7). The principles for federal statistical agencies presented here should apply to other federal agencies that carry

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition BOX 1 Federal Agencies Represented on the Interagency Council on Statistical Policy as of 2005 Office of Management and Budget (OMB), Chair Bureau of Economic Analysis (BEA), U.S. Department of Commerce Bureau of Justice Statistics (BJS), U.S. Department of Justice Bureau of Labor Statistics (BLS), U.S. Department of Labor Bureau of Transportation Statistics (BTS), U.S. Department of Transportation Census Bureau, U.S. Department of Commerce Economic Research Service (ERS), U.S. Department of Agriculture Energy Information Administration (EIA), U.S. Department of Energy National Agricultural Statistics Service (NASS), U.S. Department of Agriculture National Center for Education Statistics (NCES), U.S. Department of Education National Center for Health Statistics (NCHS), U.S. Department of Health and Human Services Office of Environmental Information, Environmental Protection Agency (EPA) Office of Research, Evaluation, and Statistics, Social Security Administration (SSA) Science Resources Statistics Division, National Science Foundation (NSF) Statistics of Income Division (SOI), Internal Revenue Service, U.S. Department of the Treasury out statistical activities, and they may find many of the detailed practices pertinent as well. Similarly, the principles and practices may be relevant to statistical units in state and local government agencies, and international audiences may also find them useful. ESTABLISHMENT OF A FEDERAL STATISTICAL AGENCY One of the most important reasons for establishing a statistical agency is to provide information that will allow for an informed citizenry. A democracy depends on an informed electorate. A citizen has a right to information that comes from a trustworthy, credible source and is relevant, accurate, and timely. Timely information of high quality is also critical to

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition policy analysts and decision makers in both the public and private sectors. (For more information on the purposes of official statistics, see the Fundamental Principles of Official Statistics of the United Nations Statistical Commission in Appendix A; see also United Nations Economic Commission for Europe, 2003; United Nations Statistical Commission, 2003.) Federal statistical agencies serve the key functions of providing a broad array of information to the public and policy makers and of ensuring the necessary quality and credibility of the data. Commercial, nonprofit, and academic organizations in the private sector also provide useful statistical information, including data they collect themselves and data they acquire from government agencies and other data sources to which they add value. However, because the benefits of statistical information are shared widely throughout society and because it is often difficult to collect payments for these benefits, private markets are not likely to provide all of the data that are needed for public and private decision making or to make data as widely available as needed for important public purposes. Government statistical agencies are established to ensure that a broad range of information is publicly available. (See National Research Council, 1999b, for a discussion of the governmental role in providing public goods, or near public goods, such as research and data.) The United States government collected and published statistics long before any distinct federal statistical agency was formed (see Duncan and Shelton, 1978; Norwood, 1995). The U.S. Constitution mandated the conduct of a decennial census of population beginning in 1790, and the census enumeration was originally conducted by U.S. marshals as one of their many duties. Legislation providing for the compilation of statistics on agriculture, education, and income was enacted by Congress in the 1860s. The Bureau of Labor (forerunner of the Bureau of Labor Statistics) was established by law in 1884 as a separate agency with a general mandate to respond to widespread public demand for information on the conditions of industrial workers. The Census Bureau was established as a permanent agency in 1902 to conduct the decennial census and related statistical activities. Many federal statistical agencies that can trace their roots back to the 19th or early 20th century, such as the National Center for Education Statistics and the National Center for Health Statistics, were organized in their current form following World War II. Several relatively new agencies have since been established, including the Energy Information Administration, the Bureau of Justice Statistics, and, most recently (in 1991), the Bureau of Transportation Statistics. In every case, the agency itself, in

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition consultation with users of its information, has major responsibility for determining its specific statistical programs and for setting priorities. Initially, many of these agencies also had responsibilities for certain policy analysis functions for their department heads. More recently, policy analysis has generally been located in separate units that are not themselves considered to be statistical agencies, a separation that helps establish and maintain the credibility of statistical agencies as providers of data and analyses that are not designed for particular policy alternatives. A statistical agency has at least two roles: (1) provider of the statistical information and analysis needed to inform policy making and program assessment by its own department and (2) source of national statistics for the public in its area of concern. It is sometimes difficult to keep these two roles distinct on policy-relevant statistics. An effective statistical agency, nevertheless, will frequently play a creative, not just reactive, role in the development of data needed for policy analysis. Sometimes federal statistical agencies play additional roles, such as monitor and consultant on statistical matters to other units within the same department (see e.g., National Research Council, 1985a) and collector of data on a reimbursable basis for other agencies. There is no set rule or guideline for when it is appropriate to establish a separate federal statistical agency, carry on statistical activities within the operating units of departments and independent agencies, or contract for statistical services from existing federal statistical agencies or other organizations. Establishment of a federal statistical agency should be considered when one or more of the following conditions prevails:1 There is a need for information on an ongoing basis beyond the capacity of existing operating units, possibly involving other departments and agencies. Such needs may require coordinating data from various sources, initiating new data collection programs to fill gaps, or developing regularly updated time series of estimates. There is a need, as a matter of credibility, to ensure that major data series are independent of policy makers’ control. 1   National Research Council (2001b:Ch.6) cited a number of these reasons in recommending the establishment or identification of a statistical unit in the U.S. Department of Health and Human Services to be assigned responsibility and authority for carrying out statistical functions and data collection for social welfare programs and the populations they serve; see also National Research Council and Institute of Medicine (2004).

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition There is a need to establish the functional separation of statistical data that are collected for statistical purposes from identifiable data that may be used for administrative, regulatory, or law enforcement uses. Such separation, recommended by the Privacy Protection Study Commission (1977), bolsters a culture and practice of respect for privacy and protection of confidentiality. Functional separation is easier to maintain when the statistical data are compiled by a unit that is separate from operating units. The Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) extended legal confidentiality protection to every statistical data collection by a federal agency, whether a statistical agency or other type of agency (see Appendix B). Nonetheless, functional separation of statistical data from other kinds of data is important because it makes promises of confidentiality protection more credible. There is a need to emphasize the principles and practices of an effective statistical agency, for example, professional practice, openness about the data provided, and wide dissemination of data. There is a need to encourage research and development of a broad range of statistics in a particular area of public interest or of government activity or responsibility. There is a need to consolidate compilation, analysis, and dissemination of statistics in one unit to encourage high-quality performance, eliminate duplication, and streamline operations. PRINCIPLES FOR A FEDERAL STATISTICAL AGENCY Principle 1: A federal statistical agency must be in a position to provide information relevant to issues of public policy. A statistical agency supplies information not only for the use of managers and policy makers in the executive branch and for legislative designers and overseers in Congress, but also for all those who require statistical information on public issues, whether the information is needed for purposes of production, trade, consumption, or participation in civic affairs. Just as a free enterprise economic system depends on the availability of economic information to all participants, a democratic political system depends on—and has a fundamental duty to provide—wide access to information on education, health, transportation, the economy, the environment, criminal justice, and other social concerns. Federal statistical agencies are responsible for providing statistics on

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition conditions in a variety of areas. The resulting information is used both inside and outside the government not only to delineate problems and sometimes to suggest courses of action, but also to evaluate the results of government activity or lack of activity. The statistics provide much of the basis on which the government itself is judged. This role places a heavy responsibility on federal statistical agencies for impartiality and objectivity. In order to provide information that is relevant to public issues, statistical agencies need to reach out to users of the data. Federal statistical agencies usually are in touch with the primary users in their own departments. Considerable energy and initiative are required to open avenues of communication more broadly to other current and potential users, including analysts and policy makers in other federal departments, state and local government agencies, academic researchers, private-sector organizations, organized constituent groups, the media, and Congress. Advisory committees are recommended as a means to obtain the views of users, as well as people with relevant technical expertise (see, e.g., National Research Council, 1993a). Many agencies have traditionally had such committees to advise them—examples include the American Statistical Association’s Committee on Energy for the Energy Information Administration, the Board of Scientific Counselors for the National Center for Health Statistics, and the Census Advisory Committee of Professional Associations for the Census Bureau. The Federal Economic Statistics Advisory Committee (FESAC), chartered in November 1999, provides substantive and technical advice to three agencies—the Bureau of Economic Analysis, the Bureau of Labor Statistics, and the Census Bureau—thereby providing an important crosscutting perspective on major economic statistics programs (Bureau of Labor Statistics, 2005a). One frequently recommended method for alerting statistical agencies to emerging statistical information needs is for the agency’s own staff to engage in analysis of its data (Martin, 1981; Norwood, 1975; Triplett, 1991). For example, relevant analysis may use the agency’s data to examine correlates of key social or economic phenomena or to study the statistical error properties of the data. Such in-house analysis can lead to improvements in the quality of the statistics, to identification of new needs, to a reordering of priorities, and to closer cooperation and mutual understanding with policy analysis units. In its work for a policy analysis unit, a statistical agency describes conditions and possibly measures progress toward some previously identified goal, but it refrains from making policy recommendations. The distinction between statistical analysis and policy analysis

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition is not always clear, and a statistical agency will need to consider carefully the extent of policy-related activities that are appropriate for it to undertake. Principle 2: A federal statistical agency must have credibility based on a relationship of mutual respect and trust with those who use its data and information. Users of a statistical agency’s data must be able to trust that the data were collected and analyzed in an objective, impartial manner and that they are as reliable as the agency can make them. An agency should make every effort to provide accurate and credible statistics that will permit policy debates to be concerned about policy, not about the credibility of the data. Credibility is enhanced when an agency fully informs users of the strengths and weaknesses of the data, makes data available widely, and consults with users about priorities for data collection and analysis. Principle 3: A federal statistical agency must have a relationship of mutual respect and trust with respondents who provide data and all data subjects whose information it obtains. The statistics programs of the federal government rely in large part on information supplied by individuals and by organizations outside the federal government, such as state and local governments, businesses, and other organizations. Some of this information is required by law or regulation (such as employers’ wage reports), some of it is related to administration of government programs (such as information provided by benefit recipients), but much of it is obtained through the voluntary cooperation of respondents in statistical surveys. Even when response is mandatory, the cooperation of respondents reduces costs and likely promotes accuracy (see National Research Council, 1995b, 2004e). Important elements in encouraging such cooperation are that respondents believe that the data requested are important and legitimate for the government to collect, that they are being collected in an impartial, competent manner, and that the confidentiality of their responses will be protected. In brief, trust in a statistical agency must be maintained. The agency must not be perceived as being swayed by political considerations. It must be perceived as working in the national interest, not the interest of a particular administration, and as taking a long view, balancing new data needs against the need for consistency with past data (Ryten, 1990). Respondent

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition trust also depends on providing respondents with realistic promises of confidentiality that the agency can reasonably expect to honor and then scrupulously honoring those promises. Finally, respondent trust depends on adopting practices that respect personal privacy, such as taking steps to minimize the intrusiveness of questions and the time and effort required to participate in a survey. PRACTICES FOR A FEDERAL STATISTICAL AGENCY Practice 1: A Clearly Defined and Well-Accepted Mission A clear understanding of the mission of an agency, the scope of its statistical programs, and its authority and responsibilities is basic to planning and evaluating its programs and to maintaining credibility and independence from political control (National Research Council, 1986, 1997b). Some agency missions are clearly spelled out in legislation; other agencies have only very general legislative authority. On occasion, very specific requirements may be set by legislation or regulation. Agencies should communicate their mission clearly to others. The use of the Internet is one means to publicize an agency’s mission to a broad audience and to provide related information, including enabling legislation, the scope of the agency’s statistical program, confidentiality provisions, operating procedures, and data quality guidelines. An agency’s mission should focus on the compilation, evaluation, analysis, and dissemination of statistical information. In addition, considerable and formal attention must be paid to setting statistical priorities (National Research Council, 1976). Advice from outside groups should be sought on the agency’s statistical program, on setting statistical priorities, on the statistical methods used, and on data products. Such advice may be sought in a variety of formal and informal ways, but it should be obtained from data users and providers as well as professional or technical experts in the subject-matter area and in statistical methods and procedures. A strong research program in the agency’s subject-matter field can assist in setting priorities and identifying ways to improve an agency’s statistical programs (Triplett, 1991). Practice 2: A Strong Position of Independence A statistical agency must be able to provide credible information that may be used to evaluate the program and policies of its own department or

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition the government as a whole. More broadly, a statistical agency must be a trustworthy source of objective, reliable information for decision makers, analysts, and others inside and outside the government who want to use statistics to understand present conditions, draw comparisons with the past, and help guide plans for the future. For these purposes, a strong position of independence for a statistical agency is essential. (See the Fundamental Principles of Official Statistics of the United Nations Statistical Commission in Appendix A.) Statistical agency independence must be exercised in a broader framework. Legislative authority usually gives ultimate responsibility to the department rather than the statistical agency head. In addition, an agency is subject to the normal budgetary processes and to various coordinating and review functions of OMB, as well as the legislative mandates, oversight, and informal guidance of Congress. Within this broader framework, a statistical agency must work to maintain its credibility as an impartial purveyor of information. In the long run, the effectiveness of an agency depends on its maintaining a reputation for impartiality; thus, an agency must be continually alert to possible infringements on its credibility and be prepared to argue strenuously against such infringements. Independence of an agency head can be strengthened by the head’s being appointed for a fixed term by the president, with approval by the Senate, as is the case with the Bureau of Labor Statistics and the National Center for Education Statistics. The heads of the Bureau of Justice Statistics, the Census Bureau, and the Energy Information Administration are presidential appointees, but their terms are not fixed and usually end with a change of administration. In the case of a fixed term, it is desirable that it not coincide with the presidential term so that incumbents need not end their leadership with changes of administration and professional considerations may more easily predominate over political aims in the appointment process. It is also desirable that a statistical agency head have direct access to the secretary of the department or the head of the independent agency in which the statistical agency is located. Such access allows the head to inform new secretaries about the appropriate role of a statistical agency and present the case for new statistical initiatives to the secretary directly. Among the agency heads with presidential appointments, such direct access currently is provided by legislation only for the Bureau of Labor Statistics. It is desirable for a statistical agency to have its own funding appropria-

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition tion from Congress and not be dependent on allocations from budgets of other agencies that may be subject to reallocation. These organizational aspects—appointment of the agency head by the president with approval by the Senate for a fixed term not coincident with that of the administration, direct access to the secretary of the agency’s department, and separate budgetary authority—are neither necessary nor sufficient for a strong position of independence for a statistical agency, but they facilitate such independence.2 Control over personnel actions, especially the selection and appointment of qualified professional staff, including senior executive career staff, is another aspect of independence. Agency staff reporting directly to the agency head should have formal education and deep experience in the substantive, methodological, operational, or management issues facing the agency as appropriate for their positions. In addition, professional qualifications are of the utmost importance for statistical agency heads, whether the profession is that of statistician or the subject-matter field of the statistical agency (National Research Council, 1997b). Relevant professional associations can be a source of valuable input on suitable candidates. The authority to ensure that information technology systems fulfill the specialized needs of the statistical agency is also an aspect of independence. A statistical agency must be able to vouch for the integrity, confidentiality, and impartiality of the information collected and maintained under its authority so that it retains the trust of its data providers and data users. Such trust is fostered when a statistical agency has control over its information technology resources, and there is no opportunity or perception that policy, program, or regulatory agencies could gain access to records of individual respondents. A statistical agency also needs control over its information technology resources to support timely and accurate release of official statistics, which are often produced under stringent deadlines. Authority to decide the scope and specific content of the data collected or compiled is yet another important element of independence. Most 2   Legislation and administrative actions have removed some of these organizational features for some statistical agencies. (For example, legislation signed on November 30, 2004, as part of reorganizing the U.S. Department of Transportation, changed the head of the Bureau of Transportation Statistics (BTS) from a presidential appointee to a career position and specified that the BTS director is to report to the administrator of the new Research and Innovative Technology Administration, not to the secretary.)

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition risks to participants from the data collection and analysis, select participants equitably with regard to the benefits and risks of the research, and seek informed consent from participants. Under the regulations, most federally funded research involving human participants must be reviewed by an independent institutional review board (IRB) to determine that the design meets the ethical requirements for protection. (See Office for Human Research Protections [2005] for information about the Common Rule and procedures for the certification of IRBs by the Office for Human Research Protections in the U.S. Department of Health and Human Services.) Data collections of federal statistical agencies are subject to IRB review within some departments. The Census Bureau, citing Title 13, has maintained an exemption from IRB review for its data collection programs under section 46.101(b.3), which permits exemption if “federal statute(s) require(s) without exception that the confidentiality of the personally identifiable information will be maintained throughout the research and thereafter.” Whether or not a statistical agency is subject to formal IRB review, it should strive to incorporate the spirit of the Common Rule regulations in the design and operation of its data collection programs. An agency that is required to obtain IRB approval for data collection should work proactively with the IRB to determine how best to apply the regulations in ways that do not unnecessarily inhibit response. For example, signed written consent is not necessary for mail surveys and is hardly ever necessary for telephone surveys of the general population: such documentation does not provide any added protection to the respondent, and it is likely to reduce participation. As noted above, an effective statistical agency will seek ways—such as sending an advance letter—to furnish information to potential respondents that will help them make an informed decision about whether to participate. Such information should include the planned uses of the data and the benefits to individuals and the public. Practice 8: Commitment to Quality and Professional Standards of Practice The best guarantee of high-quality data is a strong professional staff that includes experts in the subject-matter fields covered by the agency’s program, experts in statistical methods and techniques, and experts in data collection, processing, and other operations. A major function of an agency’s managers is to strike a balance among these groups and promote working

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition relationships that make the agency’s program as productive as possible, with each group of experts contributing to the work of the others. An effective statistical agency devotes resources to developing, implementing, and inculcating standards for data quality and professional practice. Although a long-standing culture of data quality contributes to professional practice, an agency should also seek to develop and document standards through an explicit process. The Information Quality Act of 2000 requires agencies to develop and publicize quality guidelines of a fairly general nature (see Appendix B).3 In addition, it is in an agency’s interest to develop detailed guidelines for use by its staff, contractors, and even outside researchers. The existence of explicit standards and guidelines, regularly reviewed and updated, facilitates training of new in-house staff and contractors’ staffs. An effective statistical agency keeps up to date on developments in theory and practice that may be relevant to its program, such as new techniques for imputing missing data (see, for example, National Research Council, 2004e:App. F). An effective agency is also alert to changes in the economy or in society that may call for changes in the concepts or methods used in particular data sets (see, for example, National Research Council, 1995a, on concepts of poverty, and National Research Council, 2002a, on cost-of-living concepts). Often the need for change conflicts with the need for comparability with past data series, and this issue can easily dominate consideration of proposals for change. Agencies have the responsibility to manage this conflict by initiating more relevant data series or revising existing series to improve quality while providing information to compare old and new series, such as was done when the Bureau of Labor Statistics revised the treatment of owner-occupied housing in the consumer price index. To ensure the quality of its data collection programs and reports, an effective statistical agency has mechanisms and processes for obtaining both inside and outside review of such aspects as the soundness of the data collection and estimation methods and the completeness of the documentation of the methods used and the error properties of the data. At a program or agency-wide level, mechanisms for outside review include standing advisory committees of technical experts and periodic assessments by 3   See also data quality guidelines of statistical agencies in other countries, including Canada (Statistical Reference Centre, 2005), and Great Britain (Office for National Statistics, 2005).

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition ad hoc committees (see, e.g., National Research Council, 1985a, 1986, 1993a, 1997b, 2003c, 2004c, 2004d). For individual publications and reports, formal processes are needed that incorporate review by agency technical experts and, as appropriate, by technical experts in other agencies and outside the government. (See Appendix B for a description of recent OMB guidelines for peer review of scientific information.) Practice 9: An Active Research Program Substantive Research and Analysis There are strong arguments for a statistical agency to have staff whose responsibility is to conduct objective substantive analyses of the data that the agency compiles, such as analyses that assess trends over time or compare population groups: Agency analysts are in a position to understand the need for and purposes of the data and know how the statistics will be used. Such information must be available to the agency and understood thoroughly if the survey design is to produce the data required. Those involved in analysis can best articulate the concepts that should form the basic framework of a statistical series. Agency analysts are well situated to understand and transmit the views of external users and researchers; at the same time, close working relationships between analysts and data producers are needed for the translation of the conceptual framework into the design and operation of the survey. Agency analysts have access to the complete microdata and so are in a better position than analysts outside the agency to understand and describe the limitations of the data for analysis purposes and to identify errors or shortcomings in the data that can lead to subsequent improvements. Substantive research by analysts on an agency’s staff will have credibility because of the agency’s commitment to openness about the data provided and maintaining independence from political control. Substantive research by analysts on an agency’s staff can assist in formulating the agency’s data program, suggesting changes in priorities, concepts, and needs for new data or discontinuance of outmoded or little-used series.

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition As with descriptive analyses provided by the agency, substantive analyses must be designed to be relevant to policy but not take positions on policy options or be designed with any particular policy agenda in mind. These issues are discussed in Martin (1981), Norwood (1975), and Triplett (1991). Research on Methodology and Operations For statistical agencies to be innovative in methods for data collection, analysis, and dissemination, research on methodology and operational procedures must be ongoing. Methodological research may be directed toward improving survey design, measuring error and, when possible, reducing it from such sources as nonresponse and reporting errors, reducing the time and effort asked of respondents, developing new and improved summary measures and estimation techniques, and developing innovative statistical methods for confidentiality protection. Research on operational procedures may be directed toward facilitating data collection in the field, improving the efficiency and reproducibility of data capture and processing, and enhancing the usability of Internet-based data dissemination systems. Much of current practice in statistical agencies was developed through research they conducted or obtained from other agencies. Federal statistical agencies, frequently in partnership with academic researchers, pioneered the applications of statistical probability sampling, the national economic accounts, input-output models, and other analytic methods. The U.S. Census Bureau pioneered the use of computers for processing the census, and research on data collection, processing, and dissemination operations continues to lead to creative uses of automated procedures and equipment in these areas. Several federal statistical agencies sponsor research using academic principles of cognitive psychology to improve the design of questionnaires, the clarity of data presentation, and the ease of use of electronic data collection and dissemination tools such as the Internet. The history of the statistical agencies has shown repeatedly that methodological and operations research can lead to large productivity gains in statistical activities at relatively low cost. An effective statistical agency actively partners with the academic community for methodological research. It also seeks out academic and industry expertise for improving data collection, processing, and dissemination operations. For example, a statistical agency can learn techniques and

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition best practices for improving software development processes from computer scientists (see National Research Council, 2003e, 2004d). Research on Policy Uses Much more needs to be known about how statistics are actually used in the policy-making process, both inside and outside the government. Research about how the information produced by a statistical agency is used in practice can contribute to future improvements in design, concepts, and format of data products. For example, public-use files of statistical microdata were developed in response to the growing analytic needs of government and academic researchers. Gaining an understanding of the variety of uses and users of an agency’s data is only a first step. More in-depth research on the policy uses of an agency’s information might, for example, explore the use of data in microsimulation or other economic models, or go further to examine how the information from such models and other sources is used in decision-making (see National Research Council, 1991a, 1991b, 1997a, 2001b, 2003a). Practice 10: Professional Advancement of Staff An effective federal statistical agency has personnel policies that encourage the development and retention of a strong professional staff who are committed to the highest standards of quality work. There are several key elements of such a policy: The required levels of technical and professional qualifications for positions in the agency are identified, and the agency adheres to these requirements in recruitment and professional development of staff. Position requirements take account of the different kinds of technical and other skills, such as supervisory skills, that are necessary for an agency to have a full range of qualified staff, including not only statisticians, but also experts in relevant subject-matter areas, data collection, processing, dissemination processes, and management of complex, technical operations. Continuing technical education and training of staff, appropriate to the needs of their positions, is provided by sponsoring in-house training programs and providing opportunities for external education and training. Professional activities, such as publication in refereed journals and presentations at conferences, are encouraged and recognized, including

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition presentations of technical work in progress with appropriate disclaimers. Participation in relevant statistical and other scientific associations is encouraged to promote interactions with academic researchers and other data users. Such participation is also a mechanism for openness about the data provided. Interaction with other professionals is increased through technical advisory committees, supervision of contract research and research consultants, fellowship programs of visiting researchers, exchange of staff with relevant statistical, policy, or research organizations, and opportunities for new assignments within the agency. Accomplishment is rewarded by appropriate recognition and by affording opportunity for further professional development. The prestige and credibility of a statistical agency is enhanced by the professional visibility of its staff, which may include establishing high-level non-management positions for highly qualified technical experts. An effective statistical agency considers carefully the costs and benefits—monetary and nonmonetary—of using contractor organizations, not only for data collection as most agencies do, but also to supplement inhouse staff in other areas.4 Outsourcing of functions can have benefits, such as providing experts in areas in which the agency is unlikely to be able to attract highly qualified in-house staff (e.g., some information technology functions), enabling an agency to handle an increase in its workload that is expected to be temporary or that requires specialized skills, and allowing an agency to learn from best industry practices. However, outsourcing can also have costs, including that agency staff become primarily contract managers and less qualified as technical experts and leaders in their fields. An effective statistical agency maintains and develops a sufficiently large number of in-house staff, including mathematical statisticians, who are qualified to analyze the agency’s data and to plan, design, carry out, and evaluate its core operations so that the agency maintains the integrity of its data and its credibility in planning and fulfilling its mission. Statistical agencies should also maintain and develop staff with the expertise necessary for effective management of contractor resources. 4   Only the Bureau of Labor Statistics, the Census Bureau, and the National Agricultural Statistics Service maintain their own interviewing staff.

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition An effective statistical agency has policies and practices to instill the highest possible commitment to professional ethics among its staff, as well as procedures for monitoring contractor compliance with ethical standards. When an agency comes under pressure to act against its principles—for example, if it is asked to disclose confidential information for an enforcement purpose or to support an inaccurate interpretation of its data—it must be able to rely on its staff to resist such actions as contrary to the ethical principles of their profession. An effective agency refers its staff to such statements of professional practice as the guidelines published by the American Statistical Association (2005) and the International Statistical Institute (2005), as well as to the agency’s own statements about protection of confidentiality, respect for privacy, standards for data quality, and similar matters. It endeavors in other ways to ensure that its staff are fully cognizant of the ethics that must guide their actions in order for the agency to maintain its credibility as a source of objective, reliable information for use by all. Practice 11: Coordination and Cooperation with Other Statistical Agencies The U.S. federal statistical system consists of many agencies in different departments, each with its own mission. Nonetheless, statistical agencies do not and should not conduct their activities in isolation. An effective statistical agency actively explores ways to work with other agencies to meet current information needs, for example, by seeking ways to integrate the designs of existing data systems to provide new or more useful data than a single system can provide. An effective agency is also alert for occasions when it can provide technical assistance to other agencies—including not only other statistical agencies, but also program agencies in its department—as well as occasions when it can receive such assistance in turn. Efforts to standardize concepts and definitions, such as those for industries and occupations, further contribute to effective coordination of statistical agency endeavors, as does the development of broad macro models, such as the system of national accounts (see, e.g., National Research Council, 2004a). Initiatives for sharing data among statistical agencies (including individual data and address lists when permitted by law and when sharing does not violate confidentiality promises) can be helpful for such purposes as achieving greater efficiency in drawing samples, evaluating completeness

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition of population coverage, and reducing duplication among statistical programs, as well as reducing respondent burden. The responsibility for coordinating statistical work in the federal government is specifically assigned to the Office of Information and Regulatory Affairs (OIRA) in the OMB by the Paperwork Reduction Act (previously, by the Federal Reports Act and the Budget and Accounting Procedures Act). The Statistical and Science Policy Office in OIRA, often working with the assistance of interagency committees, reviews concepts of interest to more than one agency; issues standard classification systems (of industries, metropolitan areas, etc.) and oversees their periodic revision; consults with other parts of OMB on statistical budgets; and, by reviewing the statistical program of the government as a whole, identifies gaps in statistical data, programs that may be duplicative, and areas in which interagency cooperation might lead to greater efficiency and added utility of data. The Statistical and Science Policy Office also is responsible for coordinating U.S. participation in international statistical activities.5 The Statistical and Science Policy Office encourages the use of administrative data for statistical purposes, when feasible, and works to establish common goals and norms on major statistical issues such as confidentiality. It sponsors and heads the interagency Federal Committee on Statistical Methodology (FCSM), which issues guidelines and recommendations on statistical issues common to a number of agencies (see Federal Committee on Statistical Methodology, 1978a-2004; see also http://www.fcsm.gov). It encourages the Committee on National Statistics at the National Academies to serve as an independent adviser and reviewer of federal statistical activities. The 1995 reauthorization of the Paperwork Reduction Act created a statutory basis for the Interagency Council on Statistical Policy (ICSP), formalizing an arrangement whereby statistical agency heads participate with OMB in activities to coordinate federal statistical activities. (See Box 1 for a list of agencies represented on the ICSP.) There are many forms of interagency cooperation and coordination. Some efforts are multilateral, some bilateral. Many result from common interests in specific subject areas, such as economic statistics, statistics on 5   The Statistical and Science Policy Office, previously the Statistical Policy Office, was renamed to reflect added responsibilities with respect to the Information Quality Act standards and guidelines, OMB’s guidance on peer review planning and implementation, and evaluations of science underlying proposed regulatory actions.

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition people with disabilities, or statistics on children or the elderly. (See U.S. Office of Management and Budget, 2004c:Ch.3, for a description of several interagency collaborative efforts, such as joint support for research that fosters new and innovative approaches to surveys, expansion and improvement of FedStats coverage and features, and development of the American Community Survey.) A common type of bilateral arrangement is the agreement of a program agency to provide administrative data to a statistical agency to be used as a sampling frame, a source of classification information, or a summary compilation to check (and possibly revise) preliminary sample results. The Bureau of Labor Statistics, for example, benchmarks its monthly establishment employment reports to data supplied by state employment security agencies. Such practices improve statistical estimates, reduce costs, and eliminate duplicate requests for information from the same respondents. In other cases, federal statistical agencies engage in cooperative data collection with state counterparts to let one collection system satisfy the needs of both. A number of such joint systems have been developed, notably by the Bureau of Labor Statistics, the National Agricultural Statistics Service, the National Center for Education Statistics, and the National Center for Health Statistics. Another example of a joint arrangement is the case in which one statistical agency contracts with another to conduct a survey, compile special tabulations, or develop models. Such arrangements make use of the special skills of the supplying agency and facilitate use of common concepts and methods. The Census Bureau conducts many surveys for other agencies, as do the National Center for Health Statistics and the National Agricultural Statistics Service (see U.S. Office of Management and Budget, 2004c:App. A). The major federal statistics agencies are also concerned with international comparability of statistics. Under the leadership of OMB’s Statistical and Science Policy Office, they contribute to the deliberations of the United Nations Statistical Commission, the Organization for Economic Co-operation and Development, and other international organizations; participate in the development of international standard classifications and systems; and support educational activities that promote improved statistics in developing countries. Statistical agencies also learn from and contribute to the work of established statistical agencies in other countries in such areas as survey methodology, record linkage, confidentiality protection techniques, and data quality standards. Several statistical agencies run educational programs for government statisticians in developing countries.

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition Some statistical agencies have long-term cooperative relationships with international groups, for example, the Bureau of Labor Statistics with the International Labor Organization, the National Agricultural Statistics Service with the Food and Agriculture Organization, and the National Center for Health Statistics with the World Health Organization. To be of most value, the efforts of statistical agencies to cooperate as partners with one another should involve the full range of their activities, including definitions, concepts, measurement methods, analytical tools, dissemination modes, and disclosure limitation techniques. Such efforts should also extend to policies and professional practices so that agencies can respond effectively and with a coordinated voice to such governmentwide initiatives as data quality guidelines, privacy impact assessments, performance rating criteria, institutional review board requirements, and others. Finally, coordination efforts should encompass the development of data, especially for emerging policy issues (National Research Council, 1999a). In some cases, it may be not only more efficient, but also productive of needed new data for agencies to fully integrate the designs of existing data systems, such as when one survey provides the sampling frame for a related survey. In other instances, cooperative efforts may identify ways for agencies to improve their individual data systems so that they are more useful for a wide range of purposes. Two of the more effective continuing cooperative efforts in this regard have been the Federal Interagency Forum on Aging-Related Statistics and the Federal Interagency Forum on Child and Family Statistics. The former was established in the mid-1980s by the National Institute on Aging, in cooperation with the National Center for Health Statistics and the Census Bureau. The forum’s goals include coordinating the development and use of statistical databases among federal agencies, identifying information gaps and data inconsistencies, and encouraging cross-national research and data collection for the aging population. The forum was reorganized in 1998 to include six new member agencies, and the reconfigured forum decided at its first meeting in March 1999 to focus on developing a periodic indicators chart book, which was first published the following year (the latest publication is Federal Interagency Forum on Aging-Related Statistics, 2004). The Federal Interagency Forum on Child and Family Statistics was formalized in a 1994 executive order to foster coordination and collaboration in the collection and reporting of federal data on children and families. It includes many relevant statistical and program agencies. Its reports (e.g.,

OCR for page 13
Principles and Practices for a Federal Statistical Agency: Third Edition Federal Interagency Forum on Child and Family Statistics, 2003, 2004) describe the condition of America’s children, including changing population and family characteristics, the environment in which children are living, and indicators of well-being in the areas of economic security, health, behavior, social environment, and education. No single agency, whether a statistical or program agency, could have produced the forum reports alone. Working together in this way, federal statistical agencies contribute to presenting data in a form that is more relevant to policy concerns and to a stronger statistical system overall.