Part II:
Commentary

DEFINITION OF A FEDERAL STATISTICAL AGENCY

A federal statistical agency is a unit of the federal government whose principal function is the compilation and analysis of data and the dissemination of information for statistical purposes.

A statistical agency may be labeled a bureau, center, division, or office or similar title, so long as it is recognized as a distinct entity. Statistical agencies have been established to serve several purposes, including:

• to develop new information for an area of public concern (e.g., the Bureau of Labor Statistics [BLS] and the National Center for Health Statistics);

• to conduct large statistical collection and dissemination operations specified by law (e.g., the U.S. Census Bureau);

• to compile and analyze statistics from sets of administrative records for policy purposes and public use (e.g., the Statistics of Income Division in the Internal Revenue Service [IRS]); and

• to develop broad and consistent estimates from a variety of statistical and administrative sources under a specified conceptual framework (e.g., the Bureau of Economic Analysis [BEA] in the U.S. Department of Commerce, which produces the U.S. National Income and Product Accounts).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
Part II: Commentary DEFINITION OF A FEDERAL STATISTICAL AGENCY A federal statistical agency is a unit of the federal government whose principal function is the compilation and analysis of data and the dissemination of information for statistical purposes. A statistical agency may be labeled a bureau, center, division, or office or similar title, so long as it is recognized as a distinct entity. Statistical agen- cies have been established to serve several purposes, including: • to develop new information for an area of public concern (e.g., the Bureau of Labor Statistics [BLS] and the National Center for Health Statistics); • to conduct large statistical collection and dissemination operations specified by law (e.g., the U.S. Census Bureau); • to compile and analyze statistics from sets of administrative records for policy purposes and public use (e.g., the Statistics of Income Division in the Internal Revenue Service [IRS]); and • to develop broad and consistent estimates from a variety of statistical and administrative sources under a specified conceptual framework (e.g., the Bureau of Economic Analysis [BEA] in the U.S. Department of Com- merce, which produces the U.S. National Income and Product Accounts). 25

OCR for page 25
26 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY Once established, many statistical agencies engage in all of these functions to varying degrees. This definition of a federal statistical agency does not include many statistical activities of the federal government because they are not per- formed by distinct units or because they do not result in the dissemination of statistics to others. Such activities include statistics compiled by the U.S. Postal Service to set rates or statistics developed by the U.S. Depart- ment of Defense in the testing of weapon systems (see National Research Council, 1998b, 2003b, and 2012d). Nor does it include agencies whose primary functions are the conduct or support of problem-oriented research, although their research may be based on information gathered by statisti- cal means, and they may also sponsor important surveys: examples include the National Institutes of Health, the Agency for Healthcare Research and Quality, and other agencies in the U.S. Department of Health and Human Services. This definition of a statistical agency also does not usually include agencies whose primary function is policy analysis and planning (e.g., the Office of Tax Analysis in the U.S. Department of the Treasury, the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services). Such agencies may collect and analyze statistical information, and statistical agencies, in turn, may perform some policy-related analysis (e.g., produce reports on trends in after-tax income or child care arrangements of families). However, to maintain credibility as an objective source of accurate, useful information, statistical agencies must be separate from units that are involved in developing policy and assessing policy alternatives. Statistical agencies have as their primary purpose the dissemination of information that can be used for a wide range of statistical purposes but not for administrative, enforcement, or regulatory purposes that could affect an individual (person or business) data provider. Such data are usually collected under a pledge of confidentiality. Statistical agen- cies may collect information from government agencies in which indi- vidual reporting units are identified because the data are already public i ­nformation—as, for example, in the Census Bureau’s program to collect financial and employment information for state and local governments (see National Research Council, 2007a) and the program of the National ­ Center for Science and Engineering Statistics to collect information on research and development spending from federal agencies (see National Research Council, 2010c).

OCR for page 25
PART II: COMMENTARY 27 Occasionally, statistical agencies are charged to collect information that is made available for both statistical and nonstatistical purposes. For exam­ ple, the Bureau of Transportation Statistics (BTS) maintains the Airline O ­ n-Time Statistics Program (originated by the former Civil Aeronautics Board), which identifies individual airlines.1 However, BTS does not itself use the data for administrative or regulatory purposes—those functions are carried out by the Federal Aviation Administration—and the data are not collected under a pledge of confidentiality. As another example, higher education institutions that participate in federal student aid programs are required by law (20 USC 1094(a)(17)) to respond to surveys conducted by the National Center for Education Statistics (NCES). The data collected on enrollments, graduation rates, faculty and staff, finances, institutional prices, and student financial aid feed into the Integrated Post­ econdary Education s Data System (IPEDS). The data are not collected under a pledge of confi- ­ dentiality, and NCES makes information on individual institutions available to parents and students to help them in choosing a college, as well as to re- searchers and others.2 NCES also collaborates with institutions through the National Postsecondary Education Cooperative, which works to improve the quality of reporting and dissemination of information to the public, identify modifications to definitions that are necessary to keep abreast of changes in the field, and address other aspects of the IPEDS program. Statistical agencies should carefully consider the advantages and dis­ advantages of undertaking a program with both statistical and nonstatistical purposes. One potential advantage is that there may be improved consis- tency and quality when a statistical agency collects information for its own use and that of other parts of its department. One potential disadvantage is that the program may compromise the public perception of the agency as objective and separate from government administrative, regulatory, and enforcement functions. When an agency decides to carry out a program that has both statistical and nonstatistical uses, it must take care to clearly describe that program on such dimensions as the extent of confidentiality protection, if any (for example, some but not all of the data may be collected under a pledge of confidentiality); the statutory basis for the program and the public purposes it serves, including benefits to respondents from having comparative infor- 1Available: http://www.bts.gov/xml/ontimesummarystatistics/src/index.xml [February 2013]. 2See http://nces.ed.gov/ipeds/about [February 2013].

OCR for page 25
28 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY mation available of uniform quality; and the role of the agency (for example, providing information to the public, working with respondents to improve reporting). Should an agency decide that the nature of a program is such that no amount of description or explanation is likely to make it possible for the agency to maintain its credibility as a statistical agency, it should decline to carry out the activity. The work of federal statistical agencies is coordinated through the U.S. Office of Management and Budget (OMB) Statistical and Science Policy O ­ ffice (SSP) and the Interagency Council on Statistical Policy (ICSP), which was created by OMB in the 1980s and authorized in statute in the 1995 reauthorization of the Paperwork Reduction Act (44 USC 3504(e)(8)). The ICSP is chaired by the chief statistician in OMB and currently includes representation from 14 agencies and units, which are housed in 9 cabinet departments and 3 independent agencies (see Appendix B): • Bureau of Economic Analysis (U.S. Department of Commerce) • Bureau of Justice Statistics (U.S. Department of Justice) • Bureau of Labor Statistics (U.S. Department of Labor) • Bureau of Transportation Statistics (U.S. Department of Transportation) • Census Bureau (U.S. Department of Commerce) • Economic Research Service (U.S. Department of Agriculture) • Energy Information Administration (U.S. Department of Energy) • National Agricultural Statistics Service (U.S. Department of Agriculture) • National Center for Education Statistics (U.S. Department of Education) • National Center for Health Statistics (U.S. Department of Health and Human Services) • National Center for Science and Engineering Statistics (U.S. N ­ ational Science Foundation [NSF]) • Office of Environmental Information (U.S. Environmental Protec- tion Agency) • Office of Research, Evaluation, and Statistics (Social Security Administra­ ion, U.S. Department of Health and Human Services) t • Statistics of Income Division (U.S. Department of the Treasury) In addition to these 14 agencies, OMB currently recognizes 110 other units and agencies that are not statistical agencies but that have annual

OCR for page 25
PART II: COMMENTARY 29 b ­ udgets of $500,000 or more for statistical activities (U.S. Office of Manage- ment and Budget, 2012b:Table 1). The principles for federal statistical agen- cies presented here are relevant to these other agencies that carry out statistical activities, and many of the detailed practices are also pertinent. Similarly, the principles and practices may be relevant to statistical units in state and local government agencies, as well as for international statistical agencies. ESTABLISHMENT OF A FEDERAL STATISTICAL AGENCY One of the most important reasons for establishing a statistical agency is to provide information that will allow for an informed citizenry. A d ­ emocracy depends on an informed electorate. A citizen has a right to information that comes from a trustworthy, credible source and that is relevant, accurate, and timely. Timely information of high quality is also critical to policy analysts and decision makers in both the public and private sectors. (For more information on the purposes of official statistics, see the Fundamental Principles of Official Statistics of the U.N. Statistical Com- mission in Appendix C; see also U.N. Economic Commission for Europe, 2003; U.N. Statistical Commission, 2003.) Federal statistical agencies serve the key functions of providing a broad array of information to the public and to policy makers and of ensuring the necessary quality and credibility of the data. Commercial, nonprofit, and academic organizations in the private sector also provide useful statistical information, including data they col- lect themselves and data they acquire from government agencies and other data collectors to which they add useful other information or analysis. However, because the benefits of statistical information are shared widely throughout society and because it is often difficult to garner payments for these benefits, private entities are not likely to collect all of the data that are needed for public and private decision making or to make data as widely available as needed for important public purposes. Nor are they likely to have the capacity or interest to continually work to make data comparable across geographic areas, population groups, and over time, or, in general, to continually work to provide the needed scope, scale, and quality of statisti- cal information. Government statistical agencies are established to ensure that a broad range of relevant, accurate, timely, and credible information is publicly available. (See National Research Council, 1999b, 2005b, for a discussion of the governmental role in providing public goods, or near public goods, such as research and data.)

OCR for page 25
30 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY The U.S. government collected and published statistics long before any distinct federal statistical agency was formed (see Duncan and Shelton, 1978; Norwood, 1995). The U.S. Constitution mandated the conduct of a decennial census of population; the first such censuses (beginning in 1790) were conducted by U.S. marshals as one of their many duties. Legislation providing for the compilation of statistics on agriculture, education, and income was enacted by Congress in the 1860s. The Bureau of Labor (fore- runner of the Bureau of Labor Statistics) was established by law in 1884 as a separate agency with a general mandate to respond to widespread public demand for information on the conditions of industrial workers. The C ­ ensus Bureau was established as a permanent agency in 1902 to conduct the decennial census and related statistical activities. Many federal statistical agencies that can trace their roots back to the 19th or early 20th century, such as the National Center for Education ­Statistics and the National Center for Health Statistics, were organized in their current form following World War II. Several relatively new agencies have since been established, including the Energy Information Administration, the Bureau of Justice Statistics, and the Bureau of Transportation Statistics.3 In every case, the agency itself, in consultation with users of its in- formation, has major responsibility for determining its specific statistical programs and for setting priorities. Initially, many of these agencies also had responsibilities for certain policy analysis functions for their depart- ment heads. More recently, policy analysis has generally been located in separate units that are not themselves considered to be statistical agencies, a separation that helps establish and maintain the credibility of statistical agencies as providers of data and analyses that are not designed for particular policy alternatives. Nevertheless, an effective statistical agency has a role as a creative, not just reactive, actor in the development of data needed for policy analysis. Statistical agencies may also play additional roles, such as reviewer and consultant on statistical matters for other units in the same department (see, e.g., National Research Council, 1985a) and collector of data on a reimbursable basis for other agencies. There is no set rule or guideline for when it is appropriate to establish a separate federal statistical agency, carry on statistical activities within the 3Within the past decade, the Division of Science Resources Studies in the National Science Foundation became the Division of Science Resources Statistics in 2002 and, as provided by section 505 of the America COMPETES Reauthorization Act of 2010, became the National Center for Science and Engineering Statistics in 2011.

OCR for page 25
PART II: COMMENTARY 31 operating units of departments and independent agencies, or contract for statistical services from existing federal statistical agencies or other organi- zations. Establishment of a federal statistical agency should be considered when one or more of the following conditions prevail:4 • There is a need for information on an ongoing basis beyond the c ­ apacity of existing operating units, possibly involving other departments and agencies. Such needs may require coordinating data from various sources, initiating new data collection programs to fill gaps, or developing regularly updated time series of estimates. • There is a need, as a matter of credibility, to ensure that major data series are independent of policy makers’ control. • There is a need to establish the functional separation of data on individuals and organizations that are collected for statistical purposes from data on individuals and organizations that may be used for administrative, regulatory, or law enforcement uses. Such separation, recommended by the Privacy Protection Study Commission (1977), bolsters a culture and practice of respect for privacy and protection of confidentiality. Functional separation is easier to maintain when the data to be used for statistical purposes are compiled and controlled by a unit that is separate from operat- ing units or department-wide data centers. The Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) extended legal confidentiality protection to statistical data collections that may be carried out by any federal agency, whether a statistical agency or other type of agency (see Appendix A). Nonetheless, functional separation of statistical data from other kinds of data is important because it makes promises of confidentiality protection more credible.5 4The National Research Council (2001b:Ch. 6) cited a number of these reasons in recommending to the U.S. Department of Health and Human Services that it establish or identify a statistical unit to be assigned responsibility and authority for carrying out statisti- cal functions and data collection for social welfare programs and the populations they serve (although the recommendation was not adopted); see also National Research Council and Institute of Medicine (2004). 5Under the guidance issued for CIPSEA in 2007, OMB has recognized 4 new ­ tatistical s units in addition to the 12 statistical agencies originally recognized (see ­ ppendix A). These A agencies and units are authorized to assign agent status to researchers and contractors, which permits sharing individually identifiable information with them for statistical purposes and holding them legally liable for protecting the confidentiality of the information.

OCR for page 25
32 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY • There is a need to emphasize the principles and practices of an e ­ffective statistical agency, for example, professional practice, openness about the data provided, and wide dissemination of data. • There is a need to encourage research and development of a broad range of statistics in a particular area of public interest or of government activity or responsibility. • There is a need to consolidate compilation, analysis, and dissemina- tion of statistics in one unit to encourage high-quality performance, elimi- nate duplication, and streamline operations. PRINCIPLES FOR A FEDERAL STATISTICAL AGENCY Principle 1: A federal statistical agency must be in a position to provide objective, accurate, and timely information that is relevant to issues of public policy. A statistical agency supplies information not only for the use of man­ agers and policy makers in the executive branch and for legislative designers and overseers in Congress, but also for everyone who requires objective statistical information on public issues, whether the information is needed for purposes of production, trade, consumption, or participation in civic affairs. Just as a free enterprise economic system depends on the availability of economic information to all participants, a democratic political system depends on—and has a fundamental duty to provide—wide access to in- formation on education, health, transportation, the economy, the environ- ment, criminal justice, and other social issues. Federal statistical agencies are responsible for providing statistics on conditions in a variety of areas. The resulting information is used both inside and outside the government not only to delineate problems and to guide courses of action, but also to evaluate the results of government activ- ity or lack of activity. The statistics provide much of the basis on which the government itself is judged. This role places a heavy responsibility on federal statistical agencies for impartiality and objectivity. In order to provide information that is relevant to public issues, statisti- cal agencies need to reach out to users of the data. While they are usually in touch with the primary users in their own departments, considerable energy and initiative are required to open avenues of communication more broadly to other current and potential users, including analysts and policy makers in other federal departments, state and local government agencies, academic

OCR for page 25
PART II: COMMENTARY 33 researchers, private-sector businesses and other organizations, organized constituent groups, associations that represent data users, the media, and members of Congress and their staffs.6 One way to obtain the views of users outside an agency, as well as people with relevant technical expertise, is through advisory committees (see National Research Council, 1993a, 2007c). Many agencies obtain advice from committees that are chartered under the Federal Advisory Committee Act: examples include the Advisory Committee on Agriculture Statistics for the National Agricultural Statistics Service; the Board of Sci- entific Counselors for the National Center for Health Statistics; the Data Users Advisory Committee and the Technical Advisory Committee for the Bureau of Labor Statistics; and the Scientific Advisory Committee and the National Advisory Committee on Race, Ethnic, and Other Populations for the Census Bureau. The Federal Economic Statistics Advisory Com- mittee (FESAC), chartered in November 1999, provides substantive and technical advice to three ­ gencies—the Bureau of Economic Analysis, the a Bureau of Labor Statistics, and the Census Bureau—thereby providing an important cross-cutting perspective on major economic statistics programs.7 Some agencies obtain advice from committees and working groups that are organized by an independent association, such as the American Statistical Association’s Committee on Energy Statistics for the Energy Information Administration. Other means to gather information about user priorities for fed- eral statistics include workshops and conferences, which are valuable for ­ facilitat­ ng interchange among users and agency staff (see National Research i Council, 2012a). Online mechanisms, such as blogs and web surveys, may also assist a statistical agency obtain input from users. It is also important for an agency’s own staff to engage in analysis of its data to improve them and make them more relevant to users (Martin, 1981; Norwood, 1995; Triplett, 1991). For example, relevant analysis may use the agency’s data to examine correlates of key social or economic phenomena or to study the statistical error properties of the data. Such in-house analysis can lead to improvements in the quality of the statistics, to identification of new needs, to a reordering of priorities, and to closer cooperation and 6For example, there are more policy uses of statistical data from the American Com- munity Survey (ACS) and ways to make the ACS data more relevant and accurate for these purposes than have yet been tapped: see National Research Council (2008d, 2011a, 2012g). 7See http://www.bls.gov/bls/fesac.htm [February 2013].

OCR for page 25
34 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY mutual understanding with policy analysis units. In working with a policy analysis unit, a statistical agency may describe conditions and possibly measure progress toward some previously identified goal, but it refrains from making policy recommendations. The distinction between statistical analysis and policy analysis is not always clear, and a statistical agency will need to consider carefully the extent of policy-related activities that are ap- propriate for it to undertake. Principle 2: A federal statistical agency must have credibility with those who use its data and information. Users of a statistical agency’s data must be able to trust that the data were collected and analyzed in an objective, impartial manner and that they are as relevant, accurate, and timely as the agency can make them. Without the reality and appearance of credibility, policy debates may deteriorate into attacks on the data, instead of using the information to inform policy choices. Credibility is enhanced when an agency fully informs users of the strengths and weaknesses of the data, makes data available widely, and actively engages with users about priorities for data collection and analysis. When it does so, an agency is perceived to be working in the national inter- est, not the interest of a particular administration (Ryten, 1990). Credibility is also enhanced when a statistical agency’s website has read- ily accessible information about its policies on such topics as confidentiality and privacy protection, scientific integrity, standards for data quality and for documenting sources of error in data collections and the limitations of datasets and statistical models, procedures and schedules for the release of new and continuing data series, and procedures for timely notice of errors and corrections to previously released data. Links to policies of an agency’s parent cabinet department or independent agency that clearly specify the authority that is delegated to the statistical agency also enhance credibility and build trust with users. Principle 3: A federal statistical agency must have the trust of those whose information it obtains. The statistical programs of the federal government rely on informa- tion supplied by many data providers, including not only other agencies of the federal government, but also individuals and organizations outside the federal government, such as state and local governments, businesses, and

OCR for page 25
PART II: COMMENTARY 35 other organizations. Some of this information is a by-product of data col- lections that are required by law or regulation for use in the administration of government tax and transfer programs, such as employers’ wage reports to state employment security agencies or records of payments to program beneficiaries. But much of it is obtained through the voluntary cooperation of respondents in statistical surveys. Even when response is mandatory, as in the case of statistical programs that are critical to the nation, such as the population and economic censuses, the cooperation of respondents reduces costs and likely promotes accuracy (see National Research Council, 1995b, 2004d). Important elements in encouraging such cooperation are that r ­espondents believe that the data requested are important and legitimate for the government to collect, that they are being collected in an impartial, competent manner, and that the confidentiality of their responses will be protected. With regard to confidentiality, trust depends on providing respondents with realistic promises of confidentiality that the agency can reasonably expect to honor and then scrupulously honoring those promises. Respondent trust also depends on adopting practices that respect personal privacy, such as taking steps to minimize the intrusiveness of questions and the time and effort required to participate in a survey to the maximum extent possible that is consistent with the needs for information. When data are obtained from the administrative records of other fed- eral, state, or local government agencies, the same principle of trust applies in order to secure the fullest possible cooperation of the agencies with a statistical agency’s needs for the records and their documentation. Provider agencies need to believe that their records are important and legitimate for a statistical agency to obtain, that their restrictions on data access will be honored, and that the statistical agency will make every effort to minimize their burden in responding to the agency’s requests. Principle 4: A federal statistical agency must be independent from political and other undue external influence in developing, produc- ing, and disseminating statistics. A statistical agency must be able to provide credible information that may be used to evaluate the program and policies of its own department or the government as a whole. More broadly, a statistical agency must be a trustworthy source of objective, relevant, accurate, and timely informa- tion for decision makers, analysts, and others—inside and outside the

OCR for page 25
68 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY Many of the current practices in statistical agencies were developed through research they conducted or obtained from other agencies. Federal statistical agencies, frequently in partnership with academic researchers, pioneered the applications of statistical probability sampling, the national economic accounts, input-output models, and other analytic methods. The U.S. Census Bureau pioneered the use of computers for processing the census, and research on data collection, processing, and dissemination operations continues to lead to creative uses of automated procedures and equipment in these areas. Several federal statistical agencies sponsor research using academic principles of cognitive psychology to improve the design of questionnaires (see National Research Council, 1984), the clarity of data presentation, and the ease of use of electronic data collection and dissemi- nation tools such as the Internet. The history of the statistical agencies has shown repeatedly that methodological and operations research can lead to large productivity gains in statistical activities at relatively low cost. An effective statistical agency actively partners with the academic com- munity for methodological research. It also seeks out academic and industry expertise for improving data collection, processing, and dissemination operations. For example, a statistical agency can learn techniques and best practices for improving software development processes from computer scientists (see National Research Council, 2003e, 2004c). Research on Policy Uses Much more needs to be known about how statistics are actually used in the policy-making process, both inside and outside the government. Research about how the information produced by a statistical agency is used in practice should contribute to future improvements in the design, concepts, and format of data products. For example, public-use files of s ­tatistical microdata were developed in response to the growing analytic needs of government and academic researchers. Gaining an understanding of the variety of uses and users of an agency’s data is only a first step. More in-depth research on the policy uses of an agency’s information might, for example, explore the use of data in micro- simulation or other economic models, or go further to examine how the information from such models and other sources is used in decision mak- ing (see National Research Council, 1991a, 1991b, 1997a, 2000b, 2001b, 2003a, 2010e).

OCR for page 25
PART II: COMMENTARY 69 Practice 11: Professional Advancement of Staff An effective federal statistical agency has personnel policies that encour- age the development and retention of a strong professional staff who are committed to the highest standards of quality work for their agency and in collaboration with other agencies. There are several key elements of such a policy: • The required levels of technical and professional qualifications for positions in the agency are identified, and the agency adheres to these re- quirements in recruitment and professional development of staff. Position requirements take account of the different kinds of technical and other skills, such as supervisory skills, that are necessary for an agency to have a full range of qualified staff, including not only statisticians, but also experts in relevant subject-matter areas, data collection, processing, and dissemina- tion processes, and management of complex, technical operations. • Continuing technical education and training, appropriate to the needs of their positions, is provided to staff through in-house training pro- grams and opportunities for external education and training. • Position responsibilities are structured to ensure that staff have the opportunity to participate, in ways appropriate to their experience and ex- pertise, in research and development activities to improve data quality and cost-effectiveness of agency operations. • Professional activities, such as publishing in refereed journals and presentations at conferences, are encouraged and recognized, including pre- sentations of technical work in progress with appropriate disclaimers. Par- ticipation in relevant statistical and other scientific associations, including leadership positions, is encouraged to promote interactions with researchers and methodologists in other organizations that advance the state of the art. Such participation is also a mechanism for disseminating information about an agency’s programs, including the sources and limitations of the data pro- vided. Guidance from the Office of Science and Technology Policy issued in 2010 stresses the importance of participation in professional activities as a means of ensuring a culture of scientific integrity in federal agencies (see Appendix A). • Interaction with other professionals inside and outside the agency is fostered through opportunities to participate in technical advisory commit- tee meetings, establish and be active in listservs, blogs, and wikis that take advantage of Internet technology to foster informal exchanges on technical

OCR for page 25
70 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY matters, supervise contract research and research consultants on substantive matters, interact with visiting fellows and staff detailed from other agencies, take assignments with other relevant statistical, policy, or research organiza- tions, and regularly receive new assignments within the agency. • Participation in cross-agency collaboration efforts, such as the Federal Committee on Statistical Methodology and its subcommittees, is supported. Such participation not only benefits the professional staff of an agency, but also contributes to improving the work of the statistical system as a whole. • Accomplishment is rewarded by appropriate recognition and by affording opportunities for further professional development. The prestige and credibility of a statistical agency is enhanced by the professional visibil- ity of its staff, which may include establishing high-level nonmanagement positions for highly qualified technical experts. An effective statistical agency considers carefully the costs and ­ enefits— b both monetary and nonmonetary—of using contractor organizations, not only for data collection, as most agencies do, but also to supplement in-house staff in other areas.42 Outsourcing can have benefits, such as: providing experts in areas in which the agency is unlikely to be able to attract highly qualified in-house staff (e.g., some information technology functions), enabling an agency to handle an increase in its workload that is expected to be temporary or that requires specialized skills, and allowing an agency to learn from best industry practices. However, outsourcing can also have costs, including that agency staff become primarily contract managers and less qualified as techni- cal experts and leaders in their fields. An effective statistical agency maintains and develops a sufficiently large number of in-house staff, including math- ematical statisticians, survey researchers, and subject-matter specialists, who are qualified to analyze the agency’s data and to plan, design, carry out, and evaluate its core operations so that the agency maintains the integrity of its data and its credibility in planning and fulfilling its mission. Statistical agen- cies should also maintain and develop staff with the expertise necessary for effective technical and administrative management of contractor resources. An effective statistical agency has policies and practices to instill the highest possible commitment to professional ethics among its staff, as well as procedures for monitoring contractor compliance with ethical standards. When an agency comes under pressure to act against its principles—for example, if it is asked to disclose confidential information for an enforce- 42Only BLS and the Census Bureau maintain their own interviewing staffs.

OCR for page 25
PART II: COMMENTARY 71 ment purpose or to support an inaccurate interpretation of its data—it must be able to rely on its staff to resist such actions as contrary to the ethical principles of their profession. An effective agency ensures that its staff are aware of and have access to such statements of professional practice as the guidelines published by the American Statistical Association (1999) and the International Statistical Institute (1985), as well as to the agency’s own statements about protection of confidentiality, respect for privacy, standards for data quality, and similar matters. It endeavors in other ways to ensure that its staff are fully cognizant of the ethics that must guide their actions in order for the agency to maintain its credibility as a source of objective, reliable information for use by all. Practice 12: A Strong Internal and External Evaluation Program Statistical agencies that fully follow such practices as continual develop- ment of more useful data, openness about sources and limitations of the data provided, wide dissemination of data, commitment to quality and pro- fessional standards of practice, and an active research program will likely be in a good position to make continuous assessments of and improvements in the relevance and quality of their data collection systems. Yet even the best functioning agencies will benefit from an explicit program of internal and independent external evaluations, which frequently offer fresh perspectives. Such evaluations need to address not only specific agency programs, but also the agency’s portfolio of programs considered as a whole. Evaluating Quality Evaluation of data quality for a continuing survey or other kind of data collection program begins with regular monitoring of quality indicators that are readily available to users. For surveys, such monitoring includes unit and item response rates, population coverage rates, and information on sampling error, such as coefficients of variation. In addition, in-depth assessment of quality on a wide range of dimensions—including sampling and nonsampling errors across time and among population groups and geo- graphic areas—needs to be undertaken on a periodic basis and the results made public (National Research Council, 2007c). Research on methods to improve data quality may cover such areas as alternative methods for imputing values for missing data and alternative question wordings, using cognitive methods, to reduce respondent report-

OCR for page 25
72 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY ing errors. Methods for such research may include the use of “methods panels” (small samples of respondents with whom experiments are con- ducted by using alternative procedures and questionnaires), matching with administrative records, simulations of sensitivity to alternative procedures, and the like. The goal of the research is the development of feasible, cost- effective improved procedures for implementation. In ongoing programs for which it is disruptive to implement improve- ments on a continuing basis, a common practice is to undertake major research and development activities at intervals of 5 or 10 years or longer. Agencies should ensure, however, that the intervals between major research and development activities do not become so long that data collection pro- grams deteriorate in relevance, quality, and efficiency. Regular, well-designed program evaluations, with adequate budget sup- port, are key to ensuring that data collection programs do not deteriorate. Having a set schedule for research and development efforts will enable data collection managers to ensure that the quality and usefulness of their data are maintained and help prevent the locking into place of increasingly less optimal procedures over time. Evaluating Relevance In addition to quality, it is important to assess the relevance of an agency’s data collection programs. The question in this instance is whether the agency is “doing the right thing” in contrast to whether the agency is “doing things right.” Relevance should be assessed not only for particu- lar programs or closely related sets of programs, but also for an agency’s complete portfolio in order to assist it in making the best choices among program priorities given the available resources. Keeping in close touch with stakeholders and important user constituencies—through such means as regular meetings, workshops, con- ferences, and other activities—is important to ensuring relevance. Customer surveys can be helpful on some aspects of relevance, although they typically provide only gross indicators of customer satisfaction, usually with regard to timeliness and ease of use of data products. As discussed in the next section, including other federal statistical colleagues in this communication, both as users and as collaborators, can also be valuable. Statistical agencies commonly find that it is difficult to discontinue or scale back a particular data series, even when it has largely outlived its usefulness relative to other series, because of objections by users who have

OCR for page 25
PART II: COMMENTARY 73 become accustomed to it. In the face of limited resources, however, discon- tinuing a series is preferable to across-the-board cuts in all programs, which would reduce the accuracy and usefulness of both the more relevant and less relevant data series. Regular internal and external reviews can help an agency not only reassess its priorities, but also develop the justification and support for changes to its portfolio. Types of Reviews Regular program reviews should include a mixture of internal and ex- ternal evaluation. Agency staff should set goals and timetables for internal evaluations, which should involve staff who do not regularly work on the program under review. Independent external evaluations should also be conducted on a regular basis, the frequency of which should depend on the importance of the data and on how quickly changes in such factors as respondent behavior and data collection technology may adversely affect a program. In a world in which people and organizations appear increas- ingly less willing to respond to surveys, it becomes increasingly urgent to continually monitor response and have more frequent evaluations than in a more stable environment. In addition to program evaluations, agencies should seek outside reviews to examine priorities and quality practices across the entire agency. External reviews can take many forms. They may include recommen- dations from advisory committees that meet at regular intervals (typically every 6 months). However, advisory committees should never be the sole source of outside review because the members of such committees rarely have the opportunity to become deeply familiar with agency programs. External reviews can also take the form of a “visiting committee” using the NSF model;43 an academic-type visiting committee; a special committee established by a relevant professional association (see, e.g., American Sta- tistical Association, 1984); or a study by a panel of experts.44 43For examples of evaluations of NSF programs, see http://www.nsf.gov/od/oia/­ activities/cov/covs.jsp [February 2013]. 44See, e.g., National Research Council (1985a—study of the statistical programs of the Immigration and Naturalization Service); (1986—study of NCES); (1997b—study of BTS); (2004b—study of NCSES statistics on research and development expenditures); (2009a— study of BJS); and other National Research Council reports in the references.

OCR for page 25
74 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY Practice 13: Coordination and Collaboration with Other Statistical Agencies The U.S. federal statistical system consists of many agencies in dif- ferent departments, each with its own mission. Nonetheless, statistical agencies do not and should not conduct their activities in isolation. An effective statistical agency actively explores ways to work with other agen- cies to meet current information needs, through such means as seeking ways to integrate the designs of existing data systems to provide new or more useful data than a single system can provide. An effective agency is also alert for occasions when it can provide technical assistance to other agencies—including not only other statistical agencies, but also program agencies in its department—as well as occasions when it can benefit from such assistance in turn. Efforts to standardize concepts and definitions, such as those for in- dustries, occupations, and race and ethnicity, can contribute to effective coordination of statistical agency endeavors (see, e.g., National Research Council, 2004a, 2004b; also see Appendix A), as does the development of broad macro models, such as the system of national accounts. Efforts to standardize categories on survey questionnaires among agencies can enhance data comparability, while efforts by agencies to adopt common standards for data documentation and other metadata can contribute to the ease of the use of statistical products. Initiatives for interrelating and synchronizing data among statistical agencies (including individual data and address lists when permitted by law) can be helpful for such purposes as achieving greater efficiency in drawing samples, evaluating completeness of population coverage, and reducing duplication among statistical programs, as well as reducing respondent burden. Role of OMB The responsibility for coordinating statistical work in the federal gov- ernment is specifically assigned to the Office of Information and Regulatory Affairs (OIRA) in OMB by the Paperwork Reduction Act (previously, by the Federal Reports Act and the Budget and Accounting Procedures Act— see Appendix A). The Statistical and Science Policy Office in OIRA, often working with the assistance of interagency committees, reviews concepts of interest to more than one agency; issues standard classification systems (of industries, metropolitan areas, etc.) and oversees their periodic revision;

OCR for page 25
PART II: COMMENTARY 75 consults with other parts of OMB on statistical budgets; and, by reviewing statistical information collections as well as the statistical programs of the government as a whole, identifies gaps in statistical data, programs that may be duplicative, and areas in which interagency cooperation might lead to greater efficiency and added utility of data. The Statistical and Science Policy Office also is responsible for coordinating U.S. participation in inter­ national statistical activities.45 The Statistical and Science Policy Office encourages the use of admin- istrative data for statistical purposes, when feasible, and works to establish common goals and norms on major statistical issues, such as confidentiality. It sponsors and heads the interagency FCSM, which issues guidelines and recommendations on statistical issues common to a number of agencies, typically by working through subcommittees, and also hosts conferences that facilitate professional interaction and development (see Federal Com- mittee on Statistical Methodology, 1978a–2005).46 It encourages CNSTAT at the National Research Council to serve as an independent adviser and reviewer of federal statistical activities. The 1995 reauthorization of the Paper­ ork Reduction Act created a statutory basis for the existing Inter- w agency Council on Statistical Policy, formalizing an arrangement whereby statistical agency heads participate with OMB in activities to coordinate federal statistical programs (see Appendixes A and B). Forms of Interagency Collaboration There are many forms of interagency collaboration and coordination. Some efforts are multilateral, some bilateral. Many result from common interests in specific subject areas, such as economic statistics, statistics on people with disabilities, or statistics on children or the elderly. The U.S. O ­ ffice of Management and Budget (2011:Ch. 3) describes several inter- agency collaborative efforts, such as joint support for research that fosters new and innovative approaches to surveys; the development of a statistical community of practice for agencies to share, standardize, and improve 45The Statistical and Science Policy Office, formerly the Statistical Policy Office, was renamed to reflect added responsibilities with respect to the 2001 Information Quality Act standards and guidelines, OMB’s guidance on peer review planning and implementation, and evaluations of science underlying proposed regulatory actions. 46The papers from the most recent FCSM research conference are available at: http:// www.fcsm.gov/events/papers2012 [February 2013].

OCR for page 25
76 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY statistical protocols and tools; a systemwide initiative to facilitate the sta- tistical uses of administrative records under the leadership of an FCSM sub­ ommittee; and implementation of comparable measures of disability c on major household surveys. A common type of bilateral arrangement is the agreement of a program agency to provide administrative data to a statistical agency to be used as a sampling frame, a source of classification information, or a summary compilation to check (and possibly revise) preliminary sample results. The Bureau of Labor Statistics, for example, benchmarks its monthly establish- ment employment reports to data supplied by state employment security agencies. Such practices improve statistical estimates, reduce costs, and eliminate duplicate requests for information from the same respondents. In other cases, federal statistical agencies engage in cooperative data collec- tion with state counterparts to let one collection system satisfy the needs of both. A number of such joint systems have been developed, notably by the Bureau of Labor Statistics, the National Agricultural Statistics Service, the National Center for Education Statistics, and the National Center for Health Statistics. Another example of a joint arrangement is one in which a statistical agency contracts with another to conduct a survey, compile special tabula- tions, or develop models. Such arrangements make use of the special skills of the supplying agency and facilitate use of common concepts and meth- ods. The Census Bureau conducts many surveys for other agencies, both the National Center for Health Statistics and the National Agricultural Statistics Service receive funding from other agencies in their departments to support their survey work, and the National Center for Science and Engi­ neering Statistics receives funding from agencies in other departments to support several of its surveys (see U.S. Office of Management and Budget, 2012b:Table 2). The major federal statistics agencies are also concerned with inter­ national comparability of statistics. Under the leadership of OMB’s S ­ tatistical and Science Policy Office, they contribute to the deliberations of the United Nations Statistical Commission, the OECD, and other inter­ national organizations; participate in the development of international standard classifications and systems; and support educational activities that promote improved statistics in developing countries. Statistical agencies also learn from and contribute to the work of established statistical agen- cies in other countries in such areas as survey methodology, record linkage, confidentiality protection techniques, and data quality standards. Several

OCR for page 25
PART II: COMMENTARY 77 statistical agencies run educational programs for government statisticians in developing countries. Some statistical agencies have long-term coopera- tive relationships with international groups: examples include the Bureau of Labor Statistics with the International Labor Organization, the National Agricultural Statistics Service with the Food and Agriculture Organization, the National Center for Education Statistics with the International Indica- tors of Education Systems project of the OECD, and the National Center for Health Statistics with the World Health Organization. To be of most value, the efforts of statistical agencies to collaborate as partners with one another need to involve the full range of their activities, including definitions, concepts, measurement methods, analytical tools, dissemination modes, and disclosure limitation techniques. Such efforts should also extend to policies and professional practices, so that agencies can respond effectively and with a coordinated voice to such government-wide initiatives as data quality guidelines, privacy impact assessments, institu- tional review board requirements, and others. Collaboration efforts should also encompass the development of data, especially for emerging policy issues (see, e.g., National Research Council, 1999a, 2007b). In some cases, it may be not only more efficient, but also productive of needed new data for agencies to fully integrate the designs of existing data systems, such as when one survey provides the sampling frame for a related survey. In other instances, collaborative efforts may identify ways for agencies to improve their individual data systems so that they are more useful for a wide range of purposes. Collaboration on ways and means of using alternative data sources, such as administrative records, should be pursued so that the entire statisti- cal system can move forward to improve the relevance, accuracy, timeliness, and cost-effectiveness of their data programs. Toward this goal, in 2008 the FCSM established a Subcommittee on Administrative Records, which is working to develop standards and provide guidance to statistical agencies that will facilitate not only use of administrative records, but also evalua- tion of their quality and fitness to be part of an agency’s data collection, estimation, and evaluation programs. This subcommittee has released two products from its work: one is a compilation of case studies of successful statistical uses of administrative data (Federal Committee on Statistical Methodology, 2009); the other is a checklist tool for assessing the quality of administrative data (Federal Committee on Statistical Methodology, 2013). Two continuing collaborative efforts for providing statistical informa- tion to the public in a broad area of interest are the Federal Interagency

OCR for page 25
78 PRINCIPLES AND PRACTICES FOR A FEDERAL STATISTICAL AGENCY Forum on Aging-Related Statistics and the Federal Interagency Forum on Child and Family Statistics. The former was established in the mid-1980s by the National Institute on Aging, in cooperation with the National C ­ enter for Health Statistics and the Census Bureau. The forum’s goals in- clude coordinating the development and use of statistical databases among federal agencies, identifying information gaps and data inconsistencies, and encouraging cross-national research and data collection for the aging population. The forum was reorganized in 1998 to include 6 new agencies and has grown since then to include 15 agencies. The forum develops a periodic indicators chart book, which was first published in 2000 and was most recently issued in 2012 (Federal Interagency Forum on Aging-Related Statistics, 2012). The Federal Interagency Forum on Child and Family Statistics was formalized in a 1994 executive order to foster collaboration in the collec- tion and reporting of federal data on children and families. Its membership currently includes 22 statistical and program agencies. The forum’s reports (e.g., Federal Interagency Forum on Child and Family Statistics, 2012) describe the condition of America’s children, including changing popula- tion and family characteristics, the environment in which children are liv- ing, and indicators of well-being in the areas of economic security, health, behavior, social environment, and education. No single agency, whether a statistical agency or program agency, could have produced the forum reports alone. Working together in this way, fed- eral statistical agencies contribute to presenting data in a form that is more relevant to policy concerns and to a stronger statistical system overall. Simi- lar collaborative efforts aimed at integrating not only data dissemination, but also data collection and estimation, using traditional and nontraditional data sources, are critically important to improving the relevance, accuracy, timeliness, and cost-effectiveness of the output from the nation’s federal statistical system.