A commitment to quality and professional standards of practice are two recognized elements in the effective operation of a statistical agency (National Research Council, 2001:5). The commitment should involve several key elements: use of modern statistical theory and sound statistical practice, strong staff expertise, an understanding of the validity and accuracy of the data, and quality assurance programs that include documentation of concepts and definitions, as well as data collection methodologies and measures of uncertainty and possible sources of error. In view of the importance of the overall environment for quality, early in our work we reviewed these elements of organizational and management commitment to quality.
As a benchmark, the panel turned to the earlier NRC study Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies (1999) and evaluated the progress that SRS has made to implement the recommendations for quality improvement in that report. That report found that SRS had a good track record of improving the data quality and meeting statistical standards in the recent past, and it recommended additional steps to ensure that standards are met across SRS operations. With respect to the R&D expenditure surveys, the previous NRC report recommended three specific steps to improve data quality: (1) require all contractors provide methods reports that address quality standards; (2) continue recent efforts to provide NSF staff with professional development opportunities to improve statistical skills; and (3) continue to develop and strengthen a program of methodological research, undertaking rigorous analysis of the data collected to assess the quality of the data relative to concepts they are supposed to measure. These issues will be addressed further in the panel’s final report.
Since the 1999 NRC report was published, SRS has implemented several specific improvements and has laid the groundwork for others. The change of name to the Division of Science Resources Statistics is one indication of the commitment of management to improving the statistical foundation of the R&D data. Other specific actions that have improved or will lead to improvements include the addition of mathematical statistics expertise to the SRS senior staff, the incorporation of specific contractual obligations for measuring and presenting measures of quality in contracts with the data collection organizations responsible for conducting the several survey programs, and development of a long-term plan for methodological research for each program.
The 1999 NRC report focused considerable attention on perceived staffing issues in SRS, calling for enhancing staff expertise through professional development activities, augmenting staff expertise in statistical methods and subject-matter skills, and developing a more interactive relationship with outside researchers. Over the past two years, SRS has added four full-time senior statisticians and has secured the services of experts in survey design and cognitive aspects of questionnaire design. These staff and consultant enhancements have served to focus attention on statistical methods for the R&D surveys.