The potential for obtaining additional information that is useful to NIOSH from current and future survey results.
Each of these issues was considered by the committee to constitute a task to be accomplished during its review. To the extent that information was available to it, the committee assessed each of these issues and made recommendations, when appropriate.
The objective of SRUP was “to provide information to develop educational interventions for specific populations and to increase the frequency and effectiveness of respirator use in the workplace.”3 The survey was designed to provide estimates of the number of establishments and employees who used respirators in a recent 12-month period by type of respirator and type of use, and to collect data on the characteristics of the respirator program at the establishment; medical fitness to wear respirators; characteristics of respirator training at the establishment; usefulness of NIOSH approval labels and respirator manufacturers’ instructions; substances protected against by the use of respirators; and fit-testing methods used for respirators. The target population of the survey was private-sector establishments with employment covered by unemployment insurance programs that were included in the 1999 Survey of Occupational Injuries and Illnesses (SOII). The data to illuminate these survey objectives had never been systematically collected before from such a large number of establishments covering so many industries and size classes.
The 2001 SRUP was critiqued with the thought that the analysis and recommendations would help guide a subsequent survey of this scope and nature and that the critique would be based on published documentation provided by BLS and NIOSH. This was, in many ways, a landmark survey for both agencies. Although it built on the extensive sampling capacity of BLS that came with years of experience in conducting the SOII, it was experimental in its selection of questions and sample units.
Finding 1: The survey was an important first step in collecting respiratory protection data from a probability sample. As such, it was a worthwhile learning experience for both NIOSH and BLS.
It soon became obvious to BLS management that the survey suffered from inadequate funding for its scope and size. One casualty of the lack of funding was the inability of BLS to follow up with a full set of documentation.
Finding 2: There was insufficient documentation and detail for the committee to reconstruct key aspects of the methodology and to fully understand the survey design and implementation.
The lack of documentation was particularly true for the sample design, sample weighting, content development, and handling of missing data through a type of imputation procedure known as the “hotdeck,” in which missing lines of data are replaced by sampled complete data records. Appropriate documentation and access to this documentation are essential to evaluating and reproducing a survey of this type.
Development of the questionnaire was a joint responsibility between NIOSH and BLS. NIOSH participated in the development of the survey questionnaire by providing BLS with direction on technical subjects such as regulations, respirator types and uses, and specific substances that require respirator use. Although the survey was appropriately subjected to cognitive experimentation and field testing, the resulting questions tended to be focused more on items that were measurable from the perspective of the employer respondents, and the questions tended to elicit information on regulatory compliance rather than respirator certification and use.
Finding 3: The survey questionnaire was not adequately related to the initial survey objectives.
The committee found that it was difficult to evaluate the adequacy of pretesting because, in general, the documentation about the details of the testing, the resulting instrument revisions, and the efficacy of those revisions were inadequate. Although the testing appears, overall, to have uncovered a large number of problems, it is difficult to determine the effectiveness of the solutions without explicit examples or results of their retesting.
The field test provided valuable insights that enhanced the survey operations. For example, the overall field-test response rate of 80 percent was fairly close to the reported survey response rate of 75.5 percent. However there were many issues with the pretesting, including the following:
Finding 4: The field test paid little attention to exploring validation procedures that might have provided information on the quality of data collected or motivated the need for a formal quality assessment of the data, and thus missed an opportunity to improve understanding of the quality of the SRUP data.
Finding 5: Many features of the survey were not user friendly or optimally designed to aid navigation.