7

Key Actions to Move Forward with an Ideal National Occupational Safety and Health Surveillance System

Throughout the report, the committee uses a framework that describes a surveillance system as a set of processes that are enabled by different components. In the previous chapter, promising ongoing surveillance activities were reviewed and situated within this framework. This chapter examines four actions essential for moving forward with the national occupational safety and health (OSH) surveillance system that is needed to improve worker safety and health:

1. Set forth a clear rationale and prioritization for surveillance including quantifying the economic and health burden of occupational illness and injury;
2. Coordinate surveillance strategies and operations among key agencies, organizations, and stakeholders to ensure a “system-of-systems” architecture and align strategic planning and operations;
3. Use information technology effectively to meet surveillance objectives; and
4. Enhance training and support for surveillance practitioners with appropriate skills and knowledge to conduct, analyze, and recommend worker safety improvements based on OSH surveillance.

SET FORTH A CLEAR RATIONALE AND PRIORITIZATION FOR OSH SURVEILLANCE

The committee’s deliberations were guided by the premise that a 21st-century OSH surveillance system needs to collect, interpret, and analyze

relevant data at the lowest cost feasible, and then rapidly and effectively disseminate that information to those who need to know. Meeting this goal requires a system that is cost effective; i.e., it will need to collect data that could have the biggest impact on improving worker safety and health, while minimizing the cost of collecting such data, and avoiding the collection of data that hold little value (Gold et al., 1996; Haddix et al., 1996). A cost-effective surveillance system balances the importance of collecting data about factors that can be modified to improve OSH with the costs of collecting each data element. It also measures costs and benefits from a societal perspective, consistent with the standards that have been widely accepted for cost-effectiveness analyses (Gold et al., 1996; Haddix et al., 1996; Muennig, 2008). This section examines four steps toward providing the basis and priorities for national OSH surveillance:

• quantifying the health and economic burdens,
• enhancing return from current resources,
• effectively allocating resources (evidence-based), and
• quantifying the fiscal rationale for coordination.

Quantifying the Health and Economic Burden

As defined in Chapter 1, the costs of a surveillance system include not only the costs of conducting surveillance activities, but also the costs associated with the health and productivity consequences of occupational exposures, injuries, illnesses, and mortality on workers, their families, and society. Currently, there is no regular, standardized reporting on the overall economic burden of occupational illness, injury, and death in the United States, even though the most recent independent study estimated the burden to be $250 billion annually (Leigh, 2011). Given the enormity of this recent estimate, a regular national report on the financial and health burdens of occupational illnesses, injuries, and fatalities is essential for monitoring whether the United States is making measurable and meaningful progress in improving worker safety over time.

Resources to produce a national estimate on a regular basis could overcome some of the current methodological and data limitations. For example, the method for attributing mortality and morbidity to workplace illness and injury, known as attributable fractions, can lead to very different estimates of economic and health burden relative to official counts of workplace injuries, illnesses, and deaths, which are substantially lower than the estimates based on attributable fractions (Steenland et al., 2003).

The authors of a recently published, comprehensive framework for defining and measuring the health and economic burden of occupational “injury, disease, and distress” in the United States concluded that “the

importance of documenting burden is to use it to plan investment in occupational safety and health risk prevention, risk management, and research and to prompt decision-makers to allocate funds for such investments” (Schulte et al., 2017). Likewise, the European Union (EU) has recently acknowledged the importance of regularly assessing the economic burden of occupational illnesses, injuries, and fatalities for Europe. The EU is currently implementing a process to routinely measure this burden and has recently published its first report on the availability and quality of data for developing European estimates of the burden of occupational illnesses, injuries, and fatalities (EU-OSHA, 2017).

An improved OSH surveillance system is the tool needed by policy makers, industry, workers, and other stakeholders to prioritize and target interventions that have been shown through research or practice to be most effective in improving worker safety and health. The surveillance system can then be used to monitor progress toward reducing the total economic burden of occupational illnesses, injuries, and fatalities over time.

Conclusion: Identifying the areas of greatest need for OSH interventions through use of an improved national surveillance system and then targeting effective OSH interventions based on previous research and evidence is likely to produce significant and substantial savings to employers, employees, and society and increase worker health and well-being.

If an improved national surveillance system is intended to inform actions to improve OSH outcomes, it is important to assess the value of both the investments in an improved surveillance system as well as the OSH outcomes. Cost-effectiveness analysis (CEA) is commonly used to measure the “value” of investments to improve health because it measures health outcomes in units other than dollars (Gold et al., 1996). The problem with using classical methods such as return on investment or cost-benefit analysis in health settings is that improvements in health must be valued in dollars. This requirement creates an ethical dilemma, because rather than value all lives equally, return on investment and cost-benefit calculations value lives differently, usually based on the wages or productivity of individuals in the economy. As a result, children and seniors are valued less, and high-income members of the workforce have the highest value. This limitation can be overcome by assigning average economic values to a life. For example, the Environmental Protection Agency uses mortality risk reduction, based on the value of a statistical life rather than the value of individual lives, when conducting analyses of societal costs related to exposure to environmental hazards (EPA, 2010). CEA, in contrast, places an equal value on health outcomes, regardless of the economic productivity of those who benefit, and thus avoids explicit valuation of life.

CEA generally measures health outcomes as lives saved, increased life expectancy (i.e., years of life saved), or reductions in adverse health

outcomes (e.g., reduced incidence of injury). Cost-utility analysis (CUA) is a special case of CEA in which health outcomes are measured using quality-adjusted life years (QALYs) (Gold et al., 1996). QALYs capture both improvements in life expectancy as well as the quality of life for those years—that is, QALYs account for both mortality and morbidity (Gold et al., 1996). Social return on investment (SROI) has been proposed as an alternative method for more broadly valuing public health outcomes (Banke-Thomas et al., 2015), but the scientific evidence base for this method is not nearly as well-established as the broad evidence base for CEA and CUA.

Conclusion: Because cost-benefit analysis and return on investment analysis value outcomes in dollar terms only, the committee finds that CEA and CUA are more appropriate methods for determining whether OSH surveillance systems produce value, i.e., improvement in health outcomes at a reasonable cost. CEA and CUA are preferred because they give equal weight to all workers, in contrast to other methods, which value individuals based on their salaries or wages.

An assessment of OSH surveillance systems, therefore, should measure and balance the costs of data collection, analysis, and dissemination versus measurable reductions in mortality and morbidity (including time lost from work) that can be attributed to actions taken based on OSH surveillance.

Enhancing Return from Current Resources

Regular national reporting on the economic burden of occupational illnesses, injuries, and fatalities can also assist federal agencies, state and local governments, employers, and employee groups to address surveillance objectives more effectively with existing resources. Burden-of-illness studies have been conducted in the United States for decades and have estimated the economic burden of diseases for the entire population (e.g., Rice, 1967; Cooper and Rice, 1976; Rice et al., 1985); for specific conditions (e.g., Thorpe et al., 2015); or for certain outcomes, such as fatalities (Luo and Florence, 2017). In recent years, federal agencies such as the National Institutes of Health have called for greater use of such studies in prioritizing areas for funding to combat heart disease, cancers, diabetes, etc., although empirical studies suggest at most only a moderate association between funding priorities and population disease burden (Gross et al., 1996; Gillum et al., 2011). Existing federal efforts such as the Centers for Disease Control and Prevention’s (CDC’s) Web-based Injury Statistics Query and Reporting System (WISQARS) provide another avenue for enhancing existing resources. WISQARS provides an interactive, online database of fatal and nonfatal injury, violent death, and cost-of-injury data that could be modified to report separately on occupational illnesses, injuries, and fatalities.

Finally, a regular report on the economic burden of occupational illnesses, injuries, and fatalities could serve an important role in guiding national funding priorities to ensure that current funding is being targeted cost effectively, i.e., where the burden is greatest, where effective interventions exist to reduce work-related mortality or morbidity, and where intervention costs, including the costs of regulation, are minimized in producing improved occupational health outcomes.

Effective (Evidence-Based) Allocation of Resources

Another benefit of a regular national report on the economic burden of occupational illnesses, injuries, and fatalities would be to establish funding priorities by federal agencies with OSH responsibilities. The challenge when new resources become available or resources are newly constrained is to determine the most cost-effective way to allocate changing resource levels. Agencies can use expert panels and the evidence base established from previous studies to determine where effective interventions exist for reducing adverse outcomes, and to target those conditions where health outcomes can be improved in the most cost-effective manner.

Quantification of Fiscal Rationale for Coordination

Because there is no single government agency responsible for all aspects of OSH, coordination across agencies is essential to avoid unnecessary duplication of effort and to maximize efficient and effective use of existing resources. The Occupational Safety and Health Administration (OSHA) has primary responsibility for federal OSH activities, but the National Institute for Occupational Safety and Health (NIOSH), the Bureau of Labor Statistics (BLS), other federal agencies, and state agencies play important roles as well through surveillance and the surveys they administer. The Department of Health and Human Services (HHS) maintains national health accounts that collect data to monitor trends in national spending for health care services. Some portion of the spending is attributable to occupational illnesses, injuries, and fatalities, but HHS national health accounts do not separately identify OSH costs and expenditures. Better coordination could lead to efforts to partition national health spending into occupational and non-occupational health spending estimates.

Recommendation I: NIOSH should coordinate with OSHA, BLS, and other relevant agencies to measure and report, on a regular basis, the economic and health burdens of occupational injury and disease at the national level. This report should also attempt to address the contribution of implemented interventions in reducing these burdens. The advantages

of a regular, standard report on national economic burden of occupational injury and disease include:

• focusing attention on the significant burden that already exists,
• measuring progress over time in reducing those burdens and improving worker safety and health,
• improving the allocation of existing resources to improve health outcomes, and
• establishing priorities.

Research, such as to establish the fraction of disease and injuries attributable to occupational exposures, will be necessary to continually improve the quality of burden estimates that can be produced.

COORDINATE SURVEILLANCE STRATEGIES AND OPERATIONS AMONG KEY AGENCIES, ORGANIZATIONS, AND STAKEHOLDERS

As was described in Chapter 1, surveillance entails the collection and analysis of data, followed by the interpretation and dissemination of information to relevant actors to meet public health and prevention objectives. The legal and organizational context in the United States, and to a lesser extent in other countries, is such that many aspects of OSH surveillance are under the mandate of different agencies, which may have different objectives. However, to have an efficient OSH surveillance system for the country, surveillance activities must be coordinated within and among the different agencies to allow the attainment of national objectives, while respecting and further advancing the objectives of each agency. There are three ways in which improvements could be made:

• Implement a system of systems approach,
• Improve the alignment of existing systems, and
• Coordinate communications.

Implement a System of Systems Approach

One approach to coordinating OSH surveillance at the national scale is to create a “system of systems,” which has been defined as “a collection of task-oriented or dedicated systems that pool their resources and capabilities to obtain a new, more complex ‘meta-system’ which offers more functionality and performance than simply the sum of the constituent systems” (Popper et al., 2004). Such a system is created by connecting otherwise independent systems, which have operational and managerial independence,

are geographically distributed, and are heterogeneous (Maier, 1998). In contrast to large, monolithic systems that are controlled centrally through a hierarchical structure with clear lines of reporting (Fisher, 2006), systems of systems are characterized by distributed control and cooperation (Keating et al., 2003). In response to the 2009 influenza pandemic, this strategy was used to rapidly develop a national syndromic surveillance system covering more than half of all emergency department visits in the country (Olson et al., 2011). In that example, city and state syndromic surveillance systems adopted a common model for reporting aggregated data to a central system, which then combined the data to produce summaries and identify trends at a national scale.

Conclusion: Given the current context of OSH surveillance, development of a national “system of systems” is more likely to result in rapid and sustainable gains in OSH surveillance as compared to development of a new national monolithic system, which is likely to require substantial legislative changes and substantial additional resources.

Improve the Alignment of Existing Systems

Although a “system of systems” is a compelling strategy for developing a national OSH surveillance system, creating such a system can be complicated by technical, human, and organizational differences among existing systems (Wells and Sage, 2008) as well as regulatory impediments. These complications are likely to be more pronounced if the intent is to couple individual systems tightly. However, a loose coupling is possible, where the architecture of the overall system is essentially a set of standards that allow meaningful communication or exchange of data and information among systems (Maier, 1998). For example, the influenza syndromic surveillance system described earlier used a loose coupling, requiring agreement only on a form for reporting aggregated data. The distinction between data¹ and information² is important, because data, especially at the individual level, are often difficult to share across organizations, while information is generally easier to share. Ultimately, what must drive the design of a system of systems, including which data and information should be shared, are the OSH objectives that are only possible or feasible to attain through interoperation of otherwise independent systems. The new information that the national system of systems could produce would be an “emergent property,” or a novel contribution to OSH surveillance not feasibly available from

___________________

¹ Data are raw facts, which generally afford little insight on their own (e.g., the age of a worker).

² Information is obtained by placing data in context, for example through analysis or combination with other data (e.g., counts or rates of injuries in a specific age group).

any existing system on its own but attainable through a national system. This new information should allow objectives described in Chapter 2 to be achieved more completely and more efficiently. For example, enhanced data and information sharing through coordinated planning and operations of existing surveillance systems should allow more accurate measurement of the burden of work-related illness, enable clearer identification of working populations at high risk for work-related injury or illness, and provide richer data to generate hypotheses and conduct epidemiological research.

Coordinate Communication of Information from Surveillance

An essential part of surveillance is ensuring that the information, produced by analyzing and interpreting collected data, is disseminated to stakeholders who are in positions to use that information to protect and improve worker safety and health. As described in Chapter 3, these stakeholders are many and include policy makers; federal, state, and local government agencies; individual employers and workers; industry and worker organizations; insurers; health and safety professionals; educators; and researchers, as well as the health care community. Timeliness in using surveillance results for planning and action, a key component in the evaluation of a surveillance system (CDC, 2001), means providing outputs from surveillance in usable formats that meet the needs of the information user(s).

Ongoing dissemination of surveillance information takes many forms, ranging from annual reporting of aggregated data to placing surveillance indicators on interactive websites and making them available through user queries. BLS, NIOSH, and OSHA currently have separate websites where they post surveillance findings (BLS, 2017a,b; NIOSH, 2017a,b,c,d,e,f,g,h; OSHA, 2017). Each of these sites has interactive components. BLS issues press releases and provides annual reports for the Census of Fatal Occupation Injuries (CFOI) and the Survey of Occupational Injuries and Illnesses (SOII). More extensive analyses of surveillance data are published in BLS periodicals. New findings from NIOSH and other agencies are periodically posted but with irregular time periods between postings. NIOSH findings based on more extensive analysis of surveillance data are published in peer-reviewed journals. The NIOSH surveillance home page includes links to NIOSH webpages for its various surveillance systems and functions that enable the user to query BLS employment, CFOI and SOII data, and several of the NIOSH surveillance data sources.

NIOSH also maintains a clearinghouse for surveillance reports and educational materials and surveillance tools generated by all the state-based surveillance programs. While these web-based resources represent a significant improvement in information dissemination in recent years, there is currently no central site or compilation of the information produced by the

various agencies. Even within the same agency there are separate websites for different conditions (e.g., NIOSH: respiratory disease, and state-based surveillance; NIOSH, 2017a,b,c,d,e,f,g,h). This contrasts with the dissemination of data in Great Britain by their Health and Safety Executive, where, for example, both employer and worker survey data are available through a single site (HSE, 2016).

As with any surveillance system, a national OSH system of systems must also communicate information to stakeholders, ideally in a manner that is consistent with the reporting of information by the OSH agencies that operate component surveillance systems. Therefore, in addition to coordinating the establishment of objectives and the technical activities for a national OSH surveillance system, communication from the national system should also be coordinated in an ongoing manner.

The information generated by surveillance systems includes measures, such as rates of exposures and outcomes. To be clear, centralizing information does not imply centralizing data, which is impractical and unnecessary on a national scale. Information can be centralized, for example, by having each participating system make the information it intends to disseminate available in a standard, machine-readable format, which would allow a central system to access and integrate information from different systems routinely in an automated manner. Once the information is in one location, it can be made available through a clearinghouse such as the one established by NIOSH. Once information is centralized, it is also possible to coordinate more urgent dissemination activities, such as through the health alert network maintained by the CDC. The committee returns to the topic of dissemination of surveillance information in the following section, where we consider the potential for informatics to enable OSH surveillance processes, including dissemination.

USE INFORMATION TECHNOLOGY EFFECTIVELY

An informatics perspective, where the focus is on the optimal use of data and knowledge to meet OSH surveillance objectives, allows a principled assessment of the potential benefit of new technologies to enable OSH surveillance processes. The ongoing and unprecedented gains in fundamental technologies for storing, communicating, and analyzing data have produced a dizzying array of methods and tools with the potential to automate, advance, or replace many existing processes in OSH surveillance, including data collection, data management, analysis, and dissemination. One strategy for making sense of these many opportunities is by viewing OSH surveillance through the lens of biomedical informatics, asking how data and knowledge can be used optimally to solve problems and make decisions in OSH surveillance (Shortliffe, 2014). From this perspective, the

OSH surveillance processes and objectives are central and it is possible to consider, ideally from an evidence-based perspective, which innovations in information technology are likely to make surveillance processes (data collection, analysis and interpretation, and dissemination of information) more efficient and effective. There is limited direct evidence in the biomedical literature about the effective use of existing and emerging information technologies for OSH surveillance, but it is possible to draw relevant insights from the literature on public health surveillance, from biomedical informatics more generally, and from advances in data management and analysis in domains beyond health and health care (e.g., electronic health records, machine learning, and social networks). These insights are important in identifying potential roles of new technologies in individual OSH surveillance systems and to understand how these technologies can contribute to the development of a national OSH surveillance system.

Building Capacity in Information Technology Expertise

Even when there is evidence that new information technologies can be beneficial, it can be challenging to achieve these benefits in practice. Expertise in information technology (IT) and informatics must exist within OSH surveillance agencies to realize the potential benefits. People with these skills and knowledge are critical for reviewing technical evidence, focusing attention on OSH priorities driving investments in information technology, and ensuring successful implementation of adopted technologies. Even if agencies make extensive use of contract agencies to deliver IT services, it remains essential to have a core complement of biomedical informatics expertise to set strategy and to ensure the effective management of contracted resources. Several academic institutions offer formal graduate degrees in biomedical informatics, emphasizing a range of competencies that are well matched with the needs of the OSH community (Kulikowski et al., 2012).

For a variety of reasons, however, it is difficult for OSH agencies to hire and retain staff with the knowledge and skills needed to plan, manage, evaluate, and use new information technologies. Consequently, agencies tend to employ fewer staff members with informatics expertise than they need and the turnover among these employees can be high. Turnover is problematic because most people with informatics expertise will know little about OSH when they join an agency, but over time they will acquire deeper knowledge and experience in OSH, thereby becoming increasingly valuable to the agency.

There is an acknowledged shortage of expertise in biomedical informatics, especially as it relates to public health surveillance (Edmunds et al., 2014). In this context, OSH agencies must be strategic in how they recruit, cultivate, and retain this expertise. In the longer term, supporting expansion

of training programs may be beneficial. In the medium term, increasing use of internship and fellowship opportunities, such as the public health informatics fellowships at CDC, can introduce people with biomedical expertise to the challenges and opportunities in OSH surveillance.

Recommendation J: NIOSH should build and maintain a robust internal capacity in biomedical informatics applied to OSH surveillance.

In the near term:

• Assess the need within the agency for expertise in biomedical informatics in the context of current and future demand, recognizing that it will be important to train informatics talent in OSH surveillance and then to work to retain talented individuals who develop knowledge at the intersection of the informatics discipline and OSH applications;
• Create an organizational strategy for deploying and making optimal use of expertise in biomedical informatics to support the planning and conduct of OSH surveillance;
• Develop a plan for hiring, including consideration of steps such as reaching out to academic programs, advertising in different venues, and offering internships; and
• Develop a plan for retention, including opportunities for continuing education.

Systems Architecture and Overall System Functioning

Next steps forward for OSH surveillance need to focus on technologies that can contribute to the overall functioning of a surveillance system by supporting activities across multiple processes. For example, standard controlled terminologies enable the consistent representation of data within many surveillance processes and the communication of data between processes within a system. Technologies for integrating data across systems, or allowing distributed analysis across systems, is another important technology within a national system—something that would not be possible with any single system.

Controlled Terminology Development and Standards and Harmonization

Controlled terminologies (or, data standards) can ensure that a given concept is recorded consistently by different people over time within and across systems by enumerating the accepted ways for concepts to be encoded. These terminologies enable communication between systems, with interoperability achieved more easily when different systems use the same

controlled terminologies. Interoperability between systems, however, also requires agreement on a communication or messaging standard. In other words, a controlled terminology defines what is in a message, and the messaging standard defines the structure of the message.

As reviewed in Chapter 6, many organizations have developed or adopted a number of OSH controlled terminologies in support for their activities. For the most part, these terminologies have evolved independently to meet the needs of different organizations. While each terminology represents some aspect of OSH, when taken together, the existing terminologies do not cover all of OSH in a consistent manner. These terminologies do however represent many concepts central to OSH surveillance, including the occupation of a worker (Standard Occupational Classification, or SOC) and the industry of a business establishment (the North American Industry Classification System, or NAICS). A smaller terminology was also created for use with the census by merging subsets of the SOC and the NAICS (Bureau of the Census codes). Controlled terminologies also are used to represent outcomes or events (i.e., injuries and illnesses) within OSH surveillance systems, including the International Classification of Disease (ICD, with modifier codes to represent details of an injury), the Occupational Injury and Illness Classification System (OIICS), and the Workers Compensation Insurance Organizations (WCIO) codes.

The Bureau of the Census codes are directly related to the SOC and NAICS codes, as they were explicitly derived from these other two coding systems. There are, however, no other existing mappings or explicit linkages between the various controlled terminologies used for OSH surveillance. Consequently, it is possible to encode combinations of occupation and industry which are not possible in practice. Another consequence is that outcomes coded using different controlled terminologies are not easily compared (e.g., ICD and OIICS, WCIO and OIICS), hindering the integration of data across different surveillance systems in the absence of crosswalks between coding systems (Koeman et al., 2013).

One possible solution to this problem is to develop and maintain a mapping between these related terminologies, or what is called a meta-thesaurus (Harber and Leroy, 2017). An example in the biomedical domain is the Unified Medical Language System (UMLS), a compendium of controlled terminologies developed and maintained by the National Library of Medicine of the National Institutes of Health, which links related concepts or terms across dozens of controlled terminologies. Of the controlled terminologies discussed above, only the ICD is included in the UMLS.

Recommendation K: NIOSH should work with the National Library of Medicine to incorporate core OSH surveillance terminologies, including those for industry and occupation, into the Unified Medical Language

System (UMLS). The creation and maintenance of mappings among OSH terminologies and between OSH terminologies and other relevant terminologies already included in the UMLS should be considered.

In the near term:

• Establish an inventory of relevant OSH terminologies;
• Develop use cases that benefit from the existence of mappings across OSH terminologies; and
• Prioritize terminologies in terms of the value that accrues from incorporating them into the UMLS.

In the longer term:

• Incorporate highest-priority OSH terminologies into the UMLS.

One topic for which data representation standards are not currently available is public health interventions. While terminologies do exist for recording clinical interventions that target individual patients (Hanser et al., 2009), representation standards for interventions that target workplaces or populations are not available. A standard for these interventions would allow public health officials and others to record actions taken to prevent and control occupational injury and disease in a systematic manner. While there have been calls for more consistent reporting of interventions in epidemiological studies (Des Jarlais et al., 2004), this approach has not been adopted widely by public health agencies. Efforts are now under way to develop machine-readable representations of interventions to support the consistent recording of behavioral (Michie et al., 2013) and public health interventions (Shaban-Nejad et al., 2017). Adoption and use of these ontologies to record when and where interventions are implemented should allow systematic monitoring of the effectiveness of occupational health interventions used in real-world settings. The resulting information could then be used to evaluate and continually refine these interventions in what would be a learning OSH system.

Data Integration and Storage

An important part of a system of systems is to enable access to individual, case-level data in a manner that protects confidentiality but allows the identification of new hazards, and the introduction of old hazards into new industries. Once data are collected or obtained from another source for use within a surveillance system, the data must be stored and integrated, or linked. Within a single system, the data are usually stored in one location,

often within a database or a data warehouse, where the data have been arranged to optimize regular queries. The most informative linkages between data sources are individual-level linkages (e.g., linking data on an individual’s occupation and industry with data on their health outcomes and time away from work). Linkages based on location (e.g., linking home address to census variables) or other attributes (e.g., linking occupation to an exposure matrix) may also be useful for data analysis.

Data integration and storage can pose challenges within a single surveillance system, and these challenges are greater for a system of systems. In a national system, it is not feasible to store all data in a single location due to legal barriers and agency policies that limit data sharing. However, a sufficient degree of data integration may still be possible without centralizing data. With a federated data strategy, a “virtual” database can be created by identifying the linkages between databases at each participating location. Queries made against this virtual database are answered by accessing data from the relevant locations and assembling them into a single response. It is possible with this approach for each data provider to control which data are visible to the larger system and to approve or deny any query.

Another related strategy is to perform distributed analyses, as opposed to distributed queries. With this approach, the data remain with the individual systems, and statistical analyses are distributed across participating agencies (Gini et al., 2016). For example, cohorts of workers in an industry could be identified across multiple surveillance systems and the overall effect of an exposure in that industry could be estimated by pooling the results of the same statistical analysis performed by each system against its own data.

In summary, multiple strategies exist for deriving additional value by querying and analyzing data held by different OSH agencies. The preferred strategy must be developed to realize the objectives of a system of systems, while respecting limitations around data sharing and available resources. The topic of data integration and distributed analysis is discussed later in the chapter in the section considering how informatics can be used to support data analysis.

Data Collection and Processing

The collection and processing of data in a surveillance system can benefit greatly from innovations in information technologies. Novel technologies such as environmental and personal sensors can be used to capture data. People can be empowered to collect data through crowdsourcing and data can be captured from posts to social media. Another rich source of data is the electronic health record (EHR), and novel methods for converting free text to structured codes can play an important role in processing EHR data to make them usable for OSH surveillance.

Mobile Devices and Sensors

Mobile devices are now ubiquitous with adoption at 96 percent globally in 2014 and 65 percent of U.S. adults owning a smartphone in 2015 (Pew Research Center, 2015). These devices contain sensors, which can capture location, sound, images, acceleration, and other measurements. Moreover, the devices can then transmit those data wirelessly to a central data repository. In recognition of the potential impact of such distributed sensors, NIOSH developed the Center for Direct Reading and Sensor Technologies to work with partners in advancing the development and use of sensors for OSH (NIOSH, 2016).

One application of mobile devices is to gather data on occupational exposures. Data can be captured actively by having inspectors or employers use applications that capture and process data from the device sensors, then submit results. This approach is already used for medical or exposure monitoring, but these applications generally rely on specialized devices (Evans et al., 2010), especially in settings where there is the potential for remote devices to pose a hazard due to flammability or explosion. Increasingly sophisticated monitoring and even analysis is possible using mobile devices. For example, in the context of infectious disease surveillance, routine tests such as blood smears to detect parasites can be performed with acceptable accuracy using mobile devices (Pirnstill and Coté, 2015). From a chronic disease perspective, even simple mobile phones allow tracking of healthy behaviors and environmental exposures (Donaire-Gonzalez et al., 2016).

Crowdsourcing

Crowdsourcing is a passive approach to data collection that empowers people to use apps on mobile devices to capture and then voluntarily forward data to a central site for analysis and dissemination (Brabham et al., 2014). Applications to enable crowdsourcing can be made generally available, for example to measure noise or air pollution, or they can be used by an employer for “internal crowdsourcing” (Brauch, 2015). People can be motivated to submit data using different strategies, including their engagement in what has been called participatory epidemiology (Freifeld et al., 2010) or citizen science (Pocock et al., 2017), where those who submit data are also engaged in their analysis and interpretation. Although this democratization of data access and analysis presents many opportunities, it also raises new questions such as how to derive unbiased insights from data collected through crowdsourcing (Welvaert and Caley, 2016) and how data informally collected through crowdsourcing or other means are best used by employers, employees, communities, and researchers in the absence of any authoritative interpretation.

Social Media

Social media provide a platform for people to communicate their thoughts and experiences with members of their social network and others who may be interested. Soon after the widespread adoption of social media, researchers recognized the value of systematically monitoring these public communications to generate public health intelligence. An early application of this approach to surveillance was to monitor the incidence of influenza-like illness (Chew and Eysenbach, 2010) and it is clear now that social media data can improve surveillance of seasonal influenza epidemics when combined with more traditional surveillance approaches (Mitchell and Ross, 2016). More recently, this approach has also been used to monitor the frequency of adverse effects of prescription medications. Comparisons to traditional approaches, such as spontaneous reporting, suggest that social media surveillance can identify adverse events and their frequency of occurrence (Powell et al., 2016).

Influenza-like illness and adverse drug reactions are highly prevalent events, making them ideal outcomes for surveillance through social media. Individual types of occupational injury and illness tend to occur less frequently, so additional research is required to determine the extent to which these events can be monitored through social media.

EHRs and Electronic Reporting

EHRs are increasingly used routinely in primary care and other settings, such as in emergency departments. These systems capture a range of data, which researchers have shown to be valuable for a variety of public health surveillance objectives. Syndromic surveillance is one example, where anonymized data on the reason for the encounter are collected for all patients, usually from emergency departments, and then analyzed to detect unusual increases in health care utilization for broad categories of symptoms, such as influenza-like illness and gastrointestinal disease (Mandl et al., 2004). More recently, these methods have been extended to allow automated, case-based surveillance, where more complex case definitions are applied to the multiple types of data integrated within an EHR, allowing the accurate and timely detection and reporting of cases of communicable diseases (Vogel et al., 2014).

As discussed in Chapter 6, there is an ongoing effort to increase the amount of occupational data recorded in the EHR and the growing adoption of data standards facilitates the analysis of these occupational data together with other types of data contained in the EHR. In particular, the electronic case-reporting initiative, to which NIOSH has contributed, has defined and begun to develop an infrastructure that will enable the

automated reporting of health conditions by occupation (Mac Kenzie et al., 2016). However, the collection and reporting of occupation, industry, and other OSH data in EHRs remains voluntary. The quality of the data in EHRs can be variable, so methods are needed to assess and assure the quality of OSH data extracted from EHRs.

Recommendation L: NIOSH should lead efforts to establish data standards and software tools for coding and using occupational data in electronic health records. These efforts should be coordinated with the Office of the National Coordinator for Health Information Technology (ONC) to support the establishment of a rule requiring collection and effective use of OSH data in the electronic health record.

In the near term:

• Develop a consensus within the OSH surveillance community regarding the preferred terminologies and tools for extracting data on industry and occupation from the EHR;
• Engage with ONC to communicate this consensus to other stakeholders and to establish a broader consensus among all stakeholders regarding an acceptable strategy; and
• Support ONC in the process of establishing a rule to require the capture of industry and occupation in the EHR.

In the longer term:

• Work with the occupational medicine and general medicine community to develop models and tools for using occupational data in electronic health records for clinical care and for serving the prevention needs of the clinical population.

Autocoding Software

Natural language processing (NLP) is the field of computer science concerned with the interpretation by computers of natural language, including the automated interpretation of written text. When working with written text, NLP software attempts to link words in the written text to terms in a controlled vocabulary. In other words, the NLP programs are trying to automatically assign codes to the written text. Accordingly, the term “autocoding” is often used in the OSH surveillance literature to refer to the automated assignment of codes (e.g., occupation, industry, and type of injury) to words in a textual report using NLP software.

Historically, NLP software has relied on manually developed rules to

map or link written words to standard terms. More recently, statistical methods have been used in NLP programs. The statistical methods for NLP learn a model from a set of written documents, which are already coded, and the statistical model is then used to predict the best coding for new documents. Both NIOSH and BLS have developed NLP software to support OSH surveillance. NIOSH has employed multiple strategies for autocoding, including developing and making available in 2012 a web-based NLP system (the NIOSH Industry and Occupation Computerized Coding System) that uses rules to assign industry and occupation codes (i.e., Bureau of Census codes with links to SOC and NAICS codes) to text in vital statistics, health survey, and electronic health records. Version 3 of this software is scheduled for release in 2018. In 2012, BLS began exploring the use of NLP and it developed a statistical NLP system. This system was used to assign SOC codes to text in responses to the 2014 SOII and was extended in 2015 to automatically code nature of injury and part of body affected in responses to the SOII (BLS, 2015). The Bureau of the Census has also developed autocoding strategies for employment data collected in the Current Population Survey and American Community Survey.

In addition to efforts by OSH agencies, academic groups (Patel et al., 2012; Burstyn et al., 2014; Harber and Leroy, 2017) have developed and evaluated NLP programs to code occupational history in free-text documents. Some general observations across all these efforts are that NLP programs are usually unable to code some records (30 to 50 percent) and for those that are coded, agreement with manual coding is reasonable (50 to 80 percent). Another observation is that a wide range of NLP programs have been developed, using different overall frameworks, algorithms, and lexicons. Although each approach has advantages, no single approach appears to be ideal for all types of documents in all settings and NLP is a fast-moving area of research.

Conclusion: Using natural language processing to extract data from free text has the potential to improve the efficiency of surveillance in many ways. There is also the potential to influence future rules if “industry standard” approaches can be identified for extracting OSH text from electronic health records.

Recommendation M: NIOSH and BLS, working with other relevant agencies, academic centers, and other stakeholders should coordinate and consolidate, where possible, efforts to develop and evaluate state-of-the-art computational and analytical tools for processing free-text data found in OSH surveillance records of all types. This coordination should enable rapid innovation and implementation, into OSH surveillance practice, of successful “autocoding” methods for different data sources.

In the near term:

• Conduct an inventory of activities and key stakeholders and
• Support knowledge exchange activities (symposia, competitions).

In the longer term:

• Develop open data sets that can be used to consistently evaluate methods for extracting OSH data from free text.

Analysis and Interpretation

Data collected through surveillance needs to be analyzed and interpreted appropriately to generate information that surveillance stakeholders can use to guide their actions. In practice, a range of analytical strategies are applied to surveillance data, reflecting different objectives, data, and expertise. In many surveillance settings, descriptive analyses are performed periodically to identify trends over time or unexpected patterns in population subgroups. For example, NIOSH detected an increasing trend in early coal workers’ pneumoconiosis, leading to a more detailed analysis at the state level, and ultimately preventive actions (see Box 7-1).

Periodic analysis of trends remains important, but advances in computational and statistical methods now make it possible to analyze large volumes of data and to detect meaningful variations in health outcomes that may warrant public health intervention (Lombardo and Buckeridge, 2007; Fricker, 2013). Although not widely applied in surveillance practice, researchers are also exploring the potential role for artificial intelligence in supporting decision-making based on the results of surveillance analyses (Dixon et al., 2013; Mamiya et al., 2015; Shaban-Nejad et al., 2017). It is not possible to explore the potential contribution to OSH surveillance of all these developments, but two analytical topics of direct relevance to OSH surveillance are highlighted: small-area estimation and aberration detection.

Small-Area Estimation

While several national surveys serve as essential sources of population-based information on work-related injuries, their capacity to “drill down” to specific subnational areas of the nation to facilitate comparable analyses is severely limited by cost constraints. For example, the proposed BLS household survey of occupational injuries and illnesses (see Chapter 5) specified as a supplement to the CPS will help to fill several analytical needs of a comprehensive national surveillance system for occupational safety and health. As currently envisioned by BLS, the proposed sample size

ranging from 51,000 to 57,000 individuals was specified to simultaneously satisfy the multiple objectives of sample representativeness, data quality, timeliness, and cost. While such sample specifications achieve solid levels of precision at the national level, they may result in imprecise estimates at the county or sub-county levels. There are also confidentiality constraints imposed on the release of such subnational geographic content on analytic files made available to the public.

The capacity to obtain reliable small-area estimates derived from national survey data can be substantially enhanced by application of small-area estimation methods. These techniques combine available sample data with auxiliary data using model relationships to improve the reliability

of the resulting estimates (Pfeffermann, 2013; Vaish et al., 2013; Folsom and Vaish, 2014). Effective small-area estimation methodology depends on the availability of useful predictors reasonably related to the outcome measures. These predictors are annually obtained from various sources and federal agencies such as the U.S. Census Bureau and the American Community Survey.

Aberration Detection

A concept related to small-area analysis is aberration detection, which refers to the detection of statistical anomalies, or aberrations, in surveillance

data. Aberrations can be sought along any dimension of the data, but most commonly, deviations from expectation are sought for risk factors or outcomes across time, geographical location, including workplace, and personal attributes, including occupation. As with small-area analysis, it can be challenging to reliably detect true aberrations if small amounts of data are divided into many categories, as many of the resulting cells are likely to contain a small number of events.

Historically, the tendency in public health surveillance has been to search for aberrations in time, aggregating cases to estimate rates, and then applying statistical time-series methods to the rates to detect the onset of an infectious disease epidemic or the effect of an environmental exposure in the whole population, or in subpopulations through stratified analyses. This approach remains a useful strategy as was demonstrated recently for coal dust exposure (see the case study in Box 7-1 on population surveillance). However, advances in statistics, artificial intelligence, and computing power have made it possible to automate the routine analysis of large volumes of individual-level data to detect changes across multiple aspects of time, geography, and personal attributes (Lombardo and Buckeridge, 2007). Had this type of analytical capacity been in place in 1990, the “hot spots” problem described in Box 7-1 may have been identified and results acted on a decade earlier.

Advances in analytical methods have been motivated by access to large volumes of case-level data for surveillance, for example from clinical information systems such as emergency department triage systems and EHRs. A wide range of machine learning methods have been used to detect cases in surveillance data and to identify unusual patterns among cases. More recently, researchers have developed methods to analyze temporal patterns within patient trajectories and then identify unusual subpopulations requiring closer inspection (Lange et al., 2015). There has also been progress in developing methods to integrate data from multiple sources, for example explicitly linking multiple health outcomes to measures of exposure (Morrison et al., 2016).

In occupational health, computationally intensive approaches to aberration detection could support accurate detection of unusual increases in injury or disease among types of occupations in specific workplaces (Kulldorff et al., 2003). The challenges are substantial, especially when large amounts of heterogeneous data are involved, some of which may be sought from commercial entities with proprietary interests or from entities that may offer political or privacy objections to the data use. Despite these challenges, as is done in other types of public health surveillance, OSH surveillance systems could take advantage of advanced statistical and machine learning methods together with data processing methods to automatically analyze OSH data as they are collected.

There is currently no routine review by federal or state agencies of individual, case-level data collected by various OSH surveillance systems to identify new hazards or the introduction of previously recognized hazards in new industries. NIOSH, through an interagency agreement, has access to case-based data in the BLS CFOI database, but it lacks the resources to routinely access these data. As noted earlier in this report, NIOSH does not have access to the SOII data. OSHA has a database of inspections, including fatalities, but again, it does not have the resources to routinely review these data. Without routine analysis of these data, important opportunities to identify significant concerns are missed. For example, only as a special project in 2013, after a request from a state that identified three deaths, did OSHA review its fatality investigation database and identify there had been 10 additional deaths among bathtub refinishers from 2001 to 2011 (Chester et al., 2012; see Box 7-2).

The inclusion of routine rapid analysis of case-level data as a component of the envisioned 21st-century OSH surveillance system would permit more timely identification of emergent OSH injuries, illnesses, and exposures and help facilitate concomitant rapid responses and interventions. Such coordinated analysis within a system of systems could help to identify emergent occupational illnesses and hazards that currently go unnoticed due to the lack of integration across surveillance systems and the inability to analyze data in real time.

Recommendation N: To identify emerging and serious OSH injuries, illnesses, and exposures in a timely fashion, NIOSH (in coordination with OSHA, BLS, and the states) should develop and implement a plan for routine, coordinated, rapid analysis of case-level OSH data collected by different surveillance systems, followed by the timely sharing of the findings.

In the near term:

• Develop analytical objectives, identifying the outcomes that would benefit from routine, rapid analysis and continuous monitoring across OSH surveillance systems; and
• Review technical and legal strategies for conducting analyses, including novel analytical methods and strategies for distributed analysis and ongoing analysis as the data evolve over time.

In the longer term:

• Implement routine processes for rapid data analysis, including protocols to guide the interpretation of aberrations in surveillance data.

Dissemination of Surveillance Information

Although the process of disseminating information to guide public health actions is critical to realizing the potential benefits of surveillance, this process does not always receive the attention it deserves. As with any communication strategy, dissemination of surveillance information is likely to be most effective if the audience is identified and structured into segments, and then each audience segment is targeted using appropriate media and messages. New information technologies can aid greatly in this regard as most people are now instantly accessible via mobile devices and technologies such as social media and the EHR allow messages to be tailored to specific audiences and contexts.

Mobile Devices

The ubiquitous nature of mobile devices makes them well suited to disseminating knowledge and information. Guidelines and evidence regarding risks and preventive measures are easily accessible using mobile devices. Information obtained from analyzing crowdsourced data on exposure risks could also be pushed to employers and employees, indicating nearby risks, as has been done with infectious disease exposures (HealthMap, 2017).

Social Media

As mentioned earlier, social media data have been used as a source of surveillance, for example, to detect the onset of a seasonal influenza epidemic by analyzing the frequency with which influenza symptoms are mentioned. However, social media also provide an opportunity for public health authorities to engage with people by disseminating targeted information, which may help to prevent illness or allow identification of ongoing threats to health.

In the context of influenza surveillance, researchers have used social media to notify individuals at risk of disease where influenza vaccine is available nearby (Smolinski et al., 2015). This type of feedback combines information about patient risk, patient location, and the location of prevention resources available nearby (i.e., vaccine clinics) to increase the use of evidence-based preventive maneuvers, with the aim of preventing disease. Similarly, in the context of foodborne disease, public health agencies have used automated software to identify people making posts about being ill after visiting a restaurant. These people then receive an automated message asking them to access a website and provide further details about their experience (Harris et al., 2014).

These examples have direct analogies to occupational health. For

example, people at higher risk of an occupational injury or disease could be directed toward resources, that may allow them to prevent an injury or disease such as emphasizing the use of available protection when working at heights. Automated bots could be used to flag people who may be commenting on an occupational injury of disease. Once identified, it would be possible to direct those people to a website where they could provide further data, and analysis of those data could trigger an assessment of their workplace.

Electronic Health Records

While electronic health records are a technology that can greatly facilitate the capture of data, they also present an opportunity for disseminating information produced through the analysis of surveillance data. The dissemination of surveillance information via an EHR can take different forms, including through alerts to identify patients who may be presenting with an occupational injury or disease and by providing feedback to health care providers regarding their management of patients with occupational injury or disease.

The ability to provide alerts to clinicians through an EHR has been demonstrated for communicable disease control. In that context, knowledge of an increase in infectious disease activity in a geographic region has been used to alert physicians when a patient presents from the same geographic region with symptoms consistent with the infections disease in question (Lurio et al., 2010). Similarly, researchers have shown that in emergency department encounters for some infectious disease, accounting for the prevalence of the infectious disease ascertained from surveillance data can enhance clinical decision rules, allowing more accurate diagnosis (Fine et al., 2007). Both examples have analogous applications for OSH surveillance. For example, if public health authorities recognize a cluster of illness or injury associated with an occupation or workplace, then alerts could be constructed to prompt physicians to consider an occupation etiology when patients present with a similar illness or injury. The feasibility of disseminating information in this manner was demonstrated in Massachusetts when NIOSH funded the state health department to incorporate data on occupation into electronic heath records for a major health care system in the state. The health department coded data for 26,000 patients and entered these data into the system. Based on occupation patterns alone, it was noted that a high proportion of Portuguese-speaking women were house cleaners, and Spanish-speaking men were painters. Consequently, multilingual materials on occupation-specific hazards and controls were made available on the system so that clinicians could distribute them to patients in these jobs (Brightman et al., 2013).

In some clinical settings, data captured through EHRs are pooled, analyzed to quantify variations in care across providers, and the results are then fed back to clinicians to help them situate their practice pattern in relation to their peers and clinical practice guidelines. For example, in some primary care settings, data from EHRs are collected and analyzed to determine the proportion of patients with type 2 diabetes who have had a hemoglobin A1C test performed recently. This information is then fed back to each participating primary care provider, allowing them to identify their management of such patients in relation to their peers (Seitz et al., 2011). In occupational health, a similar strategy could be employed, for example, to provide feedback to physicians regarding their management of occupational injuries or disease.

Rapid Alert Networks

As in other areas of public health surveillance, an early warning alert network could use any or all of the strategies described above to disseminate important findings to the OSH community at large. Doing so could promote fast-track situational awareness of emergent occupational illnesses and hazards, accelerate more focused analyses to determine the level of imminent risk, and stimulate decisions on prompt responses and interventions to mitigate the danger. Responses might include further targeted surveillance, OSH community messaging, and more focused research investigations making greater use of other relevant available data sources.

Recommendation O: To promote and facilitate the use of surveillance information for prevention, and to present more comprehensive information on the extent, distribution, and characteristics of OSH injuries, illnesses, and exposures, NIOSH (in coordination with and input from OSHA, BLS, and the states) should establish a coordinated strategy and mechanism for timely dissemination of surveillance information.

In the near term:

• Clarify target populations for different types of surveillance information (e.g., rapid alerts, trends, etc.);
• Establish a plan for accessing, integrating, and disseminating information from different surveillance sources; and
• Develop policies and criteria to address individuals’ and employers’ privacy and confidentiality considerations through a process that provides for stakeholder input and includes privacy experts in the development of these policies and in the design of surveillance systems.

In the longer term:

• Implement a coordinated information dissemination strategy, making use of different technologies as appropriate to communicate information to those who need it to take action for prevention.

This coordination would augment and not replace the activities and authority of individual agencies. An overall dissemination strategy will provide a better understanding of occupational injuries and illnesses to assist in the prioritization and evaluation of prevention activity.

ENHANCE TRAINING AND SUPPORT FOR OSH SURVEILLANCE PRACTITIONERS

Although a surveillance system is often thought of as a technical system, it is better conceptualized as a sociotechnical system, which is as dependent on skilled people as it is on technical components. A range of individuals is necessary to establish and effectively operate surveillance systems, but the key disciplines that contribute to the science (Thacker et al., 1989) and practice of OSH surveillance through the effective application of new information technologies are epidemiology, biostatistics, and biomedical informatics.

Trainees in OSH are likely to receive instruction in epidemiology and biostatistics, although rarely are they taught to apply methods from these disciplines to surveillance. For example, designs for evaluating surveillance systems and statistical methods for aberration detection are not taught routinely in OSH or other public health programs. The situation is more concerning for biomedical informatics. This discipline is identified as a core public health competency (CLBAPHP, 2014), however, it is taught to varying degrees across education institutions, and the specific aspects of informatics relevant to OSH surveillance are not taught routinely in many academic institutions.

Recommendation P: NIOSH, OSHA, and BLS should work together to encourage education and training of the surveillance workforce in disciplines necessary for developing and using surveillance systems, including epidemiology, biomedical informatics, and biostatistics.

In the near term:

• Identify the core competencies required for OSH surveillance and promote the science of surveillance;
• Review the curricula of existing surveillance courses;

• Collaborate with educational organizations to establish or modify training programs accordingly; and
• Require surveillance courses in all funded training programs, especially in the Education and Research Center and Program Project training grants.

In the longer term:

• Contribute to development of surveillance courses and conferences that provide training in surveillance methods.

REFERENCES

CLBAPHP (The Council on Linkages Between Academia and Public Health Practice). 2014. Core competencies for public health professionals. Available online at http://www.phf.org/resourcestools/Documents/Core_Competencies_for_Public_Health_Professionals_2014June.pdf (accessed July 18, 2017).

Cooper, B. S., and D. P. Rice. 1976. The economic cost of illness revisited. Social Security Bulletin 39(2):21-36.

Des Jarlais, D. C., C. Lyles, and N. Crepaz. 2004. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health 94(3):361-366.

Dixon, B. E., R. E. Gamache, and S. J. Grannis. 2013. Towards public health decision support: A systematic review of bidirectional communication approaches. Journal of the American Medical Informatics Association 20(3):577-583.

DOL (U.S. Department of Labor). 1996. Report of the Secretary of Labor’s Advisory Committee on the Elimination of Pneumoconiosis among Coal Miner Workers, October. Washington, DC: Mine Safety and Health Administration.

Donaire-Gonzalez, D., A. Valentín, A. de Nazelle, A. Ambros, G. Carrasco-Turigas, E. Seto, M. Jerrett, and M. J. Nieuwenhuijsen. 2016. Benefits of mobile phone technology for personal environmental monitoring. JMIR mHealth and uHealth 4(4):e126.

Edmunds, M., L. Thorpe, M. Sepulveda, C. Bezold, and D. A. Ross. 2014. The future of public health informatics: Alternative scenarios and recommended strategies. eGEMS (Washington, DC) 2(4):1156. Available online at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4371422 (accessed July 18, 2017).

EPA (U.S. Environmental Protection Agency). 2010. Valuing Mortality Risk Reductions for Environmental Policy: A White Paper. Washington, DC: EPA.

EU-OSHA (European Agency for Safety and Health at Work). 2017. Estimating the Cost of Work-Related Accidents and Ill-Health: An Analysis of European Data Sources. Luxembourg: Publications Office of the European Union.

Evans, D. E., B. K. Ku, M. E. Birch, and K. H. Dunn. 2010. Aerosol monitoring during carbon nanofiber production: Mobile direct-reading sampling. Annals of Occupational Hygiene 54(5):514-531.

Fine, A. M., L. E. Nigrovic, B. Y. Reis, E. F. Cook, and K. D. Mandl. 2007. Linking surveillance to action: Incorporation of real-time regional data into a medical decision rule. Journal of the American Medical Informatics Association 14(2):206-211.

Fisher, D. A. 2006. An Emergent Perspective on Interoperation in Systems of Systems. Pittsburgh, PA: Carnegie Mellon Software Engineering Institute.

Folsom, R. E., and A. K. Vaish. 2014. Robust small area estimation with additive random coefficients (ARC) models. Frontiers of Hierarchical Modeling in Observational Studies, Complex Surveys and Big Data Conference Honoring Professor Malay Ghosh, College Park, MD.

Freifeld, C. C., R. Chunara, S. R. Mekaru, E. H. Chan, T. Kass-Hout, A. Ayala Iacucci, and J. S. Brownstein. 2010. Participatory epidemiology: Use of mobile phones for community-based health reporting. PLoS Medicine 7(12):e1000376.

Fricker, R. 2013. Introduction to Statistical Methods for Biosurveillance. Cambridge, UK: Cambridge University Press.

Gillum, L. A., C. Gouveia, E. R. Dorsey, M. Pletcher, C. D. Mathers, C. E. McCulloch, and S. C. Johnston. 2011. NIH disease funding levels and burden of disease. PLoS ONE 6(2):e16837.

Gini, R., M. Schuemie, J. Brown, P. Ryan, E. Vacchi, M. Coppola, W. Cazzola, P. Coloma, R. Berni, G. Diallo, J. L. Oliveira, P. Avillach, G. Trifirò, P. Rijnbeek, M. Bellentani, J. van Der Lei, N. Klazinga, and M. Sturkenboom. 2016. Data extraction and management in networks of observational health care databases for scientific research: A comparison of EU-ADR, OMOP, Mini-Sentinel and MATRICE strategies. eGEMS (Washington, DC) 4(1):1189.

Gold, M. R., J. E. Siegel, L. B. Russell, and M. C. Weinstein. 1996. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press.

Gross, C. P., G. F. Anderson, and N. R. Powe. 1996. The relation between funding by the National Institutes of Health and the burden of disease. New England Journal of Medicine 340(24):1881-1887.

Haddix, A. C., S. M. Teutsch, P. A. Shaffer, and D. O. Dunet. 1996. Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation. New York: Oxford University Press.

Hanser, S., A. Zaiss, and S. Schulz. 2009. Health care procedures. Methods of Information in Medicine 48(6):540-545.

Harber, P., and G. Leroy. 2017. Feasibility and utility of lexical analysis for occupational health data. Journal of Occupational and Environmental Medicine 59(6):578-587.

Harris, J. K., R. Mansour, B. Choucair, J. Olson, C. Nissen, and J. Bhatt. 2014. Health department use of social media to identify foodborne illness - Chicago, Illinois, 2013-2014. Morbidity and Mortality Weekly Report 63(32):681-685.

HealthMap. 2017. Outbreaks near me. Available online at http://www.healthmap.org/out-breaksnearme (accessed July 18, 2017).

HSE (Health and Safety Executive). 2016. Health and safety at work. Available online at http://www.hse.gov.uk/statistics/overall/hssh1516.pdf?pdf=hssh1516 (accessed May 8, 2017).

Keating, C., R. Rogers, R. Unal, D. Dryer, A. Souza-Poza, R. Safford, W. Peterson, and G. Rabadi. 2003. System of systems engineering. Engineering Management Journal 15(3):36-45.

Koeman, T., N. S. Offermans, Y. Christopher-de Vries, P. Slottje, P. A. Van Den Brandt, R. A. Goldbohm, H. Kromhout, and R. Q. Vermeulen. 2013. JEMs and incompatible occupational coding systems: Effect of manual and automatic recoding of job codes on exposure assignment. Annals of Occupational Hygiene 57(1):107-114.

Kulikowski C. A., E. H. Shortliffe, L. M. Currie, P. L. Elkin, L. E. Hunter, T. R. Johnson, I. J. Kalet, L. A. Lenert, M. A. Musen, J. G. Ozbolt, J. W. Smith, P. Z. Tarczy-Hornoch, and J. J. Williamson. 2012. AMIA Board white paper: Definition of biomedical informatics and specification of core competencies for graduate education in the discipline. Journal of the American Medical Informatics Association 19:931–938.

Kulldorff, M., Z. Fang, and S. J. Walsh. 2003. A tree-based scan statistic for database disease surveillance. Biometrics 59(2):323-331.

Lange, J. M., R. A. Hubbard, L. Y. Inoue, and V. N. Minin. 2015. A joint model for multistate disease processes and random informative observation times, with applications to electronic medical records data. Biometrics 71(1):90-101.

Leigh, J. P. 2011. Economic burden of occupational injury and illness in the United States. Milbank Quarterly 89(4):728-772.

Lofgren, D. J., C. K. Reeb-Whitaker, and D. Adams. 2010. Surveillance of Washington OSHA exposure data to identify uncharacterized or emerging occupational health hazards. Journal of Occupational and Environmental Hygiene 7:375-388.

Lombardo, J. S., and D. L. Buckeridge, eds. 2007. Disease Surveillance: A Public Health Informatics Approach. Hoboken, NJ: John Wiley & Sons.

Luo, F., and C. Florence. 2017. State-level lifetime medical and work-loss costs of fatal injuries—United States, 2014. Morbidity and Mortality Weekly Report 66(1):1-11.

Lurio, J., F. P. Morrison, M. Pichardo, R. Berg, M. D. Buck, W. Wu, K. Kitson, F. Mostashari, and N. Calman. 2010. Using electronic health record alerts to provide public health situational awareness to clinicians. Journal of the American Medical Informatics Association 17(2):217-219.

Mac Kenzie, W. R., A. J. Davidson, A. Wiesenthal, J. P. Engel, K. Turner, L. Conn, S. J. Becker, S. Moffatt, S. L. Groseclose, J. Jellison, J. Stinn, N. Y. Garrett, L. Helmus, B. Harmon, C. L. Richards, J. R. Lumpkin, and M. F. Iademarco. 2016. The promise of electronic case reporting. Public Health Reports 131(6):742-746.

Maier, M. W. 1998. Architecting principles for systems-of-systems. Systems Engineering 1(4):267-284.

Mamiya, H., K. Schwartzman, A. Verma, C. Jauvin, M. Behr, and D. Buckeridge. 2015. Towards probabilistic decision support in public health practice: Predicting recent transmission of tuberculosis from patient attributes. Journal of Biomedical Informatics 53: 237-242.

Mandl, K. D., J. M. Overhage, M. M. Wagner, W. B. Lober, P. Sebastiani, F. Mostashari, J. A. Pavlin, P. H. Gesteland, T. Treadwell, E. Koski, L. Hutwagner, D. L. Buckeridge, R. D. Aller, and S. Grannis. 2004. Implementing syndromic surveillance: A practical guide informed by the early experience. Journal of the American Medical Informatics Association 11(2):141-150.

Michie, S., M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M. P. Eccles, J. Cane, and C. E. Wood. 2013. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: Building an international consensus for the reporting of behavior change interventions. Annals of Bevioural Medicine 46(1):81-95.

Mitchell, L., and J. V. Ross. 2016. A data-driven model for influenza transmission incorporating media effects. Royal Society Open Science 3(10):160481.

Morrison, K. T., G. Shaddick, S. B. Henderson, and D. L. Buckeridge. 2016. A latent process model for forecasting multiple time series in environmental public health surveillance. Statistics in Medicine 35(18):3085-3100.

Muennig, P. 2008. Cost-Effectiveness Analyses in Health: A Practical Approach, 2nd Ed. San Francisco, CA: Jossey-Bass.

NIOSH (National Institute for Occupational Safety and Health). 1995. Criteria for a Recommended Standard: Occupational Exposure to Coal Mine Dust. DHHS (NIOSH) Publication No. 95-106. Available online at https://www.cdc.gov/niosh/docs/95-106/pdfs/95-106.pdf (accessed December 6, 2017).

NIOSH. 2011. Coal mine dust exposures and associated health outcomes: A review of information published since 1995. Current Intelligence Bulletin 64. DHHS (NIOSH) Publication No. 2011-172. Available online at https://www.cdc.gov/niosh/docs/2011-172/pdfs/2011-172.pdf (accessed December 6, 2017).

NIOSH. 2016. Center for Direct Reading and Sensor Technologies. Available online at https://www.cdc.gov/niosh/docs/2016-132/ (accessed July 18, 2017).

NIOSH.2017a. Worker health surveillance. Available online at https://www.cdc.gov/niosh/topics/surveillance/default.html (accessed March 20, 2017).

NIOSH. 2017b. Coal workers’ health surveillance program. Available online at https://www.cdc.gov/niosh/topics/cwhsp/ (accessed June 4, 2017).

NIOSH. 2017c. Fatality Assessment and Control Evaluation (FACE) program. Available online at https://www.cdc.gov/niosh/face/default.html (accessed April 4, 2017).

NIOSH. 2017d. Firefighter fatality investigation and prevention. Available online at https://www.cdc.gov/niosh/fire/default.html (accessed January 29, 2017).

NIOSH. 2017e. NIOSH and Partners Work to Prevent Worker Deaths from Exposures to Hydrocarbon Gases and Vapors at Oil and Gas Wellsites. DHHS (NIOSH) Publication No. 2017-110. Available online at https://www.cdc.gov/niosh/docs/2017-110 (accessed May 8, 2017).

NIOSH. 2017f. Welcome to eWORLD. Available online at https://wwwn.cdc.gov/eworld (accessed June 14, 2017).

NIOSH. 2017g. Bloodborne infectious diseases: HIV/AIDS, hepatitis B, hepatitis C. Available online at https://www.cdc.gov/niosh/topics/bbp/ (accessed May 8, 2017).

NIOSH. 2017h. Health and safety practices survey of healthcare workers. Available online at https://www.cdc.gov/niosh/topics/healthcarehsps/aboutstudy.html (accessed June 14, 2017).

Olson, D. R., M. Paladini, W. B. Lober, and D. L. Buckeridge and ISDS Distribute Working Group. 2011. Applying a new model for sharing population health data to national syndromic influenza surveillance: DiSTRIBuTE project proof of concept, 2006 to 2009. PLoS Currents. 3:RRN 1251. Available online at http://currents.plos.org/influenza/article/applying-a-new-model-for-sharing-261w1jjdm6zrb-5 (accessed July 18, 2017).

OSHA (Occupational Safety and Health Administration). 2017. Data and statistics. Available online at https://www.osha.gov/oshstats/index.html (accessed July 18, 2017).

Patel, M. D., K. M. Rose, C. R. Owens, H. Bang, and J. S. Kaufman. 2012. Performance of automated and manual coding systems for occupational data: A case study of historical records. American Journal of Industrial Medicine 55(3):228-231.

Pew Research Center. 2015. U.S. smartphone use in 2015. Available online at http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015 (accessed July 18, 2017).

Pfeffermann, D. 2013. New important developments in small area estimation. Statistical Science 28(1):40-68.

Pirnstill, C. W., and G. L. Coté. 2015. Malaria diagnosis using a mobile phone polarized microscope. Scientific Reports 5:13368.

Pocock, M. J., J. C. Tweddle, J. Savage, L. D. Robinson, and H. E. Roy. 2017. The diversity and evolution of ecological and environmental citizen science. PLoS ONE 12(4):e0172579.

Popper, S., S. Bankes, R. Callaway, and D. DeLaurentis. 2004. System-of-Systems Symposium: Report on a Summer Conversation, July 21-22, 2004. Potomac Institute for Policy Studies, Arlington, VA.

Powell, G. E., H. A. Seifert, T. Reblin, P. J. Burstein, J. Blowers, J. A. Menius, J. L. Painter, M. Thomas, C. E. Pierce, H. W. Rodriguez, J. S. Brownstein, C. C. Freifeld, H. G. Bell, and N. Dasgupta. 2016. Social media listening for routine post-marketing safety surveillance. Drug Safety 39(5):443-454.

Rice, D. P. 1967. Estimating the cost of illness. American Journal of Public Health 57(3): 424-440.

Rice, D. P., T. A. Hodgson, and A. N. Kopstein. 1985. The economic costs of illness: A replication and update. Health Care Financing Review 7(1):61-80.

Schulte, P. A., R. Pana-Cryan, T. Schnorr, A. L. Schill, R. Guerin, S. Felknor, and G. R. Wagner. 2017. An approach to assess the burden of work-related injury, disease, and distress. American Journal of Public Health 107(7):1051-1057.

Seitz, P., T. Rosemann, J. Gensichen, and C. A. Huber. 2011. Interventions in primary care to improve cardiovascular risk factors and glycated haemoglobin (HbA1c) levels in patients with diabetes: A systematic review. Diabetes Obesity and Metabolism 13(6):479-489.

Shaban-Nejad, A., M. Lavigne, A. Okhmatovskaia, and D. L. Buckeridge. 2017. PopHR: A knowledge-based platform to support integration, analysis, and visualization of population health data. Annals of the New York Academy of Sciences 1387(1):44-53.

Shortliffe, E. H. 2014. Biomedical informatics: The science and the pragmatics. Pp. 3-37 in Biomedical Informatics: Computer Applications in Health Care and Biomedicine, E. H. Shortliffe and J. J. Cimino, eds. London: Springer.

Smolinski, M. S., A. W. Crawley, K. Baltrusaitis, R. Chunara, J. M. Olsen, O. Wójcik, M. Santillana, A. Nguyen, and J. S. Brownstein. 2015. Flu near you: Crowdsourced symptom reporting spanning 2 influenza seasons. American Journal of Public Health 105(10):2124-2130.

Steenland, K., C. Burnett, N. Lalich, E. Ward, and J. Hurrel. 2003. Dying for work: The magnitude of US mortality from selected causes of death associated with occupation. American Journal of Industrial Medicine 43:461-482.

Thacker, S. B., R. L. Berkelman, and D. F. Stroup. 1989. The science of public health surveillance. Journal of Public Health Policy 10(2):187-203.

Thorpe, K. E., L. Allen, and P. Joski. 2015. The role of chronic disease, obesity, and improved treatment and detection in accounting for the rise in healthcare spending between 1987 and 2011. Applied Health Economics and Health Policy 13(4):381-387.

Vaish, A. K., R. E. Folsom, K. Spagnola, N. Sathe, and A. K. Hughes. 2013. An empirical study to evaluate the performance of synthetic estimates of substance use in the National Survey on Drug Use and Health. Pp. 2538-2547 in Proceedings of the American Statistical Association, Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Vogel, J., J. S. Brown, T. Land, R. Platt, and M. Klompas. 2014. MDPHnet: Secure, distributed sharing of electronic health record data for public health surveillance, evaluation, and planning. American Journal of Public Health 104(12):2265-2270.

Wells, G. D., and A. P. Sage. 2008. Engineering of a system of systems. Pp. 44-76 in Systems of Systems Engineering: Principles and Applications, M. Jamshidi, ed. Boca Raton, FL: CRC Press.

Welvaert, M., and P. Caley. 2016. Citizen surveillance for environmental monitoring: Combining the efforts of citizen science and crowdsourcing in a quantitative data framework. SpringerPlus 5(1):1890.