3

Public Benefits of the Science to Achieve Results Program

The committee developed a framework to understand how the Environmental Protection Agency (EPA) Science to Achieve Results (STAR) program is designed to deliver its public benefits. The framework is summarized in a logic model. The logic-model approach is widely accepted for clarifying what programs must do to achieve their desired effects (Cozzens 1997; Engel-Cox et al. 2008; Liebow et al. 2009; Orians et al. 2009; McLaughlin and Jordan 2015). To develop its logic model, the committee considered one developed by an earlier National Research Council committee that evaluated EPA research efficiency (NRC 2008) and one used by the National Institute of Environmental Health Sciences (NIEHS) extramural research program (Engel-Cox et al. 2008).

The committee’s logic model for STAR includes the standard categories: inputs, activities, outputs, outcomes, and impacts. Definitions of the logic-model components are as follows.

• Inputs are resources that feed into a program. These include process inputs, such as the allocated budget, the personnel assigned to administer it, and the procedures for selection and awarding of grants and fellowships. They also include such planning inputs as the strategic research action plans (StRAPs), and the knowledge obtained through research, scientific reviews, workshops, and published literature.
• Activities are the events or actions that take place. At the EPA, activities include the awarding of grants and fellowships, monitoring grantee activity, and engagement with funded researchers. At the grantee, activities include the conduct of research, developing infrastructure for data collection and analyses, mentoring of students, engaging with EPA and other stakeholders, and submitting annual and final project reports.
• Outputs are the products of the research activities. Outputs from the STAR program include knowledge outputs (publications, presentations, tools, and methods), infrastructure outputs (improved facilities for data collection and analysis), and workforce outputs (investigator career development and student career development).

• Outcomes are the benefits or changes that result from the use of the research outputs. Short-term outcomes include synthesis products and the next generation of scientists. Intermediate outcomes include outreach and communication to business and industry, government agencies (including other EPA offices), and a strengthened environmental-research community. Long-term outcomes include an improved body of knowledge, new program initiatives, public awareness, or new guidance, regulations, standards, or technologies.
• Impacts of STAR are forms of protection of human health and the environment. They can include improvement in environmental quality through strategies to protect the environment, increased sustainability, and improved health and healthy longevity (Bozeman 2003; Engel-Cox et al. 2008). Multiple interacting influences link STAR research to those impacts, but STAR research on its own does not produce them.

METRICS

Underlying the STAR logic model are specific metrics. Important input metrics of the STAR program include program budget (described in Chapter 1) and procedures (discussed in Chapter 2). Activity metrics include the number of grants and fellowships awarded per year and the number of requests for applications (RFAs) per year. From 2003 to 2015, STAR awarded 541 individual-investigator grants, 53 center grants, and 800 fellowships.

One metric of output is the number of publications. EPA reported that its internal grants database for October 2002-April 2017 contained 5,760 journal publications supported by STAR. That is probably an underestimate in that STAR grantees are required to report publications only until the grants are closed. The committee also reviewed other STAR publication information, a bibliometric analysis that EPA provided on the work of the Safe and Sustainable Water Resources (SSWR) program by STAR grantees, and one that the committee conducted by searching Google Scholar for “Science to Achieve Results EPA OR ORD” in December 2016. The committee chose Google Scholar because it is known to include more early publications and preprints than other databases, such as the Web of Science and Scopus (Meho and Kiduk 2007).

The information EPA provided on the SSWR STAR grantees revealed that grants resulted in over 900 publications from 165 grants issued in 1998-2016, including 844 journal articles, 49 books and book sections, and 25 conference papers and proceedings. Journal articles appeared in 273 journals that are indexed in the Web of Science Core Collection. EPA also used the Thomson Reuters Web of Science and InCites products to analyze the impact of STAR publications from the SSWR program. Half the grants analyzed had at least one publication with a percentile at or below 10% (D. Winner, EPA, Washington, DC, personal communication, 2016); that is, half the grants analyzed had at least one publication that was among the most highly cited publications in their field (a lower percentile means more citations) (Thompson Reuters 2014).

The committee’s Google Scholar search yielded 71 papers published since 2000 that contained the key words and had been cited more than 100 times. The committee accessed those papers and checked their acknowledgments sections, and it confirmed that 63 resulted from research supported by STAR grants (46), fellowships (14), or a combination of grants and fellowships (three). It should be noted that such an evaluation would miss any paper that did not mention STAR in its acknowledgments or main text; this potentially reduced the number of STAR-funded papers found by the committee in that investigators might list only EPA grant numbers in the acknowledgments.

Other important output metrics considered by the committee are related to the scientific-community infrastructure. The program supports research projects nationwide. In FY 2014, the STAR program had grantees or fellows in all but two states, Vermont and South Dakota (Figure 3-2). Engagement with EPA in institutions around the United States has probably helped to create communities of scientists and engineers working in the human health and environmental sciences that would not have occurred without support from STAR grants and fellowships. Research grants also help to improve facilities for data collection and analysis within the supported grantees’ institutions.

The proportion of STAR fellows that become part of the larger scientific community is another important metric for STAR. The STAR fellowship program awarded 800 fellowships in 2003-2015. By reviewing the results of EPA’s Fellowship Information Inventory (FII), a voluntary Web-based application system through which STAR fellows could choose to report career information, the committee assessed whether these scientists were continuing careers in environmental research. The FII was developed in 2003 for program-administration purposes, to collect student applications and supporting materials, and to provide a mechanism for fellows to submit information during and after their fellowships, including information on their research projects, publications, awards, and careers. The FII ended in 2011; while it was active (2003-2011), about 33% of the STAR fellows reported on their careers. The most commonly reported positions were postdoctoral positions (34%); these were followed by teaching positions (21%) and positions as researchers (16%), in the federal government (12%), in consulting firms (5%), in state, local, or tribal governments (4%), in private industry (4%), in nonprofits (3%), and in other appointments (1%) (D. Winner, EPA, Washington, DC, personal communication, 2016).

An example of a short-term outcomes produced by STAR are synthesis reports. For many of the center grants, STAR tasked principal investigators with producing synthesis reports, many of which have been published in the scientific literature (Savage and Diallo 2005; Fanning et al. 2009; Jacob and Winner 2009; Phenrat and Lowry 2009; Weaver et al. 2009; Breysse et al. 2013; Wagstrom et al. 2014).

An example of a long-term outcome, new program initiatives, can be assessed on the basis of the new ideas assessed in strategic planning documents. The StRAPs for the national programs Air, Climate, and Energy (ACE), Chemical

Safety for Sustainability (CSS), SSWR, and Sustainable and Healthy Communities (SHC) all refer to priorities for STAR (EPA 2015a-d). In some cases, it is readily apparent that the new initiatives are informed by previous STAR initiatives. An illustration of that in the StRAP for the SHC program is the STAR priority initiative to create Environmental Health Disparities Centers, which will inform an environmental-justice roadmap in a way that is similar to how the EPA-NIEHS STAR Centers for Children’s Environmental Health have been central to informing the Children’s Environmental Health Roadmap.

An analysis provided by EPA that shows the types of organizations that are citing STAR research provides a metric of how STAR research is influencing the users of research results. In 2012, Scientific Consulting Group, an EPA contractor, identified 6,614 articles published in 2002-2012 that were funded by National Center for Environmental Research (NCER) grants. Using Thomson Reuters Essential Science Indicators and Journal Citation Reports as benchmarks, the contractor identified 252 of the 6,614 NCER articles as being in the top 1% of academic journal citations; thus, the papers had high impact and were among the most highly cited papers in their scientific fields. Another contractor, Science Applications International Corporation (SAIC), searched for citations of the 252 high-impact papers in nonacademic publications that are not indexed in bibliometric databases. It searched for such citations in three ways: using a data-mining tool that it developed to search the EPA Web sites, Google searches and manual review of results to identify regulatory and decision documents, and

searches of select federal sources and documents, such as National Center for Environmental Assessment toxicology reviews and Agency for Toxic Substances and Disease Registry toxicology profiles. SAIC found that 104 of the 252 high-impact publications were cited in federal, state, or local government documents, in international guidelines, and in other documents of academic or nonprofit organizations, such as National Research Council reports and American Public Health Association guidelines (Information provided by EPA, Washington, DC, 2016).

The committee reviewed the 104 papers; all but one were supported by STAR grants. Table 3-1 shows the 10 papers that were cited most frequently in this analysis. Nine of them are focused on human health implications of air pollution; one describes a method of sampling to evaluate natural resources. The papers are also cited in a wide variety of documents, indicating that a wide variety of entities are using the results of STAR research.

The committee looked to see whether there were any trends among the types of grants that funded this research. The 104 publications came from 55 STAR grants. Table 3-2 provides the grant number, the number of papers cited in the 2012 analysis, the year awarded, and the abstract title for each grant that led to two or more of the cited papers. The most notable trend is the year in which a grant was awarded—all these grants were awarded at least 5 years before the impact could be observed. Another notable trend is that many of the grants were center grants, which have the important inputs of larger funding than individual-investigator grants but also often allow greater collaboration between institutions. The scientific topics that the grants cover are also of note. Many of the grants have a direct human-health focus—for example, the Southern California Particle Center and Supersite and the Center for the Study of Prevalent Neurotoxicants in Children—but others aim to understand how an emerging concern may affect health—for example, “Evaluating Nanoparticle Interactions with Skin”. Other grants focused on environmental remediation, such as “Developing Functional Fe(0)-based Nanoparticles for In Situ Degradation of [Dense Non-Aqueous Phase Liquid] DNAPL Chlorinated Organic Solvents”.

The committee also evaluated the STAR program’s impact by developing a list of STAR research results that it considered beneficial to society on the basis of its own knowledge of the program. The committee found examples of STAR research that had had various types of benefits: reducing the costs of compliance with environmental regulations, providing a scientific basis for decisions required to protect public health and the environment, and improved methods for environmental management.

Some STAR research grants may lead to reductions in the cost of complying with environmental regulations. Such cost reductions could benefit regulated industries as well as states and localities that need to comply with environmental regulations. An example of STAR research that may benefit industry is the development of a tissue-based method for evaluating the thyroid effects of

TABLE 3-1 Ten STAR papers with the Highest Numbers of Citations in Documents in EPA’s 2012 Analysis

TABLE 3-2 Summary of Grants That Led to Two or More Papers Found To Be Cited in Documents in EPA’s 2012 Analysis

chemical exposures (Hutson et al. 2016). The method may reduce the cost of chemical testing compared with animal-based approaches. STAR research has also expanded the capability of air-pollution models by identifying key species and reactions occurring in cloud droplets that lead to PM formation. The improved models may reduce the costs of compliance with PM_2.5 national ambient

air quality standards (NAAQSs) (Carlton et al. 2008). Yet another research project supported by STAR discovered a cost-effective method for removing nitrate from drinking water (Berquist et al. 2016).

STAR research has supported numerous public-health decisions. The STAR program implemented several large initiatives focused on the human health effects of air pollution, such as the Particulate Matter Centers, the Clean Air Research Centers, and the Air, Climate, and Energy Centers. Studies supported by those centers showed that increased air-pollution exposure leads to a decrease in life expectancy; examples include a followup of the Harvard Six Cities Study (Laden et al. 2006) and a large epidemiologic study of PM_2.5 exposure and mortality in 51 US cities (Pope et al. 2009). The findings supported earlier research and led to the development of a more protective PM_2.5 NAAQS (EPA 2006).

Another large effective initiative is the Children’s Environmental Health and Disease Prevention Research Centers, which aim to evaluate the effects of environmental exposures on child health and development. In 2016, a research project partially supported by a STAR grant recognized that infants could be exposed to arsenic through rice cereal (Karagas et al. 2016); this discovery led the Food and Drug Administration to propose regulations to protect infant health (FDA 2016). Another example is the discovery by investigators at the University of Washington Children’s Center that farmworker children had increased exposure to the pesticide ingredient azinphos-methyl (Curl et al. 2002); this informed EPA’s decision to phase out the use of azinphos-methyl in pesticides (EPA 2006).

Examples of STAR research to improve environmental management include experiments in market-based incentives to lower emissions and studies of the potential reduction in the cost of pollution abatement (Anton et al. 2004) and auctions in which landowners and sellers compete to obtain part of a fixed budget allocated by the regulator to subsidize pollution abatement (Cason and Gangadharan 2004).

Those examples and others listed in Table 3-3 show how STAR results are contributing to a knowledge base that benefits society by improving human health and the environment.

COMMITTEE’S EVALUATION

Identifying the public benefits of the STAR program is challenging. Part of the difficulty arises from the length of time that it takes for a grant award to yield a public-health benefit; often, the benefit is a calculated or modeled benefit rather than a measured change in a health or environmental outcome. In addition, as a grant is traced through the logic model, its influence becomes more diffuse as the knowledge gained from one grant is synthesized with other information to yield public benefits. The information provided by EPA that describes the frequency of STAR citations in decision documents indicates STAR’s ability to effect public benefits. There are some flaws in the analysis, for example it is 5 years old, so there likely more than 252 high impact publications now that EPA could search for citations in decision documents. In addition, a mere citation in a

decision document does not necessarily mean that the paper drove the decision; a cited paper might have been merely critiqued within the citing document. Nonetheless, in light of the examples presented in this chapter, it is evident that the STAR program has had important implications for human health and environmental protection.

TABLE 3-3 Selected Examples of STAR Research Findings and the Public Benefits to Which They Contributed

Through the funding of research institutions throughout the United States, STAR adds to communities of science and generates reservoirs of environmental-research knowledge. Those reservoirs of knowledge represent the accumulation of understanding, knowledge, and previous research in environmental sciences and greatly contribute to the research environment. Such a “knowledge pool” encompasses both research and the collaboration of people who “interact and produce innovation and discovery through unpredictable paths and at uneven intervals” (Cozzens 1997).

The STAR fellowship program added to the knowledge community. It encouraged promising young scientists to obtain advanced degrees and pursue careers in environment-related fields. In addition, the committee found that almost 30% of the papers identified in Google Scholar as having been cited more than 100 times acknowledged support by a STAR fellowship; this suggests that these young investigators are doing high-quality work. With regard to building a research community, a major output is students trained in the methods of the research field and in analysis of complex data; these young investigators learn to thrive in interdisciplinary environments in which complex problems are tackled. The data collected by EPA through the FII show that many of the STAR fellows remained in academic or other research institutions, although the data are incomplete because of the low rate of response by former fellows and the lack of detail in the data collection. Some universities have begun to track career out-

comes of students supported by extramural grants by using internal employment records (Weinberg et al. 2014). EPA should consider investing in similar approaches and including past and present STAR fellowship holders in its analysis.

STAR has made progress in communicating findings of its programs by requiring synthesis documents from center investigators, but this approach has been inconsistent, and the committee urges EPA to invest more heavily in it.

Concrete effects of results of individual grants on health and the environment are usually difficult to characterize quantitatively; thus, the National Institutes of Health (NIH) evaluation process, for example, is actively seeking approaches to demonstrate how NIH research findings can be linked quantitatively to improvements in health outcomes (NIH 2014). Often, the links between research studies and benefits to human health are described best in case studies, which are therefore a valuable way of communicating the favorable effects of action-relevant research of the sort that the STAR program supports.

CONCLUSIONS

EPA has created a vehicle that fosters collaboration and knowledge-sharing and has produced research that contributes to public benefits. EPA should consider reporting the stories of STAR’s benefits more prominently on its Web site and blogs. It should also consider requiring grantees to report the potential influence and public benefits of their awards as part of the grantee final report and even 5-10 years after their research has been completed. However, tracking the benefits remains challenging for many organizations that support or conduct research. Evaluations like the present one would be improved if there were more robust electronic databases that could be easily searched to detect linkages between grants, fellowships, and public benefits. Through collaboration with other organizations, EPA could make strides in this regard. There is a substantial effort throughout the federal government to mine data in reports, literature, administrative records, and so forth to identify intermediate outcomes more effectively, to link federally funded projects to long-term impacts, and to track career outcomes of graduate students supported by fellowships or graduate research assistantships. NIH, for example, has created the High Impacts Tracking System. The system loads progress reports and program officers’ notes about grants into a searchable system and allows structured tagging of outputs and impacts. Another NIH example is RePARS, which allows automatic retrieval of NIH funding sources for publications in any list, such as the bibliography of a National Academies report (Drew et al. 2016). NIH recently used its new systems to show the impact of the National Toxicology Program with hexavalent chromium as a case study (Xie et al. 2016). EPA should devote personnel time to such efforts and apply the techniques to construct richer and more robust indicators to demonstrate how the results of STAR grants have improved human health and the environment.

REFERENCES

Anton, W.R.Q, G. Deltas, and M. Khanna. 2004. Incentives for environmental self-regulation and implications for environmental performance. J. Environ. Econom. Manage. 48(1):632-654.

Bergquist, A.M., J.K. Choe, T.J. Strathmann, and C.J Werth. 2016. Evaluation of a hybrid ion exchange-catalyst treatment technology for nitrate removal from drinking water. Water Res. 96(1):177-187.

Bond, T. C., S.J. Doherty, D.W. Fahey, P.M. Forster, T. Berntsen, B.J. DeAngelo, M.G. Flanner, S. Ghan, B. Kärcher, D. Koch, S. Kinne, Y. Kondo, P.K. Quinn, M.C. Sarofim, M.G. Schultz, M. Schulz, C. Venkataraman, H. Zhang, S. Zhang, N. Bellouin, S.K. Guttikunda, P.K. Hopke, M.Z. Jacobson, J.W. Kaiser, Z. Klimont, U. Lohmann, J.P. Schwarz, D. Shindell, T. Storelvmo, S.G. Warren, and C.S. Zender. 2013. Bounding the role of black carbon in the climate system: A scientific assessment. J. Geophys. Res. Atmos. 118(11):5380-5552.

Bozeman, B. 2003. Public value mapping of science outcomes: Theory and method. Pp. 3-48 in Knowledge Flows and Knowledge Collectives: Understanding the Role of Science and Technology Policies in Development, Vol. 2. Public Value Mapping for Scientific Research, B. Bozeman, D. Sarewitz, S. Feinson, G. Foladori, M. Gaughan, A. Gopta, B. Sampat, and G. Zachary, eds. Washington, DC: Center for Science Policy and Outcomes.

Breysse, P.N., R.J. Delfino, F. Dominici, A.C.P. Elder, M.W. Frampton, J.R. Froines, A.S. Geyh, J.J. Godleski, D.R. Gold, P.K. Hopke, P. Koutrakis, N. Li, G. Oberdörster, K.E. Pinkerton, J.M. Samet, M.J. Utell, and A.S. Wexler. 2013. U.S. EPA particulate matter research centers: Summary of research results for 2005-2011. Air Qual. Atmos. Health 6(2):333-355.

CARB (California Air Resources Board). 2014. Multi-Regulation Summary (MRS) Requirements for Diesel Truck and Equipment Owners [online]. Available: https://www.arb.ca.gov/msprog/onrdiesel/documents/multirule.pdf [accessed April 27, 2017].

Carlton, A.G., B.J. Turpin, K.E. Altieri, S.P. Seitzinger, R. Mathur, S.J. Roselle, and R.J. Weber. 2008. CMAQ model performance enhanced when in-cloud secondary organic aerosol is included: Comparisons of organic carbon predictions with measurements. Environ. Sci. Technol. 42 (23):8798-8802.

Cason, T.N., and L. Gangadharan. 2004. Auction design for voluntary conservation programs. Am. J. Agricult. Econ. 86 (5):1211-1217.

Cozzens, S.E. 1997. The knowledge pool: Measurement challenges in evaluating fundamental research programs. Eval. Program Plann. 20(1):77-89.

Curl, C.L., R.A. Fenske, J.C. Kissel, J.H. Shirai, T.F. Moate, W. Griffith, G. Coronado, and B. Thompson. 2002. Evaluation of take-home organophosphorus pesticide exposure among agricultural workers and their children. Environ. Health Perspect. 110(12):A787-A792.

Drew, C.H., K.G. Pettibone, F.O. Finch III, D. Giles, and P. Jordan. 2016. Automated research impact assessment: A new bibliometrics approach. Scientometrics 106(3):987-1005.

Engel-Cox, J.A., B. Van Houten, J. Phelps, and S.W. Rose. 2008. Conceptual model of comprehensive research metrics for improved human health and environment. Environ. Health Perspect. 116 (5):583-592.

EPA (U.S. Environmental Protection Agency). 2006. Interim Reregistration Eligibility Decision for Azinphos-Methyl, Case No. 0235. Office of Pesticide Protection, Washington, DC [online]. Available: https://archive.epa.gov/pesticides/reregistration/web/pdf/azinphosmethyl_ired.pdf [accessed April 27, 2017].

EPA. 2012. Provisional Assessment of Recent Studies on Health Effects of Particulate Matter Exposure. EPA/600/R-12/056F. National Center of Environmental Assessment RTP Division, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://www.epa.gov/ttn/naaqs/standards/pm/data/20121213psa.pdf [accessed April 27, 2017].

EPA. 2015a. Air, Climate, and Energy Strategic Action Plan 2016-2019. EPA 601/K-15/005. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://www.epa.gov/sites/production/files/2015-10/documents/strap_2016_ace_508.pdf [accessed January 17, 2017].

EPA. 2015b. Chemical Safety for Sustainability Strategic Research Action Plan 2016-2019. EPA 601/K-15/003. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://www.epa.gov/sites/production/files/2016-02/documents/strap_2016_css_508_compliant_final_optimized_0.pdf [accessed January 17, 2017].

EPA. 2015c. Safe and Sustainable Water Resources Strategic Research Action Plan 2016-2019. EPA 601/K-15/004. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://www.epa.gov/sites/production/files/2015-10/documents/strap_2016_sswr_508.pdf [accessed January 17, 2017].

EPA. 2015d. Sustainable and Healthy Communities Strategic Research Action Plan 2016-2019. EPA 601/K-15/006. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://www.epa.gov/sites/production/files/2015-10/documents/strap_2016_shc_508.pdf [accessed January 17, 2017].

Fanning, E.W., J.R. Froines, M.J. Utell, M. Lippmann, G. Oberdörster, M. Frampton, J. Godleski, and T.V. Larson. 2009. Particulate matter (PM) research centers (1999-2005) and the role of interdisciplinary center-based research. Environmental Health Perspect. 117(2):167-174.

FDA (U.S. Food and Drug Administration). 2016. FDA proposes limit for inorganic arsenic in infant rice cereal. FDA News Release, April 1, 2016 [online]. Available: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm493740.htm [accessed May 17, 2017].

Foran, J.A., A.D. DeWeese, M.J. Hudson, and N.E. Kmiecik. 2010. Evaluation of mercury exposure reduction through a fish consumption advisory program for anishinaabe tribal members in northern Wisconsin, Michigan, and Minnesota. J. Environ. Public Health 2010:802584.

Gauderman, W.J., H. Vora, R. McConnell, K. Berhane, F. Gilliland, D. Thomas, F. Lurmann, E. Avol, N. Kunzli, M. Jerrett, and J. Peters. 2007. Effect of exposure to traffic on lung development from 10 to 18 years of age: A cohort study. Lancet 369(9561):571-577.

GLIFWC (Great Lakes Indian Fish and Wildlife Commission). 2016. Mercury-Based Consumption Advice for Ceded Territory Muskellunge. Memo from Sara Moses, Environmental Biologist, to Neil Kmiecik, Biological Services Director, Great Lakes Indian Fish and Wildlife Commission, Odanah, WI, February 8, 2016 [online]. Available: https://www.glifwc.org/Mercury/Musky%20Consumption%20Advice%20Memo%20Feb2016.pdf [accessed April 27, 2017].

Hellerstein, D., N. Higgins, and M. Robers. 2015. Options for Improving Conservation Programs: Insights From Auction Theory and Economic Experiments. Economic Research Service Report No. 181. U.S. Department of Agriculture, Economic Research Service, Washington, DC. January 2015. [online]. Available: https://www.ers.usda.gov/webdocs/publications/45333/50532_err181.pdf?v=42502 [accessed April 27, 2017].

Hutson, M.S., P.G. Alexander, V. Allwardt, D.M Aronoff, K.L. Bruner-Tran, D.E. Cliffel, J.M. Davidson, A. Gough, D.A. Markov, L.J. McCawley, J.R. McKenzie, J.A. McLean, K.G. Osteen, V. Pensabene, P.C. Samson, N.K. Senutovitch, S.D. Sherrod, M.S. Shotwell, D.L. Taylor, L.M. Tetz, R.S. Tuan, L.A. Vernetti, and J.P. Wikswo. 2016. Organs-on-chips as bridges for predictive yoxicology. Appl. In Vitro Toxicol. 2(2):97-102.

IPCC (Intergovernmental Panel on Climate Change). 2014. Climate Change 2014: Impacts, Adaptation, and Vulnerability, Part A. Global and Sectoral Aspects, C.B. Field, V.R. Barros, D.J. Dokken, K.J. Mach, M.D. Mastrandrea, T.E. Bilir, M. Chatterjee, K.L. Ebi, Y.O. Estrada, R.C. Genova, B. Girma, E.S. Kissel, A.N. Levy, S. MacCracken, P.R. Mastrandrea, and L.L.White, eds. Cambridge, UK: Cambridge University Press. 1132 pp.

Jacob, D.J., and D.A. Winner. 2009. Effect of climate change on air quality. Atmos. Environ. 43 (1):51-63.

Johnson, J. 2016. Presentation at the First Meeting on Review of EPA’s Science to Achieve Results’ Research Grants Program, June 21, 2016, Washington, DC.

Karagas, M.R., T. Punshon, V. Sayarath, B.P. Jackson, C.L. Folt, and K.L. Cottingham. 2016. Association of rice and rice-product consumption with arsenic exposure early in life. JAMA Pediatr. 170(6):609-616.

Laden, F., J. Schwartz, F.E. Speizer, and D.W. Dockery. 2006. Reduction in fine particulate air pollution and mortality: Extended follow-up of the Harvard Six Cities study. Am. J. Respir. Crit. Care Med. 173(6):667-672.

Liebow, E., J. Phelps, B. Van Houten, S. Rose, C. Orians, J. Cohen, P. Monroe, and C.H. Drew. 2009. Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environ. Health Perspect. 117(7):1147-1154.

McConnell, R., K. Berhane, F. Gilliland, S.J. London, T. Islam, W.J. Gauderman, E. Avol, H.G. Margolis, and J.M. Peters. 2002. Asthma in exercising children exposed to ozone: A cohort study. Lancet 359(9304):386-391.

McConnell, R., K. Berhane, L. Yao, M. Jerrett, F. Lurmann, F. Gilliland, N. Künzli, J. Gauderman, E. Avol, D. Thomas, and J. Peters. 2006. Traffic, susceptibility, and childhood asthma. Environ. Health Perspect. 114(5):766-772.

McLaughlin, J.A., and G.B. Jordan. 2015. Using logic models. Pp. 62-87 in Handbook of Practical Program Evaluation, 4th Ed., K.E. Newcomer, H.P. Hatry, and J.S. Wholey, eds. Hoboken, NJ: John Wiley & Sons, Inc.

Meho, L.I., and Y. Kiduk. 2007. Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J. Am. Soc. Inf. Sci. Technol. 58(13):2105-2125.

National Academies of Sciences, Engineering, and Medicine. 2016. The Future of Atmospheric Chemistry Research: Remembering Yesterday, Understanding Today, Anticipating Tomorrow. Washington, DC: The National Academies Press.

Nel, A. 2005. Air pollution-related illness: Effects of particles. Science 308(5723):804-806.

NIH (National Institute of Health). 2014. Scientific Management Review Board Report on Approaches to Assess the Value of Biomedical Research Supported by NIH [online]. Available: https://smrb.od.nih.gov/documents/reports/VOBR%20SMRB_Report_2014.pdf [accessed May 17, 2017].

NRC (National Research Council). 2008. Evaluating Research Efficiency in the U.S. Environmental Protection Agency. Washington, DC: The National Academies Press.

NYC DEP (New York City Department fof Environmental Protection). 2011. Promulgation of Amendments to Chapter 2 of Title 15 of the Rules of the City of New York Rules Governing the Emissions from the Use of #4 and #6 Fuel Oil in Heat and Hot Water Boilers and Burners [online]. Available: http://www.nyc.gov/html/dep/pdf/air/heating_oil_rule.pdf [accessed April 27, 2017].

Oberdörster, G. 2001. Pulmonary effects of inhaled ultrafine particles. Int. Arch. Occup. Environ. Health 74(1):1-8.

Oberdörster, G., E. Oberdörster, and J. Oberdörster. 2005. Nanotoxicology: An emerging discipline evolving from studies of ultrafine particles. Environ. Health Perspect. 113(7):823-839.

Orians, C., J. Abed, C. Drew, S.W. Rose, J. Cohen, and J. Phelps. 2009. Scientific and public health impacts of the NIEHS Extramural Asthma Research Program-insights from primary data. Res. Eval. 18(5):375-385.

Patel, M.M., L. Hoepner, R. Garfinkel, S. Chillrud, A. Reyes, J.W. Quinn, F. Perera, and R.L. Miller. 2009. Ambient metals, elemental carbon, and wheeze and cough in New York City children through 24 months of age. Am. J. Respir. Crit. Care Med. 180(11):1107-1113.

Peng, R.D., H.H. Chang, M.L. Bell, A. McDermott, S.L. Zeger, J.M. Samet, and F. Dominici. 2008. Coarse particulate matter air pollution and hospital admissions for cardiovascular and respiratory diseases among Medicare patients. JAMA 299(18):2172-2179.

Peters, A., D.W. Dockery, J.E. Muller, and M.A. Mittleman. 2001. Increased particulate air pollution and the triggering of myocardial infarction. Circulation 103(23):2810-2815.

Phenrat, T., and G.V. Lowry. 2009. Physicochemistry of polyelectrolyte coatings that increase stability, mobility, and contaminant specificity of reactive nanoparticles used for groundwater remediation. Pp. 249-267 in Nanotechnology Applications for Clean Water, N. Savage, M. Diallo, J. Duncan, A. Street, and R. Sustich, eds. Norwich, NY: William Andrew Inc.

Pope, C.A., R.T. Burnett, M.J. Thun, E.E. Calle, D. Krewski, K. Ito, and G.D. Thurston. 2002. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. JAMA 287(9):1132-1141.

Pope, C.A., R.T. Burnett, G. Thurston, M. Thun, E.E. Calle, D. Krewski, and J. Godleski. 2004. Cardiovascular mortality and long-term exposure to particulate air pollution. Circulation 109(1):71-77.

Pope, C.A., M. Ezzati, and D.W. Dockery. 2009. Fine-particulate air pollution and life expectancy in the United States. N. Engl. J. Med. 360(4):376-386.

Rennings, K., A. Ziegler, K. Ankele, and E. Hoffmann. 2006. The influence of different characteristics of the EU environmental management and auditing scheme on technical environmental innovations and economic performance. Ecol. Econ. 57(1):45-59.

Savage, Nora, and Mamadou S. Diallo. 2005. Nanomaterials and water purification: opportunities and challenges. J. Nanopart. Res. 7(4):331-342.

Sinclair, R.G., J.B. Rose, S.A. Hashsham, C.P. Gerba, and C.N. Haas. 2012. Criteria for selection of surrogates used to study the fate and control of pathogens in the environment. Appl. Environ. Microbiol. 78 (6):1969-1977.

Stevens, D.L., and A.R. Olsen. 2004. Spatially balanced sampling of natural resources. J. Am. Stat. Assoc. 99(465):262-278.

Thomson Reuters. 2014. InCities Essential Science Indicators Help [online]. Available: http://ipscience-help.thomsonreuters.com/incitesLiveESI/ESIGroup/fieldBaselines/PercentilesBaselines.html [accessed April 27, 2017].

Hellerstein, D., N. Higgins, and M. Robers. 2015. Options for Improving Conservation Programs: Insights From Auction Theory and Economic Experiments. Economic Research Service Report No. 181. U.S. Department of Agriculture, Economic Research Service, Washington, DC. January 2015. [online]. Available: https://www.ers.sda.gov/webdocs/publications/45333/50532_err181.pdf?v=42502 [accessed April 27, 2017].

Johnson, J. 2016. Presentation at the First Meeting on Review of EPA’s Science to Achieve Results’ Research Grants Program, June 21, 2016, Washington, DC.

Wagstrom, K.M., K.R. Baker, A.E. Leinbach, and S.W. Hunt. Synthesizing scientific progress: Outcomes from U.S. EPA’s Carbonaceous Aerosols and Source Apportionment STAR Grants. Environ. Sci. Technol. 48(18):10561-10570.

Weinberg, B.A., J. Owen-Smith, R.F. Rosen, L. Schwarz, B. McFadden Allen, R.E. Weiss, and J. Lane. 2014. Science funding and short-term economic activity. Science. 344(6179):41-43.

Weaver, C.P., E. Cooter, R. Gilliam, A. Gilliland, A. Grambsch, D. Grano, B. Hemming, S.W. Hunt, C. Nolte, D.A. Winner, and X.Z. Liang. 2009. A preliminary synthesis of modeled climate change impacts on US regional ozone concentrations. Bull. Am. Meteorol. Soc. 90(12):1843.

Xie, Y., S. Holmgren, D.M.K. Andrews, and M.S. Wolfe. 2017. Evaluating the impact of the US National Toxicology Program: A case study on hexavalent chromium. Environ. Health Perspect. 125(2):181-188.

Zimmerman, B.D., A. Korajkic, N.E. Brinkman, A.C. Grimm, N.J. Ashbolt, and J.L. Garland. 2016. A spike cocktail approach to improve microbial performance monitoring for water reuse. Water Environ. Res. 88(9):824-837.