The Office of Data and Informatics (ODI) is an office within the Material Measurement Laboratory (MML). ODI is a service-oriented organization whose mission is to provide leadership and expertise to meet modern data challenges and leverage data-driven research opportunities for the MML and NIST scientific research data infrastructure. This helps NIST scientists optimize the discoverability, usability, and interoperability of their data products. ODI contributes to the overall mission of the MML and NIST by providing guidance in best practices and resources that optimize the creation and use of data products. They provide guidance, assistance, and resources for the biological, chemical, and materials work of the MML. In this role, ODI coordinates across MML laboratory domain experts and other data specialists at NIST.1
The ODI is focused on four functional areas: (1) distribution and curation of standard reference data (SRD); (2) research data management, preservation and dissemination; (3) in-house consultation services on informatics and analytics methods and tools; and (4) open data/open science community engagement.2 In fiscal year (FY) 2020, ODI’s budget was $4.744 million, 20 percent of which came from revenues from SRD services connected with fulfillment of 2,600 ecommerce orders.
Current staffing of ODI is 18 full-time equivalents (FTEs), which includes the director and a senior advisor, 6.5 technical staff in the Data Services Group, 6 technical staff and 2 contractors in the Data Sciences Group, 1.5 FTE of administrative support, and several affiliated staff not directly assigned to ODI.
ASSESSMENT OF TECHNICAL PROGRAMS
ODI is the principal outlet for SRD products across not only the MML but NIST as a whole. MML produces about 90 percent of SRD NIST-wide. It delivers SRD products to customers outside NIST, collect license revenue under copyright laws, and remit the funds to other NIST organizations to support the development, maintenance, and enhancement of SRD products.
It serves as the repository for SRD artifacts, safeguarding their technical and legal integrity. The repository also serves as the way for the public to browse and shop for SRD products. ODI recently started to migrate to Salesforce.com to better manage this activity.
1 Robert Hanisch, National Institute of Standards and Technology (NIST), 2020, “Office of Data and Informatics, Division 641: Overview,” presentation to the panel, September 9.
2 NIST Material Measurement Laboratory (MML), 2020, “National Academies of Sciences, Engineering, and Medicine: 2020 Assessment Read-Ahead Materials for September 9-11, 2020,” Gaithersburg, MD.
ODI has a lead role in developing and piloting laboratory information management systems (LIMS). The successfully demonstrated pilot automates many key aspects of collecting and curating data from electron microscopes. LIMS resulted in more productivity for researchers using the complex instruments and control of their data through collaboration and publishing results.
ODI plays a lead role in advocating data management plans (DMPs). Under a directive from the Office of Science and Technology Policy (OSTP),3 NIST is supposed to create DMPs for projects that summarize how the data are to be gathered, utilized, analyzed, disseminated, and preserved. ODI has been successful in socializing the concept of DMP and continues to work in securing implementation by MML projects.
ODI also plays a lead role in developing and advancing the concept of data frameworks. More ambitious than DMPs, the data framework considers the entire life cycle of data artifacts, from planning and requirements establishment, data creation/collection, use and analysis, curation, sharing, protection, access, archiving, and ultimate deaccession. ODI’s efforts for data frameworks are both internal to the MML and NIST, and external, with ODI taking a leadership role on this subject to the larger STEM community.
Challenges and Opportunities
The challenges and opportunities for the technical programs will be discussed below, jointly with those of the portfolio of scientific expertise.
PORTFOLIO OF SCIENTIFIC EXPERTISE
The accomplishments with respect to the portfolio of scientific expertise were discussed above, jointly with those of the technical programs.
Challenges and Opportunities
Data is central to virtually every activity across the MML and NIST. Best practices for data collection, curation, collaboration, transformation, analysis, protection, archiving, and life-cycle management become ever more crucial to the performance of STEM organizations. ODI’s role is to advocate, advise, consult, build and curate tools, and provide training in support of this mission.
ODI views data as a key product resulting from the activities of the MML and NIST, and therefore as a key asset of the organization. One could say that ODI has the ambition to foster a culture in which data as a pervasive asset becomes established in the research and development activities of every group, division, and laboratory at NIST, embraced and advocated at all levels of management.
There is much to be done to establish this culture of viewing and valuing data as a key asset across NIST. While ODI has a vital role, ODI alone cannot make this transformation happen. That will require attention by MML and NIST management to the issue. It will impact priorities and plans both long and short term.
The status of the LIMS initiative offers a case study for the MML and NIST on this opportunity. In collaboration with an active research group, ODI helped create a successful pilot LIMS implementation for certain electron microscopes, as noted above. That success creates the opportunity to move LIMS
3 John Holdren, Director, Office of Science and Technology Policy, 2013, “Memorandum for the Heads of Executive Departments and Agencies,” February 22, https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf.
from pilot to production status, to expand the software to cover more electron microscope types, and to build LIMS for other types of complex instruments. However, realizing this potential will necessitate changes in the priorities and programs from current plans, not just by ODI, but more broadly in other MML organizations. This is clearly something that ODI cannot be expected to accomplish on its own.
Support for fundamental data activities is not always timely, and this has led to a lack of maintenance for a number of SRD products.4 ODI was able to initiate seven SRD enhancement projects using a one-time allocation of funds. The need to assess data quality on an ongoing basis suggests that ongoing funding would better. Similarly, the Open Access to Research scientific data infrastructure work will need long-term support after this year when support from Associate Directory for Laboratory Programs ends. These instances suggest that the value of ODIs work is not universally recognized, notwithstanding some of the successes ODI has had at its own initiative to embed staff in other NIST organizational units (see discussion below in “Dissemination of Outputs”).
RECOMMENDATION 4-1: The Material Measurement Laboratory management should promote the concept of “data as an asset” and its associated culture within the Laboratory. With that understanding, management can be expected to advocate and support its adoption throughout the organization, resulting in increased professionalism within NIST, higher quality of output by NIST, increased impact of NIST products on the STEM world, and public perception of NIST as a leader.
ADEQUACY OF FACILITIES, EQUIPMENT AND HUMAN RESOURCES
ODI seemed to have made serious progress in achieving the goals in response from the 2017 report. In the response to the 2017 review,5 ODI had proposed to push for laboratory-driven science involving instrumentation and detectors and analytics. ODI has introduced efforts to embed data-driven metrology into the research workflow at NIST. LIMS capabilities at NIST have advanced significantly through establishment of a network.
The SRD program is providing a good service with a sustainable financial model for industry. The research data framework has a strong outreach program.
Challenges and Opportunities
ODI could benefit from a sustainment strategy of meeting the goals in its LIMS efforts to ensure that any progress made in the past 3 years is not lost. The challenges include the following:
- A diverse set of needs across the laboratory, with no one system meeting everyone’s needs.
- Most divisions lack a DMP and continue to work through requirements and workflows.
- LIMS is an unfunded effort.
4 NIST MML, 2020, “National Academies of Sciences, Engineering, and Medicine: 2020 Assessment Read-Ahead Materials for September 9-11, 2020,” Gaithersburg, MD, p. 52.
5 National Academies of Sciences, Engineering, and Medicine (NASEM), 2017, An Assessment of the National Institute of Standards and Technology Material Measurement Laboratory: Fiscal Year 2017, Washington, DC: The National Academies Press.
ODI needs software engineers, as there is minimal expertise across NIST for scientific data systems design and architecture. A high level of security is required for NIST, and gate keeping for deployments, requirements, and design are not available.
To help grow and sustain LIMS efforts, it may be desirable to establish informatics/data tools that enable one to fuse data across instrumentation. Another potential opportunity is to link to computational infrastructure to connect characterization and simulation studies. Further opportunities have been identified by ODI, including the completion of LIMS requirements, design, and workflows to level of support for roll-out across division projects.
DISSEMINATION OF OUTPUTS
ODI has been active in numerous outreach efforts such as the Project Open Data Cloud-First strategy, an enhanced science data portal and an open-source code base hosted on github/USNISTGOV. ODI has been active in both national and international scientific efforts, including CODATA and OSTP subcommittees. The Informatics and Analytics program area has embedded staff in different NIST collaborations and introduced seminars and sponsored membership in The Carpentries,6 a non-profit organization that provides training in coding and data science skills (i.e., “software carpentry”).7 NIST is now well positioned to meet the new Federal Data Strategy 2020 Action Plan.8
Challenges and Opportunities
The cultural acceptance of doing data-driven research is still a challenge. Scientific researchers not familiar with data analytical techniques still question why it is necessary to track and keep data. The challenges acknowledged in the 2017 report seem to persist. Initial efforts toward DMPs were uneven, and staff generally consider DMPs to be an administrative burden with no real benefit.
Unfortunately, there is still no one-size-fits-all LIMS solution for the diversity of data and operational modes present within MML; however, by building and reutilizing and by working in collaboration with technical staff in each division—perhaps by the latter assigning a data steward with a partial FTE—progress can be made.
With respect to SRD, copyright issues could interfere with the business model. NIST will need to consider how to address the challenge of how does one continue in an “open data” environment. A strategy needs to be established on how to maintain the financial model but still meet the spirit and intent of open access in the scientific community.
With the unique resource available through the NIST postdoc program, ODI may want to include informatics training of post docs by setting up laboratory rotations with the other divisions. This may help to expand the role of informatics/data science tools as more than just a service but a critical value-added component in research.
7 NIST MML, 2020, “National Academies of Sciences, Engineering, and Medicine: 2020 Assessment Read-Ahead Materials for September 9-11, 2020,” Gaithersburg, MD, p. 48.
OVERALL CONCLUSIONS AND RECOMMENDATIONS
It is widely recognized that data is central to advances in science and engineering and is critical to the competitiveness of U.S. industry. Given its position and role in the MML and NIST in general, ODI, when properly leveraged and resourced (e.g., when it has funds to modernize SRD products), is in a unique position to advance the goals of not only the MML but NIST more broadly. To achieve this potential, ODI will require an explicit effort at the laboratory level to bring about a cultural change that brings data as a product into the overall laboratory consciousness (see Recommendation 4-1). ODI’s current activities, such as development of LIMS systems to provide the institutional infrastructure, mentorship and training will accelerate this culture shift.
The following recommendations are intended to facilitate this shift.
RECOMMENDATION 4-2: The Office of Data and Informatics (ODI) should build out structures for enhancing divisional interactions. One concrete mechanism for this is to create tightly integrated multidisciplinary teams, which include ODI domain expertise as an integral part of a research team. The concept of “research software engineering” has been advocated as one such mechanism for creating research teams that can respond to the centrality of data and computation in a research activity.
FINDING: Postdoctoral fellows are being used effectively in other organizational units within the MML, however within ODI their use is limited. This is understandable given the service-orientated nature of their mission. However, even within these confines, postdocs could be used effectively both to advance the direct goals of ODI, but more importantly to act as agents of culture change across the laboratory toward an awareness of data as a product. ODI is already engaged in training with its programs in “software carpentry.”
RECOMMENDATION 4-3: The “software carpentry” program should be expanded to include rotations of postdocs through ODI for more extensive, hands-on guidance. In addition, identifying postdocs in the MML divisions with an understanding of the importance of sound computational techniques and establishing joint mentorship programs with those divisions and ODI would be beneficial.
RECOMMENDATION 4-4: The Material Measurement Laboratory should enhance engagement with creation/integration of reference materials.