The mission of the Engineering Laboratory (EL) of the National Institute of Standards and Technology (NIST) is to “promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology for engineered systems in ways that enhance economic security and improve quality of life.”1 To support this mission the EL has developed thrusts in smart manufacturing, construction, and cyberphysical systems; in sustainable and energy-efficient manufacturing materials and infrastructure; and in disaster-resilient buildings, infrastructure, and communities. The technical work of the EL is performed in five divisions—Intelligent Systems, Materials and Structural Systems, Energy and Environment, Systems Integration, and Fire Research—and in two offices—the Applied Economics Office and the Smart Grid Program Office.
In 2017, at the request of the acting director of NIST, the National Academies of Sciences, Engineering, and Medicine formed the Panel on Review of the Engineering Laboratory at the National Institute of Standards and Technology (referred to in this report as “the panel”) and established the following statement of task for the panel:
The Panel on Review of the Engineering Laboratory at the National Institute of Standards and Technology will assess the scientific and technical work performed by the National Institute of Standards and Technology (NIST) Engineering Laboratory (EL). The panel will review technical reports and technical program descriptions prepared by NIST staff, and will visit the facilities of the NIST laboratory. The visit will include technical presentations by NIST staff, demonstrations of NIST projects, tours of NIST facilities, and discussions with NIST staff. The panel will deliberate findings in a closed session panel meeting and will prepare a report summarizing its assessment findings.
The acting director of NIST requested that in 2017 the panel confine its assessment to the following smart manufacturing programs conducted at the Engineering Laboratory: Measurement Science for Additive Manufacturing (MSAM), Robotic Systems for Smart Manufacturing (RSSM), Smart Manufacturing Operations Planning and Control (SMOPAC), and Smart Manufacturing Systems Design and Analysis (SMSDA).
CONCLUSIONS AND RECOMMENDATIONS
General Conclusions and Recommendations
Smart manufacturing programs in the EL at NIST have significantly evolved in the last five years, matching increased national recognition of the importance of manufacturing in the economy, as well as international recognition of the importance of integration of digital computing and information technologies into manufacturing systems. For example, the additive manufacturing program area was
established approximately three and one-half years ago and is now an impressive effort in an area of rapid technology growth, staffed by competent early-career researchers, and facilitated by a measurement testbed of their own design with unique and powerful capabilities. Standards are being given a high priority, and there is excellent collaboration with other organizations and leadership in these efforts. There is significant focus on small and medium enterprises (SMEs), which is a valuable objective that is difficult to achieve. There are good research collaborations with industry, but perhaps an overreliance on (hoped for) trickle down from work with larger industries; more extensive outreach to SMEs, especially young SMEs, would be beneficial.
The EL’s smart manufacturing program appears to lack a roadmap and measurable milestones, as do the individual programs and projects. Without roadmaps and measurable milestones, the quality and value of NIST’s smart manufacturing programs and their management were more difficult to assess, and therefore the value, quality, and results of individual projects were more difficult to measure. Although some excellent research outputs, evinced by strong publication records in peer-reviewed journals, are being obtained and significant progress in standards is being made, many project plans did not show intermediate project milestones, and so it was difficult to determine in some cases whether projects were making the expected amount of progress and achieving the desired results. Roadmaps and measurable milestones would be invaluable tools for NIST management.
There is also, in some cases, a lack of crosscutting activities that can take advantage of the synergistic objectives of individual projects. Examining the project’s plans, together to find opportunities for cross-fertilization and integrating the testing and evaluation work across the overall smart manufacturing program, could provide more synergy across the projects, as well as an opportunity to achieve broader impact of the program work as a whole, rather than only at the project level. This is not helped by the current distance between laboratory spaces, and it would be better to have a larger and more contiguous space so that multiple projects and programs could be integrated into a larger, more complex testbed that more resembles some manufacturing environments.
The new program areas are well covered by both junior and senior staff, who have significantly and successfully adapted their efforts to new technologies. Some areas appear to be short on permanent staff and expertise, but staffing appears to be generally sufficient to support the research that is under way. The journal publication record is excellent. There has been significant external recognition of the staff and its research, but additional efforts need to be made to promote and apply for external awards for the staff. This also would promote recognition of the smart manufacturing research programs at NIST.
Measurement Science for Additive Manufacturing
The majority of national laboratories within the federal government currently pursue additive manufacturing (AM), and the significant progress that has been made within the Measurement Science for Additive Manufacturing (MSAM) program could benefit these other laboratories that, in turn, could likely help the MSAM program. Additionally, closer engagement and alignment with other U.S. government laboratories could ensure that the MSAM program is effectively leveraging the investments being made in this area.
RECOMMENDATION 1: The Engineering Laboratory should consider closer engagements with other national laboratories’ additive manufacturing groups.
A dialogue needs to be intensified with the Material Measurement Laboratory (MML) and its Polymers Processing Group in order to exploit common strengths and overlapping research interests. This dialogue could build on existing meetings with the MML and the Polymers Processing Group and evolve toward common research projects, such as feedstock characterization or the design of additive manufacturing machines.
RECOMMENDATION 2: The Engineering Laboratory should consider beginning an intensified dialogue with the Material Measurement Laboratory and the Polymers Processing Group.
There are several highly industry relevant topics within AM that are not currently pursued with sufficient emphasis by the MSAM program. These topics include support structures and their design, the effects of different build orientations, design limitations for AM, and the post-processing side of AM. Broadening the focus into post- or secondary processing will help researchers understand what limitations within the technologies cannot be overcome with post-processing methods. Expanding research veins into support structures and post-processing, and defining new procedures in this field, would allow an opportunity to develop new standards, which could help industry find common ground.
RECOMMENDATION 3: The Engineering Laboratory should consider expanding its research into the measurements and standards needed to assist industry in the following areas: support structures and their design, the effects of different build orientations, design limitations for additive manufacturing, and the post-processing side of additive manufacturing.
Robotic Systems for Smart Manufacturing
Overall, the laboratory facilities, equipment, and human resources are top-notch and sufficient for impactful standards development activities in the Robotic Systems for Smart Manufacturing (RSSM) program. The program staff has done an excellent job of identifying the top brands of robotic and sensor equipment that will likely be used by SMEs and has acquired and activated the equipment for its research. The staff has been successful in operating the equipment to develop best practice methods for using the hardware and related software and to conduct a wide variety of experiments and demonstrations to evaluate potential metrics, test artifacts, and methods for multi-robot collaboration, agility, interoperability, and integration.
The RSSM program is providing important contributions to the development of standards, metrics, and applicable technologies in the areas of performance assessment, collaborative robotics, agility, interoperability, and integration. The focus on the development of standards and performance metrics and tests in these areas aligns well with the mission of the EL. In particular, the program has made important advances in identifying weaknesses with existing metrics and standards in robotic mobility and agility, anticipating unmet needs for future industry challenges, and developing new metrics and standards to address these issues.
The RSSM program staff has an opportunity to lead in the development of a more systematic methodology for generating test and use cases. Such a methodology could include needs assessment, gap identification, research, metrics development, test methods and artifacts development, standards development, evaluation, and dissemination.
RECOMMENDATION 4: The Engineering Laboratory should consider leading the development of methodological approaches for standards development in all component areas of its robotic systems for smart manufacturing research, including the areas of needs assessment, gap identification, research, metrics development, test methods and artifacts development, standards development, evaluation, and dissemination.
The RSSM program could broaden its industrial impact through stronger interactions with the Manufacturing Extension Partnership (MEP) centers or other industrial consortia. Through these deepened interactions, the RSSM program could develop a set of use cases and obtain more in-depth feedback on standards development.
RECOMMENDATION 5: The Engineering Laboratory should consider interacting on a regular basis (e.g., twice annually) with the Manufacturing Extension Partnership (MEP) centers or other industry consortia to source use cases and receive feedback on the effectiveness of their development of metrics and standards.
Although vision sensors are used in several RSSM testbeds, the EL has an unexplored opportunity to investigate how machine vision can be used to enhance the flexibility and user-friendliness of robots for SMEs. The exploration of how to take advantage of machine vision to simplify robot programming, develop self-teaching techniques, ensure safe human-robot interaction, and adapt to the varied application environments at SME facilities is a good fit with the EL’s objectives for the RSSM program.
RECOMMENDATION 6: The Engineering Laboratory should consider exploring the use of machine vision to enhance the value of robots for small and medium enterprises (SMEs) by reducing programming complexity for short-run manufacturing applications.
There is an increasing utilization of cloud and edge resources for robotics and industrial automation. The Industry 4.0, Microsoft Azure, and Amazon Web Services (AWS) efforts are trying to integrate resources in a consistent and efficient framework, and the RSSM program has the opportunity to contribute to and lead this nascent field.
RECOMMENDATION 7: The Engineering Laboratory should consider exploring methodological approaches and standards development related to cloud robotics by contributing to activities in this area (e.g., Industry 4.0), and taking a leadership role in this field.
Given the level of expertise and contribution of the RSSM research team, additional peer recognition is possible in the form of merit-based robotics awards, elevation in professional societies, invitations to delivering opening and keynote presentations at conferences, and early-career investigator awards. This type of external recognition can serve not only to award deserving individuals but also to raise the visibility of EL in the field.
RECOMMENDATION 8: The Engineering Laboratory should be more proactive in seeking individual recognition for accomplished robotics staff personnel, rather than waiting to be discovered.
Human-machine collaboration is an important challenge in robotics for manufacturing. The RSSM program appears to have an expertise gap in human robot interaction due to lack of permanent staff in this area. Active recruiting of such researchers into NIST permanent staff is important to establish this topic as an active research area.
RECOMMENDATION 9: The Engineering Laboratory should establish permanent staff expertise in its human-robot interaction area to avoid a future expertise gap.
The RSSM could more systematically leverage and seek out expertise from other organizations, including universities and industries, and through an extended use of guest researcher programs. A consortium of industrial companies, both large and small, as well as universities that meet with the RSSM on an annual or a semi-annual basis, could serve as additional sources of expertise for needs assessment, gap identification, and the testing and validation of standards.
RECOMMENDATION 10: The Engineering Laboratory should consider systematically leveraging and seeking out expertise from other organizations, including university, industry,
and Manufacturing USA Institutes who can serve as additional sources of expertise for needs assessment, gap identification, and the testing and validation of standards.
The RSSM program has researched the capabilities of a wide variety of robotic grippers. They have developed many test methods, artifacts, and metrics to guide in the design and use of grippers for dexterous grasping. They have an opportunity to leverage this research to create community-wide test sets that can assist the research community in developing new approaches to dexterous grasping. For example, the Yale-CMU-Berkeley (YCB) Object and Model Set data set is a related contribution to the research community that helps advance robotic manipulation, and a similar contribution based on the RSSM research would be highly beneficial to the research community.
RECOMMENDATION 11: The Engineering Laboratory should consider coordinating with other organizations to develop community-wide test sets for dexterous grasping.
Although the research facilities are adequate, they are also distributed across the NIST campus and are physically disconnected across multiple buildings on the campus. Given that the RSSM research projects share many common technologies, science, and resource needs, and can benefit from knowledge exchanges, such separations between the project teams might contribute to less cohesiveness and overall impact than might be possible through co-located facilities. The lack of crosscutting activities across all the projects of the RSSM program is not helped by the distance between laboratory spaces.
RECOMMENDATION 12: The Engineering Laboratory should consider co-locating test facilities to enhance integration of Robotic Systems for Smart Manufacturing (RSSM) program activities across the entire program.
There is an opportunity to develop software infrastructure resources to facilitate the interoperability of testbeds developed in individual projects. Several of the testbeds developed for individual projects have overlapping capabilities but cannot currently be integrated because they do not have interoperable software.
RECOMMENDATION 13: The Engineering Laboratory should strive for interoperability of software infrastructure for their testbeds.
The ability of RSSM staff to share videos, images, and other information through social media is hindered by the lengthy approval process at NIST and the lack of staff with expertise in social media outreach. As a result, the program has very little visibility on social media, which is a missed opportunity to improve the impact of the RSSM program’s activities.
RECOMMENDATION 14: The Engineering Laboratory should consider adding staff who can improve dissemination of technical accomplishments through social media, trade publications, and/or popular press and should work with NIST management to identify ways to streamline approval processes for applying these methods.
Smart Manufacturing Operations Planning and Control
The Smart Manufacturing Operations Planning and Control (SMOPAC) program needs to develop an overarching, integrated roadmap with measurable milestones that shows the relationships of the various markets (automotive, aerospace, etc.) and requirements to the technologies, as well as the relationships between the technical areas themselves. The roadmap needs to be an integrated, high-level plan, not a set of semi-related programs.
RECOMMENDATION 15: The Engineering Laboratory should consider developing an overarching roadmap with measurable milestones for the Smart Manufacturing Operations Planning and Control (SMOPAC) program that integrates market needs, product requirements, and technologies.
The EL needs to examine its activities within the context of national and global activities at other organizations to identify competitive and duplicative areas.
RECOMMENDATION 16: The Engineering Laboratory should consider benchmarking its work in smart manufacturing operations planning and control against other manufacturing programs, such as those in EU Horizon 2020, to identify competitive and duplicative areas.
In order to ensure alignment with manufacturers’ needs, and to identify early adopters of advanced technologies, programs, tools, and processes, the SMOPAC program needs to engage additional manufacturers, including new entrants to the manufacturing sector, such as start-up manufacturing companies and suppliers. They could partner, for example, with Manufacturing USA to facilitate information exchange.
RECOMMENDATION 17: The Engineering Laboratory should consider engaging new manufacturers and suppliers to ensure alignment with manufacturers’ needs and to identify early adopters of advanced technologies, programs, tools, and processes.
RECOMMENDATION 18: The Engineering Laboratory should consider partnering with Manufacturing USA to establish Standard for the Exchange of Product Data (STEP)-like computer-aided design (CAD), computer-aided manufacturing (CAM), and computer-integrated manufacturing (CIM) product information exchange standards.
The SMOPAC program needs to develop and show project plans with measurable milestones to describe their 5-year projects. Such project plans can be used as communication tools to inform management, the staff working on the project, users, emerging companies, and the public about the value of the project’s goals and when the results will be made available.
RECOMMENDATION 19: The Engineering Laboratory should develop project plans with measurable milestones that describe their 5-year projects and use these plans as communication tools.
The definition of “life cycle” that the Digital Thread for Smart Manufacturing project is using shows the end of the cycle as being when the initial product is delivered to the customer. This is different from the Department of Defense (DoD) ManTech projects, which include the Army, Navy, and Air Force projects on digital thread and digital twin, where life cycle also includes the life of the aircraft. In aerospace and automotive industries (as well as with health care devices and others), the manufacturer needs to have the ability to track the vehicles and devices in the field for potential part defects as long as they are in service; and so life cycle is from cradle to grave. In all fields, products have increasingly tremendous sensory and computational capabilities, and in-the-field capabilities for diagnostics and prognostics need to be considered as part of the life cycle in the digital thread.
RECOMMENDATION 20: The Engineering Laboratory should improve and clarify the definition of “life cycle” to be more comprehensive and include manufacturing activities after the product is delivered.
The concept of the digital thread can be applied to multiple industries of various sizes. The Digital Thread for Smart Manufacturing project is targeting medium-size manufacturers. While smaller manufacturers may either already have or be able to adopt these tools and standards, the team needs to address how large-scale industries (e.g., aerospace, automotive) will adopt NIST’s unique tools.
RECOMMENDATION 21: The Engineering Laboratory should put together a business case for small-, medium-, and large-scale industries to adopt their digital thread tools.
The Prognostics, Health Management, and Control (PHMC) team has developed an efficient and effective linear axis error detection methodology based on data collected from an inertial measurement unit (IMU), which has been developed and verified on the linear axis testbed and machine tools within the PHMC project at NIST. Validation efforts are ongoing both internally within NIST and externally with several manufacturing collaborators. The developed IMU sensor is very practical to assess machine health degradation rapidly. The team has demonstrated platform-based technologies to allow diversified users to connect to the machine and share their data. The team is planning to promote its research to system levels; however, the research scope needs to be more clearly defined (e.g., type of applications, sensors, and analytics) before this happens. Collaboration with the Clean Energy Smart Manufacturing Innovation Institute (CESMII) will help to define, identify, and satisfy industry needs and requirements as well as provide efficiencies in areas of overlapping research.
RECOMMENDATION 22: The Engineering Laboratory should consider collaborating with the Clean Energy Smart Manufacturing Innovation Institute to help to identify, prioritize, and satisfy industry needs and requirements and to provide efficiencies in areas of overlapping research.
The Cybersecurity for Smart Manufacturing project has established a testbed to validate the cybersecurity framework manufacturing profile. Data obtained from this testbed will be used to develop guidance for implementation of this framework. The testbed needs to include a simulation system that can generate virtual cyber threats to evaluate the resilience of the system.
RECOMMENDATION 23: The Engineering Laboratory’s testbed for the Cybersecurity for Smart Manufacturing project should include a simulation system that can generate virtual cyber threats to evaluate the resilience of the system.
The Wireless Systems for Industrial Environments project research is just beginning, and it needs to be considered more broadly. The team needs to develop testbeds that include both cybersecurity and wireless technology and consider integrating with a cloud-based or edge-based environment to support more tether-free monitoring system applications.
RECOMMENDATION 24: The Engineering Laboratory should consider developing a testbed that includes both cybersecurity and wireless technology.
RECOMMENDATION 25: The Engineering Laboratory should consider integrating with a cloud-based or edge-based environment to support more tether-free monitoring system applications.
In some of the testbeds, there are arrangements being made to integrate with other programs, such as robotics. This is a good start; however, the integration is small. It would be better to have a larger space so that multiple projects and programs could be integrated into a bigger, more complex testbed that more resembles an actual manufacturing environment. Validation of models under these conditions would provide meaningful data.
RECOMMENDATION 26: The Engineering Laboratory should consider dedicating an area or a facility for the integration of multiple projects and programs to simulate an industrial environment.
Smart Manufacturing Systems Design and Analysis
The NIST Smart Manufacturing Systems Design and Analysis (SMSDA) program would greatly benefit from and could contribute to an overarching roadmap of smart manufacturing for the nation.
RECOMMENDATION 27: The Engineering Laboratory should consider collaborating with the Smart Manufacturing Leadership Coalition (SMLC) and the newly formed Clean Energy and Smart Manufacturing Innovation Institute (CESMII) to develop for the National Institute of Standards and Technology and the United States an overarching roadmap with measurable milestones for smart manufacturing.
Where best practices for smart manufacturing capabilities do not exist, the task is to develop structures, databases, prototypes, and techniques to define best practices. During execution of tasks of this type, the inclusion of a domain expert on the team would greatly improve the output.
RECOMMENDATION 28: Where a smart manufacturing best practice does not exist, the Engineering Laboratory should consider embedding domain experts on the project teams to ensure the quality of their output.
The SMSDA program needs to ensure that the output of projects where a smart manufacturing best practice does not exist is tested in at least two SMEs. These SMEs need to become an integral part of the project teams to ensure that the developed structures, databases, and techniques make it easier for SMEs to utilize smart manufacturing.
RECOMMENDATION 29: The Engineering Laboratory should ensure that where a smart manufacturing best practice does not exist, the output of projects is tested in at least two small and medium enterprises (SMEs).
Where there is a lack of existing standards or guidelines, the SMSDA program needs to broaden its scope to include identifying smart manufacturing best practices in a select set of industry sectors. Given that the program has been actively collaborating with various standards communities and has access to a wide range of industries, this broadening of scope appears both feasible and practical.
RECOMMENDATION 30: Where there is a lack of existing standard or guidelines, the Engineering Laboratory should consider broadening its scope to include identifying smart manufacturing best practices in a select set of industry sectors.
Project planning for multiyear projects needs to include measurable intermediate milestones so that progress and midcourse correction can easily be understood and executed. The projects need to have a clear definition of deliverables and measurable milestones. Additionally, deliverables need to include SME demonstrations and testing.
RECOMMENDATION 31: The Engineering Laboratory should consider planning multiyear projects with measurable intermediate milestones so that progress and midcourse correction needs are readily visible.