Summary
At the request of the Director of the National Institute of Standards and Technology (NIST), in 2019 the National Academies of Sciences, Engineering, and Medicine formed the Panel on Review of the Engineering Laboratory at the National Institute of Standards and Technology and established the following statement of task for the panel:
The Panel on Review of the Engineering Laboratory at the National Institute of Standards and Technology will assess the scientific and technical work performed by the National Institute of Standards and Technology (NIST) Engineering Laboratory. The panel will review technical reports and technical program descriptions prepared by NIST staff, and will visit the facilities of the NIST laboratory. The visit will include technical presentations by NIST staff, demonstrations of NIST projects, tours of NIST facilities, and discussions with NIST staff. The panel will deliberate findings in a closed session panel meeting and will prepare a report summarizing its assessment findings.
The Director of NIST requested that in 2020 the panel assess the following activities conducted at the Engineering Laboratory (EL), which conducts activities in other areas as well:
- Community Resilience Program (CRP),
- Structural Performance Under Multi-hazards (SPUMH) Program,
- Earthquake Risk Reduction in Buildings and Infrastructure (ERR) Program,
- Engineered Materials for Resilient Infrastructure Program,
- Fire Research Programs,
- Net-Zero Energy, High-Performance Buildings Program, and
- Embedded Intelligence in Buildings (EIB) Program.
The Director of NIST also suggested that the panel consider during its assessment the following factors:
- The technical merit of the current laboratory program relative to current state-of-the-art programs worldwide;
- The portfolio of scientific expertise as it supports the ability of the organization to achieve its stated objectives;
- The adequacy of the laboratory budget, facilities, equipment, and human resources, as they affect the quality of the laboratory’s technical programs; and
- The effectiveness by which the laboratory disseminates its program outputs.
To accomplish the assessment, the National Academies assembled a panel of 24 volunteers whose expertise matched that of the work performed by the EL staff.1
On September 29 through October 1, 2020, the panel assembled virtually (the COVID-19
___________________
1 See the NIST Engineering Laboratory website at https://www.nist.gov/el for information on Engineering Laboratory organization and programs.
pandemic prohibited an in-person meeting) and interacted with NIST staff for a 3-day assessment, during which it received welcoming remarks from the Director of NIST and the EL director, heard overview presentations by EL management and presentations by researchers at the EL, and virtually toured portions of the EL facility, and the panel chair attended an interactive session with the Director of NIST, the NIST associate director for Laboratory Programs, and the EL Laboratory Director. The panel also met in a closed session to deliberate on its findings and to define the contents of this assessment report. This report summarizes the panel’s findings, conclusions, and recommendations.
The choice of projects to be reviewed was made by the EL. The panel applied a largely qualitative approach to the assessment. Given the nonexhaustive nature of the review, the omission in this report of any particular EL project should not be interpreted as a negative reflection on the omitted project.
GENERAL CONCLUSIONS
The NIST programs reviewed (CRP; SPUMH Program; ERR Program, Engineered Materials for Resilient Infrastructure Program; Fire Research Programs, Net-Zero Energy, High-Performance Buildings Program; and EIB Program) are each collaboratively performed by staff from across EL Divisional Groups. These programs have contributed major advancements in measurement science, standards, and technology over the past decade, which have strongly advanced U.S. competitiveness and innovations. Current national conditions, including the international COVID-19 pandemic, major wildfires throughout the western United States, a record number of hurricanes in the Atlantic Basin, and numerous other recent high-hazard events have heightened awareness of an urgent need for NIST’s work in these specific areas. For example, the Net Zero Energy, High Performance Buildings Program quickly incorporated the CONTAMX engine in the FaTIMA tool to assist in evaluating risk due to COVID-19 transmission in buildings. The Fire Risk Reduction in Communities Program rapidly developed and disseminated its current fire assessment and prediction models and is actively working with the California Department of Forestry and Fire Protection (CAL FIRE) and other on-the-ground wildland urban interface (WUI) firefighters. Additionally, NIST is the only organization in the United States that has the authority to independently investigate structural failures of built facilities. These contributions and others are of increasingly high value to the United States, both in ensuring health and safety of communities as well as enabling production and commerce.
Several common themes arise across these programs. As the complexity of current extreme conditions increase, explicit collaboration and coordination across these seven NIST programs and the NIST community as a whole, as well as with external organizations, will be critical to achieve significant advancements in the future. Multi-disciplinary and multi-organizational teams can effectively pursue long-term, highly interdependent research roadmaps aligned with NIST’s mission to enhance economic security and improve the quality of life.
While the demands of the nation on these divisions increase, and as NIST’s research agenda expands in response, NIST staff reported that the personnel, equipment, and facilities have not substantially increased over time and that these programs are stretched thin in expertise and resources to meet growing needs. NIST staff also reported that critical equipment and campus infrastructure are now constraining necessary facility and equipment upgrades. Internationally recognized experts within these seven programs possess unique capabilities, but as needs evolve and as staff retire, NIST’s preeminence in these fields may be jeopardized without strategic human resource planning, including recruiting, mentoring, and retaining diverse engineering and technical personnel.
Additionally, emerging technologies (e.g., artificial intelligence, machine learning, and increases in computational capabilities) are likely to impact these programs, and it is important to maintain continuous assessment of the applicability of such technologies to EL programs.
While these seven programs contribute to new standards and codes, new measurement methods and standards, and new computer models and equipment, the effective dissemination and implementation of these important advancements is hindered by a lack of coordinated outreach. Accelerating the co-
development and diffusion of results through employing advanced communications and marketing approaches, particularly targeting end users, could save lives, protect the economy, and improve the quality of life in the United States. Measuring the outcomes of the work by these seven programs, focusing on demonstrated benefits to safety, quality of life, and economic security, could further ensure NIST’s future contributions in these fields through apprising the U.S. public and industry of this critical and important resource.
The necessary outreach and remote collaboration brought on by the pandemic response provides an opportunity to increase the effectiveness of NIST in the eyes of the public by making it possible for more stakeholders to participate in and observe the activities at NIST.
ACCOMPLISHMENTS, CHALLENGES, AND OPPORTUNITIES
Technical Merit of the Programs
The technical merit of the seven programs reviewed by this panel is high, and NIST remains one of the highest-credibility sources for science-based tools for measurements, data, models, protocols, and reference standards nationally and internationally. For example, the Engineered Materials for Resilience Infrastructure Program has developed standard reference materials (SRMs) for concrete that are recognized and utilized worldwide by cement manufacturers and research laboratories. The regulations based on fire performance tests for mattresses developed by the Fire Research Division has reduced the number of deaths caused by flaming ignition of mattresses by close to 80 percent since its implementation.
These programs often use a combination of laboratory tests informed by in-field studies. For example, the CRP is actively learning from investigations performed in the aftermath of the 2017 Hurricane Maria, and the SPUMH Program based its 16 recommendations on the NIST Joplin tornado investigation. The Wildfire-Urban Interface Parcel program in the FRD uses targeted laboratory and field experiments, post-wildfire-urban interface (WUI) fire data collection and analysis, and a range of models, including vegetation and structure fire models, to understand WUI exposures.
Many of the programs collaborate with external organizations to enhance their effectiveness. For example, the CRP supports the Center of Excellence for Risk-Based Community Resilience Planning (CoE), an academic consortium of 14 universities headquartered at Colorado State University. The NIST ERR Program is the lead agency for the National Earthquake Hazards Reduction Program (NEHRP) with the Federal Emergency Management Agency (FEMA), the National Science Foundation, and the U.S. Geological Survey, and often partners with those agencies as well as industry associations to develop and advance related standards and codes.
Increased collaboration within NIST as well as nationally and internationally can further advance the technical merit and contributions of these programs. For example ERR’s partner agency in the NEHRP, FEMA, has moved forward with development of next-generation performance-based design procedures that characterize building performance directly in terms of the probable economic, human, and environmental impacts of building damage, while inherently considering the uncertainties associated with such characterization, and ERR is in a unique position, as lead agency, to advance this methodology. Many of the pre-eminent earthquake engineers are at universities; some level of involvement with them would seem beneficial.
Portfolio of Expertise
The programs have competent and qualified scientific and engineering expertise for their missions and program objectives. Members of the programs have been distinguished with awards and distinctions from industry, including the staff member in ERR who was named a Housner Fellow by the
Earthquake Engineering Research Institute (the Housner Fellows Program recognizes young to mid-career professionals with promising capacity for leading efforts to reduce earthquake risk); and the staff member in the CRP who was awarded the American Society of Civil Engineers Earnest E. Howard Award (the Howard Award is made annually to a member of the ASCE who has made a definite contribution to the advancement of structural engineering). The SPUMH group members have received 12 external awards, including a Gears of Government award for development and enhancement of the national wind design maps; Precast/Prestressed Concrete Institute’s (PCI’s) George Nasser award for work in developing robust precast concrete connections (this award is for a paper published by PCI by authors 40 years of age or younger); the American Concrete Institute’s (ACI’s) Arthur J. Boase award for work in assessing the effects of alkali-silica reaction (the award recognizes personal contributions to the concrete industry); and the National Storm Shelter Association’s Kiesling award, which recognizes individuals for outstanding service and contribution to the storm shelter industry through effective initiative and leadership.
Emerging research areas, however, may increase the need to add new or different areas of expertise. For example, some programs may need strengthening in experience in performance of large and complex physical tests. In addition, the programs will need to continually assess the portfolio of expertise with respect to critical program areas as personnel retire or move on to other organizations. NIST EL may access some of these specific areas of expertise through enhanced collaboration with other organizations, and may consider adding in-house staff to ensure core or strategic expertise.
Adequacy of Resources
Many of these programs operate state-of-the-art laboratory facilities that provide unique capabilities to accomplish NIST’s missions and objectives. The National Fire Research Laboratory (NFRL) is strongly complemented by the development of the Burn Observation Bubble (BOB), which is a simple and creative implementation of commercially available gear to provide immersive visualization of severe fire environments that are otherwise unavailable. The unique Net Zero Energy Residential Test Facility (NZERTF) offers tremendous opportunities for a multitude of high-performance building studies, including equipment testing, energy efficiency, intelligent controls, monitoring, and indoor air quality (IAQ).
Recently updated and upgraded testing laboratories, such as the Performance-based Engineering Research for Multi-hazards (PERFORM) structural laboratory enhance the ability to perform testing in-house. Additional facilities (and related infrastructure) may need investment to enable the programs to achieve their key objectives. It remains important, however, that these programs maintain active and ongoing relationships with other major national and international laboratories and researchers to ensure access to special expertise or facilities.
In several areas, EL staff reported current and pending shortages in staff expertise. This represents an opportunity for EL to strategically plan its research and staff succession needs, considering research areas both likely to expand and diminish in importance as well as ways to leverage expertise through collaborations. Given financial constraints, it may not be possible to increase staffing levels within the EL. If growth is not possible, and new expertise is needed, then strategic planning will be needed to determine which programs should have a net reduction in staff. If a large number of staff are near retirement age, this is a good time for NIST to do a strategic plan to determine what the critical research areas should be for the future. It may not be necessary to replace all the expertise that will be lost with the retirements, especially if there will be need to increase the staff in new areas. Additionally, it is important to consider the role of collaborations with other organizations.
The programs have challenges related to their relatively small staff sizes, particularly in light of emerging diverse demands. In particular, the programs that are responsible for administering large-scale field investigations, such as the WUI Data Collection in the FRD and the NEHRP by the ERR, are dually responsible for both core research as well as forensic analyses. However, forensic investigations provide a
particular challenge in planning because the staff skills and level of effort are largely unpredictable. Direct funding of field investigations could put the investigation and research into better balance and would foster complementarity between the efforts.
Effectiveness of Dissemination of Outputs
Products such as software tools and models are available to the public and effectively supported by the NIST team. BACnet, in the EIB program, is a dissemination highlight, with wide adoption in industry and substantial impact. CONTAM, in the NZE program, is one of the most widely used multizone modeling tools for the study of air and contaminant flows in buildings. In addition, NIST has developed four software programs (BEES, BIRDS, BIRDS NEST, and E3) for use in valuation of economic performance (life-cycle cost) and environmental performance (life-cycle assessment).
Broader adoption of NIST outputs could have a significant impact on achieving NIST’s mission and objectives—and help save lives. For example, given the massive installed base of existing buildings, the dissemination of critical outcomes, best practices for building systems—for guidance about preventative maintenance and for other information—is critical to the success of these programs and for the greatest impact.
There is no group or organization in the United States other than NIST that has the authority to independently investigate structural failures. The lessons learned from such investigations would represent important feedback to researchers and designers. It is not clear that the Structures Group is taking full advantage of its position.
CROSSCUTTING CONCLUSIONS AND RECOMMENDATIONS
The report’s chapters present detailed conclusions and recommendations for each program reviewed. This section presents conclusions and recommendations that apply to two or more of the programs reviewed—first, in general, and second, by the factors considered by this panel (i.e., technical merit of the programs; portfolio of expertise; adequacy of budget, facilities, equipment, and human resources; and effectiveness of dissemination of program outputs).
Technical Merit of the Programs
NIST EL has a long and distinguished record of technological leadership, innovation, and experience in providing analysis and solutions to large and small problems. EL research quality is exemplified by the programs’ science-based tools and outputs. The EL programs show an evolution over the past few years, which indicates a positive attitude toward adaptation to evolving needs. However, in response to these evolving needs, outside collaborations should be sought if expertise does not exist within NIST. In addition, insularity may at times be a risk.
RECOMMENDATION: The Engineering Laboratory should establish formal procedures to ensure interaction with practicing professionals and researchers at other institutions, including federal agencies and universities, to ensure that the program does not become overly insular.
The NIST EL programs demonstrate global leadership in measurement science and its application. While successful outcomes from the individual projects advance the mission and vision of the EL, strategic thinking and planning to support long-term efforts is critical and essential.
RECOMMENDATION: The Engineering Laboratory should articulate and plan more activities around a long-term strategic research plan, developed with input from independent outside advisory panels.
There are frequently good communications with technical peers and standards groups. At the same time, EL staff did not present a clear consensus understanding of the EL’s policy on the importance of publications of varying types (e.g., peer-reviewed, technical notes, training documents, and conference proceedings) relative to other critical activities. Since NIST is not an academic institution and needs to balance publications, standards support, reference materials, and research, its productivity cannot be measured by publications alone. However, where publications are encouraged, it should be made clear to the researchers whether they are measured by the number of publications or the number of citations from their publications.
RECOMMENDATION: The Engineering Laboratory should communicate to staff clear goals for the rate of annual publications of various types.
Portfolio of Expertise
The high quality of the EL portfolio of expertise is evident in its impact on national and international codes and standards and its strong industry reputation. As programs evolve and new programs are introduced, additional areas of expertise will be needed to enable significant EL contributions and thought leadership. NIST EL contributions address a full range of issues directed to improved human productivity, safety, and quality of life, and the EL portfolio of expertise could be enriched to explicitly incorporate those competencies. For example, the areas of human behavior and health in residential and community settings do not appear to receive the same attention as other technical aspects of programs.
RECOMMENDATION: The Engineering Laboratory should assess gaps in its expertise and add new competencies as needed, such as social scientists and medical scientists to the technical staff.
Adequacy of Resources
In some areas, the resources of programs seem to be stretched very thin. The prospect of future retirements suggests a challenge for ensuring the longevity of programs.
RECOMMENDATION: The Engineering Laboratory should ensure longevity of programs through succession planning to ensure areas of expertise are not lost within NIST.
EL resources also have challenges related to the laboratory’s relatively limited size and the expanding diverse demands on its program areas. Because of the imbalance between potential demands and available resources, it is difficult for the EL to have a profound effect on the industry.
RECOMMENDATION: The budget, human resources, facilities, and equipment resources required for both continuity and growth of the Engineering Laboratory programs should be reviewed, and adequate resources should be provided to ensure they continue to develop.
A diverse staff often yields insights and perspectives that enhance the quality of research.
RECOMMENDATION: The Engineering Laboratory should assure an appropriate level of diversity of qualified researchers and managers as older members retire and new talent is brought on board.
The EL has long been the industry leader in metrication and standardization materials, equipment, systems, and processes. To maintain this leadership position, the research and testing facilities need to be maintained and upgraded to match the innovation occurring in the industry.
RECOMMENDATION: The Engineering Laboratory should seek adequate funding of facilities maintenance to ensure that NIST can continue its industry leadership position in measurement and standardization.
Effectiveness of Dissemination of Outputs
Greater awareness of the programs’ products outside of NIST and its immediate stakeholders would significantly benefit the recognition of the value of the products and the awareness of the important role NIST can play nationally and globally. Working more directly with stakeholders and end users would provide valuable feedback to NIST’s efforts, which can be used iteratively to improve NIST EL outputs and products.
The lack of a clear Internet communications strategy for NIST programs is a serious one that needs to be addressed. This includes information about the programs, their roles, the impact they have had and intend to have, and opportunities for industry and academia to engage the programs. There is also a need for a strategy for using social media.
In light of the value of reaching out to external communities of researchers, designers, and localities, it is important that the EL continually assess the opportunities for external collaboration. Such assessment would mitigate against duplication of effort, identify and enlist individuals and organizations possessing expertise and/or facilities not resident at NIST, and assist in strategic planning to address in-house staffing concerns.
RECOMMENDATION: The Engineering Laboratory should develop and actively promote a Stakeholder Engagement and Dissemination Strategic Plan that makes use of a broad range of traditional and emergent media to report and interpret results and solicit user input.
RECOMMENDATION: The Engineering Laboratory should consider establishing a primary point of contact for outreach and dissemination to its diverse stakeholders, including companies, professional organizations, communities, regional and state agencies, and universities and community colleges.
RECOMMENDATION: The Engineering Laboratory should consider an enterprise evaluation system that would promote a holistic approach toward product development, implementation, and user feedback and would promote community engagement early and throughout the process of design through dissemination.
RECOMMENDATION: The Engineering Laboratory should consider developing programs and tools that are user-driven, not developer-driven.
RECOMMENDATION: The Engineering Laboratory should develop stronger relationships with diverse portions of industry, such as design firms, utilities, manufacturing firms, facility owners, local governments, and other agencies to help ensure
that its products are responding to the needs and cultures of different types of organizations.
RECOMMENDATION: The Engineering Laboratory should distribute its products directly to actual and potential users, and the outcomes of these dissemination efforts should be evaluated.
RECOMMENDATION: The Engineering Laboratory should increase its development of partnerships with international organizations.
Dissemination of critical outcomes and best practices for building systems—for guidance about preventative maintenance and for other information—is critical to the success of the program and for greater impact. Given the massive installed base of residential buildings, homeowners and other occupants can play a critical role in intelligent building operation.
This is best done in coordination with other federal agencies (e.g., the Department of Energy and the Environmental Protection Agency are active in this area also). Without coordination, the end result would duplicate work done in different ways.
RECOMMENDATION: The Engineering Laboratory should work toward broader interface with homeowners and other end users to collect and review requirements, disseminate information from its programs, influence improved user interfaces, and help end users get the best results in building operations.
