3
Software and Systems Division
The main purpose of the Software and Systems Division (SSD) is to inspire and cultivate trust and confidence in software, systems, and their measurements. The division comprises the following four groups: Software Quality Group, Information Systems Group, Systems Interoperability Group, and Cyberinfrastructure Group. Application domains within the division include digital forensics, health care, imaging, biosciences, voting, smart grid, Internet of Things (IoT), cloud computing, materials genome initiative, and scalable computing.
TECHNICAL QUALITY OF THE WORK
The work of SSD demonstrates multiple areas of unique resources and competencies. These include digital forensics, voting, and characterization of imaging processes covering multiple modalities. In addition, there is promising, newer work in several additional areas, including category theory—foundations of systems semantics, SSD Bugs Framework; software metrology; and artificial intelligence (AI) in imaging and natural language processing across application domains that include health, science, and engineering.
Assessment of Individual Projects
Digital forensics comprises retrieving, storing, and analyzing electronic data from computers, hard drives, mobile phones, and other storage devices that can be useful in criminal investigations. This is a unique niche with multiple notable accomplishments. The National Software Reference Library won the SSD Judson French Award for work combating child sexual exploitation. The library enables digital forensics examiners to eliminate files of non-interest during analysis, enabling them to focus on user artifacts; it is much easier to find a needle in a haystack if that haystack can typically be reduced by 90 percent. The library serves as a unique source for the digital evidence community.
The Federated Testing Project aids laboratories with a method to test tools (e.g., forensic string search, mobile forensic data extraction) and facilitates the sharing of tool test reports that follow the SSD test protocol with the forensics community. This is a highly effective resource for increasing the quality assurance in digital laboratories that seek to produce quality results but lack the resources to support a dedicated, internal testing team.
Computer Forensic Reference Data Sets (CFReDS) is a highly effective tool for examiners and laboratories. CFReDS posts extractions from computers and related devices that can be used in a myriad of ways, including competency testing, proficiency testing, and training.
There is not a minimum performance threshold for tools tested by the Computer Forensics Tool Testing Program. This presents an opportunity for the digital forensics program to collaborate with digital forensics community organizations to set a threshold for performance rather than simply enumerating issues. Potential collaborators include the Scientific Working Group on Digital Evidence and the Digital Evidence Subcommittee and the Organization of Scientific Area Committees for Forensic Science.
The work of the Software Quality Group spans a broad range of well-chosen topics, and evinces impressive results. Especially noteworthy is the work on preparing a response to Executive Order on Improving the Nation’s Cybersecurity,1 which was issued May 12, 2021. The draft response can have a significant impact on software development practice. SSD itself is mentioned no less than 21 times in the executive order, so it clearly has a major role to play.
Progress of the Software Quality Group in some areas seems to have stalled. An example of this is the work on the static analysis tool exposition (SATE) initiative, which is a non-competitive study of static analysis tool effectiveness aimed at improving tools and increasing public awareness and adoption. The initiative provides sample test suites to vendors of static code analysis tools, who then apply their tools to the test suites and report results. The results are then discussed at workshops. Between 2008 and 2014, five such workshops were held, but since then just one more event was held in 2019. With the growing importance of static code analysis as part of a routine software development practice, resuming the pace of regular SATE tool exposition events guided by SSD can be beneficial. The tool assessment could also benefit from a more quantitative approach, using measurable benchmarks for static analysis tools.
Similarly, the last update to the Software Assurance Reference Dataset (SARD) seems to have occurred in 2017.2 Several cases were added in 2020. SARD provides users, researchers, and software security assurance tool developers with a set of known security flaws. Comparable data sets maintained—for example, for CVE (enumerating known cybersecurity vulnerabilities)3—have seen significant growth in the past few years, so it is surprising that the SARD data set has not experienced similar growth. The creation of this data set is itself an important initiative that deserves praise. The SARD data set itself, if kept up to date, would also be an important source of input for the tool evaluation workshops (SATE). The data set could help in the creation of a reference set of code that can be used as a target for an objective, quantitative assessment of the performance of software analysis tools, challenging tool vendors to gradually improve over time.
Work on the development of a framework for classifying software bugs (Bugs Framework) that started in 2016 can similarly have significant impact. It could provide new insights if applied to large reference sets, such as CVE or SARD, to quantify the frequency of different categories of bugs. This would be a Herculean task without substantial community support, given the large size of the data sets and the relatively small size of the internal SSD team. This could be a valuable opportunity for public outreach and stronger external collaborations.
Assessment Across the Division
Given that inspiring trust in software tools and measurements is among its key objectives, SSD could take a leading role in the development of a neutral platform for tool evaluation (e.g., SATE) and in the articulation of a vendor-neutral standard output format for static analysis tools in general. An emerging standard that is currently being developed, called SARIF (static analysis results interchange format),4 is supported by a subset of the commercial tool vendors but does not seem so far to have benefited from the leadership or participation of NIST. There is a clear role for NIST to play here.
Described in 1985 as the integration of people, processes, and technology with connectable devices and sensors to enable remote monitoring, status, manipulation, and evaluation of trends of such
___________________
1 Executive Office of the President, 2021, “Improving the Nation’s Cybersecurity,” E.O. 14028, May 12.
2 NIST Software Assurance Reference Dataset Project website, https://samate.nist.gov/SARD/index.php, accessed July 12, 2021.
3 CVE, “CVE List Home,” https://cve.mitre.org/cve/, accessed July 12, 2021.
4 Static Analysis Results Interchange Format (SARIF) website, https://sarifweb.azurewebsites.net/, accessed July 12, 2021.
devices,5 the IoT continues to evolve through the integration of advanced control systems, real-time analytics, machine learning, commodity sensors, high-speed wireless communications, and embedded systems with high computational capabilities. Real-world applications of IoT include smart homes, wearables, smart/connected cars, industrial controls/automation, smart cities, smart retail, health care, energy management, and agriculture—each of which require some unique approaches to technology integration to support effective and efficient operations.6
Many of the tasks performed by an IoT network require devices and systems to have a synchronized method of time for their correct operation. A main reason for a highly accurate synchronized method of time is to provide highly accurate data collection that can be correlated from different sensors and actuators. This highly accurate time synchronization will support reliable and accurate analysis of events, accurate real-time actuation of controls across highly distributed embedded systems, and efficient data communications across a complex networked system. Given the diversity of IoT applications, there are many trade-offs to consider when developing a time-synchronization solution. There is no single time synchronization methodology that satisfies all IoT application requirements, making the development of technologies, standards, and testing methodologies challenging.
The scientific, engineering, and commercial development communities agree with SSD in concluding that integrating state-of-the-art time-transfer methods into modern cyber-physical infrastructure is needed for data and system synchronization. SSD is correct in concluding that timing infrastructures need to continuously evolve to transfer a common timescale across all nodes at ever increasing accuracy as demanded by many types of complex systems, including power systems, robotics, advanced manufacturing, and quantum networks. Even though timing technology is evolving quickly, there is still a lag in the research, development, integration, and standardization of new technologies to support synchronized time references for advanced IoT deployments. Some emerging IoT environments required picosecond timing stability for accurate operation. Manufacturing, energy, transportation, health care, and advanced research environments are pushing the limits of existing time synchronization methods.
SSD is working with several organizations (e.g., the Institute of Electrical and Electronics Engineers [IEEE], Electric Power Research Institute, North American SynchroPhasor Initiative, American Society for Testing Materials, and Department of Homeland Security) on time synchronization methods, standards, and related testing in support of high-resolution calibration techniques, data collection, data analysis, and device control in real-time environments. The key for these advanced real-time environments is the need for highly accurate correlation of events. SSD is contributing knowledge to many standards development activities, including the IEEE 1588 Precision Timing Protocol. SSD is also engaging with stakeholders to design and establish accurate infrastructure to provide measurement time references capable of under 200ps timing stability.
SSD is continuing to make significant contributions to IoT technologies and standards. However, demand for more advanced time synchronization methods is outstripping supply. The need for sub-picosecond time synchronization is nearing, and effective methodologies must be developed. SSD has the opportunity to provide the necessary leadership in the development of these methodologies and their resulting standards. The main challenges are allocation of the resources, identifying partnerships, and establishing the focus needed to address the needs of the diverse IoT applications.
SSD laboratories have multiple separate IoT facilities. To be able to handle the expected growth in IoT systems requires an increase in the available SSD IoT infrastructure, including a common IoT research testbed with advanced capabilities that includes a core setup plus contributing IoT systems from
___________________
5 “History of IoT,” Applications of Internet of Things (IoT) in the Market, https://sites.google.com/site/whatisiot2017v3/history, accessed July 12, 2021.
6 Analytics Vidhya, 2016, “10 Real World Internet of Things (IoT) Explained in Videos,” August 26, https://www.analyticsvidhya.com/blog/2016/08/10-youtube-videos-explaining-the-real-world-applications-ofinternet-of-things-iot.
each laboratory. This would allow SSD groups to leverage the sum total of the resources available to tackle bigger projects.
RECOMMENDATION: SSD should establish a common Internet of Things research testbed with advanced capabilities (e.g., commercial-off-the-shelf devices, high-speed network, and state-of-the-art test/measurement equipment).
Without standardization, performance and integration of IoT systems will be limited in performance, have suboptimal functionality, and be filled with vulnerabilities.
RECOMMENDATION: SSD should drive more standardization across the emerging Internet of Things applications.
RECOMMENDATION: SSD should collaborate with other laboratories and research teams to develop time synchronization methods to support near sub-picosecond performance.
The Systems Interoperability Group develops advanced testing infrastructures and contributes to standards development for ensuring the robustness and interoperability of health information technology (IT) systems, thus removing technical obstacles to implementation and interoperability and accelerating the adoption of cost-effective health IT. The quality of its portfolio is variable. Some projects are very innovative, but a few are unlikely to bring the return on investment consistent with the Information Access Division’s (IAD’s) expectations for increasing value and making a significant impact on the nation. Some projects focus on theory relating to important dimensions of managing the computing and communications environment. Another portion of the portfolio is more practically focused. One part of the interoperability program, which helped the nation deal with COVID-19, is commendable, influential, and meaningful despite having limited longer-range value once the nation has established the technical infrastructure to monitor and manage public health emergencies, including pandemics.
NIST provides the necessary conformance tests, test tools and techniques to advance health care IT standards that are complete and testable. The focus on the automatic generation of artifacts for conformance testing is appropriate and valuable. Establishing and validating approaches that reduce the requirement for human intervention is critical for adopting standards at scale.
However, that portion of the portfolio focused on health, and health care does not seem to have evolved since the National Academies of Sciences, Engineering, and Medicine review of 20157 despite adding a nurse informatician to the workforce and supporting a physician through the University of Maryland Professional Research Experience Program (PREP) program. While adding this expertise to the team was a good step, the division still has not undertaken a comprehensive review of all health systems–related work to take advantage of the new perspectives, knowledge, and skills they brought.
Supplying a substantial voluntary workforce for Health Level (HL) 7 activities cuts into the time and talent available for work offering a more significant impact. HL 7 undertakes many initiatives, and some are likely to have a transformative effect on the health care system, while others are less likely to be influential. The current health care activities seem unlikely to contribute to the strength and visibility of SSD.
Substantial opportunities exist for SSD to impact health care. Three examples illustrate how SSD could refocus, extend, and integrate its health system–oriented priorities and activities to more relevant emerging issues. Given the visibility and importance of SSD, its programmatic work could make a profound and national impact.
___________________
7 National Academies of Sciences, Engineering, and Medicine, 2015, Review of Three Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2015, Washington, DC: The National Academies Press.
The first example relates to the health impact of global warming. There are sure to be significant challenges relating to needed measures and standards as this existential issue is addressed. Recently, the combined National Academies have given climate change a very high priority, and the likelihood of a significant federal investment in infrastructure could supply substantial multi-year funding.
A second example relates to the impact that the Internet has had on the availability of scientific and non-scientific information becoming a ubiquitous resource for citizens as well as health care professionals. Citizen science is motivating society to focus on social determinants (i.e., upstream factors) of health rather than simply addressing medical care issues. For example, SSD could creatively support metrics and software to allow citizens to partner with universities and scientists to apply valid and timely scientific information to create innovative technology-based solutions for addressing social determinants of health. This approach would also fit with the IoT work in SSD.
A third example comes directly from health care delivery. What was already a crisis was made more acute due to the demands and pressures presented by the COVID-19 pandemic. Over the past few years, the National Academy of Medicine, Center for Medicare and Medicaid Services (CMS), Office of the National Coordinator for Health Information Technology, the American Medical Association, the American Nurses Association, and the American Medical Informatics Association have each focused on factors that influence clinician burnout, including clinical documentation, which is three times more demanding of time in the United States than in other developed health care systems. Part of this relates to demands for administrative chores rather than care-related work. Similarly, an effort to deal with care authorization, billing, and payment data could be easier to automate effectively and efficiently. Key enabling domains—such as privacy preserving activity recognition and natural language understanding, clinical activity categorization, and approaches to the management of unstructured clinical data—could be advanced by SSD. This example also serves to illustrate how SSD might use a central theme on which to center its systems interoperability efforts to magnify the effects of work being undertaken and create synergies between otherwise disparate efforts. Consistent with SSD’s identification of support for smart health care as a target for expansion of SSD competence and scope among issues within the National Security Commission on AI report,8 such an effort could tie together work in privacy, IoT, AI methods, and more with systems interoperability.
During the past decade, CMS has supported innovation by establishing the CMS Innovation Center, which oversees a portfolio testing various payment and service delivery models designed to achieve better care for patients, smarter spending, and healthier communities.9 Consideration might be given to the value of having such an explicitly designed and operated unit within SSD to achieve the goals of influencing health care system quality and cost in a way that is not occurring or likely to occur given the current activities. Partnerships between SSD and CMS might approach issues like those highlighted above.
Our society today is undergoing a massive technological shift toward automation at scale. AI and machine learning (ML) have been widely viewed as among the most transformative technologies that will revolutionize the way we live. Given the potential impact on the economy and national security, these technologies must be developed and used in a trustworthy and responsible way. Characteristics to support trustworthiness include accuracy, explainability and interpretability, reliability, privacy, robustness, safety, security (resilience), and mitigation of harmful bias. Principles such as transparency, fairness, and accountability need to be considered during deployment and use.
SSD has had a relatively light investment in AI historically, but SSD leadership has identified AI to be an important area to focus on for both opportunities and challenges it presents. In a relatively short period of time, SSD has made some important accomplishments in AI, at least from the perspectives of leading initiatives and setting standards and requirements for AI systems. SSD plays an important role in
___________________
8 National Security Commission on Artificial Intelligence, 2021, Final Report, https://www.nscai.gov/wpcontent/uploads/2021/03/Full-Report-Digital-1.pdf.
9 Centers for Medicare and Medicaid Services, “Where Innovation Is Happening,” https://innovation.cms.gov/, accessed July 12, 2021.
co-chairing the National Science and Technology Council Machine Learning and AI Subcommittee and participating at policy discussions with its senior technical member on detail to the National Security Commission on AI. These are important services to the United States and can help in creating requirements that are flexible enough to be meaningful for applications, while sufficiently concrete to have meaningful impacts. SSD has also participated in congressional testimonies and hearings.
In collaboration with other federal agencies and academic and industry communities, SSD has led the way to develop guidance for assurance, governance, and practice improvements, as well as techniques for enhancing communication among different stakeholder groups on bias in AI. Bias is not unique to AI. The goal is to identify, understand, measure, manage, and reduce bias in AI systems. Standards and guides are needed for terminology, measurement, and evaluation of bias. SSD can play an important role in this aspect and has already made initial and outstanding efforts to develop trustworthy AI. Trustworthiness and interpretability of machine learning and artificial intelligence is also a topic worthy of attention.
For AI applications, the choices of natural language processing (NLP), especially text processing and imaging, are timely, and the quality of work is excellent with immediate and broad social implications.
Given that AI and ML have been widely regarded as some of the most key technologies to invest in across different fields, one of the biggest challenges faced by SSD will be in recruiting a more diverse workforce that is well trained across multiple disciplines for foundational research and use-inspired development. Although the NIST PREP with Morgan State University, designated a historically Black college and university, is a good idea, it may not be adequate to address this issue. Partnering with other nearby academic institutions, along with scholarships or fellowships for the underrepresented groups that are coupled with internship opportunities at SSD, could be considered to broaden the pool of potential candidates. SSD and IAD could collaborate in this area to further expand the potential impact they can make together on foundational research in AI and ML, in terms of setting the guidelines and standards for data sets, reliability measurements, AI trustworthiness, system requirements, and mass communication to the broader public.
RECOMMENDATION: SSD should partner with academic institutions and collaborate with other communities on recruiting future workforce, expanding impact, and setting guidelines on artificial intelligence/machine learning system requirements and data sets.
The Imaging Group has a long tradition of leading the field in the space of novel instruments and calibration protocols covering optical, electron, neutron, and magnetic resonance modalities. Although small, the group is very visible and succeeds in producing highly relevant work by focusing on its core expertise.
In image analytics, the focus of the group is on trusted and reproducible measurements over terabyte-sized images. Image sources can cover a wide range of physical scales—nano to centimeter—corresponding to different underlying physical or biological processes. As a result, it is often difficult to establish measurement accuracy, uncertainty, reproducibility, and interoperability, making the work of the SSD imaging group critical to many in industry and academia. With the emergence of AI-based measurement models, the complexity is increased, and the validation of models becomes a critical component of any imaging system.
SSD’s work has inspired, for example, the QUAREP-LiMi initiative to establish guidelines for quality assessment and reproducibility for instruments and images in light microscopy.10 A very relevant work quantifies the variability in microscopy image analysis for COVID-19 drug discovery.11
The TrojAI program is an example of the high-quality-imaging AI work pursued in the division. It is focused on methods of detecting hidden behavior in AI models, prior to widespread deployment. The topic is very relevant for the current status of AI, and the TrojAI team shows a mastery of the work, including partnerships, impact, and dissemination of results.12
The larger the image data, the more important is the role of the underlying computing infrastructure. The AI-based processing, training, and validation of measurement methods on large image collections require specialized computing infrastructure covering both the parallel computational power (graphics processing units [GPUs]) and the ability to store and transfer data with a very high bandwidth (flash storage). It seems that the scientific productivity of the group would be greatly supported by an advanced computing infrastructure, which is ideally specialized for imaging and related data.
The group has the opportunity to broaden and grow the testing and validation of emerging AI-based multimodal image measurements, an activity that is very much needed by industry. Related to this, the group has the opportunity to drive and stimulate the formation of emerging standards in AI for imaging, a topic that is well aligned with SSD’s mission.
The high-performance computing (HPC) team has a track record of delivering high-impact capabilities in image processing using HPC and GPU-based systems, reducing the time to process images by up to 4 orders of magnitude. This has a substantial impact on the SSD science mission through the team’s close collaboration with internal SSD stakeholders in materials science and biology that use its software. The team’s current work on the development of an abstraction layer aimed at improving the productivity of HPC software developers for this domain is well-thought out, with the data flow design informed by members’ experience in image processing. The team’s approach is correctly scoped to its resources, since it is leveraging HPC numerical kernels from the vendors and focusing its development effort on the data pipeline, which has been a key component of past success. It is also collaborating with the very high-quality and well-established University of Utah HPC research group to extend the abstraction layer, using the latter’s Uintah framework to provide support for distributed computing based on the message passing interface standard.
One facilities issue for the HPC team. Exacerbated by the current global supply shortage of GPU hardware, is the difficulty in obtaining early access to the latest GPU hardware. This lack of access is a significant handicap, causing delays in deploying the software to users when new systems become more widely available.
RECOMMENDATION: SSD management should consider obtaining early access to systems through the large supercomputer centers housed at the Department of Energy, the Department of Defense, and the National Science Foundation, which generally have better access to such systems.
___________________
10 U. Boehm, G. Nelson, C.M. Brown, S. Bagley, P. Bajcsy, J. Bischof, A. Dauphin, et al., 2021, QUAREP-LiMi: A community endeavor to advance quality assessment and reproducibility in light microscopy, Nature Methods, https://doi.org/10.1038/s41592-021-01162-y.
11 M. Simon, S. Yu, J. Nagarajan, P. Bajcsy, N.J. Schaub, M. Ouladi, S. Prativadi, N. Hotaling, 2021, “Quantifying Variability in Microscopy Image Analyses for COVID-19 Drug Discovery,” in pp. 3801-3809 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 2021, https://openaccess.thecvf.com.
12 P. Bajcsy, N.J. Schaub, and M. Majurski, 2021, Designing Trojan detectors in neural networks using interactive simulations, Applied Sciences 11(4): 1865.
TECHNICAL EXPERTISE OF THE STAFF
Two of SSD’s strategic goals are related to technical expertise of the staff—(1) expanding competence and scope to new domains consistent with SSD’s and the Information Technology Laboratory’s strategic investments and (2) leveraging shared competencies with other SSD units and external organizations.
SSD highlighted the potential to become a leader in addressing the research and development issues identified in the 2021 National Security Commission on AI report.13 There is a high level of competency across programs. Nimbleness/agility for new approaches was identified by SSD as a core competency for staff. SSD targeted six issues within the National Security Commission on AI report as plausible areas for expansion of SSD competence and scope—big data, massive information, and large knowledge bases; innovative approaches to software quality; support for smart health care; engineering biology; mobile computing; and cyber-physical social systems.
SSD is a valued venue for guest researchers, Ph.D. students, and postdoctoral trainees. Recruitment of such personnel is facilitated by ongoing relationships with multiple universities in the United States and internationally. Successful recruitment of several recent postdoctoral fellows into permanent positions suggests that the mentoring efforts of the permanent staff benefit the broader community and also facilitate a pipeline of talented researchers for hire at SSD.
Competition with industry for talent is a challenge for maintaining and expanding the technical expertise of the staff. There is a need for general data science competencies across all areas. The program developed by the National Library of Medicine to expand the data science competencies of its staff may provide a model and resources for SSD staff wishing to expand their data science competencies.
ADEQUACY OF RESOURCES
A third SSD Strategic Goal is to ensure that the division has adequate human and computing resources. SSD has 45 full-time staff, including two National Research Council postdoctoral fellows, and 57 associates; the latter includes students, guest researchers, contractors, and others. In response to a perceived insufficiency of clinical expertise in the Systems Interoperability Group, SSD has hired a clinical informatician with a nursing background and a physician. SSD identified the need for full-time staff and funding in the areas of advanced information modeling, including semantics; software testing, system verification, and formal methods; AI; computational science; and biomedical informatics.
In recognition of the growing demands in the division, SSD has contributed to a proposal that delineates the short- and long-term plans to increase computational resources and infrastructure across SSD.
The diversity of projects in the SSD portfolio strains existing human and computational resources. In terms of human resources, recruitment of highly qualified full-time staff has been identified as an overall challenge at SSD due to competition with industry. Within SSD, it may be difficult for small teams to be effective due to lack of critical mass in some areas. The relatively large proportion (about one third) of the permanent workforce at retirement age raises the potential need for succession planning to ensure maintenance of competence in core areas while meeting the demand for new areas of competency.
Current and future SSD needs cannot be met without the addition of new competencies to complement existing strengths.
RECOMMENDATION: SSD should establish exchange programs with relevant government laboratories (e.g., Army Research Laboratory), academic institutions, and
___________________
13 National Security Commission on Artificial Intelligence, 2021, Final Report, https://www.nscai.gov/wpcontent/uploads/2021/03/Full-Report-Digital-1.pdf.
industry consortia to stimulate new ideas and problem areas, enhance competencies, and facilitate collaboration.
RECOMMENDATION: SSD should plan and implement effective ways to recruit and retain a diverse workforce to ensure the appropriate staffing in areas of significant interest to national welfare and security, and to address severe competition from industry in areas such as artificial intelligence, cybersecurity, and the Internet of Things.
Existing projects in several areas are outgrowing current computational resources. For example, for IoT, expressed needs include a common IoT research testbed, a collection of commercial off-the-shelf devices that can be shared and reused across the laboratories, a common set of hardware and software applications that can be used by interested parties at SSD, a high-speed network, and a pool of available general computing resources.
There are opportunities for innovative approaches, such as public-private partnerships and sharing computational resources across national laboratories or other agencies with supercomputers, to complement the purchase of relevant high-performance resources.
RECOMMENDATION: SSD should establish a multi-faceted, multi-phased approach to enhance computational resources and infrastructure (e.g., programmable logic for optimization and acceleration) through public-private partnerships, sharing with other national laboratories, and direct purchase.
Partnering with government agencies such as the Department of Energy (DOE), Department of Defense (DoD), and the National Science Foundation would ensure early access for developers to every high-performance computing platform to be ahead of the industry curve. It would open the door to a larger opportunity for both traditional HPC platforms, but also for collaboration on emerging technologies in computing that are showing promise in HPC settings, and also in other computing areas such as AI, ML, and related data science technologies such as encryption and large-scale storage and retrieval. DoD and DOE already collaborate successfully on both the high-precision supercomputing and the lower-precision ML/AI computational technologies.
EFFECTIVENESS OF DISSEMINATION OF OUTPUTS
The volume of outputs is impressive across the groups and projects and includes scholarly products, code, software, and participation or leadership in standards development organizations. Dissemination of outputs includes press releases, technical reports, books, journal publications, conference presentations, events such as Connectathons, and repositories such as GitHub for code and software.
For scholarly products, code, and software, the effectiveness of dissemination is evaluated through measures such as frequency of downloads. Effectiveness of leadership and participation in standard development organizations such as HL 7, IEEE, and International Standards Organization is evaluated by volume of leadership, participation, and standards to which SSD has contributed. High
potential impact of Executive Order on Improving Nation’s Cybersecurity14 gave SSD several tasks, including developing minimum standards for software testing to support federal procurements.
SSD outputs are clearly delineated, but it is more difficult to ascertain effectiveness and impact for multiple reasons, including missing metrics or lack of clarity in metrics. In the area of leadership and participation in standards development organizations, it is difficult to ascertain the cost-versus-benefit ratio of such work related to other job responsibilities, although SSD participation is clearly valued due to its perceived neutral stance.
Matching the needs of the market and users to product development is an essential foundation for effectiveness and impact. A tool or other product may be considered high quality from a technical perspective but not meet specific market or user needs. For example, in terms of system interoperability in health care, FHIR (Fast Healthcare Interoperability Resources), a draft standard at HL 7, is driven by policy and market factors that promote its use, while some SSD tools remain focused on the technical aspects if conformance testing in the HL 7 version 2 context.
The strategies for dissemination of SSD outputs appear to be limited to primarily scientific and technical audiences. To increase the effectiveness and impact of SSD efforts, a broader variety of tailored communication strategies is needed to reach other target audiences such as policy makers.
The effectiveness and impact of SSD’s substantial outputs is suboptimal.
RECOMMENDATION: Building on its current strengths in engagement with target communities, SSD should establish strategies to enhance its understanding of market and user needs to ensure that products are not only of high technical quality, but also useful to relevant stakeholders.
RECOMMENDATION: SSD should consider using the National Security Commission on AI target issue of support for smart health care as an organizing framework to increase the impact of their health-related work. This could be facilitated by establishing a center of innovation.
RECOMMENDATION: SSD should implement a broader variety of tailored communication strategies for dissemination of outputs to reach important target audiences (e.g., policy makers) beyond scientific and technical audiences.
GENERAL CONCLUSIONS
Consistent with its purpose of inspiring and cultivating trust and confidence in software, systems, and their measurements, the overall technical quality of SSD’s unique and promising new programs is excellent, given human and computational resources. There is a high-level of technical competency among staff across SSD programs. There are four areas requiring additional focus. First, SSD’s capacity to handle the expected growth in IoT systems is insufficient. Second, SSD performance is increasingly limited in several areas by lack of adequate computational resources and infrastructure. Third, current and future SSD needs cannot be met without the addition of new competencies to complement existing strengths. Fourth, the effectiveness and impact of SSD’s substantial outputs is suboptimal.
___________________
14 Executive Office of the President, 2021, “Improving the Nation’s Cybersecurity,” E.O. 14028, May 12.
