2
Information Access Division
The mission of the Information Access Division (IAD) is to foster trust in technologies that make sense of complex information relating to human action, human behavior, and human characteristics. The IAD includes the following four groups: the Image Group, Multimodal Information Group, Retrieval Group, and Visualization and Usability Group. IAD is an effective organization with prodigious output in terms of reports, publications, standards development, and other outputs.
TECHNICAL QUALITY OF THE WORK
IAD personnel provide leadership in various information technology (IT) fields through a collaborative process that helps define and quantify problems and then build communities of interest that can address them. Some of the efforts lead to new evaluation metrics or data sets and proposed standards. IAD has led in expanding NIST’s approaches beyond traditional metrology to understanding human factors in IT success and failures. Overall, IAD research is of very high quality.
IAD benefits from broad community interactions and pushes into emerging areas involving support of other agencies. Through collaborative processes, IAD has visibly driven the creation or advancement of important technical fields. IAD has facilitated benchmarking, using best known methods for measurement to accurately assess the state of the art in several fields.
Assessment of Individual Projects
NIST Information Technology Laboratory (ITL) voting work, centered on the development of the Voluntary Voting Systems Guidelines (VVSG), including the much-desired 2021 release of VVSG 2.0 for human factors, accessibility, and usability, is an example of a NIST project requiring assembling expertise from multiple computer science divisions and working with multiple outside constituencies, including community outreach via public working groups. The division contributed high-quality human factors technical expertise in improving the accessibility and usability of voting systems with critical work on improving system understandability by poll workers and work on providing equal access to all voters, including voters with disabilities, through the use of universal design methods. While voting system security receives more public focus, these human-computer interaction aspects are equally vital to trouble-free and fair elections.
The Retrieval Group and its longstanding flagship Text Retrieval Conference (TREC) are the best-in-the-world group for evaluating information access from unstructured data. An IAD TREC organizer recently received external recognition as a fellow of the Association for Computing Machinery (ACM) and of the Washington Academy of Sciences, and by election to ACM SIGIR’s (Special Interest Group on Information Retrieval’s) Academy as well as internal recognition by appointment as a NIST fellow in 2021. In the past 2 years, the group has continued to innovate with well-chosen new evaluation foci, such as Health Misinformation and Fair Ranking, and there are plans for a new Trustworthy AI
Conference (TRUC), suggested at a workshop at the National Academies of Sciences, Engineering, and Medicine.
Sister benchmark evaluations to TREC take place under the Text Analysis Conference (TAC) and the video retrieval evaluation tracks of TREC (TRECvid). TAC defines and runs evaluations for advanced natural language processing technologies such as question answering, knowledge base population, and summarization. This activity has played a beneficial role in codifying new NLP tasks and supporting evaluations for the Defense Advanced Research Projects Agency (DARPA) and the National Institutes of Health, although perhaps not achieving the standout recognition of TREC. TRECvid has jumpstarted a large amount of academic research in content-based video retrieval, for which the researchers received an Institute of Electrical and Electronics Engineers (IEEE) PAMI prize in 2018. In 2021, TRECvid supported the following six major tracks: ad hoc video search, activities in extended video, instance search, video to text, video summarization, and disaster scene description and indexing. Disaster scene description is an example of IAD focusing its resources in a way that exploits government data sources, focuses on an area with lagging performance not of commercial interest, and builds important national capabilities.
The Multimodal Information Group has a long history and deep expertise in evaluating spoken language and multimedia technologies, machine translation, and knowledge representation. It has an excellent reputation for statistical rigor in the design and implementation of evaluations, which has led to a large demand from outside agencies. It has a decades-long history of important contributions to evaluation of speech technology, which has helped drive the tremendous advances made in speech and speaker recognition. The work has evolved to include language processing, with considerable impact on machine translation, leading to important efforts on cross-lingual language processing tasks. While many of these efforts are initiated by outside agencies, work continues with open evaluations in several areas.
With the goal of developing a secured reference architecture for big data applications, the Big Data Public Working Group has focused on big data definitions and taxonomies, use cases and requirements, security and privacy, specification of a reference architecture, and a standards roadmap. A comprehensive list of 437 requirements were extracted from 51 use cases, and 35 aggregated general requirements have been divided into 7 categories. Various NIST Big Data Interoperability Framework (NBDIF) documents describing the proposed reference architecture, its interfaces, and the issues associated with adoption and modernization have been published. Five out of seven of the NBDIF documents have become ISO/IEC standards.
The group working on explainable artificial intelligence (AI) has released a white paper and has conducted a NIST workshop with the goal of endowing AI algorithms with the capacity to provide explanations regarding their various inferences and outputs. Principles of explainable AI have been proposed and are in the process of garnering community consensus. Connections between users, policy makers, industry, and academia have been nurtured. This is a prime example of NIST’s capacity to provide thought leadership.
The Image Group continues to provide best-in-the-world technology evaluations and to support the development of a variety of biometric technologies. The group has long hosted multiple highly impactful evaluations in face and iris recognition, with worldwide participation and impact. The face recognition studies benchmark an order of magnitude decrease in error rates over the past 3 years. The group’s prominent study on differential performance of face recognition across age, ethnicity, and gender has been presented approximately 160 times in multiple venues (industry and academic forums, congressional testimony, press coverage), including being featured on the CBS show 60 Minutes. For fingerprints, technologies associated with credentialing, law enforcements, and forensics have been supported via standardization of template representations as well as routines associated with fingerprint image segmentation. The use of standardized minutia records has resulted in a significant reduction in storage and truly effortless swapping between formats. The group also supports technologies associated with iris recognition, contactless fingerprint capture, and tattoo recognition. All of these efforts are supported by a high-performance computing infrastructure at the Biometrics Research Laboratory (BRL).
The group working on biometric forensic science continues to advance this frontier of forensic science capabilities. With the goal of anchoring the latent fingerprint community, the team has established a new latent fingerprint dataset with 200 subjects and are working on minutiae correspondences. The team has performed experiments with human examiners focused on facial imagery captured in uncontrolled circumstances. A critical finding is again evidence of cross-race differential bias. The team has also conducted experiments to support the efficacy of forensic iris recognition without the need for specialized image capture devices. The societal and legal implications of this work are significant.
A collection of public safety communication research projects has focused on the technology needs of the first responder community. A survey with 7,182 respondents clearly identifies the communication needs of this community. Automated stream analysis methods have been vetted, and mechanisms that allow end users to evaluate such technology using their data in a secure manner have been developed. This illustrates IADs drive to enable the development of technology and support its adoption by society.
IAD’s usable security research is prominent within the security research community, and it has the potential for broader impacts beyond experts. For example, the researchers have established the concept of security fatigue. Their Smart Home work led to a recent publication in a top-tier cybersecurity venue. Fifteen government and business organizations expressed interest in evaluating or implementing Phish Scale, a method for rating human phishing detection difficulty; the method has also appeared in academic journals and the technology press. NIST Special Publication 800-63,1 which includes a data-driven usability chapter covering this work, has been downloaded nearly 420,000 times since its update in June 2017. This publication is well known in both usable security research and password compliance communities.
Assessment Across the Division
It does not appear that IAD has specific, defined objectives to measure success against. Rather, IAD has a purpose statement and a collection of individual projects. IAD is under-focusing on strategic and stakeholder communications aspects of technical program management.
RECOMMENDATION: IAD needs to define answers to the following questions: Who are our stakeholders and what do they need? How do we organize projects (individually and collectively) to meet those needs? How do we assess how well we’re meeting those needs? How are we communicating our findings to those entities in understandable and usable ways?
Bias in AI is becoming increasingly recognized as a key issue in trustworthiness and is an opportunity for IAD. Bias is only one aspect of trustworthy AI—the entire set of considerations of trustworthy AI needs to be embraced by NIST, including a quality-controlled process of making standards using data and in context of real-world deployments, with stability and robustness to the human judgment calls as a central concern.
IAD has made commendable recent investments in social science expertise, leading to the very recent publication of a NIST Special Publication on identifying and managing bias in AI,2 and NIST has done definitive work in characterizing demographic differentials in face recognition. However, other examples of differential performance that have been highlighted for voice and language have not yet been addressed; IAD needs to take a more proactive role in providing data that facilitates more such analyses.
___________________
1 National Institute of Standards and Technology (NIST), 2017, Digital Identity Guidelines, SP 800-63-3, June, https://pages.nist.gov/800-63-3.
2 NIST, 2021, A Proposal for Identifying and Managing Bias in Artificial Intelligence, Draft NIST Special Publication 1270, June, https://doi.org/10.6028/NIST.SP.1270-draft.
Moreover, at this stage, IAD seems insufficiently engaged with bias and fairness stakeholders, such as groups who participate in the ACM FaCCT (Fairness, Accountability, and Transparency) community. There is still a need to more extensively complement IAD’s excellent technical metrology work with social science expertise focusing on the societal dangers of potentially biased AI tools. While IAD has turned the corner in valuing qualitative research, there is still more to do here.
Trustworthy AI is an open problem. IAD has done a commendable job in laying out cross-cutting, high-level elements of trustworthiness, and there are additional cross-cutting contributions that it can make in terms of guidelines. However, standards will be highly dependent on the application area, and getting community trust depends on incorporating communities and their input at the design phase. Even if the division grows, the number of people in IAD working on this will be limited, and it will be important to focus efforts, taking into account both stakeholder needs and areas of IAD expertise.
IAD is well known for its work on the design and implementation of open evaluations for technologies grounded in human characteristics and behavior. As individual research groups, companies, and collectives of researchers are increasingly posting open challenges and shared tasks to evaluate related technologies, IAD will need to continue to evolve to maintain its leadership. The Image Group has organized online and ongoing evaluations whereby participants can submit at any time and a leaderboard tracks performance. The Retrieval and Multimodal Information Groups also need to move in this direction, where appropriate, in areas such as old data sets or already well-defined tasks or opportunities making for data sets openly downloadable. The IAD could do more to extend the reach of its metrology expertise to the broader community—by consulting, hosting tutorials, or playing an important role in establishing best practices for setting up informative evaluations.
Human-agent interaction and human-in-the-loop systems represent a growing area of AI, particularly for speech, language, and multimodal technologies. Evaluation in this space is an open problem, and there is an opportunity for IAD to have an impact here by drawing on the expertise in multiple groups. It would be a significant undertaking if staff are already spread thin, so where to focus needs to be carefully considered.
TECHNICAL EXPERTISE OF THE STAFF
IAD staff has done an excellent job in building the division research projects. They have published that research broadly, including at top-tier venues. The face and fingerprint recognition technology evaluation within the biometrics group is one of the best in the world. Research teams bring together social scientists, engineers, and computer scientists, producing stronger research through this combination of disciplines. In the past, the social scientists found it a challenge to explain the importance of qualitative research to their engineer and scientist colleagues. However, now they are receiving requests for that expertise from several initiatives.
Many of the challenges faced by ITL’s mission would benefit from broader application of the social science expertise of IAD. While that expertise is being leveraged in the trustworthy AI and public safety projects, it needs to be recognized and made use of more broadly across other projects. As other projects recognize their need for social science expertise, it seems likely that more demands will be put on those resources.
Given the importance of the research, IAD needs the ability to compete for top talent, who are offered generous salaries for their expertise in industry.
RECOMMENDATION: IAD should explore compensating benefits such as flexibility and work/life balance.
RECOMMENDATION: IAD should consider more external exposure of its ongoing diversity, equity, and inclusion initiatives to benefit hiring.
ADEQUACY OF RESOURCES
IAD has unique facilities that support the breadth of its activities, including the BRL, the Usability Testing Laboratory, the Assessor Laboratory, and the Sequestered Data Testing Laboratory. In particular, the BRL is an excellent resource for the work involving controlled unclassified information. The Usability Testing Laboratory was probably underutilized during the pandemic, but it will be an important resource moving forward, particularly given that usability is important for trust in AI.
IAD researchers provide expertise in multiple disciplines, including mathematics, engineering, computer science, social science, and IT. This is critical for supporting the different work that those researchers do. In areas where they need additional expertise, IAD staff collaborate with staff in other divisions in ITL and across NIST. ITL is actively supporting continuous learning for staff members, including online learning and graduate work.
IAD significantly contributed to research in reports aimed at understanding shortcomings in the areas of diversity, equity, and inclusion (DEI) at NIST. IAD is responding by instituting new procedures for performance assessments, hiring, and leadership training. Since 2016, the percentage of women has increased to roughly a third of their staff, including women in leadership roles. They also have a collaboration with Morgan State University for recruiting Professional Research Experience Program students.
Work on visualization was missing from the presentations and division documentation provided to the panel, which raises the question of a gap in expertise of the Visualization and Usability Group. IAD leadership explained that this was a historical artifact associated with challenges in changing the name. It does not focus on visualization work, but instead relies on the expertise of colleagues in other divisions. Visualization is an effective means to communicate information in an accessible manner to most people. This gap needs to be filled at NIST.
Like many organizations, IAD recognizes that there will be challenges and opportunities in moving back to in-person work after an extended period of remote work during the pandemic. Allowing continued teleworking for some staff members may be attractive to some staff and helpful for recruiting and retention, and it offers the potential for expanding efforts within existing space. However, it will be important to consider ways to include remote workers in opportunities for socialization that are important for team building and generating new ideas.
There are growing needs in the areas of privacy, usable security, bias, and AI, and this work will be especially important in the next few years.
Given its expertise, IAD is already getting more requests than it can take on, and such requests are likely to increase. The groups are spread thin with the many projects they currently have.
RECOMMENDATION: To maintain their high standards, IAD should prioritize projects with all of its stakeholders in mind.
Expanding the projects that it takes on will require expanding the facilities and increased human resources.
There is increasing recognition that it is important to assess demographic differential performance. While differences in face recognition performance have attracted a lot of media attention, similar differences have been observed for speech recognition. Assessing such performance differences can involve tracking sensitive information. Further, as face and speaker recognition technology has improved, much audio and video data has become personally identifiable information and thus potentially sensitive.
RECOMMENDATION: In order to continue its important work on audio and video data, IAD should expand the facilities for handling controlled unclassified information and should consider streamlining the privacy process for obtaining and providing access to sensitive data.
In the vision and language processing communities, there is a trend of having leaderboards for assessing performance, since technology is evolving so quickly. IAD could beneficially consider whether it wants to provide more support for ongoing assessment, and if so, it is important to be forward thinking about computing resources.
All of ITL will be facing challenges in hiring, given the high demand for IT expertise in industry now and the fact that government salaries are not competitive. This will be a particular challenge for IAD if it is to expand its staff to meet the growing demands for its expertise in privacy, security, and trustworthy AI. It will be important to identify strategic advantages that IAD might have. Offering flexibility of telecommuting is important, but many companies are recognizing this. The support for continuing education, the potential for societal impact, and the development of an inclusive community are possible selling points. Building relationships with people by supporting interns, postdoctoral researchers, and visiting scholars can also support the recruiting process.
IAD has acknowledged that there are DEI issues to address. It has taken the first steps in doing so, but it would be helpful to develop an explicit strategy for recruiting and retention with measurable objectives.
The large percentage of other agency funding is an opportunity and a challenge, since it is risky to commit to permanent employees.
The BRL may need to be expanded to handle growing privacy concerns related to speech and video.
Providing reference data is an important aspect of standards for performance measurement. IAD plays a vital role in providing biometric data sets and in reporting demographic differences in system performance. Efforts in speech and multimedia standards and reference material could be expanded.
EFFECTIVENESS OF DISSEMINATION OF OUTPUTS
IAD’s technical products include the depth and specificity required to drive capability advancement, and there are numerous examples that demonstrate how its work positively impacts the nation.
IAD has identified “researchers who are developing technology” as its primary stakeholder, and their research and dissemination of findings are clearly driven by and support this group. Their work appears to be in the appropriate intersection of investigations that this community requires, and which is appropriate for NIST to perform. The range of outputs provided, such as published papers, data sets, technical briefings, tools, and guidelines, vary properly to meet the needs of individual technology areas so that IAD is positively impacting their advancements to support government and non-government needs. No substantive changes are necessary in these commendable efforts.
It is apparent that IAD staff inherently monitor how researchers use their outputs (as evidenced by the selective use of a range of outputs and how these evolve over time), but it does not appear that IAD has taken on such assessments formally. Doing so would provide insights to further improve the scoping of their activities and how results are presented, as well as provide management benchmarks on which to measure progress and IAD’s influence on this stakeholder’s priorities.
IAD identified Congress and policy makers as its secondary stakeholder and recipients, users, and consumers of technology as its third. For the most part, IAD’s outputs are not driven by, nor are they effective for, these stakeholders. Core visualization and usability work is driven by stakeholders and users.
Indeed, these entities are usually required to analyze IAD’s technical outputs themselves and determine their relevance and meaning, often resulting in misinterpretations, because these very technical outputs are not understandable by a variety of non-expert audiences.
RECOMMENDATION: IAD should take action to better support nonexpert stakeholders, including policy makers and the general public.
The gap in priority between IAD’s primary and other stakeholders is not as wide as its current focus suggests, and it will further narrow in the future. Actions taken to address this important gap ought not diminish the technical depth and specificity of IAD’s reports, which need to continue for the division to support capability advancement. Appendices within technical reports, or adjunct publications, devoted to meet the specific needs of each stakeholder group (policy makers in Congress or the White House and those that attempt to influence them, the public and press, technology-specific communities) need to be considered instead.
In all cases, measuring effectiveness and impact needs to be modernized and assessed against each stakeholder group. The metrics currently used are artifacts (numbers of papers published, awards received, or briefings provided) or are indicators of a community’s advancement (performance improvements over time, more participants in technology evaluations, adoption of the technology) rather than IAD’s specific influence. Developing and using metrics that assess IAD’s impacts for each of its stakeholder constituencies will drive needed cultural changes, help overcome the division’s strategic planning gaps, and provide data-driven evidence for use in promoting IAD to its stakeholders.
GENERAL CONCLUSIONS
The work and staff across IAD can generally be characterized as excellent. IAD researchers address ITL objectives. The subject matter that IAD is charged with is of great importance to the nation, and IAD provides high-quality research and service to the nation. This effort will be enhanced if IAD can integrate the individual projects with a common thread and recruit the proper staff who can address IAD’s evolving objectives. IAD has a purpose statement and a collection of outstanding individual projects. However, IAD is not adequately focusing on strategic and stakeholder communications and needs to identify stakeholder needs, organize projects to address those needs, communicate findings to stakeholders in understandable and usable ways, and assess how well it meets stakeholder needs.
The fact that there is no integrated objective does not give the whole IAD the credit that it otherwise deserves. For example, bias in AI is becoming increasingly recognized as a key issue in trustworthiness and is an opportunity for IAD. IAD has made commendable recent investments in social science expertise, leading to the publication of a NIST Special Publication on identifying and managing bias in AI, and IAD has done definitive work in characterizing demographic differentials in face recognition. However, other examples of differential performance that have been highlighted for voice, language have not yet been addressed, and IAD needs to take a more proactive role in providing data that facilitates more such analyses. IAD seems insufficiently engaged with bias and fairness stakeholders, such as groups who participate in the ACM FaCCT community. There is still a need to more extensively complement IAD’s excellent technical metrology work with social science expertise focusing on the societal dangers of potentially biased AI tools. While IAD has turned the corner in valuing qualitative research, there is still more to do.
Trustworthy AI is an open problem. IAD has done a commendable job in laying out cross-cutting, high-level elements of trustworthiness, and there are additional cross-cutting contributions that IAD can make in terms of guidelines. However, standards will be highly dependent on the application area, and getting community trust crucially depends on incorporating communities and their input at the design phase. Even if the division grows, the number of people in IAD working on this will be limited, and it will be important to focus efforts, taking into account both stakeholder needs and areas of IAD expertise.
IAD is well known for its work on the design and implementation of open evaluations for technologies grounded in human characteristics and behavior. As individual research groups, companies, and collectives of researchers are increasingly posting open challenges and shared tasks to evaluate related technologies, IAD will need to continue to evolve to maintain its leadership. The Image Group has organized online and ongoing evaluations, whereby participants can submit at any time and a leaderboard tracks performance. The Retrieval Group and Multimodal Information Group could also move in this
direction, where this is appropriate. There may be more opportunities for data sets to be openly downloadable, such as old data sets or already well-defined tasks. IAD groups could do more to extend the reach of their metrology expertise to the broader community, such as by consulting, hosting tutorials, or playing an important role in establishing best practices for setting up informative evaluations. Human-agent interaction and human-in-the-loop systems represent a growing area of AI, particularly for speech, language, and multimodal technologies. Evaluation in this space is an open problem, and there is an opportunity for IAD to have an impact here, drawing on the expertise in multiple groups. It would be a significant undertaking when staff are already spread thin, so where to focus needs to be carefully considered.
