Review of Three Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2015 (2015)

2

Information Access Division

INTRODUCTION

The activities of the Information Access Division (IAD), which has about 90 personnel, are organized along four themes: multimedia information, information retrieval, image processing, and visualization and usability. The research teams enjoy broad peer recognition while supporting the NIST’s vision of information technology (IT) in measurement science overall and in measurement science for information technology. The core competencies and facilities in this division are strong, and interactions with other parts of NIST and the ITL are evident, if not always formalized. The following sections in this chapter critique the work in the IAD along the requested dimensions of technical programs, scientific expertise, facilities and resources, and dissemination of outputs. Examples of the division’s work are discussed as are ways that the division could enhance the effectiveness, relevance, and recognition of its program while continuing its fine record of success.

ASSESSMENT OF TECHNICAL PROGRAMS

Accomplishments

Community Leadership

IAD personnel provide leadership in the field through a collaborative process that helps define and quantify problems, so that communities of interest can address them. Some of the efforts lead to proposed standards and to standards accepted by outside standards development organizations (SDOs), such as the International Organization for Standardization (ISO) and the Institute of Electrical and Electronics Engineers (IEEE). In some cases, as in the exchange of forensic transactions, ITL laboratory is itself the SDO. This is a valuable service that the IAD carries out in an excellent manner.

There are examples in this collaborative program where ITL and the IAD visibly drive the creation or advancement of important fields. One such example is the Text Retrieval Conference (TREC). Over the past 25 years, the IAD has used the TREC platform to advance the metrics of performance in text retrieval. In particular, the community has moved from simple measures of precision estimates based on simple pooling of test run data to bootstrapping of test runs that consider interactions among query, system, and documents.

In the case of TREC for Video (TRECVID), the IAD has facilitated the identification of key tasks in video retrieval for defining system performance. Today these include semantic indexing, instance search, surveillance event detection, multimedia event detection, and event recounting. These are all tasks that the video retrieval community has accepted as a valid testing methodology for video retrieval systems.

In collaborating with the community, the IAD has repeatedly demonstrated its ability to bring about shared and sustainable leadership, affected by collaboration with the relevant community. Examples are the TREC steering committee, Interoperable Iris Exchange (IREX) iris studies, and ISO SC37

standards for biometrics. These collaborations cover the range of problem definition, creation of data sets with ground truths, and performance metrics. The IAD has helped carry the problem areas to the point where the community can take over the effort, when NIST involvement needs to decrease or end. In this connection, the IAD tends to maintain an effective balance between ongoing areas and new areas that emerge or are mandated.

Competitions, Metrics, and Usability

The IAD has facilitated competitions with results that demonstrate state of the art in the field. Many competitions have high barriers to entry such as cost of equipment or hours of tests required. The IAD has successfully lowered the barriers to entry in some instances.

In the speaker recognition evaluation, the IAD has contributed by providing compact i-vectors rather than terabytes of raw speech data, which would have required substantial computational processing as well as storage to reach the point where analysis could be performed. IAD built the Speaker Recognition I-Vector challenge. In machine translation (MT), the IAD instigated the creation of open-source MT toolkits to lower barriers for entry. The IAD also created “light” track alternatives to large government evaluations (such as at the Defense Advanced Research Projects Agency [DARPA]) so that small laboratories and researchers can participate at some level and contribute diverse and valuable ideas. For example, since 2006, the IAD has run open-MT evaluations parallel to the Global Autonomous Language Exploitation (GALE) and Broad Operational Language Translation (BOLT) evaluations by DARPA.

The IAD has driven the development of a very broad range of new metrics, practices, and tools in information science and technology. Examples are numerous: methods for assessing the strengths of passwords, ways to estimate recall, methods of handling large data sets, human-targeted translation error rate, identification of information intrinsic to latent fingerprints, quality covariates for biometrics, and biometric aging. To explain one contribution: the Metrics for Machine Translation Challenge (MetricsMaTr) evaluations evaluated machine translation metrics that led to the development of alternative automated MT metrics. In summary, the IAD has played a unique and sustained role in focusing worldwide efforts on new metrics and in building machine translation metrics through launching and managing metrics competitions. The multimedia technology contributions of the IAD are complemented by leadership in the topic of visualization and usability. As an example, the IAD was instrumental in creating the biometrics usability field through studies such as those that discovered optimal heights and angles for security scanners at port of entry kiosks.

Opportunities and Challenges

Data-Driven Information Science

A data-driven approach is critical to advancing the field of information science and technology. The IAD team is starting to make an important contribution in this regard. The team is actively engaged in generating appropriate data sets and ground truths, running meaningful tests, and facilitating new web-based data sites. This direction is very positive. There is also an opportunity for the IAD to expand its vision beyond labeled data to include live data and to establish a methodology where larger numbers of candidate algorithms can be investigated through continuous evaluations. This is an industry-wide opportunity and trend, with implications for how to address ground truths and atypical events.

Baseline Implementations

A useful service that the IAD provides is harvesting and providing baseline implementation information to spur further research—supporting the concept of “building on the shoulders of others.” The challenge is how to extend this from a closed environment to a larger community of researchers, thus lowering the bar for introducing novel ideas into the mix. On a related theme, many researchers now offer open-source or matrix laboratory (MATLAB^®) implementations that allow for quick and easy testing. However, the licensing arrangements can be foggy, and there is a wide variance with respect to the quality of implementation and documentation. The IAD could take a role in helping to harvest these implementations with an eye toward persistence and reliability, and in leveraging open-source environments such as Python that offer useful ways to bundle together data, access methods, and explanatory text.

Tools for Security Policy

IAD’s work on the creation of tools for security policy and the representation of these tools in a form that permits agencies to simplify their password definition rules represents a very promising opportunity. Currently, the rules are ambiguous and too complex for system users to comprehend, and this seems to be an obstacle to the application of appropriate security practices government-wide. The IAD tool to clarify password rules promises to reduce this kind of obstacle. A further development of this tool, with appropriate explanatory text, could become an industry resource for citizen-facing government and institutional services, and for consumer-facing commercial firms.

Common Industry Format-Plus

The Common Industry Format (CIF) [ISO/IEC 25062:2006 Software Engineering–Software Product Quality Requirements and Evaluation (SQuaRE)] for usability test reports has been successful. The CIF for summative usability test reports is now in use across industry for safety-related usability certification of electronic health record systems, for usability and accessibility testing and certification of voting systems, and for usability testing of biometric systems. However, there is the question of whether that tool needs to be updated. The CIF focuses on speed, accuracy, and satisfaction with how systems support human work. In the past decade, other important aspects of user experience have gained commercial importance, such as completeness (rather than only speed) of learning, comprehension, and engagement (rather than only accuracy) in game-playing. As design becomes more important in commercial and recreational applications and educational settings for both formal schooling and lifelong learning, factors such as psychophysiological indicators of cognitive load, enjoyment, and other hedonic qualities are becoming increasingly relevant. There is rich measurement literature for these topics, but the unique capabilities of the IAD and ITL are needed to transform research into a more complete metrology of user experience, as measured through the holistic testing of usability and user engagement.

Health Informatics

The health informatics arena represents both an opportunity and a challenge. The Electronic Health Record (EHR) program is not yet quantitatively driven. This was attributed by the IAD to the circumstance that real data arrived late in the process, after a 4-year procurement and Institutional Review Board (IRB) delay. This is a mandated project, and if process indeed hinders its progress, ITL needs to address those problems, in collaboration with NIST leadership, perhaps at the level of the U.S. Congress. If the delay continues, the credibility of NIST could suffer.

Conclusions

The technical programs in the IAD are in general well organized, well staffed, and appropriate to the mission of ITL and IAD.

It is important that IAD maintain its momentum and leadership in its many successful activities but also consider the new opportunities that have been described above. This will be important to ensure success in a rapidly changing science and technology landscape.

It is also important that the division management review programs to identify those that need additional methodology or rigor and those that face challenging adoption paths. Such critical review could lead to the identification of needed resources not found at the division level.

The EHR program, for example, would benefit from a critical review. The addition of new emphases to the Text Retrieval Conference (TREC) Clinical Decision Support tracks would add value to the division’s EHR work. Developing health data sets that can be used by the health informatics community would link the 25-year impact of TREC to increasingly important challenges in electronic health records.

PORTFOLIO OF SCIENTIFIC EXPERTISE

Accomplishments

The IAD is a recognized leader in impactful standards activities and conferences. Evidence of recognition appears in many forms, such as program committee selection, conference leadership, and awards. As one example, TREC methods and results have been applauded by the search industry (Google’s chief economist), been credited with a 5:1 economic return in an RTI International study,¹ and cited in a federal case.²

The IAD personnel have deep expertise in creating metrics and best practices for many classes of information applications. They have the core competency for continuing their mission in the next generation of multimedia and multimodal technologies. It is impressive that the IAD has been able to attract and retain personnel who are among the best in the world in their fields. The fairly recent creation of new programs in open data, big data, data science, privacy, and video is noteworthy and indicative of a broad intellectual base in the IAD.

Opportunities and Challenges

There is an opportunity to enhance core capabilities in several areas where the IAD aspires to make a major contribution. The Data Science program affords an opportunity to develop expertise in big data, linked data, unintended relations, and data mining. In the Privacy program, there is the continuing challenge of deidentification and usable interface controls for privacy settings. In Public Safety, where there is a great societal need and infusion of significant funding in the pipeline, current IAD staff will need to be augmented in terms of scope as well as agility in new scientific areas.

The IAD and the Statistical Engineering Division (SED) have recent, ongoing, and planned collaborations in the following data science projects: identifying face quality and factor measures for video using the point-and-shoot challenge (PaSC) video data, fingerprint database mining for performance assessment of non-parametric likelihood ratios for expressing weight of evidence, statistical analysis methods for evaluating automated text translation systems, measuring the usability and security

_________________________

¹ B.R. Rowe, D.W. Wood, A.N. Link, and D.A. Simoni, Economic Impact Assessment of NIST's Text REtrieval Conference (TREC) Program: Final Report, RTI Project Number 0211875, RTI International, Research Triangle Park, N.C., December 2010, http://www.rti.org/publications/abstract.cfm?pubid=16086.

² Stanley v. Pipe, D. Md. (2008). Memorandum and order.

of permuted passwords on mobile platforms, and advanced statistical metrology for information retrieval. Taking advantage of all opportunities for collaboration between these divisions would be beneficial to both.

Addressing outcome measures beyond the CIF, discussed above, is a fundamental scientific opportunity. It is also a necessary evolution of the technical program if the IAD wants to be a player in topics such as educational effectiveness, game-playing experience, social engagement in media, social networking statistics, and citizen engagement in government services. These represent a reasonable extrapolation of IAD activity, but the core expertise will need to be enhanced with respect to expertise and number of researchers.

Conclusions and Recommendations

The portfolio of scientific expertise is appropriate and impressive for the traditional areas that the IAD focuses on, but core capabilities need to be enhanced in areas where the IAD aspires to make an important contribution.

Supporting the standards process places resource and attention-span demands on personnel. One option to reduce the demands could be to turn over standards materials to ISO and other SDOs. However, this approach is sometimes associated with update cycles that are too long and possibly too onerous to meet the original intent of the standard, and it entails risk to IAD credibility.

ADEQUACY OF FACILITIES, EQUIPMENT, AND HUMAN RESOURCES

Accomplishments

There are notable examples of good infrastructure available for the IAD, such as the Biometric Research Laboratory. Also impressive is a well-cultivated mechanism to expanded reach and impact through off-campus centers of excellence such as the effort with the University of Maryland and the use of cross-organizational virtual teams within ITL and across NIST.

Opportunities and Challenges

The IAD team plays a significant role as leaders and shepherds of their communities of expertise. To fulfill this role with timeliness and excellence, seamless interaction and agility are critical success factors. As such, operational latencies in IRB or travel approval processes can be first-order detriments.

Maximization of impact depends on continued cross-division collaboration and resource allocations. In a strategy that was described by IAD during the review, there appeared to be a major focus in the first one or two years of a new business area, followed by a reliance on the momentum and eventual allocation of resources to the budget. This might be a process worthy of more attention. Ideally, practices would be formalized in a way that permits their reuse for subsequent cross-division projects.

Conclusions and Recommendations

The facilities, equipment, and human resources are strong assets of the IAD. As areas such as health informatics gain importance, there seems to be a tendency to make do with the core talent areas rather than hiring specialists (trained biomedical informaticians and health data scientists) permanently or even for shorter times. This practice needs to be reconsidered in the strategic planning of the division. A prime example of this need is the EHR team.

Collaborative teamwork with scientists and engineers inside and outside of IAD, including virtual teamwork and in-person interactions, is of fundamental importance to the work of the IAD staff.

It would be helpful to create an IT sandbox, using computational resources not connected to the core NIST infrastructure, to leverage infrastructure in creative and flexible ways that a general information security policy might inhibit.

DISSEMINATION OF OUTPUTS

Accomplishments

The IAD has played key roles in creating standards for almost 30 years. Examples cover a broad range, from usability test reporting to biometric information exchange. The CIF for software usability testing and reporting, which started in what is now the IAD, has become an ISO standard (SC7 (CIF) ISO/IEC 25062:2005). The standard for the exchange of fingerprint data, which also started in the IAD (when NIST was the National Bureau of Standards) as a Federal Bureau of Investigation (FBI)-funded project, has remained an American National Standards Institute (ANSI) standard since it was first approved in 1986. This standard has been updated several times to the current version (ANSI/NIST-ITL 1-2011 Update 2013). It has been used internationally, in at least 50 countries and by Interpol, for the exchange of biometric and forensic data in and among criminal justice systems, applicant screening systems, and border-crossing applications. IAD has also worked with the Election Assistance

Commission to develop usability and accessibility standards for voting systems (Chapter 3 of the Voluntary Voting System Standards) as part of the NIST Voting Program.

In further support of standards, the IAD has developed and made available software trackers of standard conformance that are also widely used.

The IAD has also developed broadly usable IT tools for parsing and displaying password policies.

Opportunities and Challenges

Web-based tools can be used to broaden participation in workshops and short conferences and to disseminate the results and proceedings. This has been done recently for the NIST conference “Improving Biometric and Forensic Technology: The Future of Research Datasets.”

The culture of IAD seems to emphasize highlighting the successes of others. This is very conducive to building a community spirit, but it needs to coexist with better marketing of the fundamental value that IAD personnel contribute.

Conclusions and Recommendations

The IAD value proposition with standards and conformance tool sets is outstanding and deserves continued nurturing and support. Its dissemination process would be enhanced by greater use of Internet-based methods.

The IAD has a fine record of recognition and engagement. There is an opportunity to gain even broader recognition for the value of the IAD and to engage the even larger community that could benefit from IAD contributions. This will need an effort on the part of the IAD leadership to engage in assertive marketing, including continuously updated website presentations of citations and praise. Some existing examples of such citations come from diverse constituencies such as the private sector and federal court case literature.