An Assessment of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2007 (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

General Assessment of the Information Technology Laboratory A central part of the charge to the panel was to assess the quality of the work being done at ITL. The panel found that in general the appropriate peer group for this quality assessment was the U.S. national laboratories. The work at ITL generally ranks at or near the top of the work being done by peer institutions. There are some exceptions, principally cases where another laboratory has strong motivation to develop in a specialty area (e.g., laboratories concerned with nuclear weapons research have developed outstanding capabilities in high-performance computing). RESEARCH STRATEGIES This section identifies several impressive approaches to research at ITL. These approaches are quite innovative and in many cases unique to NIST. Community Building ITL has a long tradition of promoting research by serving as an honest broker for competitions. ITL scientists work very hard to learn the goals of a research community, and they present that community with appropriate challenges in the form of data, metrics, tasks, and evaluation protocols by which the community can test the performance of their system. Often, these challenges are offered in an annual competition. A repeatable methodology has evolved for involving the community, measuring performance, refining the challenges, and making the test data public after the fact, allowing new teams to join the community. The original competition of this class is the ongoing challenge of speech recognition. ITL now also offers competitions in information retrieval, cryptography, face recognition, and language translation, for example. Several instances of this process have been created for external research agencies, which find the methodology to be an important tool for driving research. Interoperability Testing Making complex software systems interoperate is a difficult challenge. Thus, another important theme of research at ITL is the evaluation of commercial products for the ability of the results of one to be consumed by another. Examples include the US- Visit program, XML validation, and cryptographic device validation. The US-Visit program uses devices that record fingerprints and compare them with their supposed matches. Two devices may not produce the same electronic representation of the same finger, even though each device conforms to the standard for representation. When the differences are inherent in the design of the device, it becomes impossible for the two devices to be used on the same person, because the probability of false negative becomes too high. There is an ITL effort to specify best practices for defining and using the XML 4 

Schema. ITL staff have written documents and tools to help assure that schema are well formed and that software processing the XML conforms to specifications (through published examples). The work also addresses schema reusability by developing guidelines for meta tools that control the naming and design of schema. Cryptographic device validation is a mature initiative for accrediting laboratories that test cryptographic hardware and software. This process ensures that laboratories implement algorithms correctly and that they protect sensitive information, such as keying material, according to best practices. ITL provides testing software to the laboratories and defines the testing interface to implementors, streamlining both the testing process and the confidence in the results. External Collaborations The impact of ITL’s work with external agencies can be found across the laboratory. There are some good collaborations, both with other laboratories at NIST and with external agencies. In the latter case, ITL staff are often supported by contracts to perform necessary work. In general, the contracted tasks are appropriate to the overall NIST mission and often enhance the overall research capabilities of ITL. There are, for instance, external collaborations on health care with the Department of Health and Human Services, the Integrating the Healthcare Enterprise project, and the American Telemedicine Association. There is work on radiation detection for the Department of Homeland Security and work on speech processing for the National Security Agency. The Boulder group of ITL’s Statistical Engineering Division (SED) works with colleagues at the NIST Center for Neutron Research on a project involving the imaging of hydrogen fuel cells. Water management in the fuel cells, where water is a by-product, is crucial, and neutron imaging is a uniquely good way to monitor water in the operating fuel cell. Statisticians have worked on strategies for improved image processing to assist in monitoring the water. SED personnel were approached by the Department of Homeland Security just before a radiation-detector evaluation experiment comparing different vendors was to be conducted at the Department of Energy’s Nevada Test Site. The evaluation had already been designed, and SED was originally approached to assist only with data analysis. The resulting report was adopted as the sole analytic input to the vendor selection process. In addition, the work of SED was valued so highly that the division has now been incorporated into the design and analysis for subsequent phases of the project. OPPORTUNITIES ITL is beginning several interesting directions for research, which ITL management should promote and grow. A number of the opportunities described by ITL are identified below. Medical Informatics There is a huge national challenge in making health systems interoperate. These 5 

systems include patient medical records and telemedicine. The problems are addressable by the traditional strengths of ITL—they will almost always involve the creation of standards and the validation of interoperability among different commercial systems. ITL has made some effort in these directions, such as HL7 message conformance and telemedicine standards. However, there is much more to be done. ITL should expand this effort, including establishing the links needed with other agencies and other stakeholders, creating relevant projects, and organizing interdisciplinary teams to execute the projects. As a first step, it should develop a strategic plan for research in this area and should augment its leadership capabilities here. Special attention should be given to identifying people in biomedical informatics who could provide such leadership and/or scientific capabilities. Credible performance in this area may require M.D.s in leadership positions. Metrology Metrology science is the bread and butter of NIST. There are a number of ways in which the work of ITL creates new challenges in this area, and ITL staff should tackle them. In a number of recent activities, standards or evaluation procedures involve a human in the loop—that is, a human has to examine each submission and make a subjective decision as to its quality. In some cases this is analogous to defining the weight of a kilogram by what an expert on weights thinks should be a kilogram. While there may be cases where nothing better can be done to eliminate subjective judgment, ITL staff should think carefully about these procedures. Removing human judgment from an evaluation, even partially, can have two advantages: It usually reduces cost, and it avoids the arguments that arise when there are subjective evaluations. A good example is the Mathematical and Computational Sciences Division’s recent algorithms work, which replaced equipment adjustments previously done by forensic laboratory technicians. ITL should also further develop measures for correct translation among natural languages. As for another example, the routine way to evaluate translation of a natural language is to give each translation to a human to evaluate. This process is expensive and does not scale. Another approach, which ITL is looking into, involves a human setting down criteria for correctness of the translation of a small passage, and then having a machine check the criteria without having to consult the human on every submission. Standards are generally assumed to be beneficial, but the proof for this assumption is sometimes lacking. As an example, the Federal Information Security Management Act (FISMA) has been applied broadly, but its benefits are not known. ITL should consider investigating the effects of standards and should develop a broadly applicable methodology for such investigations. Statistics There has been attrition in SED over the past 2 years, and a new division head has been appointed. Like the Mathematical and Computational Sciences Division in ITL, SED’s primary focus is on collaborative research with other groups at NIST, primarily outside ITL. 6 

While growth is justified in many of ITL’s divisions, there is an especially urgent need to expand activities in statistical engineering. Most significantly, there are a number of subdisciplines of modern statistics where in-house expertise is lacking and practitioners of these arts are needed. Secure Networking There is an opportunity to serve ITL’s own needs for secure access to external computing resources and at the same time do research in an interesting new direction. The section on computing infrastructure, below, discusses this opportunity in further detail. PLANNING FOR GROWTH NIST’s budget is projected to double over the next 5 years. Even if that projection is not realized, there will undoubtedly be substantial opportunity for growth in the near future. ITL should be prepared with a roadmap for that growth. Demographics The average age of laboratory members is rather high, and the change over the past years has not been good. The average age in one division rose by 4.1 years over the past 4 years. The age distribution shows a dearth of 30- and 40-year-olds, the age at which scientists and engineers tend to mature and increase their effectiveness. Temporary Versus Permanent Hires ITL should identify the areas in which it would be most useful to grow and should develop a strategic plan for such growth. The plan needs to take into account long-term versus temporary hires. There is an advantage to being able to hire someone for a limited time and then turn to someone else as priorities change. Yet competition with industry is severe, and talented people are unlikely to accept the risk of a short-term appointment in addition to a salary structure that cannot compete with what industry offers. A significant fraction of the ITL budget comes from other federal agencies. Some of these arrangements have proved to be reliable over the years, while others soon disappear. Giving a potential hire the impression that his or her position is temporary and that he or she would have to compete against all comers for a permanent position would be sufficiently negative to deter top people. It is encouraging that none of the divisions reported such an event and that it is apparently possible to avoid competing a position when converting temporary staff to permanent. However, the risks are real, and ITL should look carefully at how temporary positions are used. Hiring in Nontraditional Areas Some interesting directions are best served by nontraditional hires. In some observed cases the appropriate background could be different from that of the 7 

stereotypical hard scientist (e.g., a physicist or chemist). For example, assessing the usability of voting machines requires thinking carefully not only about people with recognized disabilities (e.g., blindness) but also about people with other limitations (e.g., arthritis). A physical therapist or other relevant health-care professional could add an important perspective to this kind of evaluation. As another example, the realization that a translator of natural languages would be an essential part of a team for evaluating machine-translation software proved to be problematic because at NIST translators are classified as office staff rather than scientists. ITL management should consider hiring in nontraditional areas when appropriate to the mission, and NIST management should try to remove impediments to such hires. Status of the Information Technology Staff The data from ITL suggest that the laboratory has a rather small proportion of the senior scientific and technical positions (“SES” equivalent) that have been allocated to NIST. One reason for the disparity may be the awards garnered by scientists in other NIST laboratories. There are Nobel winners among the scientists of NIST; the panel is not aware of analogous ITL staff (e.g., Turing award winners or Fields medalists). On the other hand, information technology is vital to U.S. competitiveness. More than a third of all new jobs created in the United States in recent years are in information technology. Moreover, these jobs tend to be “good” jobs. In his book The World Is Flat Thomas Friedman reminds us that the wealth of a nation is today determined by the strength of its IT. Attracting more top people to ITL, given the areas it covers, makes sense in the light of this reminder. THE RESEARCH CULTURE There are two ways, discussed next, in which the reputation of ITL can be enhanced. Marketing There is no one model for outreach that applies to all the divisions of ITL, and it may be appropriate for different divisions to market their work in somewhat different ways. However, in general there are two ways to have influence: through publications and through outreach—formal and informal interaction with other groups. Some divisions publish regularly; some do not. Some have external visibility; others do not. Both modes of enhancing external visibility should be considered seriously by every division. If a division is not doing both, then management should at least understand why not. Seeking Research Opportunities It is important that a large proportion of the scientific and engineering staff be alert to new opportunities. Ideally, many on the ITL staff would be searching actively for the next important problem to work on, and certainly some are. However, more ITL staff 8 

should be doing so, and management should facilitate a more adventuresome approach to research. THE PROJECT/PROGRAM MATRIX A recent ITL innovation is the establishment of crosscutting projects that have their own leadership and budgets. The creation of interdisciplinary projects is a good thing for ITL. Yet, things can go wrong if their introduction and management are not handled carefully. The plan for matrix management at ITL is not likely to succeed unless there is more enthusiasm for the plan among the staff. The following is an outline of some of the risks: It is hard to will research projects with impact into existence. A successful research project requires buy-in—a belief by the participants in the importance of the project. Ideally, the project is created by the staff of the project itself. It is necessary to find leaders who are enthusiastic advocates for the project. They need to be able to inspire people on the team and to support the shared vision with management above. Project leaders should have strong technical or scientific credentials. If policies are not made very clear, there will be reason for staff to worry about how they fit in. Staff may be concerned about whether they will be evaluated by their project leader, their division chief, or both. They may wonder whether everyone will eventually be part of a project or whether those who want to will be able to continue working in their own preferred discipline. The funding model may distort operations. For example, the initial budget for the ITL projects came from an approximately equal “tax” on each of the divisions. However, the current projects are not now, and may never be, equally suitable for members of all of the divisions. As a result, there will be unintended winners and losers. There is an opportunity to use the anticipated growth to bring in leaders and key staff for some exciting new projects. ITL management should communicate more with the staff to identify and allay some of the concerns that have been raised and to provide assurances that their work and status do not depend on whether or not they are participating in a project. While the stability of government programs should not depend on the personnel assigned to them, maintenance of a stable group of mature professionals is necessary for the efficient execution of complicated technical challenges. When converting to a matrix management structure, ITL should avoid disrupting the established challenge problems of the Information Access Division and seek to keep the experienced expert staff in place. 9 

COMPUTING INFRASTRUCTURE The equipment resources are adequate to serve the needs of the research. However, there is a limitation of another kind. For obvious and valid reasons, the Department of Commerce has instituted strong new controls over computer and Internet use to minimize the risk of intrusion or other attacks on its computer systems and data. The risks of circumventing or ignoring these controls are high. However, the new policies have created some serious impediments to research at NIST, and at ITL in particular. One example illustrates the problem: The Advanced Network Technologies Division has been mandated by Congress to provide a roadmap whereby the entire U.S. government will adapt to IPv6 (Internet Protocol Version 6), yet it has been unable to gain access to an IPv6 experimental network. Such networks exist outside NIST and could be used, but not without violating policy. The problem described is not unique to NIST and will become more widespread. Moreover, the issues surrounding how to work around the problem largely involve standards. Therefore ITL itself should undertake the research necessary to provide a next generation of security standards that will facilitate rather than inhibit the activities that a research organization needs to conduct. 10