Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 53
Biometric Recognition: Challenges and Opportunities 2 Engineering Biometric Systems The preceding chapter described many of the fundamental concepts that underlie biometric systems. Of equal importance is the engineering of these systems. Moreover, while design, engineering and development of component parts of the systems are important, it is the development of a biometric system as a whole that is most critical to successful system deployment. The scope of a biometric system is broad and includes not only basic operations such as enrollment or matching but also user training and the adjudication process for dealing with contested results and exception handling in general. A holistic view that accounts for human interaction, and not simply the combination of sensors and matchers, is needed. Biometric systems are best considered in their deployment context, including all their particularities such as function, environment, and user population. A systems engineering view is especially important when the systems are to be used on a large scale, such as for border control or social service entitlement, when all the best practices associated with system design and management are called for. While the evolution of sensor devices, matching technologies, and human factors can dominate the attention of system designers, the ultimate success of the overall system generally relies on attention to conventional system development issues. These include: A clear understanding of the system’s functional objectives, including the user population and environmental constraints of the system’s deployment context.
OCR for page 54
Biometric Recognition: Challenges and Opportunities A model for accommodating the evolution of technology over time. An understanding of the effectiveness of the system through frequent or continuous operational testing. A model for validating the financial value of the system over time. A strong understanding of the human interactions with the system and how they change with time. A holistic security model that addresses the application security context and provides a cogent analysis of the potential for fraudulent presentation of biometric samples. Successful biometric applications require top-down conceptualization, with clear delineation of purpose within a systems context. Technology-driven implementations could inadvertently target a secondary rather than a primary objective and fail to anticipate mission-critical aspects of the context in which this human-centric technology is applied, including sources of challenges to the system. This chapter starts with an overview of biometric system operations. It then turns to the application requirements of biometric systems, including application parameters that can affect system performance. A brief discussion of test and evaluation concludes the chapter. BASIC BIOMETRIC SYSTEM OPERATIONS The operations performed by a generic biometric system are the capture and storage of enrollment (reference) biometric samples and the capture of new biometric samples and their comparison with corresponding reference samples. Figure 2.1 depicts the operation of a generic biometric system and is a more detailed version of the sample operational diagram from Chapter 1, although some systems will differ in their particulars. Enrollment Operations Enrollment of a new subject into a biometric system is achieved by performing the functions denoted in the upper processes A, B, C, and E of Figure 2.1. The samples are analyzed to ensure their quality is consistent with the matching algorithms to be used later for comparison with opera-
OCR for page 55
Biometric Recognition: Challenges and Opportunities FIGURE 2.1 Idealized operations of a generic biometric system. tionally obtained samples.1 The data samples are then processed to form references that are stored for future comparison in a database or on transportable media such as a smart card.2 Depending on the requirements of 1 However, quality assessment algorithms for biometric enrollment are not universally available and may need to be specific to the proprietary feature extraction and comparison algorithms used later in the process. In some applications, such as use of facial recognition with images stored by passport issuance agencies on electronic passports, there will be no knowledge during the enrollment process of what algorithms might be used later to compare the stored image. This is true for visas as well. 2 In other applications, such as facial recognition biometrics with images stored on passports, processing of the reference sample is deferred until the matching process is initiated.
OCR for page 56
Biometric Recognition: Challenges and Opportunities the system, functions D and F might also be performed against previously stored references to detect attempts at duplicate enrollment. Ensuring that the reference format generated by function C conforms to pertinent standards such as those issued by ANSI or ISO3 would aid data interoperability between systems or within an evolving system over time. If sample quality is inadequate to create a reference likely to be correctly matched in future comparisons, the biometric characteristic might be resampled until adequate quality is obtained. However, the ability to resample to control quality of the enrollment sample will depend upon both system function and policy. If adequate quality is unattainable, then policy will dictate either that the best of the samples captured is retained or that the subject is declared a “failure to enroll.” In the latter case, a fallback approach might be established for the individual. For example, in an employee access control system, the particular employee might be allowed entry with only a card and a PIN (two-factor authentication) or by requiring that a security guard verify the individual’s facial photo. For enrollment in a watch list system, however, system management would have to balance the impact on future accuracy of retaining poor-quality references against having no reference for a person of interest. Such balancing has been a major challenge for systems used in forensic, military, and intelligence applications. Enrollment of a subject also properly links the digital representation of the biometric reference with identity attributes established for use in this application.4 Biometric traits acquired at later times are then recognized by comparing with the enrolling reference.5 Hence, the quality and representativeness of the enrolling reference are crucial to later success, and variation in these can contribute to errors in the recognition process. Most of the factors that affect the quality and representativeness of the enrollment reference involve the interaction of the subject with the collection device or irregularities in or absence of the biometric trait in 3 See the Registry of U.S. Government Recommended Biometric Standards (Version 1.0, approved June 5, 2008) for a catalog of suitable standards for interoperability of U.S. federal biometric applications. Available at http://www.biometrics.gov/standards/StandardsRegistry.pdf. 4 In the general case, no assumptions are made about the presumed “true identity” of the enrolling subject. The process of enrolling the subject’s biometric data allows for subsequent retrieval of identity attributes. The legitimacy of the subject’s association with those attributes, however, must be established by means outside the biometric enrollment process. See FIPS PUB 201-1 for an example of a process for verifying identity claims. Available at http://csrc.nist.gov/publications/fips/fips201-1/FIPS-201-1-chng1.pdf. 5 Some systems may update enrolled references over time as a side effect of matching actions to minimize the drift of a biometric trait. In general, controls comparable to those applied at initial enrollment should be in place to minimize the opportunity to inject fraudulent data into the biometric database.
OCR for page 57
Biometric Recognition: Challenges and Opportunities the subject, with some involving the device itself. Generally, the operational conditions under which both references and subsequent biometric samples are collected for a criminal justice system (such as booking a crime suspect) allow more control of quality than other types of systems. For image-based biometric modalities these may include the stability and positioning relative to a desired standard during imaging, the presence of distracting or camouflaging appurtenances, calibration and cleanliness of the sensing device, the image review process and criteria for certifying successful enrollment, and, indirectly, factors that affect the ability to control these. Such indirect but important factors may include the degree to which the acquisition of these biometric references is supervised and the staffing levels, throughput goals and pressure to achieve them, aspects of the environment that can affect sensor and human performance, the degree to which the subject is aware of and cooperative with the acquisition, and the efficiency of handling special exceptions, such as subjects who are physically or mentally challenged in their ability to present the required biometric or whose references are repeatedly rejected. For some systems the reference representation is too large or complicated for efficient storage and high-throughput computer processing. An alternative is to extract mathematical abstractions known as “features” (or models of those features) from the reference samples and store only those, discarding the reference samples. Another alternative is to store reference samples using some standard compression technique (such as JPEG), not processing the sample to produce features until they are needed for comparison. In principle the choice of biometric features, the degree of independent information and hence distinctiveness conveyed by each additional feature, and the balance between feature multiplicity and storage efficiency can have major effects on recognition error rates. This is borne out by the large performance differences exhibited in tests that compare products using the same biometric trait.6 Storage of the reference, whether in the form of samples, features, or models, completes the enrollment process. Subsequently, samples provided by the enrollee and others claiming or claimed to be the enrollee will be compared against the reference. In some cases, for example, in the case of a law enforcement database and an enrollee who is the object of a security or law enforcement search, the reference may be added to a watch list containing many millions of samples that might ultimately be checked. 6 For example, the NIST Minutiae Interoperability Exchange Test. See http://fingerprint.nist.gov/minex/.
OCR for page 58
Biometric Recognition: Challenges and Opportunities Capture and Matching Operations The processes A, B, C, D, E, and F in Figure 2.1 are carried out to capture one or more biometric samples and compare them against the reference(s). When possible, a sample to be used for comparison against a stored reference is analyzed to ensure adequate quality consistent with the feature extraction and matching algorithms used.7 When the user is cooperative and the sample quality is inadequate, acquisition attempts may be repeated up to a permitted maximum number of times until a suitable sample is obtained. If adequate quality is not attained, depending on policy, either the best sample is used or “failure to acquire” is declared and a fallback procedure invoked, as noted earlier. Acceptable samples are converted to a format suitable for comparing against the stored references, generally following the same methods used to create the corresponding references. If verification is the desired operation, the reference for the identity claimed is retrieved from the database (perhaps triggered by the user presenting a PIN or a proximity card), a comparison is made, and a comparison score is assigned. If this score exceeds a threshold, a match is declared. Otherwise a nonmatch is declared or possibly, for an intermediate window of scores, an indeterminate result is declared and a cooperative subject may be asked to resubmit the same biometric sample, an alternative biometric sample, or take additional action, such as contacting a security guard to execute a manual fallback procedure. If the desired operation is identification, then the sample features are compared against a portion or all of the reference database. In this case, one or more matches are indicated if the comparison score exceeds a predetermined threshold value. The top scoring match exceeding a threshold may be indicated, or all matches may be adjudicated by a human examiner. This discussion has outlined the structure of a process that cannot be expected to work perfectly, deterministically providing the “right” answers, because it has inherent opportunities for error and uncertainty. It should be noted that the accuracy associated with declaring a match or nonmatch at a given score threshold is probabilistic and associated with metrics such as the false match rate (FMR) and false nonmatch rate (FNMR). So it is important to think carefully not only about what thresh- 7 Adequate sample quality is a somewhat vague characterization of what is needed. In this context we are concerned with the utility of the sample in the biometric system. This utility can be maximized when a clearly expressed trait is captured with high fidelity. If the data subject’s traits are poorly expressed owing to, for example, highly abraded finger friction ridges, then repeated attempts to capture a sample useful for comparison are unlikely to succeed.
OCR for page 59
Biometric Recognition: Challenges and Opportunities olds are appropriate but also about all the various ways uncertainties can enter the system, as described in Chapter 1. OPERATIONAL CONTEXT Many factors affect the success or failure of a biometric system in its implementation. Table 2.1 illustrates the common parameters of the data subject, application, and technology contexts that affect both system design decisions and operational performance. Some parameter labels in Table 2.1 are shortened to fit and are then elaborated on below. Parameter intensities are directionally ordered left to right by increasing difficulty of implementation. For example, biometric systems of which participating subjects are aware and for which they are cooperatively motivated are easier to implement successfully than systems in which sample acquisition is surreptitious and subjects are hostile to the purpose. This list is incomplete but intended to spur the thinking of prospective system implementers. Each row is discussed below, grouped by user, application, technology, and performance contexts. For any given class of applications or, more precisely, for any target deployment, one can begin an analysis of where on this table the demands TABLE 2.1 Parameters That Affect System Design Decisions and System Effectiveness Parameter Degree or Intensity from High to Low User context Data subject awareness Very Not very Data subject motivated Very Not very Data subject well-trained Very Not very Data subject habituated Very Not very Who benefits? Both User/consumer Owner/agency Application context Application supervised Very Not very Application type Positive claim Negative claim Application type Verification One to few matching Identification Data interoperability Closed Supposed to be closed Open Technology context Environment controlled Very Not very Passive versus active Active Passive w/cooperation Passive Covert versus overt Overt Covert Performance context Throughput requirements Low Medium High Sensitivity to error rate requirements Low Medium High
OCR for page 60
Biometric Recognition: Challenges and Opportunities of the application lie. For example, consider a system to allow entrance to a gym, where data subject awareness and motivation would be high, the training of data subjects would be medium to low, the system probably requires active cooperation, the environment is controlled, and throughput is probably not a significant consideration. By contrast, a border control system using biometrics will be faced with users who are not well trained and perhaps not well motivated but will nonetheless have high throughput and stringent error rate requirements and so on. Clearly, stating that a system is a biometric system or uses “biometrics” does not provide much information about what the system is for or how difficult it is to successfully implement. Moreover, the parameters listed here are limited to factors that arise in day-to-day operations. There are broader system requirements (such as security, availability, and so on) that will also vary a great deal depending on the application and will also bear on the design, development, and, ultimately, operation of the system. User Context Data Subject Awareness Does the data subject know that a biometric sample is being collected? Watch-list applications may use references gathered under different circumstances and from other systems, so even if the data subject is aware that a biometric characteristic is being collected, he or she might not know that the data are being collected for the purpose of biometric recognition. For example, a police station might take a mug shot photo when booking a suspect. Although the subject would be very aware that a photo was being taken, he or she might not know that the photo would later become part of a facial recognition watch list in an airport. Data Subject Motivation Does the data subject want to present the feature to the system in a way that is repeatable? Some subjects—such as in a typical physical or logical access control application where the subject is enrolled in the system, has valid rights, and wants access privileges—are motivated because matching is needed to perform a job function or to access to an entitlement, a privilege, or money. Other data subjects, such as prisoners, may be very unmotivated to interact with the system. This lack of motivation could be due to the adverse outcomes of a correct match or the desire to deliberately attempt to deceive the system by faulty interaction. Motivation and cooperation are closely aligned but not identical. A data subject can be cooperating with a system but still not be motivated to use it. In
OCR for page 61
Biometric Recognition: Challenges and Opportunities other cases, such as use of facial images on passports, the data subject may be motivated to present an image deemed “attractive” rather than one that is truly representative and repeatable. Data Subject Training Has the subject been instructed in the proper use of the system? Is there an opportunity in the enrollment process to provide feedback on correct or incorrect feature presentation? Many applications may not give data subjects a chance to have human interaction with system staff during enrollment or subsequent uses. Is there an opportunity to inform the subject before exposure to the system? Different system types may require more training than others for correct usage. A more general question is, What kind of training or operational cues or feedback to subjects will improve system usability and performance? Data Subject Habituation How often does the subject use the system? Does the subject use the system and get feedback on a regular basis for a long time (for example, long enough to develop muscle memory if any is required)?8 Some systems that require active participation offer more assistance than others, guiding subjects to intuitively perform the functions necessary to give high-quality biometric measures. Infrequent usage, long gaps between uses, or poor ergonomic design can lead to poor positioning, which affects system accuracy and thereby throughput (see the discussion of technology context below). Who Benefits? Who receives the benefits of implementing a biometric recognition system in a particular application scenario? If the system is used in place of an existing system (for example, replacing verification using a driver’s license), then both the deployer of the system and the data subjects may expect to obtain the same benefits as from the existing system. A system meant to offer convenience to users and save time for its deployers may also save time for the users and may be more convenient for the deployers. Both parties gain each other’s intended benefits. In other situations 8 A NIST study on the effects of habituation on fingerprint quality found that habituation in the absence of feedback failed to affect the quality of fingerprint collection. Habituation with feedback did lead to improved fingerprint quality over time from a cooperative population. Available at http://zing.ncsl.nist.gov/biousa/docs/WP302_Theofanos.pdf.
OCR for page 62
Biometric Recognition: Challenges and Opportunities the benefits may be one-sided, especially where the biometric system is employed as an additional layer in an existing system. Take the example of adding a biometric system as a third factor to an existing card-plus-PIN physical access control system. A subject must spend additional time at a portal and faces an increased chance of being denied access because of the additional challenge and appears, moreover, to gain no direct benefit. The deployer of the system, however, substantially increases its confidence that the person entering is in fact the authorized user, with all other things being equal. So the need for security on the part of the system deployer is transferred to increased burden on the user. Even in a replacement implementation, this situation could exist. In a prison application, the use of a biometric system in lieu of an alternative system of prisoner identification offers no particular benefits from a prisoner perspective, and the prisoner is probably not interested in being verified as him or herself; however, the operators of the prison have a very high interest in proper prisoner identification relative to many other factors such as convenience and speed. Accuracy is the main benefit sought, but the data subject may have limited stake in this and little motivation to cooperate. Application Context Supervision Is the system staffed? Is there someone present in the immediate vicinity who could give instructions or advice, help with questions, or solve problems with system usage, or is the system unstaffed? There may be a middle ground where there are persons nearby who can help but who are not immediately accessible or who have limited knowledge about the system and can help only to a certain degree. A system also may be self-service, with no opportunity for assistance other than, perhaps, a phone call. Positive vs. Negative Claim Systems Systems whose purpose is to verify a claim that a data subject is known to the system are considered “positive claim” systems. These systems often require a claim by the data subject to a specific reference and by extension to the corresponding enrollment record. However, alternative examples of positive claim systems, needing the unspecific claim “I am enrolled,” have existed since the early 1990s. These systems were previously called “PINless verification” systems. Their function is to verify the claim without returning any information on which enrollment record (“who”) the data subject corresponds to. Negative claim systems verify
OCR for page 63
Biometric Recognition: Challenges and Opportunities the claim that a data subject is not enrolled in the system. Examples of such systems include systems to detect fraudulent multiple enrollments in social service entitlement programs or driver’s license registration systems. Note that some systems could be viewed and used as both positive and negative claim systems, but generally not simultaneously. A driver’s license or an identification card system could verify a negative claim at the time of enrollment and then verify a positive claim later when the identification card is used to assert an identity. Some systems, such as watch lists, might on the surface appear to be ambiguous with regard to negative or positive claim—with the nature of the claim depending on whether the point of view is that of the data subject or the system administrators. However, such systems are nearly always classified from the point of view of the data subject, because his or her viewpoint will determine the nature of system vulnerabilities. Verification vs. Identification Systems Verification systems are used to check an individual’s claim to be a subject enrolled in the system. These systems perform a so-called one-to-one type of match, usually by the data subject providing a name, number, token, or password that points to or unlocks the subject’s enrollment reference.9 Identification systems generally scan all references in a database to see if there is a match to the sample presented. Typically, verification systems are positive claim systems;10 identification systems can be either positive or negative claim systems; however, due to the nature of those applications (see discussion of positive vs. negative claim systems above) most negative-claim systems (to verify that a subject is not already in the system) are for identification. A middle-ground situation exists where certain members of a user group can choose to be enrolled together (linked). Subsequent system uses by any of the members will cause the system to do comparisons only within that user group, also known as one-to-few matching. 9 A one-to-one relationship in a verification comparison isn’t always strictly true. In a verification system one or more candidate samples may be compared against a collection of references that were previously provided by the nominated subject or may be compared to a larger “cohort” set of references from many individuals. 10 Negative-claim verification systems are theoretically possible, even if rare or nonexistent in practice. An example would be a system to verify that an individual is not a known terrorist.
OCR for page 65
Biometric Recognition: Challenges and Opportunities Passive vs. Active Technologies Passive technologies do not require the data subject to undergo a controlled interaction with the system although they may require cooperation from a data subject whether the subject is knowingly or unknowingly providing that cooperation, such as looking in a particular direction. Most facial and iris recognition technologies are considered passive but require a certain amount of cooperation by users, since at present they must look in a particular direction to be scanned successfully. On the other hand, gait recognition might be considered completely passive. Voice recognition applications could also be considered passive if they are based on regular conversational speech without subject interaction. Active technologies are those that require direct human interaction such as speaking a particular phrase or positioning a finger, hand, or head in the correct location. Some technologies might be considered semipassive, active in that a subject must position a feature in a certain orientation but passive in that there is no need to actually contact any surface and orientation need not be at all precise. Covert vs. Overt Technologies Covert technologies are those that are designed to be used without the user’s knowledge. Overt technologies actively involve the user. Performance Context Throughput Requirements Throughput refers to how quickly data subjects can be successfully processed by the system. This includes both successful and unsuccessful attempts at recognition. How long will a cooperative subject be allowed to present a feature before a timeout is called? How many retries will a user be allowed? Most applications have some degree of throughput required before the system becomes economically or procedurally prohibitive, and all applications should include the time required to process both correct and false matches that lead to application exception processing, since these cases tend to take more time and manual intervention than application acceptances. Error Rate Requirements Error rates, technically more properly referred to as error proportions, measure the frequencies of errors relative to the number of adjudications that could produce errors of the corresponding type. This term receives
OCR for page 66
Biometric Recognition: Challenges and Opportunities by far the most public attention and is likely also the most misused term. There are multiple components that contribute to system inaccuracies, and only a few are technology dependent. These are typically contributions from signal noise and background noise, but typically the largest components are the human interaction and environmental components. Error rates have direct impact on throughput, since false rejections take significantly longer to process than acceptances (whether true or false). Applications that require high throughput should aim for low false rejections; however, care must be taken in the application’s security analysis to balance low false rejections with the potential for an increase in false acceptances. INTEROPERABILITY Interoperability is a factor to consider when designing almost any kind of system. It must of course be considered when discussing data exchange between systems but must also be considered for evolving systems that retain and reuse data collected over time. In other words, even closed systems might have to interoperate with multiple generations and instantiations of themselves as they change over time. Interoperability also plays a role at the subsystem level, when systems are composed of vendor software or hardware. In the biometric system context, such components may include fingerprint matcher components, segmentation software, and minutiae detectors. In general, standards help to promote interoperability. However, there are times when the use of a standard format in preference to a proprietary format can be detrimental and potentially limit functionality or flexibility.12 For biometric systems, sensor interoperability, discussed below, poses some specific challenges. Sensor Interoperability Sensor interoperability refers to the compatibility between an enrolled biometric reference and a test sample, acquired using different sensors. In some systems, it is assumed that the two samples to be compared were 12 For example, the NIST Minutiae Interoperability Exchange Test 2004 compared the performance of proprietary systems against the performance of template data in standard format generated and matched by different vendors. In these tests the use of standards-based formats for single-finger matches was inferior to proprietary systems. Changing the system to use two-finger matching adequately compensated for the reduced accuracy but remained inferior to proprietary two-finger operation. Of course, the operational issues encountered when switching from collection of one fingerprint to two fingerprints may militate against the use of the latter two-fingerprint method to improve accuracy.
OCR for page 67
Biometric Recognition: Challenges and Opportunities acquired using the same sensor—or at least the same type and vintage of sensor. However, improvements in sensor technology and reduction in sensor costs means that enrollment and test samples are often obtained using different sensor types. This may also happen if a sensor manufacturer goes out of business and support is no longer available for a line of sensors. In a large, distributed system such as the FBI’s Integrated Automated Fingerprint Identification System (IAFIS), remote sites use a variety of end-point systems and may collect samples using a variety of certified sensors. In short, it is possible in almost any biometric system that test samples will not be collected using the same sensor as used in enrollment. (In criminal justice systems, there is no known change in error rates when each booking site selects whichever certified scanning system it wants.) It has been observed that the matching performance drops when the reference and test samples for fingerprint, iris, and voice are acquired using different sensors rather than the same.13 There are several reasons for this degradation in matching performance: (1) change in the sensor resolution and its operating behavior; (2) change in the sensor technology; (3) change in the user interface, and (4) changes in operational environment that have an impact on sensor performance. In the first case, while the underlying sensing technology remains the same—in, for example, an optical total internal reflection (TIR) fingerprint sensor—the image resolution (say from 300 by 300 to 500 by 500 pixels per inch) and/or the signal to noise ratio (SNR) of the sensor may change. In the second case, the two sensors providing reference and test samples may be based on completely different technologies (for example, one may be an optical TIR fingerprint sensor and one a capacitive solid state sensor). Of the factors listed, the second is more problematic because the change in sensor technology may decrease compatibility between enrolled references and test samples. There are multiple examples of different sensing technologies for a given modality. For example, sensors for fingerprints can be based on optical, capacitive, thermal, pressure, ultrasound, or multispectral technologies. Some touchless three-dimensional fingerprint sensors are being developed as well. Differences in how a user must interact with a sensor may introduce variations in sample coverage area and distortion. Similarly, two-dimensional images for face recognition can be captured in visible color, infrared, and thermal, as well as range (depth). Three-dimensional face images that 13 See, for example, A. Ross and R. Nadgir, A calibration model for fingerprint sensor interoperability, P. Flynn and S. Pankanti, eds., Proceedings of SPIE Volume 6202, Biometric Technology for Human Identification III (2006); Modi et al., Statistical analysis of fingerprint sensor interoperability performance, Biometrics: Theory, Application, and Systems (2009); and International Biometric Group, Independent testing of iris recognition technology, Report to Department of Homeland Security (May 2005).
OCR for page 68
Biometric Recognition: Challenges and Opportunities capture the depth image are also being used for face recognition; these so-called range sensors capture a different face modality than the usual two-dimensional intensity or texture images captured by charge-coupled device (CCD) cameras. Sensor interoperability is a major concern in large biometric installations, since it is expensive and time consuming to re-enroll a large number of subjects as the technology evolves and access to the subject population may become limited. In some cases, re-enrollment may be unavoidable and should be viewed as part of upgrading the infrastructure. To the extent possible, of course, representations of traits that perform well across existing and anticipated technologies are desirable. Human Interface Interoperability One aspect of interoperability is the development of standardized human interfaces that would allow the data subject to know what to expect when interacting with a biometric system and how to control the recognition process. Although other technology interfaces such as are found in automatic teller machines, automobiles, televisions, and self-service gasoline pumps have a level of standardization that allows transferring experience gained with one system to other systems, little has been done in this area for biometrics, and these mass-market interfaces can confuse even experienced users on occasion. More standardized user interfaces coupled with broader human factors testing would contribute to greater maturity in all biometric applications.14 SYSTEM LIFE-CYCLE ISSUES Biometric systems that are large in scale and that are expected to persist and be used for more than a short period of time face the same challenges as other large-scale technology implementations.15 Software and 14 For more on usability and biometric systems, see “Usability and Biometrics: Ensuring Successful Biometric Systems,” available at http://zing.ncsl.nist.gov/biousa/docs/Usability_and_Biometrics_final2.pdf. In particular, the report notes: In order to improve the usability of biometric systems, it is critical to take a holistic approach that considers the needs of users as well as the entire experience users will have with a system, including the hardware, software and instructional design of a system. Adopting a user-centric view of the biometric process is not only beneficial to the end users, but a user-centric view can also help to improve the performance and effectiveness of a system. 15 In the case of biometric systems, scale can refer to the number of sensors in the system, the number of comparisons being performed for a given unit time, the number of users (administrators and data subjects), the geographic distribution of the system, the potential number of data subjects, or any combination of these factors. The point here is that there is
OCR for page 69
Biometric Recognition: Challenges and Opportunities hardware have limited lifetimes and become obsolete. The maintenance and technology refresh requirements of biometric systems can be poorly understood and neglected as a result. Most biometric systems have basic needs, such as cleaning of the lenses or plates, sensor recalibration when the components or the environment changes, software updates to keep pace with hardware changes, and so on. Evolution of large-scale systems while in use requires careful pretesting to verify the ability to migrate from the old technology to the new and have both coexist in the system simultaneously. Some system components may change without maintaining backward compatibility. Technologies will change significantly over the expected lifespan of a system, and biometric components may need to be updated. In general, biometric systems may be similar to other computer-based systems in that useful lifetimes cannot be expected to surpass 5 to 7 years without becoming obsolete. Unlike some computer-based systems, however, biometric systems are likely to have a critical hardware component, making upgrades and replacements more logistically challenging than, say, pushing software updates to a networked information system. For example, consider the simple case of a single fingerprint sensor, deployed to provide data security (rather than convenience) by controlling access to a laptop computer. In normal operation, this application will be limited to repeated private interactions between the owner and machine for the machine’s full life. However, a sensor can fail from simply wearing out, from physical damage due to rough usage, from a dirty or otherwise inimical environment, or from intentional damage by an unauthorized user who obtains control of the machine. The fingerprint acquisition and/or fingerprint matching software, or the file with the enrolled biometric template—a software issue—can also be corrupted. What happens if the system can no longer be used to recognize the user? Is the user simply cut off, or are there alternative access options? And how are those alternative access options made clear to the user later in the system’s life cycle in a way that does not compromise the security that the system was deployed for in the first place? Quality control, especially in large-scale systems, is critically important. When engineering a biometric system, planning how to ensure continued high-quality performance is key. At the same time, mechanisms are needed to detect and accommodate degraded performance, should it occur. Related to this are issues of scalability, as mentioned previously. a significant difference between a very localized, small-scale application (such as facilitating quick entry for members of a local gym) and much larger systems (such as border control or social welfare systems). This section focuses primarily on the technology and engineering challenges associated with large-scale biometric systems.
OCR for page 70
Biometric Recognition: Challenges and Opportunities Can performance be maintained as the system scales up either in terms of the size of the user population, number of sensors, geographic distribution, size of the search space, or some other dimension? For a given system, what characteristics must be monitored and compensated for as a system grows? Possible areas of improvement include matcher improvement, collecting more data—for example, 10 fingers instead of 2 or other types of multibiometrics—and so on. Security is also a life-cycle consideration. As described in Chapter 1, security considerations are integral to the design of a biometrics system. Risks are not static, and changes in attack methods, potential exposure or compromise of biometric data, and the emergence of new vulnerabilities to threats need to be assessed periodically. Consider the example of access to a laptop controlled by a fingerprint sensor cited above. In high-confidence systems, sensor replacement, for example, poses a risk because a compromised sensor used to replace a legitimate sensor could allow an attacker to access the system. A draconian solution to this security problem would be to automatically monitor sensor performance, letting sensor failure or replacement initiate a data overwrite or physical destruction of the storage medium or of cryptographic keys used to encrypt storage on the machine. However, this approach could be detrimental to the owner unless exceptionally stringent backup systems were in place. Alternatively there could be a backup computer activation mechanism and password authentication for the owner or a computer locksmith. This would partially spoil the purpose of the fingerprint device, however, and the entrusted locksmith would have to be trustworthy. Furthermore, as with computers that are not biometrically protected, stored data lacking sufficient encryption might be accessed by physical removal of the data storage medium. The idea is not to examine or critique various approaches for dealing with such contingencies but to point out that even in this simple situation, numerous issues that would arise during the life cycle need to be resolved during the system design and engineering phase. Future-proofing, to the extent possible, and paths to technology refreshment can encompass issues such as software and hardware modularity—including sensors and matchers—as well as common interfaces and interoperable data formats. TEST AND EVALUATION Biometric systems are almost always components of larger systems designed to perform a business process. Testing and evaluating these larger systems depends on the biometric system, application, concept of operations, functional and performance requirements and on many deployment-specific factors that must be considered, including confor-
OCR for page 71
Biometric Recognition: Challenges and Opportunities mance to standards. (See Appendix E for a brief overview of the biometrics standards landscape.) The factors include the environment, maintenance, subject population, subject education, subject cooperation, and policy. Testing and evaluating of biometric systems can serve multiple purposes, including estimating performance under real-world usage of a biometric system; developing or examining metrics related to biometric systems; determining conditions that affect performance; supporting system procurement; accepting a system at the time of delivery to an operational site; and letting developers and program managers know about opportunities for system improvement. Testing can occur at many levels and might involve presenting original samples to a biometric system and recording the results, or might involve only presenting a standardized database of biometric characteristics to feature extraction and matching algorithms. More generally, testing and evaluation are broad terms. Aspects of systems that could be measured include technical performance and accuracy, throughput, interoperability, conformance (to standards or requirements), reliability, availability, maintainability, security and robustness against vulnerabilities, safety, usability, and public perceptions and acceptance. As this chapter describes, work in all of these areas is needed, although progress on the first three is further along than the others. The testing and evaluation of particular aspects of systems can be aimed at either assessing the performance of a particular system in isolation or comparing the performance of similar systems. For examples of test and evaluation practice in three real-world examples (the FBI’s IAFIS, Disney’s entrance control system, and the U.S. Army’s BAT system), see Appendix D. NIST divides testing of biometric systems into four general categories: conformance, scenario, interoperability, and technology testing. In addition, a full testing and analysis regimen includes operational and usability testing.16 Conformance testing addresses technical interface and data interchange standards. Scenario evaluations use volunteers to model data subject interaction with a system in a laboratory environment designed to model a target application. These scenario tests include system level testing focused on performance as well as human centered biometric sample collection testing. Interoperability testing addresses sensor interoperability and modality-specific templates and matchers. Technology testing evaluates feature/model extraction and comparison algorithms using a 16 See the NIST National Voluntary Laboratory Accreditation Program (NVLAP) Handbook 150-25 on Biometric Testing, which is available at http://ts.nist.gov/Standards/Accreditation/upload/NIST-HB150-25-2009.pdf and http://ts.nist.gov/Standards/Accreditation/bio-lap.cfm. As of July 2009, laboratory accreditation under NVLAP is available only for conformance testing and aspects of system testing under scenario testing.
OCR for page 72
Biometric Recognition: Challenges and Opportunities standardized database of biometric samples; sensors are tested using standardized test materials. Operational testing looks at an existing system in situ. Usability testing evaluates the effectiveness of interaction between the users and operators of a biometric system. Evaluation of biometric systems includes all of the factors noted above as well as the analysis and assessment of technology, scenario, and operational test results and of the training process. An evaluation plan generally begins with the purpose, application, concept of operations, functional and performance requirements, and development of relevant metrics. The data used in test and evaluation of biometric systems is an important piece of the T&E process. Biometric subsystem testing can be conducted using found data that mirrors some aspects of the application or conducted on an operationally deployed system (including recording the operational data for later evaluation).17 Depending on the purpose, there are best practices, such as the ISO/IEC 19795 series of standards for conducting tests and evaluations of biometric systems.18 Testing and evaluation are important throughout a system’s life cycle. While it may seem obvious to test typical deployment use of a biometric system, other phases of system use merit evaluation as well. Some biometric systems require a training phase—one where biometric samples are presented to the system and models are built to be used during recognition. This phase might be evaluated with a properly designed test. Another type of evaluation is a core technology evaluation to compare various matching algorithms on a common task using common data. Yet another example is to predict real-world performance by evaluating performance in a laboratory scenario. Regardless of which aspects of the system or system life cycle are under scrutiny, the testing discussed in this report is considerably more sophisticated than brute force tests aimed at quickly stressing a system or device to the point of failure. The evaluation approaches discussed here are intended to scientifically evaluate the performance of presumably properly working devices in various dimensions. At the same time, as 17 “Found data” is a term that applies to data collected for some other purpose by someone who is willing to make the data available for testing. Use of “found” biometric data is generally problematic because of Institutional Review Board concerns over limiting data use to the application for which it was collected. Nonetheless, there are several “found” databases available for testing. 18 Work along these lines includes efforts undertaken by the British government since 1999, NIST Speaker Recognition Evaluation since 1994, and ISO/IEC JTC1 SC37 since 2002. There is also NIST’s emerging National Voluntary Laboratory Accreditation Program (NVLAP) for biometrics (see the NIST NVLAP Draft Handbook 150-25), which establishes a mechanism for biometric testing laboratories that can be certified for different types of product tests; however, the methods for different products and applications had not yet been specified at the time of this writing.
OCR for page 73
Biometric Recognition: Challenges and Opportunities will be seen in the brief descriptions of the biometric system tests below, abstracting to general principles for evaluating a biometric system and developing appropriate operational tests and evaluative techniques based on those principles can be a challenge. Even with significant testing, great caution is warranted in generalizing from one system to another or extrapolating behavior of a system across environments or user populations. Usability Evaluations Biometric system evaluations have historically been centered on error rate and throughput estimations, and they have tended to neglect usability considerations and acceptance testing. However, for all systems, even covert, both measured error rates and throughput are dependent on human interaction with the system. For example, the NIST testing taxonomy mentioned previously emphasizes algorithm testing in technical tests and does not significantly focus on human interaction. NIST’s recently initiated assessment of human interactions with the biometric system offers a real opportunity to enhance biometric system testing.19 The NIST efforts focus on the user as data subject, that is, on the individual who is presenting a sample to the system for recognition. This could be broadened to include system operators and system administrators, and, in some cases, systems owners as users.20 Test and Evaluation Standards Biometric testing standards have evolved to address various forms of testing. Biometric performance testing and reporting of international standards, published as the ISO/IEC 19795 series of standards, evaluate biometric systems in terms of error rates and throughput rates.21 Metrics for the various error rates in biometric enrollment, verification, and identification are specified. Recommendations are given for evaluating performance through planning the evaluation; collection of enrollment, 19 Results of the NIST usability tests are publicly available on the biometrics and usability website at http://zing.ncsl.nist.gov/biousa/. 20 A recent workshop hosted by the NRC Computer Science and Telecommunications Board examined the broader issue of usability, security, and privacy. See National Research Council, Toward Better Usability, Security, and Privacy of Information Technology: Report of a Workshop, Washington, D.C.: The National Academies Press (2010). 21 The introduction to the standard notes that “technical performance testing seeks to determine error and throughput rates, with the goal of understanding and predicting the real-world error and throughput performance of biometric systems. The error rates include both false positive and false negative decisions, as well as failure-to-enroll and failure-to-acquire rates across the test population.” It should be noted that these standards do not fit the AFIS/ABIS system benchmarking and acceptance testing that governments perform.
OCR for page 74
Biometric Recognition: Challenges and Opportunities verification, or identification transaction data; analysis of error rates; and reporting and presentation of results. The principles apply to a wide range of biometric modalities, applications, and test purposes and to both offline and online testing methodologies. These principles are aimed at (1) avoiding bias due to inappropriate data collection or analytic procedures; (2) providing better estimates of field performance for the expended effort; and (3) clarifying the extent to which test results are applicable.22 Performance Assessment and Evaluation Performance—which goes beyond whether the system returns adequate recognition results for a given application—can be a critical feature of biometric systems, particularly at scale. In this sense, performance encompasses not just error rates but also throughput, reliability, and other features crucial to system success. For example, systems that are engineered so that performance can be dynamically monitored during testing and deployment should offer system administrators performance data throughout the operational life cycle. Similarly, the system development and testing communities would have an opportunity to work on ways to tune the operation of a biometric system to maintain performance metrics at acceptable levels in the face of a changing load and environment. The performance of systems depends on the performance and interrelationships of their components (including the data subjects and human operators). Component performance is best measured and understood by studies that limit the number of factors involved through experimentation or tightly constrained observation that adequately model the target application. Modeling, including simulation, also may be useful, but it must be grounded in observation. Even when the performance of a system depends in a simple manner on the performance of its components operating independently, only holistic study of the system can confirm that dependence. An understanding of performance and realistic planning for improvement require both analytical studies of individual system components and holistic studies of full system operation. Consequently, performance statistics cannot be accurately predicted for systems orders of magnitude larger than those previously studied or that have very different hardware/software/user interfaces. Characteristics that limit system performance or opportunities to improve it may reside with individual components or with the manner in which they interact within the system. However, improving the perfor- 22 Several organizations and institutions perform biometric testing, and there are several periodic large-scale tests in which many biometric vendors participate.
OCR for page 75
Biometric Recognition: Challenges and Opportunities mance of one component sometimes degrades the performance of a larger system in which it resides.23 There is no way to definitively determine the impact of component changes on system-level performance until the components have been inserted and the system is tested. System performance may depend critically on factors such as the current performance levels of other components, whether the performance characteristics of some components depend upon the outputs of other components, the nature and quality of system input material, and the characteristics of the environment in which the system operates. 23 Upgrading a face image detector in a facial recognition system is one example of how improving performance of a component can degrade the system. If the system can find more faces in more poses, then the number of off-angle and partial faces sent into the comparison process increases. This suboptimal data situation can lead to poor performance. Moving from lossy to loss-less compression is another example where a local improvement can degrade overall system effectiveness. See, for example, NIST, Effect of image size and compression on one-to-one fingerprint matching, NISTIR 7201 (February 2005), and J. Daugman and C. Downing, Effect of severe image compression on iris recognition performance, IEEE Transactions on Information Forensics and Security 3:1 (2008).