National Biometric Security Project

Performance Testing

Performance Testing

NBSP Class 1 Performance tests utilize scenario test procedures in a setting comparable to an office environment. Two copies of the product are placed in the Biometric Test Center (BTC) for testing by volunteer operators in a simulated operating environment. To help determine how the product will perform in its intended normal application environment. The Performance Test evaluates the ability of a biometric device to operate against a set of performance levels, also known as Common Performance Standards (CPS).

To conduct these tests NBSP recruits and maintains a Volunteer Operator pool of 200-500 people to provide as reasonable a demographic and ethnic cross section of the general population as possible for ages 18-80 based on the US National population (as reported by the US Census Bureau, 2000 in the below graphs). Each Operator visits the BTC at least once each week for several rounds of testing on actual products installed on a wall or suitable surface as they would be in actual operation. During this period each device is activated a minimum of 10,000 times. The performance measures, determined by actual activations as opposed to theoretical computer analyses, measure False Accept, False Reject, Failure to Acquire , Failure to Enroll and Throughput rates.

Proprietary test management software controls the test flow to assure accurate data collection and to minimize data collection errors. The following are the four main CPS each device will be tested against:

Table 1: Common Performance Standards (CPS)

CPS	How Computed	How Reported	Target Value
False Accept Rate	A false accept occurs when a person not enrolled on a particular device is "recognized" or accepted by it. The FAR is computed as the number of false accepts observed on a device by the total number of imposter decisions made during the test period. For a family of devices, it is the total number of false accepts on all such devices divided by the total number of imposter decisions made by all the devices.	As a percentage of total imposter decisions made by the device. As a percentage of all imposter decisions made on all devices in the tested family.	≤0.5%
False Reject Rate* + Failure To Acquire Rate	A false reject occurs when a person, enrolled on a particular device and submitting a correct PIN, is rejected or not recognized. A failure to acquire rate is the frequency that the device fails to acquire biometric information required for a decision. For a family of devices, it is the total of false rejects plus the failed attempts to acquire on all such devices divided by the total number of genuine decisions made by all the devices.	As a percentage of total genuine decisions made by the device. As a percentage of all genuine decisions made on all devices in the tested family.	≤5.0%
Failure To Enroll Rate	A failure to enroll rate is the failure of the biometric system to form a proper enrolment template for an end-user. For a family of devices, it is the total of failed attempts to enroll on all such devices divided by the total number of enrollment attempts made by users on all the devices.	As a percentage of all user enrollments attempted on an individual device. As a percentage of all user enrollments attempted on all devices in the tested family.	≤5.0%
Throughput Rate	The rate, in elapsed time (seconds or decimal seconds), at which a biometric sub-system or component will process one individual for access or transaction. The time starts when the individual presents his biometric feature for processing, including any action required to retrieve the file for matching, and stops when the match/non-match decision signal is indicated. The throughput rate is the mean time computed for a device for all transactions during the test period. The throughput rate for a family of devices is the mean time of all devices for all transactions during the test period.	For a device, the mean ±_ throughput time for that device. For a family of devices, the mean ±? throughput time for all such devices for the duration of the test period.	<10

*For the purposes of the QPL, the FRRs will be assumed to include instances of Failure to Acquire. It is beyond the means of the Class 1 testing to distinguish between the two types of errors. All possible measures will be taken to minimize FTAs.