|
Comparison of Types of Testing
There are three commonly recognized categories of biometric tests:
Technology Tests, Scenario Tests and Operational Tests.
Technology tests (back to top)
are performed in a laboratory environment and generally focus on product software, rather than the product itself. They are designed to validate the underlying theory of biometric identification methods, and to identify areas of improvement for future research and design. Technology testing is the most common form of biometric testing and is normally used to produce a repeatable and scalable assessment of an algorithm/sensor using offline data processing, producing FRR, FAR, & EER results that can be illustrated readily in graphics such as the DET curve.
Technology tests can consist of, but are not limited to testing:
Algorithms: The output generated from enrollment or matching algorithms can be tested based on defined criteria. A specific example for a test of an algorithm is a conformance test of biometric data to published standards.
Sensors: Specific quantitative and qualitative values can be defined and tested on the output data of biometric sensors.
Comparison tests: Two or more devices can be compared and reviewed for specific criteria.
Testing is conducted using offline processing of data. This data will be representative of the data the biometric device acquires; however the data will be independent of the device and never seen before by the developers. Such data can be acquired from large databases. Some key attributes of technology tests are:
Large population size: Large sample population sizes can be obtained from databases to provide reliable testing results.
Repeatability: Since a defined set of data is used all technology tests can be repeated.
Research: There are no real-time demands for output in technology testing; time can be used in research and development. Exploratory tests can be conducted to record the effect design changes have to the output.
| Detection Error Trade-off (DET) Curve |
Receiver-Operator Curve |
 |
 |
Scenario Testing (back to top)
is carried out in an environment that models a real-world target application of interest. Each tested system will have its own acquisition sensor and will receive slightly different data. Consequently, if multiple systems are being compared, care will be required that data collection across all tested systems is in the same environment with the same population using the same policies. Test results will be repeatable only to the extent that the modeled scenario and the variables associated with operator group selection can be carefully controlled.
NBSP's enhanced scenario test procedures include a volunteer operator group of 500+ individuals available year round to evaluate the efficiency of biometric products. Demographic information can be modeled after the clients required user group to create a more accurate test scenario.The standard for scenario testing only requires that a group of testers come in for an initial enrollment and then come back some weeks later for another activation. Meanwhile, the bulk of the analysis is still derived from technology testing's computer simulations. In NBSP's approach he statistical method used by the test team to determine the number of volunteer operators required to achieve the most accurate results are those developed by Dr. Michael Schuckers of St. Lawrence University. NBSP's enhanced scenario testing employs 200-400 testers who come in six to eight times during the course of a QPL test. The resulting 10,000+ activations shifts the burden of analysis from theoretical computer simulations to thousands of live operator activations.
Operational Testing (back to top)
services provided by the Testing Services Division are an evaluation of a complete biometric system in a real world environment. Problems often arise once a new technology is taken out of the lab and placed in a real world environment. During the operational testing the NBSP Testing Services Division will be able to monitor and document these problems and provide expert recommendations. This test will determine if the overall performance of the system meets the requirements of the client.
To better serve the client, the Testing Service Division will act as the program manager and oversee all installation and configuration of the biometric equipment in the operational environment. In addition, the team will monitor and document every aspect of the biometric system, the environment, and generate a report that includes expert recommendations.
Practical Experience
In July 2004, the TRDC was the site for the U.S. Department of Homeland Security (DHS) e-passport interoperability testing session. Tests were conducted under the auspices of the DHS US-VISIT program to evaluate the interoperability of e-passports and passport readers. Over 50 companies and 18 countries were represented. The TRDC is equipped with the appropriate tools for testing biometric systems and products, and evaluating the results of such testing under published protocols that are fair, reasonable, and predictable with regard to the practices employed. The testing program is supported by a reference library housing all relevant technical and operational data related to the technologies and products under test.
|
