This Article 
 Bibliographic References 
 Add to: 
Meta-Recognition: The Theory and Practice of Recognition Score Analysis
August 2011 (vol. 33 no. 8)
pp. 1689-1695
In this paper, we define meta-recognition, a performance prediction method for recognition algorithms, and examine the theoretical basis for its postrecognition score analysis form through the use of the statistical extreme value theory (EVT). The ability to predict the performance of a recognition system based on its outputs for each match instance is desirable for a number of important reasons, including automatic threshold selection for determining matches and nonmatches, and automatic algorithm selection or weighting for multi-algorithm fusion. The emerging body of literature on postrecognition score analysis has been largely constrained to biometrics, where the analysis has been shown to successfully complement or replace image quality metrics as a predictor. We develop a new statistical predictor based upon the Weibull distribution, which produces accurate results on a per instance recognition basis across different recognition problems. Experimental results are provided for two different face recognition algorithms, a fingerprint recognition algorithm, a SIFT-based object recognition system, and a content-based image retrieval system.

[1] J. Flavell and H. Wellman, "Metamemory," Perspectives on the Development of Memory and Cognition, J.R.V. Kail and J.W. Hagen, eds., pp. 3-33, Lawrence Erlbaum Assoc., 1988.
[2] M. Cox, "Metacognition in Computation: A Selected Research Review," Artificial Intelligence, vol. 169, no. 2, pp. 104-141, 2005.
[3] T. Riopka and T. Boult, "Classification Enhancement via Biometric Pattern Perturbation," Proc. Int'l Conf. Assoc. for Pattern Recognition Audio- and Video-Based Biometric Person Authentication, vol. 3546, pp. 850-859, 2005.
[4] W. Scheirer, A. Bendale, and T. Boult, "Predicting Biometric Facial Recognition Failure with Similarity Surfaces and Support Vector Machines," Proc. IEEE Workshop Biometrics, 2008.
[5] E. Tabassi, C. Wilson, and C. Watson, "Fingerprint Image Quality, NFIQ," NISTIR 7151, Nat'l Inst. of Standards and Tech nology, 2004.
[6] P. Grother and E. Tabassi, "Performance of Biometric Quality Evaluations," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 4, pp. 531-543, Apr. 2007.
[7] J.R. Beveridge, G. Givens, P.J. Phillips, and B. Draper, "Focus on Quality, Predicting FRVT 2006 Performance," Proc. Int'l Conf. Automatic Face and Gesture Recognition, 2008.
[8] P. Phillips and J.R. Beveridge, "An Introduction to Biometric-Completeness: The Equivalence of Matching and Quality," Proc. IEEE Third Int'l Conf. Biometrics: Theory, Applications, and Systems, pp. 1-5, Sept. 2009.
[9] J.R. Beveridge, "Face Recognition Vendor Test 2006 Experiment 4 Covariate Study," Proc. First Multiple Biometrics Grand Challenge Kick-Off Workshop, 2008.
[10] S. Furui, "Recent Advances in Speaker Recognition," Pattern Recognition Letters, vol. 18, no. 9, pp. 859-872, 1997.
[11] S. Tulyakov, Z. Zhang, and V. Govindaraju, "Comparison of Combination Methods Utilizing T-Normalization and Second Best Score Models," Proc. IEEE Workshop Biometrics, 2008.
[12] G. Aggarwal, N. Ratha, R. Bolle, and R. Chellappa, "Multi-Biometric Cohort Analysis for Biometric Fusion," Proc. IEEE Conf. Acoustics, Speech and Signal Processing, 2008.
[13] R. Auckenthaler, M. Carey, and H. Lloyd-Thomas, "Normalization for Text-Independent Speaker Verification Systems," Digital Signal Processing, vol. 10, pp. 42-54, 2000.
[14] N. Poh, A. Merati, and J. Kittler, "Adaptive Client-Impostor Centric Score Normalization: A Case Study in Fingerprint Verification," Proc. IEEE Third Int'l Conf. Biometrics: Theory, Applications, and Systems, 2009.
[15] N. Poh, A. Merati, and J. Kittler, "Making Better Biometric Decisions with Quality and Cohort Information: A Case Study in Fingerprint Verification," Proc. European Signal Processing Conf., 2009.
[16] Z. Shi, F. Kiefer, J. Schneider, and V. Govindaraju, "Modeling Biometric Systems Using the General Pareto Distribution (GDP)," Proc. SPIE Conf., 2008.
[17] J. Broadwater and R. Chellappa, "Adaptive Threshold Estimation via Extreme Value Theory," IEEE Trans. Signal Processing, vol. 58, no. 2, pp. 490-500, Feb. 2010.
[18] W. Li, X. Gao, and T. Boult, "Predicting Biometric System Failure," Proc. IEEE Int'l Conf. Computational Intelligence for Homeland Security and Personal Safety, pp. 57-64, 2005.
[19] W. Scheirer and T. Boult, "A Fusion-Based Approach to Enhancing Multi-Modal Biometric Recognition System Failure Prediction and Overall Performance," Proc. IEEE Second Int'l Conf. Biometrics: Theory, Applications, and Systems, 2008.
[20] D. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[21] G. Shakhnarovich, J. Fisher, and T. Darrell, "Face Recognition from Long-Term Observations," Proc. Seventh European Conf. Computer Vision, pp. 851-868, 2002.
[22] P. Grother and P. Phillips, "Models of Large Population Recognition Performance," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 68-75, 2004.
[23] S. Kotz and S. Nadarajah, Extreme Value Distributions: Theory and Applications, first ed., World Scientific Publishing Co., 2001.
[24] E. Gumbel, Statistical Theory of Extreme Values and Some Practical Applications. US Govt. Printing Office, 1954.
[25] NIST, NIST/SEMATECH e-Handbook of Statistical Methods. US Govt. Printing Office, 2008.
[26] S. Berman, "Limiting Distribution of the Maximum Term in Sequences of Dependent Random Variables," Annals of Math. Statistics, vol. 33, no. 3, pp. 894-908, 1962.
[27] NIST Biometric Scores Set, biometricscores /, 2004.
[28] K. Okada, J. Steffans, T. Maurer, H. Hong, E. Elagin, H. Neven, and C. von der Malsburg, "The Bochum/USC Face Recognition System and How It Fared in the FERET Phase III Test," Face Recognition: From Theory to Applications, H. Wechsler, P. Phillips, V. Bruce, F.F. Soulie, and T. Huang, eds., pp. 186-205, Springer-Verlag, 1998.
[29] D. Bolme, J.R. Beveridge, M. Teixeira, and B. Draper, "The CSU Face Identification Evaluation System: Its Purpose, Features, and Structure," Proc. Conf. Vision Systems, pp. 304-313, 2003.
[30] J. Geusebroek, G. Burghouts, and A. Smeulders, "The Amsterdam Library of Object Images," Int'l J. Computer Vision, vol. 61, no. 1, pp. 103-112, 2005.
[31] J. Almeida, A. Rocha, R. Torres, and S. Goldenstein, "Making Colors Worth More than a Thousand Words," Proc. ACM Symp. Applied Computing, pp. 1179-1185, 2008.
[32] R. Stehling, M. Nascimento, and A. Falcão, "A Compact and Efficient Image Retrieval Approach Based on Border/Interior Pixel Classification," Proc. ACM Conf. Information and Knowledge Management, pp. 102-109, 2002.
[33] W. Scheirer, A. Rocha, R. Micheals, and T. Boult, "Robust Fusion: Extreme Value Theory for Recognition Score Normalization," Proc. 11th European Conf. Computer Vision, pp. 481-495, 2010.

Index Terms:
Weibull distribution,content-based retrieval,face recognition,fingerprint identification,image fusion,image retrieval,object recognition,content-based image retrieval system,meta-recognition,performance prediction method,postrecognition score analysis,statistical extreme value theory,statistical EVT,automatic threshold selection,automatic algorithm selection,multialgorithm fusion,biometrics,image quality metrics,Weibull distribution,face recognition algorithm,fingerprint recognition algorithm,SIFT-based object recognition system,Portfolios,Probes,Weibull distribution,Prediction algorithms,Data models,Image recognition,Face recognition,extreme value theory.,Meta-recognition,performance modeling,multialgorithm fusion,object recognition,face recognition,fingerprint recognition,content-based image retrieval,similarity scores
"Meta-Recognition: The Theory and Practice of Recognition Score Analysis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 8, pp. 1689-1695, Aug. 2011, doi:10.1109/TPAMI.2011.54
Usage of this product signifies your acceptance of the Terms of Use.