Pattern Recognition, International Conference on (2004)
Aug. 23, 2004 to Aug. 26, 2004
Ruud M. Bolle , IBM Thomas J. Watson Research Center, Yorktown Heights, NY
Nalini K. Ratha , IBM Thomas J. Watson Research Center, Yorktown Heights, NY
Sharath Pankanti , IBM Thomas J. Watson Research Center, Yorktown Heights, NY
Reporting the accuracy performance of pattern recognition systems (e.g., biometrics ID system) is a controversial issue and perhaps an issue that is not well understood [An introduction to evaluating biometric systems, Confidence interval and test size estimation for biometric data]. This work focuses on the research issues related to the oft used confidence interval metric for performance evaluation. Using a biometric (fingerprint) authentication system, we estimate the False Reject Rates and False Accept Rates of the system using a real fingerprint dataset. We also estimate confidence intervals of these error rates using a number of parametric (e.g., see [Confidence interval and test size estimation for biometric data]) and non-parametric (e.g., bootstrapping [1, 3, 6]) methods. We attempt to assess the accuracy of the confidence intervals based on estimate and verify strategy applied to repetitive random train/test splits of the dataset. Our experiments objectively verify the hypothesis that the traditional bootstrap and parametric estimate methods are not very effective in estimating the confidence intervals and magnitude of interdependence among data may be one of the reasons for their ineffective estimates. Further, we demonstrate that the resampling the subsets of the data samples (inspired from moving block bootstrap [Moving blocks Jackknife and Bootstrap capture weak dependence]) may be one way of replicating interdependence among the data; the bootstrapping methods using such subset resampling may indeed improve the accuracy of the estimates. Irrespective of the method of estimation, the results show that the (1 - α) 100% confidence intervals empirically estimated from the training set capture significantly smaller than (1 - α) fraction of the estimates obtained from the test set.
S. Pankanti, N. K. Ratha and R. M. Bolle, "An Evaluation of Error Confidence Interval Estimation Methods," Pattern Recognition, International Conference on(ICPR), Cambridge UK, 2004, pp. 103-106.