This Article 
 Bibliographic References 
 Add to: 
Estimation of Classifier Performance
October 1989 (vol. 11 no. 10)
pp. 1087-1101

An expression for expected classifier performance previously derived by the authors is applied to a variety of error estimation methods and a unified and comprehensive approach to the analysis of classifier performance is presented. After the error expression is introduced, it is applied to three cases: (1) a given classifier and a finite test set; (2) given test distributions a finite design set; and (3) finite and independent design and test sets. For all cases, the expected values and variances of the classifier errors are presented. Although the study of Case 1 does not produce any new results, it is important to confirm that the proposed approach produces the known results, and also to show how these results are modified when the design set becomes finite, as in Cases 2 and 3. The error expression is used to compute the bias between the leave-one-out and resubstitution errors for quadratic classifiers. The effect of outliers in design samples on the classification error is discussed. Finally, the theoretical analysis of the bootstrap method is presented for quadratic classifiers.

[1] D. M. Foley, "Considerations of sample and feature size,"IEEE Trans. Inform. Theory, vol. IT-18, pp. 618-626, 1972.
[2] P. A. Lachenbruch and R. M. Mickey, "Estimation of error rates in discriminant analysis,"Technometrics, vol. 10, no. 1, pp. 1-11, 1968.
[3] B. Efron, "Bootstrap methods: Another look at the jackknife,"Ann. Statist., vol. 7, pp. 1-26, 1979.
[4] L. M. Novak, "On the sensitivity of Bayes and Fisher classifiers in radar target detection," inProc. 18th Asilomar Conf. Circuits, Systems, and Computers, Nov. 5-7, 1984.
[5] S. Raudys and V. Pikelis, "On dimensionality, sample size, classification error, and complexity of classification algorithm in pattern recognition,"IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-2, no. 3, pp. 242-252, May 1980.
[6] G. T. Toussaint, "Bibliography on estimation of misclassification,"IEEE Trans. Inform. Theory, vol. 20, pp. 472-479, 1974.
[7] D. J. Hand, "Recent advances in error rate estimation,"Pattern Recog. Lett., vol. 5, pp. 335-346, 1986.
[8] A. K. Jain and B. Chandrasekaran, "Dimensionality and sample size considerations in pattern recognition practice,"Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 835-855.
[9] C. P. Han, "Distribution of discriminant function in circular models,"Inst. Statist., Math. Ann., vol. 22, no. 1, pp. 117-125, 1970.
[10] G. J. McLachlan, "Some expected values for the error rates of the sample quadratic discriminant function,"Australian J. Statist., vol. 17, no. 3, pp. 161-165, 1975.
[11] S. John, "Errors in discrimination,"Ann. Math. Statist., vol. 32, pp. 1125-1144, 1961.
[12] K. Fukunaga and R. R. Hayes, "Effects of sample size in classifier design,"IEEE Trans. Pattern Anal. Machine Intell., vol. 11, no. 8, pp. 873-885, Aug. 1989.
[13] K. Fukunaga,Introduction to Statistical Pattern Recognition. New York: Academic, 1972.
[14] S. Raudys, "Comparison of the estimates of the probability of misclassification," inProc. VIJCPR, Kyoto, Japan, 1978, pp. 280-282.
[15] A. K. Jain, R. C. Dubes, and C. C. Chen, "Bootstrap techniques for error estimation,"IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-9, no. 9, pp. 628-636, 1987.

Index Terms:
pattern recognition; performance analysis; error estimation; classifier; finite test set; error expression; error analysis; estimation theory; pattern recognition
K. Fukunaga, R.R. Hayes, "Estimation of Classifier Performance," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 10, pp. 1087-1101, Oct. 1989, doi:10.1109/34.42839
Usage of this product signifies your acceptance of the Terms of Use.