|
| This Article | ||
| ||
| Share | ||
| Bibliographic References | ||
| Add to: | ||
 Digg  Furl  Spurl  Blink  Simpy  Google  Del.icio.us  Y!MyWeb | ||
| Search | ||
| ||
| ASCII Text | x | ||
| K. Fukunaga, R.R. Hayes, "Estimation of Classifier Performance," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 10, pp. 1087-1101, October, 1989. | |||
| BibTex | x | ||
| @article{ 10.1109/34.42839, author = {K. Fukunaga and R.R. Hayes}, title = {Estimation of Classifier Performance}, journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence}, volume = {11}, number = {10}, issn = {0162-8828}, year = {1989}, pages = {1087-1101}, doi = {http://doi.ieeecomputersociety.org/10.1109/34.42839}, publisher = {IEEE Computer Society}, address = {Los Alamitos, CA, USA}, } | |||
| RefWorks Procite/RefMan/Endnote | x | ||
| TY - JOUR JO - IEEE Transactions on Pattern Analysis and Machine Intelligence TI - Estimation of Classifier Performance IS - 10 SN - 0162-8828 SP1087 EP1101 EPD - 1087-1101 A1 - K. Fukunaga, A1 - R.R. Hayes, PY - 1989 KW - pattern recognition; performance analysis; error estimation; classifier; finite test set; error expression; error analysis; estimation theory; pattern recognition VL - 11 JA - IEEE Transactions on Pattern Analysis and Machine Intelligence ER - | |||
An expression for expected classifier performance previously derived by the authors is applied to a variety of error estimation methods and a unified and comprehensive approach to the analysis of classifier performance is presented. After the error expression is introduced, it is applied to three cases: (1) a given classifier and a finite test set; (2) given test distributions a finite design set; and (3) finite and independent design and test sets. For all cases, the expected values and variances of the classifier errors are presented. Although the study of Case 1 does not produce any new results, it is important to confirm that the proposed approach produces the known results, and also to show how these results are modified when the design set becomes finite, as in Cases 2 and 3. The error expression is used to compute the bias between the leave-one-out and resubstitution errors for quadratic classifiers. The effect of outliers in design samples on the classification error is discussed. Finally, the theoretical analysis of the bootstrap method is presented for quadratic classifiers.
[1] D. M. Foley, "Considerations of sample and feature size,"IEEE Trans. Inform. Theory, vol. IT-18, pp. 618-626, 1972.
[2] P. A. Lachenbruch and R. M. Mickey, "Estimation of error rates in discriminant analysis,"Technometrics, vol. 10, no. 1, pp. 1-11, 1968.
[3] B. Efron, "Bootstrap methods: Another look at the jackknife,"Ann. Statist., vol. 7, pp. 1-26, 1979.
[4] L. M. Novak, "On the sensitivity of Bayes and Fisher classifiers in radar target detection," inProc. 18th Asilomar Conf. Circuits, Systems, and Computers, Nov. 5-7, 1984.
[5] S. Raudys and V. Pikelis, "On dimensionality, sample size, classification error, and complexity of classification algorithm in pattern recognition,"IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-2, no. 3, pp. 242-252, May 1980.
[6] G. T. Toussaint, "Bibliography on estimation of misclassification,"IEEE Trans. Inform. Theory, vol. 20, pp. 472-479, 1974.
[7] D. J. Hand, "Recent advances in error rate estimation,"Pattern Recog. Lett., vol. 5, pp. 335-346, 1986.
[8] A. K. Jain and B. Chandrasekaran, "Dimensionality and sample size considerations in pattern recognition practice,"Handbook of Statistics, vol. 2, P. R. Krishnaiah and L. N. Kanal, Eds. Amsterdam, The Netherlands: North-Holland, 1982, pp. 835-855.
[9] C. P. Han, "Distribution of discriminant function in circular models,"Inst. Statist., Math. Ann., vol. 22, no. 1, pp. 117-125, 1970.
[10] G. J. McLachlan, "Some expected values for the error rates of the sample quadratic discriminant function,"Australian J. Statist., vol. 17, no. 3, pp. 161-165, 1975.
[11] S. John, "Errors in discrimination,"Ann. Math. Statist., vol. 32, pp. 1125-1144, 1961.
[12] K. Fukunaga and R. R. Hayes, "Effects of sample size in classifier design,"IEEE Trans. Pattern Anal. Machine Intell., vol. 11, no. 8, pp. 873-885, Aug. 1989.
[13] K. Fukunaga,Introduction to Statistical Pattern Recognition. New York: Academic, 1972.
[14] S. Raudys, "Comparison of the estimates of the probability of misclassification," inProc. VIJCPR, Kyoto, Japan, 1978, pp. 280-282.
[15] A. K. Jain, R. C. Dubes, and C. C. Chen, "Bootstrap techniques for error estimation,"IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-9, no. 9, pp. 628-636, 1987.