This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
On the Behavior of Artificial Neural Network Classifiers in High-Dimensional Spaces
May 1996 (vol. 18 no. 5)
pp. 571-574

Abstract—It is widely believed in the pattern recognition field that when a fixed number of training samples is used to design a classifier, the generalization error of the classifier tends to increase as the number of features gets large. In this paper, we will discuss the generalization error of the artificial neural network (ANN) classifiers in high-dimensional spaces, under a practical condition that the ratio of the training sample size to the dimensionality is small. Experimental results show that the generalization error of ANN classifiers seems much less sensitive to the feature size than 1-NN, Parzen and quadratic classifiers.

[1] R.P. Lippmann, "An Introduction to Computing with Neural Nets," IEEE Acoustics, Speech, and Signal Processing Magazine, vol. 4, pp. 4-22, Apr. 1987.
[2] R.P. Lippmann, "Pattern Classification Using Neural Networks," IEEE Communications Magazine, pp. 47-64, Nov. 1989.
[3] S.J. Raudys and A.K. Jain, "Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 252-264, 1991.
[4] A.K. Jain and B. Chandrasekaran, "Dimensionality and Sample Size Considerations in Pattern Recognition Practice," Handbook of Statistics 2, P.R. Krishnaiah and L.N. Kanal, eds., pp. 835-855. NorthHolland, 1982.
[5] S. Raudys and A.K. Jain, "Small Sample Size Problems in Designing Artificial Neural Networks," Artificial Neural Networks and Statistical Pattern Recognition Old and New Connections, I.K. Sethi and A.K. Jain, eds., pp. 33-50. Elsevier Science Publishers, 1991.
[6] S. Raudys, "On Dimensionality, Learning Sample Size and Complexity of Classification Algorithms," Proc. Third Int'l Conf. Pattern Recognition, pp. 166-169, 1976.
[7] K. Fukunaga, Introduction to Statistical Pattern Recognition, second edition. Academic Press, 1990.
[8] L. Kanal and B. Chandrasekaran, "On Dimensionality and Sample Size in Statistical Pattern Recognition," Pattern Recognition, pp. 238-255, 1971.
[9] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning Internal Representations by Error Propagation," Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1: Foundations, D.E. Rumelhart and J.L. McClelland et al., eds., chapter 8, pp. 318-362.Cambridge, Mass.: MIT Press, 1986.
[10] A.K. Jain and M.D. Ramaswami, "Classifier Design with Parzen Windows," Pattern Recognition and Artificial Intelligence, E.S. Gelsema and L.N. Kanal, eds., pp. 211-228. Elsevier Science Publishers, 1988.
[11] K. Fukunaga and D.M. Hummels, "Bayes Error Estimation Using Parzen and k-NN Procedures," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, pp. 634-643, 1987.
[12] P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach. Prentice Hall, 1982.
[13] W.Y. Huang and R.P. Lippmann, "Comparisons Between Neural Net and Conventional Classifiers," Proc. IEEE Int'l Conf. Neural Networks, pp. IV485-IV493, 1987.
[14] A. Khotanzad and J.-H. Lu, "Distortion Invariant Character Recognition by a Multi-Layer Perceptron and Back-Propagation Learning," Proc. IEEE Int'l Conf. Neural Networks, pp. I625-I632, 1988.
[15] J. Van Ness, "On the Dominance of Non-Parametric Bayes Rule Discriminant Algorithms in High Dimensions," Pattern Recognition, vol. 12, pp. 355-368, 1980.
[16] K. Fukunaga and D.M. Hummels, "Bias of Nearest Neighbor Estimates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, pp. 103-112, 1987.
[17] K. Fukunaga and R.R. Hayes, "Effects of Sample Size in Classifier Design," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 8, pp. 873-885, Aug. 1989.
[18] R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis. John Wiley&Sons, 1973.
[19] A. Khotanzad and J.-H. Lu, "Shape and Texture Recognition by a Neural Network," Artificial Neural Networks and Statistical Pattern Recognition Old and New Connections, I.K. Sethi and A.K. Jain, eds., pp. 109-131. Elsevier Science Publishers, 1991.
[20] S. Amari, N. Fujita, and S. Shinomoto, "Four Types of Learning Curves," Neural Computation, vol. 4, pp. 605-618, 1992.
[21] E.B. Baum, "When Are k-Nearest Neighbor and Back Propagation Accurate for Feasible Sized Sets of Examples?" Proc. EURASP Workshop Neural Networks, L.B. Almeida and C.J. Wellekens, eds., Delft Univ. of Tech nology, pp. 2-25, 1990.
[22] D. Cohn and G. Tesauro, "How Tight are the Vapnik-Chervonenkis Bounds?" Neural Computation, vol. 4, pp. 249-269, 1992.
[23] S. Raudys, "Why do Multilayer Perceptrons Have Favorable Small Sample Properties?" Pattern Recognition in Practice 4, E.S. Gelsema and L.N. Kanal, eds., pp. 287-298. Elsevier Science B.V., 1994.
[24] R.P.W. Duin, "Superlearning Capabilities of Neural Networks," Proc. Eighth Scandinavian Conf. Image Analysis, Tromso, pp. 547-554, 1993.
[25] S. Raudys, "On the Effectiveness of Parzen Window Classifier," Informatica, vol. 2, no. 3, pp. 434-454, 1991.
[26] S. Raudys, "On Shape of Pattern Error Function. Initializations and Intrinsic Dimensionality in ANN Classifier Design," Informatica, vol. 4, no. 3-4, pp. 360-383, 1993.
[27] S. Raudys, "Unexpected Small Sample Properties of the Linear Perceptrons," Technical report on research activities in LAFORIA-IBP, Univ. PARIS VI, LAFORIA 95/17, 1995.
[28] G.F. Hughes, "On the Mean Accuracy of Statistical Pattern Recognizers," IEEE Trans. Information Theory, vol. 14, pp. 55-63, 1968.

Index Terms:
Artificial neural networks, generalization error, dimensionality, training sample size, peaking phenomenon, 1-NN classifier, Parzen classifier.
Citation:
Yoshihiko Hamamoto, Shunji Uchimura, Shingo Tomita, "On the Behavior of Artificial Neural Network Classifiers in High-Dimensional Spaces," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, no. 5, pp. 571-574, May 1996, doi:10.1109/34.494648
Usage of this product signifies your acceptance of the Terms of Use.