This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Best-Case Results for Nearest-Neighbor Learning
June 1995 (vol. 17 no. 6)
pp. 599-608

Abstract—In this paper we propose a theoretical model for analysis of classification methods, in which the teacher knows the classification algorithm and chooses examples in the best way possible. We apply this model using the nearest-neighbor learning algorithm, and develop upper and lower bounds on sample complexity for several different concept classes. For some concept classes, the sample complexity turns out to be exponential even using this best-case model, which implies that the concept class is inherently difficult for the NN algorithm. We identify several geometric properties that make learning certain concepts relatively easy. Finally we discuss the relation of our work to helpful teacher models, its application to decision tree learning algorithms, and some of its implications for current experimental work.

[1] B.V. Dasarathy,NN (NN) Norms: NN Pattern Classification Techniques.Los Alamitos, Calif.: IEEE CS Press, 1991.
[2] D.W. Aha, D. Kibler, and M.K. Albert, “Instance-Based Learning Algorithms,” Machine Learning, vol. 6, pp. 37-66, 1991.
[3] S. Salzberg,Learning with Nested Generalized Exemplars,Norwell, Mass.: Kluwer Academic Publishers, 1990.
[4] S. Cost and S. Salzberg, "A Weighted Nearest Neighbor Algorithm for Learning with Symbolic Features," Machine Learning, Vol. 10, No. 1, Jan. 1993, pp. 57-78.
[5] T.M. Cover and P. Hart, "Nearest Neighbor Pattern Classification," Proc. IEEE Trans. Information Theory, pp. 21-27, 1967.
[6] P.A. Devijver,“An overview of asymptotic properties of NN rules,” Pattern Recognition in Practice.New York: Elsevier Science Publishers B.V., pp. 343-350, 1980.
[7] T. Cover,“Geometric and statistical properties of systems of linear inequalities,” IEEE Trans. Computers, vol. 14, pp. 326-334, 1965.
[8] M. Minsky and S. Papert,Perceptrons.Cambridge, Mass.: MIT Press, 1969.
[9] S. Salzberg,A. Delcher,D. Heath,, and S. Kasif,“Learning with a helpful teacher,” Tech. rep. 90/14, Dept. of Computer Science, Johns Hopkins Univ., 1990 (revised 1992).
[10] S. Salzberg,A. Delcher,D. Heath,, and S. Kasif,“Learning with a helpful teacher,” Proc. 12th Int’l Joint Conf. Artificial Intelligence, pp. 705-711,Sydney, Australia, Aug. 1991, Morgan Kaufmann.
[11] E.F. Fix and J. Hodges,“Discriminatory analysis: Small sample performance,” Tech. rep. project 21-49-004, report no. 11, USAF School of Aviation Medicine, Randolph Field, Tex., Aug. 1952.
[12] L. Devroye, “Automatic Pattern Recognition: A Study of the Probability of Error,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, no. 4, pp. 530-543, 1988.
[13] S. Salzberg, “A Nearest Hyperrectangle Learning Method,” Machine Learning, vol. 6, pp. 251-276, 1991.
[14] D. Angluin,“Learning regular sets from queries and counter-examples,” Information and Computation, vol. 75, pp. 87-106, 1987.
[15] S. Goldman and M. Kearns,“On the complexity of teaching,” Proc. Fourth Ann. Workshop Computational Learning Theory, pp. 303-314,Santa Cruz, Calif., Aug. 1991, Morgan Kaufmann.
[16] K. Romanik and S. Salzberg,“Testing orthogonal shapes,” Computational Geometry: Theory and Applications, vol. 5, pp. 33-49, 1995.
[17] P. Hart,“The condensed NN rule,” IEEE Trans. Information Theory, vol. 14, no. 3, May 1968.
[18] C. Swonger,“Sample set condensation for a condensed NN decision rule for pattern recognition,” Frontiers of Pattern Recognition, S. Watanabe, ed. New York: Academic Press, pp. 511-519, 1972.
[19] D. Wilson,“Asymptotic properties of NN rules using edited data,” IEEE Trans. Systems, Man, and Cybernetics, vol. 2, no. 3, pp. 408-421, July 1972.
[20] C.-L. Chang,“Finding prototypes for NN classifiers,” IEEE Trans. Computers, vol. 23, no. 11, pp. 1,179-1,184, Nov. 1974.
[21] G.L. Ritter, H.B. Woodruff, and S.R. Lowry, “An Algorithm for a Selective Nearest Neighbor Decision Rule,” IEEE Trans. Information Theory, vol. 21, no. 6, pp. 665-669, 1975.
[22] G. Toussaint,B. Bhattacharya,, and R. Poulsen,“The application of Voronoi diagrams to nonparametric decision rules,” Computer Science and Statistics: Proc. 16th Symp. on the Interface, L. Billard, ed., pp. 97-108,New York, Elsevier Science Publishers, 1984.
[23] B. Bhattacharya,R. Poulsen,, and G. Toussaint,“Application of proximity graphs to editing NN decision rules,” Tech. rep. SOCS 92.19, School of Computer Science, McGill University, Montreal, Dec. 1992.
[24] F.P. Preparata and M.I. Shamos, Computational Geometry. Springer-Verlag, 1985.
[25] B. Baker,E. Grosse,, and C. Rafferty,“Nonobtuse triangulation of polygons,” Discrete Computational Geometry, vol. 3, pp. 147-168, 1988.
[26] M. Bern and D. Eppstein,“Polynomial-size nonobtuse triangulation of polygons,” Proc. Seventh Ann. Symp. Computational Geometry, pp. 342-350,New York, 1991.
[27] M. Bern,S. Mitchell,, and J. Ruppert,“Linear-size nonobtuse triangulation of polygons,” Proc. 10th Ann. ACM Symp. Computational Geometry, pp. 221-230,Stony Brook, N.Y., 1994.
[28] D. Heath,A Geometric Framework for Machine Learning, PhD thesis, Johns Hopkins Univ., 1992.
[29] M. Bern,H. Edelsbrunner,D. Eppstein,S. Mitchell,, and S. Tan,“Edge insertion for optimal triangulations,” LATIN’92: First Latin American Symp. Theoretical Informatics, I. Simon, ed., pp. 46-60,Berlin, 1992, Springer-Verlag.
[30] J.R. Quinlan, C4.5: Programs for Machine Learning,San Mateo, Calif.: Morgan Kaufman, 1992.

Index Terms:
Machine learning, nearest-neighbor, geometric concepts.
Citation:
Steven Salzberg, Arthur L. Delcher, David Heath, Simon Kasif, "Best-Case Results for Nearest-Neighbor Learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 6, pp. 599-608, June 1995, doi:10.1109/34.387506
Usage of this product signifies your acceptance of the Terms of Use.