This Article 
 Bibliographic References 
 Add to: 
Shape Matching and Object Recognition Using Shape Contexts
April 2002 (vol. 24 no. 4)
pp. 509-522

We present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by 1) solving for correspondences between points on the two shapes, 2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest-neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. Results are presented for silhouettes, trademarks, handwritten digits, and the COIL data set.

[1] Y. Amit, D. Geman, and K. Wilder, “Joint Induction of Shape Features and Tree Classifiers,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1300-1305, Nov. 1997.
[2] S. Belongie, J. Malik, and J. Puzicha, “Matching Shapes,” Proc. Eighth Int'l. Conf. Computer Vision, pp. 454-461, July 2001.
[3] S. Belongie, J. Malik, and J. Puzicha, “Shape Context: A New Descriptor for Shape Matching and Object Recognition,” Advances in Neural Information Processing Systems 13: Proc. 2000 Conf., T.K. Leen, T.G. Dietterich, and V. Tresp, eds.. pp. 831-837, 2001.
[4] P.J. Bickel, “A Distribution Free Version of the Smirnov Two-Sample Test in the Multivariate Case,” Annals of Math. Statistics, vol. 40, pp. 1-23, 1969.
[5] L. Bookstein, "Principal Warps: Thin-Plate Splines and the Decomposition of Deformations," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 6, pp. 567-585, June 1989.
[6] F.L. Bookstein, Morphometric Tools for Landmark Data: Geometry and Biology. Cambridge Univ. Press, 1991.
[7] C. Burges and B. Schölkopf, “Improving the Accuracy and Speed of Support Vector Machines,” Advances in Neural Information Processing Systems 9: Proc. 1996 Conf., D.S. Touretzky, M.C. Mozer, and M. E. Hasselmo, eds. pp. 375-381, 1997.
[8] S. Carlsson, “Order Structure, Correspondence and Shape Based Categories,” Int'l Workshop Shape, Contour and Grouping, May 1999.
[9] H. Chui and A. Rangarajan, “A New Algorithm for Non-Rigid Point Matching,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 44-51, June 2000.
[10] T.F. Cootes, C.J. Taylor, D.H. Cooper, and J. Graham, "Active Shape Models—Their Training and Application," Computer Vision and Image Understanding, vol. 61, no. 1, pp. 38-59, Jan. 1995.
[11] C. Cortes and V. Vapnik, "Support Vector Networks," Machine Learning, vol. 20, no. 3, Sept. 1995, pp. 1-25.
[12] Nearest Neighbor (NN) Norms: (NN) Pattern Classification Techniques. B.V. Dasarathy, ed., IEEE Computer Soc., 1991.
[13] D. DeCoste and B. Schölkopf, “Training Invariant Support Vector Machines,” Machine Learning, to appear in 2002.
[14] J. Duchon, “Splines Minimizing Rotation-Invariant Semi-Norms in Sobolev Spaces,” Constructive Theory of Functions of Several Variables, W. Schempp and K. Zeller, eds., pp. 85-100, Berlin: Springer-Verlag, 1977.
[15] M. Fischler and R. Elschlager, “The Representation and Matching of Pictorial Structures,” IEEE Trans. Computers, vol. 22, no. 1, pp. 67-92, 1973.
[16] D. Gavrila and V. Philomin, Real-Time Object Detection for Smart Vehicles Proc. Seventh IEEE Int'l Conf. Computer Vision, pp. 87-93, 1999.
[17] Y. Gdalyahu and D. Weinshall, “Flexible Syntactic Matching of Curves and its Application to Automatic Hierarchical Classification of Silhouettes,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 12, pp. 1312-1328, Dec. 1999.
[18] F. Girosi, M. Jones, and T. Poggio, “Regularization Theory and Neural Networks Architectures,” Neural Computation, vol. 7, no. 2, pp. 219-269, 1995.
[19] S. Gold, A. Rangarajan, C.-P. Lu, S. Pappu, and E. Mjolsness, “New Algorithms for 2D and 3D Point Matching: Pose Estimation and Correspondence,” Pattern Recognition, vol. 31, no. 8, 1998.
[20] E. Goldmeier, “Similarity in Visually Perceived Forms,” Psychological Issues, vol. 8, no. 1, pp. 1-135, 1936/1972.
[21] Y.S. Chow, U. Grenander, and D.M. Keenan, HANDS. A Pattern-Theoretic Study Of Biological Shapes.New York: Springer-Verlag, 1991.
[22] M. Hagedoorn, “Pattern Matching Using Similarity Measures,” PhD thesis, Universiteit Utrecht, 2000.
[23] D.P. Huttenlocher, G.A. Klanderman, and W.J. Rucklidge, “Comparing Images Using the Hausdorff Distance,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 850-863, Sept. 1993.
[24] D. Huttenlocher, R. Lilien, and C. Olson, “View-Based Recognition Using an Eigenspace Approximation to the Hausdorff Measure,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 9, pp. 951-955, Sept. 1999.
[25] S. Jeannin and M. Bober, “Description of Core Experiments for MPEG-7 Motion/Shape,” Technical Report ISO/IEC JTC 1/SC 29/WG 11 MPEG99/N2690, MPEG-7, Seoul, Mar. 1999.
[26] A.E. Johnson and M. Hebert, “Recognizing Objects by Matching Oriented Points,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 684-689, 1997.
[27] D. Jones and J. Malik, “Computational Framework to Determining Stereo Correspondence from a Set of Linear Spatial Filters,” Image and Vision Computing, vol. 10, no. 10, pp. 699-708, Dec. 1992.
[28] R. Jonker and A. Volgenant, “A Shortest Augmenting Path Algorithm for Dense and Sparse Linear Assignment Problems,” Computing, vol. 38, pp. 325-340, 1987.
[29] D. Kendall, “Shape Manifolds, Procrustean Metrics and Complex Projective Spaces,” Bull. London Math. Soc., vol. 16, pp. 81-121, 1984.
[30] J.J. Koenderink and A.J. van Doorn, “The Internal Representation of Solid Shape with Respect to Vision,” Biological Cybernetics, vol. 32, pp. 211-216, 1979.
[31] M. Lades, J.C. Vorbruggen, J. Buhmann, J. Lange, C. von der Malsburg, R.P. Wurtz, and W. Konen, “Distortion Invariant Object Recognition in the Dynamic Link Architecture,” IEEE Trans. Computers, vol. 42, no. 3, pp. 300-311, Mar. 1993.
[32] Y. Lamdan, J.T. Schwartz, and H.J. Wolfson, "Affine invariant model-based object recognition," IEEE Trans. Robotics and Automation, vol. 6, pp. 578-589, 1990.
[33] L.J. Latecki, R. Lakämper, and U. Eckhardt, “Shape Descriptors for Non-Rigid Shapes with a Single Closed Contour,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 424-429, 2000.
[34] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[35] T. Leung, M. Burl, and P. Perona, "Finding Faces in Cluttered Scenes Using Labeled Random Graph Matching," Int'l Conf. Computer Vision, 1995, pp. 637-644.
[36] D.G. Lowe, “Object Recognition from Local Scale-Invariant Features,” Proc. Seventh Int'l. Conf. Computer Vision, pp. 1150-1157, Sept. 1999.
[37] J. Meinguet, “Multivariate Interpolation at Arbitrary Points Made Simple,” J. Applied Math. Physics (ZAMP), vol. 5, pp. 439-468, 1979.
[38] B. Moghaddam, T. Jebara, and A. Pentland, “Bayesian Face Recognition,” Pattern Recognition, vol. 33, no. 11, pp. 1771-1782, Nov, 2000.
[39] F. Mokhtarian, S. Abbasi, and J. Kittler, “Efficient and Robust Retrieval by Shape Content Through Curvature Scale Space,” Image Databases and Multi-Media Search, A.W.M. Smeulders and R. Jain, eds., pp. 51-58, World Scientific, 1997.
[40] H. Murase and S.K. Nayar, “Visual Learning and Recognition of 3-D Objects from Appearance,” Int'l J. Computer Vision, vol. 14, pp. 5-24, 1995.
[41] M. Oren, C. Papageorgiou, P. Sinha, E. Osuna, and T. Poggio, Pedestrian Detection Using Wavelet Templates Proc. Computer Vision and Pattern Recognition, pp. 193-199, June 1997.
[42] C.H. Papadimitriu and K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity. Prentice Hall, 1987.
[43] E. Persoon and K. Fu, “Shape Discrimination Using Fourier Descriptors,” IEEE Trans. Systems, Man and Cybernetics, vol. 7, no. 3, pp. 170-179, Mar. 1977.
[44] M.J.D. Powell, “A Thin Plate Spline Method for Mapping Curves into Curves in Two Dimensions,” Computational Techniques and Applications (CTAC '95), 1995.
[45] B.D. Ripley, “Modelling Spatial Patterns,” J. Royal Statistical Society, Series B, vol. 39, pp. 172-212, 1977.
[46] B.D. Ripley, Pattern Recognition and Neural Networks. Cambridge Univ. Press, 1996.
[47] E. Rosch, “Natural Categories,” Cognitive Psychology, vol. 4, no. 3, pp. 328-350, 1973.
[48] E. Rosch, C.B. Mervis, W.D. Gray, D.M. Johnson, and P. Boyes-Braem, “Basic Objects in Natural Categories,” Cognitive Psychology, vol. 8, no. 3, pp. 382-439, 1976.
[49] C. Schmid and R. Mohr, “Local Grayvalue Invariants for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 5, pp. 530-535, May 1997.
[50] S. Sclaroff and A.P. Pentland, Modal Matching for Correspondence and Recognition IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 6, pp. 545-561, 1995.
[51] G. Scott and H. Longuet-Higgins, “An Algorithm for Associating the Features of Two Images,” Proc. Royal Soc. London, vol. 244, pp. 21-26, 1991.
[52] L.S. Shapiro and J.M. Brady, “Feature-Based Correspondence: An Eigenvector Approach,” Image and Vision Computing, vol. 10, pp. 283-288, 1992.
[53] D. Sharvit, J. Chan, H. Tek, and B. Kimia, “Symmetry-Based Indexing of Image Databases,” J. Visual Comm. and Image Representation, vol. 9, no. 4, pp. 366-380, Dec. 1998.
[54] L. Sirovich and M. Kirby, “Low Dimensional Procedure for the Characterization of Human Faces,” J. Optical Soc. Am. A, vol. 4, no. 3, pp. 519-524, 1987.
[55] D.W. Thompson, On Growth and Form, Cambridge Univ. Press, 1917.
[56] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-96, 1991.
[57] S. Umeyama, “An Eigendecomposition Approach to Weighted Graph Matching Problems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, no. 5, pp. 695-703, Sept. 1988.
[58] R.C. Veltkamp and M. Hagedoorn, “State of the Art in Shape Matching,” Technical Report UU-CS-1999-27, Utrecht, 1999.
[59] T. Vetter, M.J. Jones, and T. Poggio, “A Bootstrapping Algorithm for Learning Linear Models of Object Classes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 40-46, 1997.
[60] G. Wahba, Spline Models for Observational Data. Soc. Industrial and Applied Math., 1990.
[61] A. Yuille, “Deformable Templates for Face Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 59-71, 1991.
[62] C. Zahn and R. Roskies, “Fourier Descriptors for Plane Closed Curves,” IEEE Trans. Computers, vol. 21, no. 3, pp. 269-281, Mar. 1972.

Index Terms:
shape, object recognition, digit recognition, correspondence problem, MPEG7, image registration, deformable templates
S. Belongie, J. Malik, J. Puzicha, "Shape Matching and Object Recognition Using Shape Contexts," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 4, pp. 509-522, April 2002, doi:10.1109/34.993558
Usage of this product signifies your acceptance of the Terms of Use.