This Article 
 Bibliographic References 
 Add to: 
Visual Image Retrieval by Elastic Matching of User Sketches
February 1997 (vol. 19 no. 2)
pp. 121-132

Abstract—Effective image retrieval by content from database requires that visual image properties are used instead of textual labels to properly index and recover pictorial data. Retrieval by shape similarity, given a user-sketched template is particularly challenging, owing to the difficulty to derive a similarity measure that closely conforms to the common perception of similarity by humans. In this paper, we present a technique which is based on elastic matching of sketched templates over the shapes in the images to evaluate similarity ranks. The degree of matching achieved and the elastic deformation energy spent by the sketch to achieve such a match are used to derive a measure of similarity between the sketch and the images in the database and to rank images to be displayed. The elastic matching is integrated with arrangements to provide scale invariance and take into account spatial relationships between objects in multi-object queries. Examples from a prototype system are expounded with considerations about the effectiveness of the approach and comparative performance analysis.

[1] P.M. Anselone and P.J. Laurent, "A General Method for the Construction of Interpolating or Smoothing Spline-Functions," Numerische Matematik, vol. 12, pp. 66-82, 1968.
[2] E. Binaghi, I. Gagliardi, and R. Schettini, "Indexing and Fuzzy Logic-Based Retrieval of Color Images," IFIP Trans. A-7, Visual Database Systems II. Elsevier Pub., 1992.
[3] S.K. Chang, Q.Y. Shi, and C.W. Yan, "Iconic Indexing by 2D Strings," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, no. 3, Mar. 1987.
[4] S.K. Chang and E. Jungert, "Pictorial Data Management Based upon the Theory of Symbolic Projections," Journal of Visual Languages and Computing, vol. 2, no. 2, June 1991.
[5] S.K. Chang, C.W. Yan, D.C. Dimitroff, and T. Arndt, “An Intelligent Image Database System,” IEEE Trans. Software Eng., vol. 14, no. 5, pp. 681-688, May 1988.
[6] A. Del Bimbo, E. Vicario, and D. Zingoni, "A Spatial Logic for Symbolic Description of Image Contents," J. Visual Languages and Computing, vol. 5, no. 4, 1994.
[7] A.D. Bimbo, M. Campanai, and P. Nesi, “A Three-Dimensional Iconic Environment for Image Database Querying,” IEEE Trans. Software Eng., vol. 19, no. 10, pp. 997-1011, Oct. 1993.
[8] A. Del Bimbo, P. Pala, and S. Santini, "Visual Image Retrieval by Elastic Deformation of Object Shapes," Proc. IEEE VL'94, Int'l Symp. Visual Languages,St. Louis, Mo, Oct. 1994.
[9] R. Durbin and D. Willshaw, "An Analogue Approach to the Traveling Salesman Problem Using an Elastic Net Method," Nature, vol. 326, Apr. 1987.
[10] C. Faloutsos, M. Flickner, W. Niblack, D. Petkovic, W. Equitz, and R. Barber, "Efficient and Effective Querying by Image Content," Research Report 9453, IBM Research Div., Almaden Research Center, Aug. 1993.
[11] K. Hirata and T. Kato, "Query by Visual Example, Content-Based Image Retrieval," Advances in Database Technology—EDBT'92, A.Pirotte, C.Delobel, and G.Gottlob, eds., Lecture Notes on Computer Science, vol. 580.
[12] W.I. Grosky, R. Mehrotra, "Index-Based Object Recognition in Pictorial Data Management," Computer Vision Graphics and Image Processing, vol. 52, no. 3, pp. 416-436, 1990.
[13] E. Jungert, "The Observer's Point of View, an Extension of Symbolic Projections," Proc. Int'l Conf. Theories and Methods of Spatio-Temporal Reasoning in Geographic Space, Pisa, Sept. 1992, Lecture Notes in Computer Science. Springer-Verlag, 1992.
[14] C. Jacobs, A. Finkelstein, and D.H. Salesin, "Fast Multiresolution Image Querying," Proc. SIGGRAPH '95,Los Angeles, Calif., Aug. 1995.
[15] A. Kankanhalli, H.J. Zhangh, and C.Y. Low, "Using Texture for Image Retrieval," IEEE Proc. Int'l Conf. Automation Robotics and Computer Vision, 1994.
[16] S.Y. Lee and F.J. Hsu, "Spatial Reasoning and Similarity Retrieval of Images Using 2D-C String Knowledge Representation," Pattern Recognition, vol. 25, no. 3, 1992.
[17] S. Lee, M.K. Shan, and W.P. Yang, "Similarity Retrieval of Iconic Image Database," Pattern Recognition, vol. 22, no. 6, 1989.
[18] S.Y. Lee, M.C. Yang, and J.W. Chen, "Signature File as a Spatial Filter for Iconic Image Database," J. Visual Languages and Computing, vol. 3, pp. 373-397, 1992.
[19] R. Mehrotra and J.E. Gary, “Similar-Shape Retrieval in Shape Data Management,” Computer, vol. 28, no. 9, pp. 57-62, Sept. 1995.
[20] W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubino, "The QBIC Project, Querying Images by Content Using Color, Texture and Shape," Storage and Retrieval for Images and Video Databases, pp. 173-181, SPIE, Feb. 1993.
[21] M. Ohlssono, "Extensions and Explorations of the Elastic Arms Algorithm," Technical Report LU TP 92-28, Dept. of Theoretical Physics, Univ. of Lund, Sölvegatan 14A S-22362 Lund, Sweden, Dec. 1992.
[22] R.W. Picard, "The Society of Models for Video and Image Libraries," Technical Report 360, MIT Media Lab. Perceptual Computing Section, 1995.
[23] R.W. Picard and F. Liu, "A New Wold Ordering for Image Similarity," Proc. IEEE Conf. Acoustics, Speech, and Signal Processing, pp. V-129-V-132,Adelaide, Australia, Apr. 1994.
[24] M.J. Swain and D.H. Ballard, "Color Indexing," Int'l J. Computer Vision, vol. 7, no. 1, 1991.
[25] S.L. Tanimoto, "An Iconic/Symbolic Data Structuring Scheme," Pattern Recognition and Artificial Intelligence, C.H. Chen, ed. New York: Academic, 1976.
[26] A.N. Tihonov, "Regularization +of Incorrectly Posed Problems," Soviet Math. Doklady, vol. 4, pp. 1,624-1,627, 1963.
[27] A.L. Youille, K. Honda, and C. Petersono, "Particle Tracking by Deformable Templates," Proc. Int'l Joint Conf. Neural Networks, 1991.

Index Terms:
Image database, image retrieval by sketch, shape similarity-based retrieval, elastic matching.
Alberto Del Bimbo, Pietro Pala, "Visual Image Retrieval by Elastic Matching of User Sketches," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 121-132, Feb. 1997, doi:10.1109/34.574790
Usage of this product signifies your acceptance of the Terms of Use.