This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Content-Based Image Retrieval at the End of the Early Years
December 2000 (vol. 22 no. 12)
pp. 1349-1380

Abstract—The paper presents a review of 200 references in content-based image retrieval. The paper starts with discussing the working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap. Subsequent sections discuss computational steps for image retrieval systems. Step one of the review is image processing for retrieval sorted by color, texture, and local geometry. Features for retrieval are discussed next, sorted by: accumulative and global features, salient points, object and shape features, signs, and structural combinations thereof. Similarity of pictures and objects in pictures is reviewed for each of the feature types, in close connection to the types and means of feedback the user of the systems is capable of giving by interaction. We briefly discuss aspects of system engineering: databases, system architecture, and evaluation. In the concluding section, we present our view on: the driving force of the field, the heritage from computer vision, the influence on computer vision, the role of similarity and of interaction, the need for databases, the problem of evaluation, and the role of the semantic gap.

[1] P. Aigrain, H. Zhang, and D. Petkovic, “Content-Based Representation and Retrieval of Visual Media: A State of the Art Review,” Multimedia Tools and Applications, vol. 3, pp. 179-202, 1996.
[2] S. Aksoy and R. Haralick, “Graph-Theoretic Clustering for Image Grouping and Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. 63-68, 1999.
[3] R. Alferez and Y-F. Wang, “Geometric and Illumination Invariants for Object Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 6, pp. 505-536, June 1999.
[4] D. Androutsos, K.N. Plataniotis, and A.N. Venetsanopoulos, “A Novel Vector-Based Approach to Color Image Retrieval Using a Vector Angular-Based Distance Measure,” Image Understanding, vol. 75, nos. 1-2, pp. 46-58, 1999.
[5] E. Angelopoulou and L.B. Wolff, “Sign of Gaussian Curvature from Curve Orientation in Photometric Space,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 10, pp. 1,056-1,066, Oct.. 1998.
[6] L. Armitage and P. Enser, “Analysis of User Need in Image Archives,” J. Information Science, vol. 23, no. 4, pp. 287-299, 1997.
[7] S. Arya, D.M. Mount, N.S. Netanyahu, R. Silverman, and A.Y. Wu, “An Optimal Algorithm for Approximate Nearest Neighborhood Searching,” Proc. Symp. Discrete Algorithms, pp. 573-582, 1994.
[8] F.G. Ashby and N.A. Perrin, “Toward a Unified Theory of Similarity and Recognition,” Psychological Rev., vol. 95, no. 1, pp. 124-150, 1988.
[9] D. Ashlock and J. Davidson, “Texture Synthesis with Tandem Genetic Algorithms Using Nonparametric Partially Ordered Markov Models,” Proc. Congress on Evolutionary Computation, pp. 1,157-1,163, 1999.
[10] J. Assfalg, A. del Bimbo, and P. Pala, “Using Multiple Examples for Content Based Retrieval,” Proc. Int'l Conf. Multimedia and Expo, 2000.
[11] R. Basri, L. Costa, D. Geiger, and D. Jacobs, “Determining the Similarity of Deformable Shapes,” Vision Research, vol. 38, nos. 15-16, pp. 2,365-2,385, 1998.
[12] S. Berretti, A. Del Bimbo, and E. Vicario, “Modeling Spatial Relationships between Color Sets,” Proc. IEEE Int'l Workshop Content Based Access of Image and Video Libraries (CBAIVL '00), June 2000.
[13] Database Techniques for Pictorial Applications, Lecture Notes in Computer Science, A. Blaser, ed., vol. 81, Springer Verlag GmbH, 1979.
[14] T. Bozkaya and M. Ozsoyoglu, “Distance-Based Indexing for High-Dimensional Metric Spaces,” Proc. SIGMOD Int'l Conf. Management of Data, pp. 357-368, 1997.
[15] L. Brown and L. Gruenwald, “Tree-Based Indexes for Image Data,” J. Visual Comm. and Image Representation, vol. 9, no. 4, pp. 300-313, 1998.
[16] R. Brunelli, O. Mich, and C.M. Modena, “A Survey on the Automatic Indexing of Video Data,” J. Visual Comm. and Image Representation, vol. 10, pp. 78-112, 1999.
[17] G. Bucci, S. Cagnoni, and R. De Dominicis, “Integrating Content-Based Retrieval in a Medical Image Reference Database,” Computerized Medical Imaging and Graphics, vol. 20, no. 4, pp. 231-241, 1996.
[18] H. Burkhardt and S. Siggelkow, “Invariant Features for Discriminating between Equivalence Classes,” Nonlinear Model-Based Image Video Processing and Analysis, John Wiley and Sons, 2000.
[19] D. Campbell and J. Stanley, Experimental and Quasi-Experimental Designs for Research. Rand McNally College Publishing, 1963.
[20] C. Carson, S. Belongie, H. Greenspan, and J. Malik, “Region-Based Image Querying,” Proc. Int'l Workshop Content-Based Access of Image and Video libraries, 1997.
[21] S.-F. Chang, J.R. Smith, M. Beigi, and A. Benitez, “Visual Information Retrieval from Large Distributed Online Repositories,” Comm. ACM, vol. 40, no. 12, pp. 63-71, 1997.
[22] S.-K. Chang and A.D. Hsu, “Image-Information Systems—Where Do We Go from Here?“ IEEE Trans. Knowledge and Data Eng., vol. 4, no. 5, pp. 431-442, Oct. 1992.
[23] S.K. Chang, Q.Y. Shi, and C.W. Yan, “Iconic Indexing by 2-D Strings,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, no. 3, pp. 413-427, July 1987.
[24] H. Chen, B. Schatz, T. Ng, J. Martinez, A. Kirchhoff, and C. Lim, “A Parallel Computing Approach to Creating Engineering Concept Spaces for Semantic Retrieval: The Illinois Digital Library Initiative Project,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 771-782, Aug. 1996.
[25] H. Choi and R. Baraniuk, “Multiscale Texture Segmentation Using Wavelet-Domain Hidden Markov Models,” Proc. 32nd Asilomar Conf. Signals, Systems, and Computers, vol. 2, pp. 1,692-1,697, 1998.
[26] C.K. Chui, L. Montefusco, and L. Puccio, Wavelets: Theory, Algorithms, and Applications. Academic Press, 1994.
[27] P. Ciaccia, M. Patella, and P. Zezula, “M-Tree: An Efficient Access Method for Similarity Search in Metric Spaces,” Proc. Int'l Conf. Very Large Data Bases, 1997.
[28] G. Ciocca and R. Schettini, “Using a Relevance Feedback Mechanism to Improve Content Based Image Retrieval,” Proc. Visual '99: Information and Information Systems, pp. 107-114, 1999.
[29] P. Correira and F. Pereira, “The Role of Analysis in Content-Based Video Coding and Indexing,” Signal Processing, vol. 66, no. 2, pp. 125-142, 1998.
[30] J.M. Corridoni, A. del Bimbo, and P. Pala, “Image Retrieval by Color Semantics,” Multimedia Systems, vol. 7, pp. 175-183, 1999.
[31] I.J. Cox, M.L. Miller, T.P. Minka, and T.V. Papathomas, “The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Pychophysical Experiments,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 20-37, 2000.
[32] G. Csurka and O. Faugeras, “Algebraic and Geometrical Tools to Compute Projective and Permutation Invariants,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 1, pp. 58-65, Jan. 1999.
[33] J.F. Cullen, J.J. Hull, and P.E. Hart, “Document Image Database Retrieval and Browsing Using Texture Analysis,” Proc. Fourth Int'l Conf. Document Analysis and Recognition, pp. 718-721, 1997.
[34] I. Daubechies,“Ten lectures on wavelets,” SIAM CBMS-61, 1992.
[35] J.S. De Bonet and P. Viola, “Texture Recognition Using a Non-Parametric Multi-Scale Statistical Model,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1998.
[36] A. del Bimbo and P. Pala, “Visual Image Retrieval by Elastic Matching of User Sketches,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 121-132, Feb. 1997.
[37] A. Dempster, N. Laird, and D. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc., vol. 39, no. 1, pp. 1-38, 1977.
[38] D. Dubois and H. Prade, “A Review of Fuzzy Set Aggregation Connectives,” Information Sciences, vol. 36, pp. 85-121, 1985.
[39] J.P. Eakins, J.M. Boardman, and M.E. Graham, “Similarity Retrieval of Trademark Images,” IEEE Multimedia, vol. 5, no. 2, pp. 53-63, Apr.-June 1998.
[40] B. Eberman, B. Fidler, R. Ianucci, C. Joerg, L. Kontothanassis, D.E. Kovalcin, P. Moreno, M.J. Swain, and J.-M. van Thong, “Indexing Multimedia for the Internet,” Proc. Visual '99: Information and Information Systems, pp. 195-202, 1999.
[41] F. Ennesser and G. Medioni, “Finding Waldo, or Focus of Attention Using Local Color Information,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp. 805-809, Aug. 1995.
[42] P.G.B. Enser, “Pictorial Information Retrieval,” J. Documentation, vol. 51, no. 2, pp. 126-170, 1995.
[43] C. Esperanca and H. Samet, “A Differential Code for Shape Representation in Image Database Applications,” Proc. Int'l Conf. Image Processing, 1997.
[44] L.M. Kaplan et al., “Fast Texture Database Retrieval Using Extended Fractal Features,” Storage and Retrieval for Image and Video Databases, VI, vol. 3,312, pp. 162-173, SPIE Press, 1998.
[45] R. Fagin, “Combining Fuzzy Information from Multiple Systems,” J. Computer Systems Science, vol. 58, no. 1, pp. 83-99, 1999.
[46] C. Faloutsos and K.I. Lin, “Fastmap: A Fast Algorithm for Indexing, Data-Mining and Visualization of Traditional and Multimedia Datasets,” Proc. SIGMOD, Int'l Conf. Management of Data, pp. 163-174, 1995.
[47] G.D. Finlayson, “Color in Perspective,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 10, pp. 1,034-1,038, Oct. 1996.
[48] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” IEEE Computer, 1995.
[49] D.A. Forsyth, “A Novel Algorithm for Color Constancy,” Int'l J. Computer Vision, vol. 5, no. 1, pp. 5-36, 1990.
[50] D.A. Forsyth and M.M. Fleck, “Automatic Detection of Human Nudes,” Int'l J. Computer Vision, vol. 32, no. 1, pp. 63-77, 1999.
[51] G. Frederix, G. Caenen, and E.J. Pauwels, “PARISS: Panoramic, Adaptive and Reconfigurable Interface for Similarity Search,” Proc. Int'l Conf. Image Processing, 2000.
[52] G. Frederix and E.J. Pauwels, “Automatic Interpretation Based on Robust Segmentation and Shape Extraction,” Proc. Visual '99: Information and Information Systems, pp. 769-776, 1999.
[53] C.-S. Fuh, S.-W. Cho, and K. Essig, “Hierarchical Color Image Region Segmentation for Content-Based Image Retrieval System,” IEEE Trans. Image Processing, vol. 9, no. 1 pp. 156-163, 2000.
[54] B. Funt and G. Finlayson, "Color Constant Color Indexing," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 5, pp. 522-529, May 1995.
[55] J.M. Geusebroek, A.W.M. Smeulders, and R. van den Boomgaard, “Measurement of Color Invariants,” Proc. Computer Vision and Pattern Recognition, 2000.
[56] T. Gevers and A.W.M. Smeulders, “Content-Based Image Retrieval by Viewpoint-Invariant Image Indexing,” Image and Vision Computing, vol. 17, no. 7, pp. 475-488, 1999.
[57] T. Gevers and A.W.M. Smeulders, “Pictoseek: Combining Color and Shape Invariant Features for Image Retrieval,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 102-119, 2000.
[58] G.L. Gimel'farb and A.K. Jain, “On Retrieving Textured Images from an Image Database,” Pattern Recognition, vol. 29, no. 9, pp. 1,461-1,483, 1996.
[59] C.C. Gottlieb and H.E. Kreyszig, “Texture Descriptors Based on Co-Occurrences Matrices,” Computer Vision, Graphics, and Image Processing, vol. 51, 1990.
[60] W.I. Grosky, “Multi-Media Information Systems.” IEEE Multimedia, vol. 1, no. 1, Mar. 1994.
[61] A. Gupta and R. Jain, “Visual Information Retrieval,” Comm. ACM, vol. 40, no. 5, pp. 70-79, May 1997.
[62] J. Hafner, H.S. Sawhney, W. Equitz, M. Flickner, and W. Niblack, “Efficient Color Histogram Indexing for Quadratic Form Distance Functions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 7, pp. 729-736, July 1995.
[63] M. Hagendoorn and R.C. Veltkamp, “Reliable and Efficient Pattern Matching Using an Affine Invariant Metric,” Int'l J. Computer Vision, vol. 35, no. 3, pp. 203-225, 1999.
[64] S. Hastings, “Query Categories in a Study of Intellectual Access to Digitized Art Images,” Proc. 58th Ann. Meeting Am. Soc. Information Science, 1995.
[65] H. Hatano, “Image Processing and Database System in the National Museum of Western Art,” Int'l J. Special Libraries, vol. 30, no. 3, pp. 259-267, 1996.
[66] G. Healey and D. Slater, “Computing Illumination-Invariant Descriptors of Spatially Filtered Color Image Regions,” IEEE Trans. Image Processing, vol. 6, no. 7 pp. 1,002-1,013, 1997.
[67] K. Hirata and T. Kato, “Rough Sketch-Based Image Information Retrieval,” NEC Research and Development, vol. 34, no. 2, pp. 263-273, 1992.
[68] A. Hiroike, Y. Musha, A. Sugimoto, and Y. Mori, “Visualization of Information Spaces to Retrieve and Browse Image Data,” Proc. Visual '99: Information and Information Systems, pp. 155-162, 1999.
[69] N.R. Howe and D.P. Huttenlocher, “Integrating Color, Texture, and Geometry for Image Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. 239-247, 2000.
[70] C.C. Hsu, W.W. Chu, and R.K. Taira, “A Knowledge-Based Approach for Retrieving Images by Content,” IEEE Trans. Knowledge and Data Eng., vol. 8, no. 4, pp. 522-532, 1996.
[71] F.J. Hsu, S.Y. Lee, and B.S. Lin, “Similairty Retrieval by 2D C-Trees Matching in Image Databases,” J. Visual Comm. and Image Representation, vol. 9, no. 1, pp. 87-100, 1998.
[72] M.K. Hu, “Pattern Recognition by Moment Invariants,” Proc. IRE Trans. Information Theory, vol. 8, pp. 179-187, 1962.
[73] J. Huang, S.R. Kumar, M. Mitra, W.-J. Zhu, and R. Zabih, “Spatial Color Indexing and Applications,” Int'l J. Computer Vision, vol. 35, no. 3, pp. 245-268, 1999.
[74] B. Huet and E.R. Hancock, “Line Pattern Retrieval Using Relational Histograms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 12, pp. 1,363-1,371, Dec. 1999.
[75] F. Idris and S. Panchanathan, “Image Indexing Using Wavelet Vector Quantization,” Proc. Digital Image Storage and Archiving Systems, vol. 2,606, pp. 269-275, 1995.
[76] L. Itti, C. Koch, and E. Niebur, “A Model for Saliency-Based Visual Attention for Rapid Scene Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1,254-1,259, Nov. 1998.
[77] C.E. Jacobs and A. Finkelstein, S.H. Salesin, “Fast Multiresolution Image Querying,” Proc. SIGGRAPH, 1995.
[78] A.K. Jain and A. Vailaya, “Image Retrieval Using Color and Shape,” Pattern Recognition, vol. 29, no. 8, pp. 1,233-1,244, 1996.
[79] A.K. Jain and A. Vailaya, “Shape-Based Retrieval: A Case Study with Trademark Image Databases,” Pattern Recognition, vol. 31, no. 9, pp. 1,369-1,390, 1998.
[80] A.K. Jain, R.P.W. Duin, and J. Mao, Statistical Pattern Recognition: A Review IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[81] Proc. US NSF Workshop Visual Information Management Systems, R. Jain, ed., 1992.
[82] R. Jain, “InfoScopes: Multimedia Information Systems,” Multimedia Systems and Techniques, Kluwer Academic Publishers, 1996.
[83] L. Jia and L. Kitchen, “Object-Based Image Similarity Computation Using Inductive Learning of Contour-Segment Relations,” IEEE Trans. Iamge Processing, vol. 9, no. 1, pp. 80-87, 2000.
[84] D.W. Joyce, P.H. Lewis, R.H. Tansley, M.R. Dobie, and W. Hall, “Semiotics and Agents for Integrating and Navigating through Multimedia Representations,” Proc. Storage and Retrieval for Media Databases, vol. 3972, pp. 120-131, 2000.
[85] D. Judd, P. McKinley, and A.K. Jain, “Large-Scale Parallel Data Clustering,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 871-876, Aug. 1998.
[86] T. Kakimoto and Y. Kambayashi, “Browsing Functions in Three-Dimensional Space for Digital Libraries,” Int'l J. Digital Libraries, vol. 2, pp. 68-78, 1999.
[87] N. Katayama and S. Satoh, “The SR-Tree: An Index Structure for High-Dimensional Nearest Neighbor Queries,” Proc. SIGMOD, Int'l Conf. Management of Data, pp. 369-380, 1997.
[88] T. Kato, T. Kurita, N. Otsu, and K. Hirata, “A Sketch Retrieval Method for Full Color Image Database—Query by Visual Example,” Proc. ICPR, Computer Vision and Applications, pp. 530-533, 1992.
[89] A. Kontanzad and Y.H. Hong, “Invariant Image Recognition by Zernike Moments,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 489-497, May 1990.
[90] F. Korn, N. Sidiropoulos, C. Faloutsos, E. Siegel, and Z. Protopapas, “Efficient and Effective Nearest Neighbor Search in a Medical Image Database of Tumor Shapes,” Image Description and Retrieval, pp. 17-54, 1998.
[91] S. Krishnamachari and R. Chellappa, “Multiresolution Gauss-Markov Random Field Models for Texture Segmentation,” IEEE Trans. Image Processing, vol. 6, no. 2, pp. 251-267, 1997.
[92] A. Laine and J. Fan, “Texture Classification by Wavelet Packet Signature,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1,186-1,191, Nov. 1993.
[93] L.J. Latecki and R. Lakämper, “Contour-Based Shape Similarity,” Proc. Visual '99: Information and Information Systems, pp. 617-624, 1999.
[94] L.J. Latecki and R. Lakämper, “Convexity Rule for Shape Decomposition Based on Discrete Contour Evolution,” Image Understanding, vol. 73, no. 3, pp. 441-454, 1999.
[95] T.K. Lau and I. King, “Montage: An Image Database for the Fashion, Textile, and Clothing Industry in Hong Kong,” Proc. Asian Conf. Computer Vision, pp. 575-582, 1998.
[96] M. Leissler, M. Hemmje, and E.J. Neuhold, “Supporting Image Retrieval by Database Driven Interactive 3D Information-Visualization,” Visual Information and Information Systems, pp. 1-14, Springer Verlag, 1999.
[97] M.S. Lew and N. Sebe, “Visual Websearching Using Iconic Queries,” Proc. Computer Vision and Pattern Recognition, pp. 788-789, 2000.
[98] C.-S. Li and V. Castelli, “Deriving Texture Feature Set for Content-Based Retrieval of Satellite Image Database,” Proc. Int'l Conf. Image Processing, 1997.
[99] H.C. Lin, L.L. Wang, and S.N. Yang, “Color Image Retrieval Based on Hidden Markov Models,” IEEE Trans. Image Processing, vol. 6, no. 2, pp. 332-339, 1997.
[100] T. Lindeberg and J.O. Eklundh, “Scale Space Primal Sketch Construction and Experiments,” Image Vision Computing, vol. 10, pp. 3-18, 1992.
[101] F. Liu and R.W. Picard, “Periodicity, Directionality, and Randomness: Wold Features for Image Modelling and Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 7, pp. 722-733, July 1996.
[102] W.Y. Ma and B.S. Manjunath, “Edge Flow: A Framework of Boundary Detection and Image Segmentation,” Proc. Computer Vision and Pattern Recognition, pp. 744-749, 1997.
[103] M.K. Mandal, F. Idris, and S. Panchanathan, “Image and Video Indexing in the Compressed Domain: A Critical Review,” Image and Vision Computing, 2000.
[104] J. Mandel, The Statistical Analysis of Experimental Data. Interscience Publishers, 1964. Republished in 1984, Mineola, N.Y.: Dover.
[105] B.S. Manjunath and W.Y. Ma, “Texture Features for Browsing and Retrieval of Image Data,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 837-842, Aug. 1996
[106] J. Mao and A.K. Jain, “Texture Classification and Segmentation Using Multiresolution Simultaneous Autoregressive Models,” Pattern Recognition, vol. 25, no. 2, 1992.
[107] J. Matas, R. Marik, and J. Kittler, “On Representation and Matching of Multi-Colored Objects,” Proc. Fifth Int'l Conf. Computer Vision, pp. 726-732, 1995.
[108] R. Mehrotra and J.E. Gary, “Similar-Shape Retrieval in Shape Data Management,” Computer, vol. 28, no. 9, pp. 57-62, Sept. 1995.
[109] B.M. Mehtre, M.S. Kankanhalli, and W.F. Lee, “Shape Measures for Content Based Image Retrieval: A Comparison,” Information Procesing Management, vol. 33, no. 3, pp. 319-337, 1997.
[110] C. Meilhac and C. Nastar, “Relevance Feedback and Category Search in Image Databases,” Proc. Int'l Conf. Multimedia Computing and Systems, pp. 512-517, 1999.
[111] T.P. Minka and R.W. Picard, “Interactive Learning with a‘Society of Models,’” Pattern Recognition, vol. 30, no. 4, pp. 565-582, 1997.
[112] M. Mirmehdi and M. Petrou, “Segmentation of Color Texture,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 2, pp. 142-159, Feb. 2000.
[113] B. Moghaddam and A. Pentland, “Probabilistic Visual Learning for Object Representation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 696-710, July 1997.
[114] B. Moghaddam, W. Wahid, and A. Pentland, “Beyond Eigenfaces: Probabilistic Matching for Face Recognition,” Proc. Third Int'l Conf. Automatic Face and Gesture Reognition, 1998.
[115] A. Mojsilovic, J. Kovacevic, J. Hu, R.J. Safranek, and S.K. Ganapathy, “Matching and Retrieval Based on the Vocabulary and Grammar of Color Patterns,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 38-54, 2000.
[116] F. Mokhtarian, “Silhouette-Based Isolated Object Recognition through Curvature Scale-Space,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 5, pp. 539-544, May 1995.
[117] Applications of Invariance in Computer Vision, Lecture Notes in Computer Science, J.L. Mundy, A. Zissermann, D. Forsyth, eds., vol. 825, Springer Verlag, 1994
[118] H. Murase and S.K. Nayar, “Visual Learning and Recognition of 3-D Objects from Appearance,” Int'l J. Computer Vision, vol. 14, pp. 5-24, 1995.
[119] D.A. Narasimhalu, M.S. Kankanhalli, and J. Wu, “Benchmarking Multimedia Databases,” Multimedia Tools and Applications, vol. 4, no. 3, pp. 333-355, 1997.
[120] V.E. Ogle, “CHABOT—Retrieval from a Relational Database of Images,” Computer, vol. 28, no. 9, pp. 40-48, Sept. 1995.
[121] S. Ornager, “Image Retrieval: Theoretical and Empirical User Studies on Accessing Information in Images,” Proc. 60th Am. Soc. Information Science Ann. Meeting, vol. 34, pp. 202-211, 1997.
[122] P. Pala and S. Santini, “Image Retrieval by Shape and Texture,” Pattern Recognition, vol. 32, no. 3, pp. 517-527, 1999.
[123] M.L. Pao and M. Lee, Concepts of Information Retrieval. Libraries Unlimited, 1989.
[124] T.V. Papathomas, E. Conway, I.J. Cox, J. Ghosn, M.L. Miller, T.P. Minka, and P.N. Yianilos, “Psychophysical Studies of the Performance of an Image Database Retrieval System,” Proc. Symp. Electronic Imaging: Conf. Human Vision and Electronic Imaging III, 1998.
[125] G. Pass and R. Zabith, “Comparing Images Using Joint Histograms,” Multimedia Systems, vol. 7, pp. 234-240, 1999.
[126] E.J. Pauwels and G. Frederix, “Nonparametric Clustering for Image Segmentation and Grouping,” Image Understanding, vol. 75, no. 1, pp. 73-85, 2000.
[127] A. Pentland and T. Choudhury, “Face Recognition for Smart Environments,” Computer, vol. 33, no. 2, pp. 50-, Feb. 2000.
[128] A. Pentland, R.W. Picard, and S. Sclaroff, “Photobook: Content-Based Manipulation of Image Databases,” Int'l J. Computer Vision, vol. 18, no. 3, pp. 233-254, 1996.
[129] E.G.M. Petrakis and C. Faloutsos, “Similarity Searching in Medical Image Databases,” IEEE Trans. Knowledge and Data Eng., vol. 9, no. 3, pp. 435-447, May/June 1997.
[130] R.W. Picard and T.P. Minka, “Vision Texture for Annotation,” Multimedia Systems, vol. 3, pp. 3-14, 1995.
[131] J. Puzicha, T. Hoffman, and J.M. Buhmann, “Non-Parametric Similarity Measures for Unsupervised Texture Segmentation and Image Retrieval,” Proc. Computer Vision and Pattern Recognition, 1997.
[132] W. Qian, M. Kallergi, L.P. Clarke, H.D. Li, D. Venugopal, D.S. Song, and L.P. Clark, “Tree-Structured Wavelet Transform Segmentation of Microcalcifications in Digital Mammography,” J. Medical Physiology, vol. 22, no. 8, pp. 1,247-1,254, 1995.
[133] T. Randen and J.H. Husoy, “Filtering for Texture Classification: A Comparative Study,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 4, pp. 291-310, Apr. 1999.
[134] A. Rao, R.K. Srihari, and Z. Zhang, “Geometric Histogram: A Distribution of Geometric Configurations of Color Subsets,” Internet Imaging, vol. 3,964, pp. 91-101, 2000.
[135] E. Riloff and L. Hollaar, “Text Databases and Information Retrieval,” ACM Computing Surveys, vol. 28, no. 1, pp. 133-135, 1996.
[136] E. Rivlin and I. Weiss, “Local Invariants for Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 3, pp. 226-238, Mar. 1995.
[137] R. Rodriguez-Sanchez, J.A. Garcia, J. Fdez-Valdivia, and X.R. Fdez-Vidal, “The RGFF Representational Model: A System for the Automatically Learned Partitioning of‘Visual Pattern’in Digital Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 1,044-1,073, Oct. 1999.
[138] P.L. Rosin, “Edges: Saliency Measures and Automatic Thresholding,” Machine Vision Applications, vol. 9, no. 7, pp.139-159, 1997.
[139] D. Roth, M.-H. Yang, and N. Ahuja, “Learning to Recognize Objects,” Proc. Computer Vision and Pattern Recognition, pp. 724-731, 2000.
[140] I. Rothe, H. Suesse, and K. Voss, “The Method of Normalization of Determine Invariants,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 4, pp. 366-376, Apr. 1996.
[141] Y. Rui, T.S. Huang, and S.-F. Chang, “Image Retrieval: Current Techniques, Promising Directions, and Open Issues,” J. Visual Comm. and Image Representation, vol. 10, no. 1, pp. 39-62, 1999.
[142] Y. Rui, T.S. Huang, M. Ortega, and S. Mehrotra, “Relevance Feednack: A Power Tool for Interactive Conten-Based Image Retrieval,” IEEE Trans. Circuits, and Video Technology, Sept. 1998.
[143] G. Salton, Automatic Text Processing. Addison-Wesley, 1988.
[144] H. Samet and A. Soffer, “MARCO: MAp Retrieval by Content,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 783-798, Aug. 1996.
[145] S. Santini, “Evaluation Vademecum for Visual Information Systems,” Storage and Retrieval for Image and Video Databases VIII, vol. 3,972, 2000.
[146] S. Santini, A. Gupta, and R. Jain, “User Interfaces for Emergent Semantics in Image Databases,” Proc. Eighth IFIP Working Conf. Database Semantics (DS-8), 1999.
[147] S. Santini and R. Jain, “Similarity Measures,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 9, pp. 871-883, Sept. 1999.
[148] C. Schmid and R. Mohr, “Local Grayvalue Invariants for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 5, pp. 530-535, May 1997.
[149] M. Schneier and M. Abdel-Mottaleb, “Exploiting the JPEG Compression Scheme for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, 849 -853, Aug. 1996.
[150] S. Sclaroff, “Deformable Prototypes for Encoding Shape Categories in Image Databases,” Pattern Recognition, vol. 30, no. 4, pp. 627-641, 1997.
[151] S. Sclaroff, M. La Cascia, and S. Sethi, “Using Textual and Visual Cues for Content-Based Image Retrieval from the World Wide Web,” Image Understanding, vol. 75, no. 2, pp. 86-98, 1999.
[152] S. Sclaroff, L. Taycher, and M. La Cascia, “Imagerover: A Content-Base Image Browser for the World Wide Web,” Proc. Workshop Content-Based Access to Image and Video Libraries, pp. 1,000-1,006, 1997.
[153] D. Sharvit, J. Chan, H. Tek, and B.B. Kimia, “Symmetry-Based Indexing of Image Databases,” J. Visual Comm. and Image Representation, vol. 9, no. 4, pp. 366-380, 1998.
[154] R.N. Shepard, “Toward a Universal Law of Generalization for Physical Science,” Science, vol. 237, pp. 1,317-1,323, 1987.
[155] R.H. Shrihari, “Automatic Indexing and Content-Based Retrieval of Captioned Images,” Computer, vol. 28, no. 9, Sept. 1995.
[156] C.-R. Shyu, C.E. Brodley, A.C. Kak, and A. Kosaka, “ASSERT: A Physician in the Loop Content-Based Retrieval System for HCRT Image Databases,” Image Understanding, vol. 75,nos. 1/2, pp. 111-132, 1999.
[157] K. Siddiqi and B.B. Kimia, “Parts of Visual Form: Computational Aspects,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 3, Mar. 1995.
[158] D. Slater and G. Healey, "The Illumination-Invariant Recognition of 3D Objects Using Local Color Invariants," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 2, pp. 206-210, Feb. 1996.
[159] A.W.M. Smeulders, T. Gevers, J.M. Geusebroek, and M. Worring, “Invariance in Content-Based Retrieval,” Proc. Int'l Conf. Multimedia and Expo, 2000.
[160] A.W.M. Smeulders, M.L. Kersten, and T. Gevers, “Crossing the Divide between Computer Vision and Data Bases in Search of Image Databases,” Proc. Fourth Working Conf. Visual Database Systems, pp. 223-239, 1998.
[161] A.W.M. Smeulders, S.D. Olabariagga, R. van den Boomgaard, and M. Worring, “Interactive Segmentation,” Proc. Visual '97: Information Systems, pp. 5-12, 1997.
[162] J.R. Smith and S.F. Chang, “Automated Binary Feature Sets for Image Retrieval,” Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, 1996.
[163] J.R. Smith and S.-F. Chang, “Integrated Sspatial and Feature Image Query,” Multimedia Systems, vol. 7, no. 2, pp. 129-140, 1999.
[164] J.R. Smith and C.-S. Li, “Image Retrieval Evaluation,” Proc. Workshop Content-Based Access of Image and Video Libraries, 1998.
[165] S.M. Smith and J.M. Brady, “SUSAN—A New Approach to Low Level Image Processing,” Int'l J. Computer Vision, vol. 23, no. 1, pp. 45-78, 1997.
[166] M. Stricker and M. Orengo, “Similarity of Color Images,” Storage and Retrieval of Image and Video Databases III, vol. 2,420, pp. 381-392, 1995.
[167] M. Stricker and M. Swain, “The Capacity of Color Histogram Indexing,” Proc. Computer Vision and Pattern Recognition, pp. 704- 708, 1994.
[168] M.J. Swain, “Searching for Multimedia on the World Wide Web,” Proc. Int'l Conf. Multimedia Computing and Systems, pp. 33-37, 1999.
[169] M.J. Swain and B.H. Ballard, “Color Indexing,” Int'l J. Computer Vision, vol. 7, no. 1, pp. 11-32, 1991.
[170] D.J. Swets and J. Weng, “Hierarchical Discriminant Analysis for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 386-401, May 1999.
[171] T.F. Syeda-Mahmood, “Location Hashing: An Efficient Method for Locating Object Queries in Image Databases,” Storage and Retrieval in Image and Video Databases, vol. 3,656, pp. 366-378, 1999.
[172] H.D. Tagare, F.M. Vos, C.C. Jaffe, and J.S. Duncan, “Arrangement—A Spatial Relation between Parts for Evaluating Similarity of Tomographic Section,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 9, pp. 880-893, Sept. 1995.
[173] T. Tan, “Rotation Invariant Texture Features and Their Use in Automatic Script Identification,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 7, pp. 751-756, July 1998.
[174] P.M. Tardif and A. Zaccarin, “Multiscale Autoregressive Image Respresentation for Texture Segmentation,” Image Processing VIII, vol. 3,026, pp. 327-337, 1997.
[175] J. Tatemura, “Browsing Images Based on Social and Content Similarity,” Proc. Int'l Conf. Multimedia and Expo, 2000.
[176] K. Tieu and P. Viola, “Boosting Image Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. pp. 228-235, 2000.
[177] A. Treisman, P. Cavanagh, B. Fisher, V.S. Ramachandran, and R. von der Heydt, “Form Perception and Attention-Striate Cortex and Beyond,” Visual Perception: The Neurophysiological Foundation, pp. 273-316, 1990.
[178] T. Tuytelaars and L. van Gool, “Content-Based Image Retrieval Based on Local Affinely Invariant Regions,” Proc. Visual '99: Information and Information Systems, pp. 493-500, 1999.
[179] A. Vailaya, M. Figueiredo, A. Jain, and H. Zhang, “Content-Based Hierarchical Classification of Vacation Images,” Proc. Int'l Conf. Multimedia Computing and Systems, 1999.
[180] G.W.A.M. van der Heijden and M. Worring, “Domain Concept to Feature Mapping for a Plant Variety Image Database,” Image Databases and Multimedia Search, vol. 8, pp. 301-308, 1997.
[181] V.N. Vapnik, Statistical Learning Theory, John Wiley&Sons, 1998.
[182] N. Vasconcelos and A. Lippman, “A Probabilistic Architecture for Content-Based Image Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. pp. 216-221, 2000.
[183] R.C. Veltkamp and M. Hagendoorn, “State-of-the-Art in Shape Matching,” Multimedia Search: State of the Art, Springer-Verlag, 2000.
[184] J. Vendrig, M. Worring, and A.W.M. Smeulders, “Filter Image Browsing: Exploiting Interaction in Retrieval,” Proc. Visual '99: Information and Information Systems, 1999.
[185] L.Z. Wang and G. Healey, “Using Zernike Moments for the Illumination and Geometry Invariant Classification of MultiSpectral Texture,” IEEE Trans. Image Processing, vol. 7, no. 2, pp. 196-203, 1991.
[186] M. Weber, M. Welling, and P. Perona, “Towards Automatic Discovery of Object Categories,” Proc. Computer Vision and Pattern Recognition, pp. 101-108, 2000.
[187] J. Weickert, S. Ishikawa, and A. Imiya, “Linear Scale Space Has First Been Proposed in Japan,” J. Math., Imaging and Vision, vol. 10, pp. 237-252, 1999.
[188] M. Werman and D. Weinshall, “Similarity and Affine Invariant Distances between 2D Point Sets,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp. 810-814, Aug. 1995.
[189] D. White and R. Jain, “Similarity Indexing with the SS-Tree,” Proc. 12th Int'l Conf. Data Eng., 1996.
[190] D.A. White and R. Jain, “Algorithms and Strategies for Similarity Retrieval,” Storage and Retrieval in Image, and Video Databases, vol. 2,060, pp. 62-72, 1996.
[191] R.C. Wilson and E.R. Hancock, “Structural Matching by Discrete Relaxation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 6, pp. 634-648, June 1997.
[192] H.J. Wolfson and I. Rigoutsos, “Geometric Hashing: An Overview,” IEEE Trans. Computational Science Eng., vol. 4, no. 4, pp. 10-21, 1997.
[193] J.K. Wu, A.D. Narasimhalu, B.M. Mehtre, C.P. Lam, and Y.J. Gao, “CORE: A Content Based Retrieval System for Multimedia Information Systems,” Multimedia Systems, vol. 3, pp. 25-41, 1995.
[194] Y. Wu, Q. Tian, and T.S. Huang, “Discriminant-EM Algorithm with Application to Image Retrieval,” Proc. Computer Vision and Pattern Recognition, pp. 222-227, 2000.
[195] P.N. Yanilos, “Data Structures and Aalgorithms for Nearest Neighbor Search in General Metric Spaces,” Proc. Third Ann. Symp. Discrete Algorithms, pp. 516-523, 1993.
[196] N. Yazdani, M. Ozsoyoglu, and G. Ozsoyoglu, “A Framework for Feature-Based Indexing for Spatial Databases,” Proc. Seventh Int'l Working Conf. Scientific and Statistical Database Management, pp. 259-269, 1994.
[197] P.C. Yuen, G.C. Feng, and D.Q. Tai, “Human Face Image Retrieval System for Large Database,” Proc. 14th Int'l Conf. Pattern Recognition, vol. 2, pp. 1,585-1,588, 1998.
[198] Q.L. Zhang, S.K. Chang, and S.S.T. Yau, “A Unified Approach to Iconic Indexing, Retrieval and Maintenance of Spatial Relationships in Image Databases,” J. Visual Comm. and Image Representation, vol. 7, no. 4, pp. 307-324, 1996.
[199] R. Zhao and W. Grosky, “From Features to Semantics: Some Preliminary Results,” Proc. Int'l Conf. Multimedia and Expo, 2000.
[200] Y. Zhong, K. Karu, and A.K. Jain, “Locating Text in Complex Color Images,” Pattern Recognition, vol. 28, no. 10, pp. 1,523-1,535, 1995.
[201] P. Zhu and P.M. Chirlian, “On Critical Point Detection of Digital Shapes,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp. 737-748, Aug. 1995.

Index Terms:
Review, content based, retrieval, semantic gap, sensory gap, narrow domain, broad domain, weak segmentation, accumulative features, salient features, signs, structural features, similarity, semantic interpretation, query space, display space, interactive session, indexing, architecture, evaluation, image databases.
Citation:
Arnold W.M. Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, Ramesh Jain, "Content-Based Image Retrieval at the End of the Early Years," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000, doi:10.1109/34.895972
Usage of this product signifies your acceptance of the Terms of Use.