The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2008 vol.30)
pp: 1919-1932
Xin-Jing Wang , Microsoft Research Asia, Beijing
Lei Zhang , Microsoft Research Asia, Beijing
Xirong Li , University of Amsterdam, Amsterdam
Wei-Ying Ma , Microsoft Research Asia, Beijing
ABSTRACT
In this paper, we propose a novel attempt of model-free image annotation which annotates images by mining their search results. It contains three steps: 1) the search process to discover visually and semantically similar search results; 2) the mining process to identify salient terms from textual descriptions of the search results; and 3) the annotation rejection process to filter out noisy terms yielded by step 2). To ensure real time annotation, two key techniques are leveraged - one is to map the high dimensional image visual features into hash codes, the other is to implement it as a distributed system, of which the search and mining processes are provided as Web services. As a typical result, the entire process finishes in less than 1 second. Our proposed approach enables annotating with unlimited vocabulary, and is highly scalable and robust to outliers. Experimental results on both real web images and a bench mark image dataset show the effectiveness and efficiency of the proposed algorithm.
INDEX TERMS
Computer vision, Applications, Pattern Recognition, Computing Methodologies, Retrieval models, Clustering, Information Search and Retrieval, Information Storage and Retrieval, Information Technology a
CITATION
Xin-Jing Wang, Lei Zhang, Xirong Li, Wei-Ying Ma, "Annotating Images by Mining Image Search Results", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.30, no. 11, pp. 1919-1932, November 2008, doi:10.1109/TPAMI.2008.127
REFERENCES
[1] K. Barnard et al., “Matching Words and Pictures,” J. Machine Learning Research, no. 3, pp. 1107-1135, 2003.
[2] K. Barnard et al., “Recognition as Translating Images into Text,” Internet Imaging IX, Electronic Imaging, 2003.
[3] K. Barnard et al., “Clustering Art,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 434-439, 2001.
[4] D. Blei and M.I. Jordan, “Modeling Annotated Data,” Proc. 26th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 127-134, 2003.
[5] D. Cai et al., “Hierarchical Clustering of WWW Image Search Results Using Visual, Textual and Link Information,” Proc. ACM Int'l Conf. Multimedia, pp. 952-959, 2004.
[6] G. Carneiro and N. Vasconcelos, “A Database Centric View of Semantic Image Annotation and Retrieval,” Proc. 28th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 559-566, 2005.
[7] C. Carson et al., “Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp.1026-1038, Aug. 2002.
[8] Z. Chen et al., “iFind: a Web Image Search Engine,” Proc. 24th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, p. 450, 2001.
[9] E. Chang et al., “CBSA: Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machines,” IEEE Trans. Circuits and Systems for Video Technology, vol. 13, no. 1, pp.26-38, 2003.
[10] I.J. Cox et al., “The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Psychophysical,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 20-37, 2000.
[11] R. Datta et al., “Content-Based Image Retrieval—Approaches and Trends of the New Age,” Proc. ACM Multimedia Workshop Multimedia Information Retrieval, pp. 253-262, 2005.
[12] P. Duygulu et al., “Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary,” Proc. European Conf. Computer Vision, pp. 97-112, 2002.
[13] X. Fan et al., “Photo-to-Search: Using Multimodal Queries to Search the Web from Mobile Devices,” Proc. ACM Multimedia Workshop Multimedia Information Retrieval, pp. 143-150, 2005.
[14] J.P. Fan et al., “Multi-Level Annotation of Natural Scenes Using Dominant Image Components and Semantic Concepts,” Proc. ACM Int'l Conf. Multimedia, pp. 540-547, 2004.
[15] J.P. Fan et al., “Automatic Image Annotation by Using Concept-Sensitive Salient Objects for Image Content Represent,” Proc. 27th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 361-368, 2004.
[16] WordNet: An Electronic Lexical Database, C. Fellbaum, ed. MIT Press, 1998.
[17] A. Ghoshal et al., “Hidden Markov Models for Automatic Annotation and Content-Based Retrieval of Images and Video,” Proc. 28th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 544-551, 2005.
[18] Google Image, images.google.com, 2008.
[19] J. Hays and A.A. Efros, “Scene Completion Using Millions of Photographs,” Proc. ACM SIGGRAPH, 2007.
[20] J. Huang et al., “Image Indexing Using Color Correlograms,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, p. 762, 1997.
[21] J. Jeon et al., “Automatic Image Annotation and Retrieval Using Cross-Media Relevance Models,” Proc. 26th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 119-126, 2003.
[22] J. Jeon and R. Manmatha, “Automatic Image Annotation of News Images with Large Vocabularies and Low Quality Training Data,” Proc. ACM Int'l Conf. Multimedia, 2004.
[23] I.T. Joliffe, Principal Component Analysis. Springer-Verlag, 1986.
[24] V. Lavrenko et al., “A Model for Learning the Semantics of Pictures,” Proc. 16th Conf. Neural Information Processing Systems, pp. 553-560, 2003.
[25] B.T. Li, K. Goh, and E. Chang, “Confidence-Based Dynamic Ensemble for Image Annotation and Semantics Discovery,” Proc. ACM Int'l Conf. Multimedia, pp. 195-206, 2003.
[26] J. Li et al., “OPTIMOL: Automatic Object Picture CollecTion via Incremental MOdel Learning,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2007.
[27] J. Li and J. Wang, “Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1075-1088, Sept. 2003.
[28] J. Li and J.Z. Wang, “Real-Time Computerized Annotation of Pictures,” Proc. ACM Int'l Conf. Multimedia, pp. 911-920, 2006.
[29] W.Y. Liu et al., “Semi-Automatic Image Annotation,” Proc. Human-Computer Interaction, pp. 326-333, 2001.
[30] X. Li et al., “Image Annotation by Large-Scale Content-Based Image Retrieval,” Proc. ACM Int'l Conf. Multimedia, pp. 607-610, 2006.
[31] X. Li et al., “SBIA: Search-Based Image Annotation by Leveraging Web-Scale Images,” Proc. ACM Int'l Conf. Multimedia, pp. 467-468, 2007.
[32] F. Monay and P. Gatica-Perez, “On Image Auto Annotation with Latent Space Models,” Proc. ACM Int'l Conf. Multimedia, pp. 275-278, 2003.
[33] F. Monay and P. Gatica-Perez, “PLSA-Based Image Auto-Annotation: Constraining the Latent Space,” Proc. ACM Int'l Conf. Multimedia, pp. 348-351, 2004.
[34] Y. Mori et al., “Image-to-Word Transformation Based on Dividing and Vector Quantizing Images with Words,” Proc. First Int'l Workshop Multimedia Intelligent Storage and Retrieval Management, 1999.
[35] M. Naphade et al., “Large-Scale Concept Ontology for Multimedia,” IEEE Multimedia, vol. 13, no. 3, pp. 86-91, July-Sept. 2006.
[36] J.-Y. Pan et al., “GCap: Graph-Based Automatic Image Captioning,” Proc. Conf. Computer Vision and Pattern Recognition Workshop, vol. 9, p. 146, 2004.
[37] J.-Y. Pan et al., “Automatic Multimedia Cross-Modal Correlation Discovery,” Proc. 10th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 653-658, 2004.
[38] K.R. Rao and P. Yip, Discrete Cosine Transform: Algorithms, Advantages, Applications. Academic, 1990.
[39] S.E. Robertson et al., “Okapi at TREC-3,” Proc. Third Text Retrieval Conf., pp. 109-126, 1995.
[40] Y. Rui et al., “Image Retrieval: Current Techniques, Promising Directions, and Open Issues,” J. Visual Comm. and Image Representation, vol. 10, no. 1, pp. 39-62, 1999.
[41] B. Shevade and H. Sundaram, “Incentive Based Image Annotation,” Technical Report AME-TR-2004-02, Arizona State Univ. 2004.
[42] “Search Result Clustering Toolbar in Microsoft Research Asia,” SRC, http:/rwsm.directtaps.net/, 2006.
[43] A.W.M. Smeulders et al., “Content-Based Image Retrieval at the End of the Early Years,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
[44] H. Toda and R. Kataoka, “A Search Result Clustering Method Using Informatively Named Entities,” Proc. Seventh ACM Int'l Workshop Web Information and Data Management, pp. 81-86, 2005.
[45] A. Torralba et al., “Tiny Images,” Technical Report MIT-CSAIL-TR-2007-024, Massachusetts Inst. of Tech nology, 2007.
[46] Vivisimo(2008), http:/www.vivisimo.com, 2008.
[47] B. Wang et al., “Large-Scale Duplicate Detection for Web Image Search,” Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 353-356, 2006.
[48] X.-J. Wang et al., “Data-Driven Approach for Bridging the Cognitive Gap in Image Retrieval,” Proc. IEEE Int'l Conf. Multimedia and Expo, no. 3, pp. 2231-2234, 2004.
[49] X.-J. Wang et al., “Multi-Model Similarity Propagation and Its Application for Web Image Retrieval,” Proc. ACM Int'l Conf. Multimedia, pp. 944-951, 2004.
[50] X.-J. Wang et al., “AnnoSearch: Image Auto-Annotation by Search,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 1483-1490, 2006.
[51] J. Weijer et al., “Learning Color Names from Real-World Images,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2007.
[52] Yahoo! News Search, http://news.search.yahoo.comnews, 2008.
[53] C. Yang et al., “Region Based Image Annotation through Multiple-Instance Learning,” Proc. ACM Int'l Conf. Multimedia, pp. 435-438, 2004.
[54] T. Yeh et al., “Searching the Web with Mobile Images for Location Recognition,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, no. 2, pp. 76-81, 2004.
[55] H.J. Zeng et al., “Learning to Cluster Web Search Results,” Proc. 27th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 210-217, 2004.
[56] L. Zhu et al., “Keyblock: An Approach for Content-Based Image Retrieval,” Proc. ACM Int'l Conf. Multimedia), pp. 157-166, 2000.
32 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool