The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.25)
pp: 2257-2270
Yuxin Chen , ETH Zurich, Zurich
Hariprasad Sampathkumar , The University of Kansas, Lawrence
Bo Luo , The University of Kansas, Lawrence
Xue-wen Chen , Wayne State University, Detroit
ABSTRACT
With the development of Internet and Web 2.0, large-volume multimedia contents have been made available online. It is highly desired to provide easy accessibility to such contents, i.e., efficient and precise retrieval of images that satisfies users' needs. Toward this goal, content-based image retrieval (CBIR) has been intensively studied in the research community, while text-based search is better adopted in the industry. Both approaches have inherent disadvantages and limitations. Therefore, unlike the great success of text search, web image search engines are still premature. In this paper, we present iLike, a vertical image search engine that integrates both textual and visual features to improve retrieval performance. We bridge the semantic gap by capturing the meaning of each text term in the visual feature space, and reweight visual features according to their significance to the query terms. We also bridge the user intention gap because we are able to infer the "visual meanings" behind the textual queries. Last but not least, we provide a visual thesaurus, which is generated from the statistical similarity between the visual space representation of textual terms. Experimental results show that our approach improves both precision and recall, compared with content-based or text-based image retrieval techniques. More importantly, search results from iLike is more consistent with users' perception of the query terms.
INDEX TERMS
Visualization, Feature extraction, Semantics, Image retrieval, Tagging, Image color analysis, Search engines, specialized search, Visualization, Feature extraction, Semantics, Image retrieval, Tagging, Image color analysis, Search engines, vertical search engine, CBIR
CITATION
Yuxin Chen, Hariprasad Sampathkumar, Bo Luo, Xue-wen Chen, "iLike: Bridging the Semantic Gap in Vertical Image Search by Integrating Text and Visual Features", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 10, pp. 2257-2270, Oct. 2013, doi:10.1109/TKDE.2012.192
REFERENCES
[1] Y. Chen, N. Yu, B. Luo, and X.-w. Chen, "iLike: Integrating Visual and Textual Features for Vertical Search," Proc. ACM Int'l Conf. Multimedia, 2010.
[2] B. Luo, X. Wang, and X. Tang, "A World Wide Web Based Image Search Engine Using Text and Image Content Features," Proc. IS&T/SPIE, vol. 5018, pp. 123-130, 2003.
[3] J. Cui, F. Wen, and X. Tang, "Real Time Google and Live Image Search Re-Ranking," Proc. 16th ACM Int'l Conf. Multimedia, 2008.
[4] J. Cui, F. Wen, and X. Tang, "Intentsearch: Interactive On-Line Image Search Re-Ranking," Proc. 16th ACM Int'l Conf. Multimedia, 2008.
[5] X. Tang, K. Liu, J. Cui, F. Wen, and X. Wang, "Intentsearch: Capturing User Intention for One-Click Internet Image Search," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 34, no. 7, pp. 1342-1353, July 2012.
[6] F. Jing, C. Wang, Y. Yao, K. Deng, L. Zhang, and W.-Y. Ma, "IGroup: Web Image Search Results Clustering," Proc. 14th ACM Int'l Conf. Multimedia, 2006.
[7] S. Wang, F. Jing, J. He, Q. Du, and L. Zhang, "IGroup: Presenting Web Image Search Results in Semantic Clusters," Proc. SIGCHI Conf. Human Factors in Computing Systems, 2007.
[8] X. Wang, K. Liu, and X. Tang, "Query-Specific Visual Semantic Spaces for Web Image Re-Ranking," Proc. IEEE Conf. Computer Vision Pattern Recognition (CVPR), June 2011.
[9] A.W.M. Smeulders, S. Member, M. Worring, S. Santini, A. Gupta, and R. Jain, "Content-Based Image Retrieval at the End of the Early Years," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
[10] M.S. Lew, N. Sebe, C. Djeraba, and R. Jain, "Content-Based Multimedia Information Retrieval: State of the Art and Challenges," ACM Trans. Multimedia Computing, Comm., and Applications, vol. 2, no. 1, pp. 1-19, 2006.
[11] R. Datta, D. Joshi, J. Li, James, and Z. Wang, "Image Retrieval: Ideas, Influences, and Trends of the New Age," ACM Computing Surveys, vol. 39, article 5, 2006.
[12] M. Stricker and M. Orengo, "Similarity of Color Images," Proc. SPIE, vol. 2420, pp. 381-392, 1995.
[13] R.M. Haralick, K. Shanmugam, and I. Dinstein, "Textural Features for Image Classification," IEEE Trans. Systems Man and Cybernetics, vol. SMC-3, no. 6, pp. 610-621, Nov. 1973.
[14] H. Tamura, S. Mori, and T. Yamawaki, "Textural Features Corresponding to Visual Perception," IEEE Trans. Systems Man and Cybernetics, vol. SMC-8, no. 6, pp. 460-473, June 1978.
[15] S.A. Dudani, K.J. Breeding, and R.B. McGhee, "Aircraft Identification by Moment Invariants," IEEE Trans. Computers, vol. C-26, no. 1, pp. 39-46, Jan. 1977.
[16] A. Vijay and M. Bhattacharya, "Content-Based Medical Image Retrieval Using the Generic Fourier Descriptor with Brightness," Proc. Int'l Conf. Machine Vision, 2009.
[17] W.-Y. Ma and H.-J. Zhang, "Content-Based Image Indexing and Retrieval," Handbook of Multimedia Computing, CRC Press, 1998.
[18] B. Manjunath, J.-R. Ohm, V. Vasudevan, and A. Yamada, "Color and Texture Descriptors," IEEE Trans. Circuits and Systems for Video Technology, vol. 11, no. 6, pp. 703-715, June 2001.
[19] S. Raimondo, S. Simone, C. Claudio, and C. Gianluigi, "Prosemantic Features for Content-Based Image Retrieval," Proc. Seventh Int'l Workshop Adaptive Multimedia Retrieval, 2009.
[20] J. Jeon, V. Lavrenko, and R. Manmatha, "Automatic Image Annotation and Retrieval Using Cross-Media Relevance Models," Proc. ACM SIGIR Conf. Research and Development in Information Retrieval, 2003.
[21] G. Carneiro, A.B. Chan, P.J. Moreno, and N. Vasconcelos, "Supervised Learning of Semantic Classes for Image Annotation and Retrieval," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 29, no. 3, pp. 394-410, Mar. 2007.
[22] J. Li and J.Z. Wang, "Real-Time Computerized Annotation of Pictures," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 30, no. 6, pp. 985-1002, June 2008.
[23] K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D.M. Blei, and M.I. Jordan, "Matching Words and Pictures," J. Machine Learning Research, vol. 3, pp. 1107-1135, Mar. 2003.
[24] X.-J. Wang, L. Zhang, F. Jing, and W.-Y. Ma, "Annosearch: Image Auto-Annotation by Search," Proc. IEEE CS Conf. Computer Vision Pattern Recognition (CVPR), 2006.
[25] L.S. Kennedy, S.-F. Chang, and I.V. Kozintsev, "To Search or to Label?: Predicting the Performance of Search-Based Automatic Image Classifiers," Proc. ACM Int'l Workshop Multimedia Information Retrieval (MIR), 2006.
[26] X. Li, L. Chen, L. Zhang, F. Lin, and W.-Y. Ma, "Image Annotation by Large-Scale Content-Based Image Retrieval," Proc. 14th Ann. ACM Int'l Conf. Multimedia, 2006.
[27] Z.-H. Zhou and H.-B. Dai, "Exploiting Image Contents in Web Search," Proc. 20th Int'l Joint Conf. Artificial Intelligence (IJCAI), 2007.
[28] H. Lieberman, E. Rozenweig, and P. Singh, "Aria: An Agent for Annotating and Retrieving Images," Computer, vol. 34, no. 7, pp. 57-62, July 2001.
[29] L. von Ahn and L. Dabbish, "Labeling Images with a Computer Game," Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI), 2004.
[30] L. Wu, L. Yang, N. Yu, and X.-S. Hua, "Learning to Tag," Proc. 18th Int'l Conf. World Wide Web (WWW), Apr. 2009.
[31] N. Sawant, R. Datta, J. Li, and J.Z. Wang, "Quest for Relevant Tags Using Local Interaction Networks and Visual Content," Proc. Int'l Conf. Multimedia Information Retrieval (MIR), 2010.
[32] Y.A. Aslandogan, C. Thier, C.T. Yu, J. Zou, and N. Rishe, "Using Semantic Contents and Wordnet in Image Retrieval," Proc. ACM SIGIR Conf. Research and Development in Information Retrieval, 1997.
[33] H.T. Shen, B.C. Ooi, and K.-L. Tan, "Giving Meanings to WWW Images," Proc. ACM Eighth Int'l Conf. Multimedia (Multimedia '00), 2000.
[34] R. Lempel and A. Soffer, "PicASHOW: Pictorial Authority Search by Hyperlinks on the Web," Proc. Int'l Conf. World Wide Web (WWW), 2001.
[35] D. Cai, X. He, Z. Li, W.-Y. Ma, and J.-R. Wen, "Hierarchical Clustering of WWW Image Search Results Using Visual, Textual and Link Information," Proc. 12th ACM Int'l Conf. Multimedia, 2004.
[36] I. Kompatsiaris, E. Triantafyllou, and M. Strintzis, "A World Wide Web Region-Based Image Search Engine," Proc. 11th Int'l Conf. Image Analysis and Processing (ICIAP), 2001.
[37] C. Frankel, M.J. Swain, and V. Athitsos, "Webseer: An Image Search Engine for the World Wide Web," technical report, 1996.
[38] S. Mukherjea, K. Hirata, and Y. Hara, "Amore: A World Wide Web Image Retrieval Engine," J. World Wide Web, vol. 2, no. 3, pp. 115-132, 1999.
[39] S. Sclaroff, L. Taycher, and M.L. Cascia, "ImageRover: A Content-Based Image Browser for the World Wide Web," Proc. IEEE Workshop Content-Based Access of Image and Video Libraries (CAIVL), 1997.
[40] Z. Chen, L. Wenyin, C. Hu, M. Li, and H.-J. Zhang, "iFind: A Web Image Search Engine," Proc. 24th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, 2001.
[41] C. Wang, L. Zhang, and H.-J. Zhang, "Learning to Reduce the Semantic Gap in Web Image Retrieval and Annotation," Proc. 31st Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, 2008.
[42] L. Zhang, L. Chen, F. Jing, K. Deng, and W.-Y. Ma, "Enjoyphoto: A Vertical Image Search Engine for Enjoying High-Quality Photos," Proc. ACM Int'l Conf. Multimedia, 2006.
[43] L. Zhang, Y. Hu, M. Li, W. Ma, and H. Zhang, "Efficient Propagation for Face Annotation in Family Albums," Proc. 12th ACM Ann. Int'l Conf. Multimedia (Multimedia), 2004.
[44] J. Cui, F. Wen, R. Xiao, Y. Tian, and X. Tang, "Easyalbum: An Interactive Photo Annotation System Based on Face Clustering and Re-Ranking," Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI), 2007.
[45] Z. Wang, Z. Chi, and D. Feng, "Fuzzy Integral for Leaf Image Retrieval," Proc. IEEE Int'l Conf. Fuzzy Systems (FUZZ), 2002.
[46] J.-X. Dua, X.-F. Wang, and G.-J. Zhang, "Leaf Shape Based Plant Species Recognition," Applied Math. and Computation, vol. 185, pp. 883-893, 2007.
[47] K.-P. Yee, K. Swearingen, K. Li, and M. Hearst, "Faceted Metadata for Image Search and Browsing," Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI), 2003.
[48] P. Kovesi, "Image Features from Phase Congruency," J. Computer Vision Research, vol. 1, no. 3, 1999.
[49] M.R. Teague, "Image Analysis via the General Theory of Moments∗," J. Optical Soc. Am., vol. 70, no. 8, pp. 920-930, Aug. 1980.
[50] T. Deselaers, D. Keysers, and H. Ney, "Features for Image Retrieval - A Quantitative Comparison," Proc. DAGM Symp. Pattern Recognition, 2004.
[51] P. Kakumanu, S. Makrogiannis, and N. Bourbakis, "A Survey of Skin-Color Modeling and Detection Methods," Pattern Recognition, vol. 40, no. 3, pp. 1106-1122, 2007.
[52] W.J. Conover, Practical Nonparametric Statistics. John Wiley & Sons, Dec. 1998.
[53] T.-W. Chang, Y.-P. Huang, and F. Sandnes, "Efficient Entropy-Based Features Selection for Image Retrieval," Proc. IEEE Int'l Conf. Systems, Man and Cybernetics (SMC), pp. 2941-2946, Oct. 2009.
[54] A. Sohail, P. Bhattacharya, S. Mudur, and S. Krishnamurthy, "Selection of Optimal Texture Descriptors for Retrieving Ultrasound Medical Images," Proc. IEEE Int'l Symp. Biomedical Imaging (ISBI), 2011.
[55] M. Dash and H. Liu, "Handling Large Unsupervised Data via Dimensionality Reduction," Proc. ACM SIGMOD Workshop Research Issues in Data Mining (DMKD), 1999.
[56] S. Pal and B. Chakraborty, "Intraclass and Interclass Ambiguities (Fuzziness) in Feature Evaluation," Pattern Recognition Letters, vol. 2, no. 5, pp. 275-279, 1984.
[57] T. Gonzalez, S. Sahni, and W.R. Franta, "An Efficient Algorithm for the Kolmogorov-Smirnov and Lilliefors Tests," ACM Trans. Math. Software, vol. 3, no. 1, pp. 60-64, 1977.
[58] I.K. Sethi and I.L. Coman, "Mining Association Rules Between Low-Level Image Features and High-Level Concepts," Proc. SPIE, vol. 4384, pp. 279-290, 2001.
[59] C. Town and D. Sinclair, "Content Based Image Retrieval Using Semantic Visual Categories," technical report, 2001.
[60] L. Zhang, F. Lin, and B. Zhang, "Support Vector Machine Learning for Image Retrieval," Proc. IEEE Int'l Conf. Image Processing (ICIP), 2001.
[61] A. Vailaya, A. Member, M.A.T. Figueiredo, A.K. Jain, H.-J. Zhang, and S. Member, "Image Classification for Content-Based Indexing," IEEE Trans. Image Processing, vol. 10, no. 1, pp. 117-130, Jan. 2001.
[62] D. Cai, X. He, Z. Li, W.-Y. Ma, and J.-R. Wen, "Hierarchical Clustering of WWW Image Search Results Using Visual, Textual and Link Information," Proc. 12th ACM Int'l Conf. Multimedia, 2004.
[63] J. Luo and A. Savakis, "Indoor vs Outdoor Classification of Consumer Photographs Using Low-Level and Semantic Features," Proc. IEEE Int'l Conf. Image Processing (ICIP), vol. 2, pp. 745-748, Oct. 2001.
[64] H. Feng, R. Shi, and T.-S. Chua, "A Bootstrapping Framework for Annotating and Retrieving WWW Images," Proc. 12th ACM Int'l Conf. Multimedia, 2004.
16 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool