The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2008 vol.30)
pp: 1933-1944
Roger C.F. Wong , Hong Kong Baptist University, Hong Kong
ABSTRACT
As the number of web images is increasing at a rapid rate, searching them semantically presents a significant challenge. Many raw images are constantly uploaded with little meaningful direct annotations of semantic content, limiting their search and discovery. In this paper, we present a semantic annotation technique based on the use of image parametric dimensions and metadata. Using decision trees and rule induction, we develop a rule-based approach to formulate explicit annotations for images fully automatically, so that by the use of our method, semantic query such as " sunset by the sea in autumn in New York" can be answered and indexed purely by machine. Our system is evaluated quantitatively using more than 100,000 web images. Experimental results indicate that this approach is able to deliver highly competent performance, attaining good recall and precision rates of sometimes over 80%. This approach enables a new degree of semantic richness to be automatically associated with images which previously can only be performed manually.
INDEX TERMS
Image/video retrieval, Scene Analysis
CITATION
Roger C.F. Wong, "Automatic Semantic Annotation of Real-World Web Images", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.30, no. 11, pp. 1933-1944, November 2008, doi:10.1109/TPAMI.2008.125
REFERENCES
[1] Y. Sun, S. Shimada, and M. Morimoto, “Visual Pattern Discovery Using Web Images,” Proc. Eighth ACM Int'l Workshop Multimedia Information Retrieval (MIR '06), pp. 127-136, 2006.
[2] B.L. Saux and G. Amato, “Image Recognition for Digital Libraries,” Proc. Sixth ACM SIGMM Int'l Workshop Multimedia Information Retrieval (MIR '04), pp. 91-98, 2004.
[3] C.F. Tsai, K. McGarry, and J. Tait, “Claire: A Modular Support Vector Image Indexing and Classification System,” ACM Trans. Information Systems, vol. 24, no. 3, pp. 353-379, 2006.
[4] R. Krishnapuram, S. Medasani, S.H. Jung, Y.S. Choi, and R. Balasubramaniam, “Content-Based Image Retrieval Based on a Fuzzy Approach,” IEEE Trans. Knowledge and Data Eng., vol. 16, no. 10, pp. 1185-1199, Oct. 2004.
[5] Y. Chen, J.Z. Wang, and R. Krovetz, “Content-Based Image Retrieval by Clustering,” Proc. Fifth ACM SIGMM Int'l Workshop Multimedia Information Retrieval (MIR '03), pp. 193-200, 2003.
[6] N. Vasconcelos, “From Pixels to Semantic Spaces: Advances in Content-Based Image Retrieval,” Computer, vol. 40, no. 7, pp. 20-26, 2007.
[7] M. Jian, J. Dong, and R. Tang, “Combining Color, Texture and Region with Objects of User's Interest for Content-Based Image Retrieval,” Proc. Seventh ACIS Int'l Conf. Software Eng., Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD '07), vol. 01, pp. 764-769, 2007.
[8] R. Datta, J. Li, and J.Z. Wang, “Content-Based Image Retrieval: Approaches and Trends of the New Age,” Proc. Seventh ACM SIGMM Int'l Workshop Multimedia Information Retrieval (MIR '05), pp. 253-262, 2005.
[9] A. Perina, M. Cristani, and V. Murino, “Natural Scenes Categorization by Hierarchical Extraction of Typicality Patterns,” Proc. 14th Int'l Conf. Image Analysis and Processing (ICIAP '07), pp. 801-806, 2007.
[10] J. Li and J.Z. Wang, “Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1075-1088, Sept. 2003.
[11] V. Athitsos, J. Alon, S. Sclaroff, and G. Kollios, “Boostmap: A Method for Efficient Approximate Similarity Rankings,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR '04), vol. 2, pp. 268-275, 2004.
[12] J. Amores, N. Sebe, and P. Radeva, “Context-Based Object-Class Recognition and Retrieval by Generalised Correlograms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp.1818-1833, Oct. 2007.
[13] R. Pawlicki, I. Kókai, J. Finger, R. Smith, and T. Vetter, “Navigating in a Shape Space of Registered Models,” IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1552-1559, Nov./Dec. 2007.
[14] J.S. Cho and J. Choi, “Contour-Based Partial Object Recognition Using Symmetry in Image Databases,” Proc. ACM Symp. Applied Computing (SAC '05), pp. 1190-1194, 2005.
[15] J. Gausemeier, J. Fruend, C. Matysczok, B. Bruederlin, and D. Beier, “Development of a Real Time Image Based Object Recognition Method for Mobile Ar-Devices,” Proc. Second Int'l Conf. Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa (AFRIGRAPH '03), pp. 133-139, 2003.
[16] T. Zöller and J.M. Buhmann, “Robust Image Segmentation Using Resampling and Shape Constraints,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 7, pp. 1147-1164, July 2007.
[17] D. Cremers, M. Rousson, and R. Deriche, “A Review of Statistical Approaches to Level Set Segmentation: Integrating Color, Texture, Motion and Shape,” Int'l J. Computer Vision, vol. 72, no. 2, pp. 195-215, 2007.
[18] J. Vogel, A. Schwaninger, C. Wallraven, and H.H. Bülthoff, “Categorization of Natural Scenes: Local versus Global Information and the Role of Color,” ACM Trans. Applied Perception, vol. 4, no. 3, p. 19, 2007.
[19] D. Liu and T. Chen, “Content-Free Image Retrieval Using Bayesian Product Rule,” Proc. IEEE Int'l Conf. Multimedia and Expo (ICME '06), pp. 89-92, 2006.
[20] P. Over, C.H.C. Leung, H. Ip, and M. Grubinger, “Multimedia Retrieval Benchmarks,” IEEE Multimedia, vol. 11, no. 80-84, Apr. 2004.
[21] J. Wang, B. Thiesson, Y. Xu, and M. Cohen, “Image and Video Segmentation by Anisotropic Kernel Mean Shift,” Proc. European Conf. Computer Vision (ECCV '04), vol. 2, pp. 238-249, 2004.
[22] T. Rohlfing, D.B. Russakoff, and C.R. Maurer Jr., “Performance-Based Classifier Combination in Atlas-Based Image Segmentation Using Expectation-Maximization Parameter Estimation,” IEEE Trans. Medical Imaging, vol. 23, no. 8, pp. 983-994, 2004.
[23] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, and T. Poggio, “Robust Object Recognition with Cortex-Like Mechanisms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp.411-426, Mar. 2007.
[24] J. Li and J.Z. Wang, “Real-Time Computerized Annotation of Pictures,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 985-1002, June 2008.
[25] R. Mendoza and M.A. Williams, “Ontology Based Object Categorisation for Robots,” Proc. Australasian Ontology Workshop (AOW '05), pp. 61-67, 2005.
[26] H. Lieberman, H. Liu, P. Singh, and B. Barry, “Beating Common Sense into Interactive Applications,” AI Magazine, vol. 25, no. 4, pp. 63-76, 2004.
[27] P. Duygulu, M. Bastan, and D.A. Forsyth, “Translating Images to Words for Recognizing Objects in Large Image and Video Collections,” Toward Category-Level Object Recognition, pp. 258-276, 2006.
[28] K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. Jordan, “Matching Words and Pictures,” J. Machine Learning Research, vol. 3, pp. 1107-1135, 2003.
[29] K. Barnard, P. Duygulu, N. de Freitas, and D. Forsyth, Exploiting Text and Image Feature Co-Occurrence Statistics in Large Datasets, R.Veltkamp, H. Kriegel, and L. Shapiro, eds., Chapter in Trends and Advances in Content-Based Image and Video Retrieval, Springer, 2005.
[30] D.M. Blei and M.I. Jordan, “Modeling Annotated Data,” Proc. 26th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR '03), pp. 127-134, 2003.
[31] M. Johnson, G.J. Brostow, J. Shotton, O. Arandjelovic, V. Kwatra, and R. Cipolla, “Semantic Photosynthesis,” Computer Graphics Forum, vol. 25, no. 3, pp. 407-413, 2006.
[32] H. Feng, R. Shi, and T.S. Chua, “A Bootstrapping Framework for Annotating and Retrieving WWW Images,” Proc. 12th Ann. ACM Int'l Conf. Multimedia (Multimedia '04), pp. 960-967, 2004.
[33] M.L. Kherfi, D. Ziou, and A. Bernardi, “Image Retrieval from the World Wide Web: Issues, Techniques, and Systems,” ACM Computing Surveys, vol. 36, no. 1, pp. 35-67, 2004.
[34] I.A. Azzam, C.H.C. Leung, and J.F. Horwood, “Implicit Concept-Based Image Indexing and Retrieval,” Proc. 10th Int'l Conf. Multimedia Modeling (MMM '04), pp. 354-359, Jan. 2004.
[35] A.P. Natsev, A. Haubold, J. Tešić, L. Xie, and R. Yan, “Semantic Concept-Based Query Expansion and Re-Ranking for Multimedia Retrieval,” Proc. 15th Ann. ACM Int'l Conf. Multimedia (Multimedia '07), pp.991-1000, 2007.
[36] J. Bidner, Amphoto's Complete Book of Photography. Amphoto Books, 2004.
[37] R. Lenman, The Oxford Companion to the Photograph. Oxford Univ. Press, 2005.
[38] Exchangeable Image File Format for Digital Still Cameras: Exif Version 2.2 JEITA CP-3451, Technical Standardization Committee on AV and IT Storage Systems and Equipment and Standard of Japan Electronics and Information Technology Industries Assoc., Apr. 2002.
[39] S. Ruggieri, “Efficient c4.5,” IEEE Trans. Knowledge and Data Eng., vol. 14, no. 2, pp. 438-444, Mar./Apr. 2002.
[40] R. Venkatesh, C. Rowland, H. Huang, O.T. Abar, and J. Sninsky, “Robust Model Selection Using Cross Validation: A Simple Iterative Technique for Developing Robust Gene Signatures in Biomedical Genomics Applications,” Proc. Int'l Conf. Machine Learning and Applications (ICMLA '06), pp. 193-198, 2006.
[41] A. Kalousis, J. Prados, J.C. Sanchez, L. Allard, and M. Hilario, “Distilling Classification Models from Cross Validation Runs: An Application to Mass Spectrometry,” Proc. 16th IEEE Int'l Conf. Tools with Artificial Intelligence (ICTAI '04), pp. 113-119, 2004.
[42] A.M. Tam and C.H.C. Leung, Semantic Content Retrieval and Structured Annotation: Beyond Keywords, ISO/IEC JTC1/SC29/WG11 MPEG00/M5738, Mar. 2000.
[43] P. van Beek et al., Multimedia Content Description Interface—Part 5. Multimedia Description Schemes, ISO/IEC JTC1/SC29/WG11 MPEG01/N3966, Mar. 2001.
[44] I. Azzam, A.G. Charlapally, C.H.C. Leung, and J.F. Horwood, “Content-Based Image Indexing and Retrieval with XML Representations,” Proc. Int'l Symp. Intelligent Multimedia, Video and Speech Processing (ISIMP '04), pp. 181-185, 2004.
[45] A.M. Tam and C.H.C. Leung, “Structured Natural-Language Descriptions for Semantic Content Retrieval of Visual Materials,” J. Am. Soc. for Information Science and Technology, vol. 52, no. 11, pp.930-937, Mar. 2001.
[46] S. Jeong and R.M. Gray, “Histogram-Based Image Retrieval Using Gauss Mixture Vector Quantization,” Proc. Int'l Conf. Multimedia and Expo (ICME '03), pp. 397-400, 2003.
[47] A. Williams and P. Yoon, “Content-Based Image Retrieval Using Joint Correlograms,” Multimedia Tools and Applications, vol. 34, no. 2, pp. 239-248, 2007.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool