The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan. (2013 vol.25)
pp: 47-61
Y. Yildirim , Dept. of Comput. Eng., Middle East Tech. Univ., Ankara, Turkey
A. Yazici , Dept. of Comput. Eng., Middle East Tech. Univ., Ankara, Turkey
T. Yilmaz , Dept. of Comput. Eng., Middle East Tech. Univ., Ankara, Turkey
ABSTRACT
Recent increase in the use of video-based applications has revealed the need for extracting the content in videos. Raw data and low-level features alone are not sufficient to fulfill the user 's needs; that is, a deeper understanding of the content at the semantic level is required. Currently, manual techniques, which are inefficient, subjective and costly in time and limit the querying capabilities, are being used to bridge the gap between low-level representative features and high-level semantic content. Here, we propose a semantic content extraction system that allows the user to query and retrieve objects, events, and concepts that are extracted automatically. We introduce an ontology-based fuzzy video semantic content model that uses spatial/temporal relations in event and concept definitions. This metaontology definition provides a wide-domain applicable rule construction standard that allows the user to construct an ontology for a given domain. In addition to domain ontologies, we use additional rule definitions (without using ontology) to lower spatial relation computation cost and to be able to define some complex situations more effectively. The proposed framework has been fully implemented and tested on three different domains. We have obtained satisfactory precision and recall rates for object, event and concept extraction.
INDEX TERMS
video retrieval, feature extraction, fuzzy set theory, knowledge based systems, ontologies (artificial intelligence), domain ontology, automatic semantic content extraction system, ontology-based fuzzy video semantic content model, rule-based model, video-based applications, video content extraction, querying capability, low-level representative features, high-level semantic content, object retrieval, event retrieval, concept retrieval, spatial-temporal relations, metaontology definition, wide-domain applicable rule construction standard, Semantics, Videos, Ontologies, Feature extraction, Data mining, Data models, Visualization, ontology, Semantic content extraction, video content modeling, fuzziness
CITATION
Y. Yildirim, A. Yazici, T. Yilmaz, "Automatic Semantic Content Extraction in Videos Using a Fuzzy Ontology and Rule-Based Model", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 1, pp. 47-61, Jan. 2013, doi:10.1109/TKDE.2011.189
REFERENCES
[1] M. Petkovic and W. Jonker, "An Overview of Data Models and Query Languages for Content-Based Video Retrieval," Proc. Int'l Conf. Advances in Infrastructure for E-Business, Science, and Education on the Internet, Aug. 2000.
[2] M. Petkovic and W. Jonker, "Content-Based Video Retrieval by Integrating Spatio-Temporal and Stochastic Recognition of Events," Proc. IEEE Int'l Workshop Detection and Recognition of Events in Video, pp. 75-82, 2001.
[3] L.S. Davis, S. Fejes, D. Harwood, Y. Yacoob, I. Haratoglu, and M.J. Black, "Visual Surveillance of Human Activity," Proc. Third Asian Conf. Computer Vision (ACCV), vol. 2, pp. 267-274, 1998.
[4] G.G. Medioni, I. Cohen, F. Brémond, S. Hongeng, and R. Nevatia, "Event Detection and Analysis from Video Streams," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 23, no. 8, pp. 873-889, Aug. 2001.
[5] S. Hongeng, R. Nevatia, and F. Brémond, "Video-Based Event Recognition: Activity Representation and Probabilistic Recognition Methods," Computer Vision and Image Understanding, vol. 96, no. 2, pp. 129-162, 2004.
[6] A. Hakeem and M. Shah, "Multiple Agent Event Detection and Representation in Videos," Proc. 20th Nat'l Conf. Artificial Intelligence (AAAI), pp. 89-94, 2005.
[7] M.E. Dönderler, E. Saykol, U. Arslan, Ö. Ulusoy, and U. Güdükbay, "Bilvideo: Design and Implementation of a Video Database Management System," Multimedia Tools Applications, vol. 27, no. 1, pp. 79-104, 2005.
[8] T. Sevilmis, M. Bastan, U. Güdükbay, and Ö. Ulusoy, "Automatic Detection of Salient Objects and Spatial Relations in Videos for a Video Database System," Image Vision Computing, vol. 26, no. 10, pp. 1384-1396, 2008.
[9] M. Köprülü, N.K. Cicekli, and A. Yazici, "Spatio-Temporal Querying in Video Databases," Information Sciences, vol. 160, nos. 1-4, pp. 131-152, 2004.
[10] J. Fan, W. Aref, A. Elmagarmid, M. Hacid, M. Marzouk, and X. Zhu, "Multiview: Multilevel Video Content Representation and Retrieval," J. Electronic Imaging, vol. 10, no. 4, pp. 895-908, 2001.
[11] J. Fan, A.K. Elmagarmid, X. Zhu, W.G. Aref, and L. Wu, "Classview: Hierarchical Video Shot Classification, Indexing, and Accessing," IEEE Trans. Multimedia, vol. 6, no. 1, pp. 70-86, Feb. 2004.
[12] L. Bai, S.Y. Lao, G. Jones, and A.F. Smeaton, "Video Semantic Content Analysis Based on Ontology," IMVIP '07: Proc. 11th Int'l Machine Vision and Image Processing Conf., pp. 117-124, 2007.
[13] R. Nevatia and P. Natarajan, "EDF: A Framework for Semantic Annotation of Video," Proc. 10th IEEE Int'l Conf. Computer Vision Workshops (ICCVW '05), p. 1876, 2005.
[14] A.D. Bagdanov, M. Bertini, A. Del Bimbo, C. Torniai, and G. Serra, "Semantic Annotation and Retrieval of Video Events Using Multimedia Ontologies," Proc. IEEE Int'l Conf. Semantic Computing (ICSC), Sept. 2007.
[15] R. Nevatia, J. Hobbs, and B. Bolles, "An Ontology for Video Event Representation," Proc. Conf. Computer Vision and Pattern Recognition Workshop, p. 119, http://ieeexplore.ieee.org/xplsabs_all. jsp?arnumber=1384914 , 2004.
[16] U. Akdemir, P.K. Turaga, and R. Chellappa, "An Ontology Based Approach for Activity Recognition from Video," Proc. ACM Int'l Conf. Multimedia, A. El-Saddik, S. Vuong, C. Griwodz, A.D. Bimbo, K.S. Candan, and A. Jaimes, eds., pp. 709-712, http://dblp.uni-trier.de/db/conf/mmmm2008.html#AkdemirTC08 , 2008.
[17] Y. Yildirim, "Automatic Semantic Content Extraction in Video Using a Spatio-Temporal Ontology Model," PhD dissertation, Computer Eng. Dept., METU, Turkey, 2009.
[18] T. Yilmaz, "Object Extraction from Images/Videos Using a Genetic Algorithm Based Approach," master's thesis, Computer Eng. Dept., METU, Turkey, 2008.
[19] Y. Yildirim and A. Yazici, "Ontology-Supported Video Modeling and Retrieval," Proc. Fourth Int'l Conf. Adaptive Multimedia Retrieval: User, Context, and Feedback (AMR), pp. 28-41, 2006.
[20] Y. Yildirim, T. Yilmaz, and A. Yazici, "Ontology-Supported Object and Event Extraction with a Genetic Algorithms Approach for Object Classification," Proc. Sixth ACM Int'l Conf. Image and Video Retrieval (CIVR '07), pp. 202-209, 2007.
[21] V. Mezaris, I. Kompatsiaris, N.V. Boulgouris, and M.G. Strintzis, "Real-Time Compressed-Domain Spatiotemporal Segmentation and Ontologies for Video Indexing and Retrieval," IEEE Trans. Circuits Systems Video Technology, vol. 14, no. 5, pp. 606-621, May 2004.
[22] D. Song, H.T. Liu, M. Cho, H. Kim, and P. Kim, "Domain Knowledge Ontology Building for Semantic Video Event Description," Proc. Int'l Conf. Image and Video Retrieval (CIVR), pp. 267-275, 2005.
[23] W. Chen and D.S. Warren, "C-logic of Complex Objects," PODS '89: Proc. Eighth ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems, pp. 369-378, 1989.
[24] J.F. Allen, "Maintaining Knowledge about Temporal Intervals," Comm. ACM, vol. 26, no. 11, pp. 832-843, 1983.
[25] M.J. Egenhofer and J.R. Herring, "A Mathematical Framework for the Definition of Topological Relationships," Proc. Fourth Int'l Symp. Spatial Data Handling, pp. 803-813, 1990.
[26] M. Vazirgiannis, "Uncertainty Handling in Spatial Relationships," SAC '00: Proc. ACM Symp. Applied Computing, pp. 494-500, 2000.
[27] P.-W. Huang and C.-H. Lee, "Image Database Design Based on 9D-SPA Representation for Spatial Relations," IEEE Trans. Knowledge and Data Eng., vol. 16, no. 12, pp. 1486-1496, Dec. 2004.
[28] I. Horrocks, P.F. Patel-Schneider, H. Boley, S. Tabet, B. Grosof, and M. Dean, "Swrl: A Semantic Web Rule Language," technical report, W3C, http://www.w3.org/SubmissionSWRL/, 2004.
[29] "Protégé Ontology Editor," http:/protege.stanford.edu/, 2012.
[30] "Jena: A Semantic Web Framework," http://www.hpl.hp.comsemweb/, 2012.
[31] C. Xu, J. Wang, K. Wan, Y. Li, and L. Duan, "Live Sports Event Detection Based on Broadcast Video and Web-Casting Text," MULTIMEDIA '06: Proc. 14th Ann. ACM Int'l Conf. Multimedia, pp. 221-230, 2006.
[32] Y. Zhang, C. Xu, Y. Rui, J. Wang, and H. Lu, "Semantic Event Extraction from Basketball Games Using Multi-Modal Analysis," Proc. IEEE Int'l Conf. Multimedia and Expo (ICME '07), pp. 2190-2193, 2007.
22 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool