This Article 
 Bibliographic References 
 Add to: 
Video Data Mining: Semantic Indexing and Event Detection from the Association Perspective
May 2005 (vol. 17 no. 5)
pp. 665-677
Advances in the media and entertainment industries, including streaming audio and digital TV, present new challenges for managing and accessing large audio-visual collections. Current content management systems support retrieval using low-level features, such as motion, color, and texture. However, low-level features often have little meaning for naive users, who much prefer to identify content using high-level semantics or concepts. This creates a gap between systems and their users that must be bridged for these systems to be used effectively. To this end, in this paper, we first present a knowledge-based video indexing and content management framework for domain specific videos (using basketball video as an example). We will provide a solution to explore video knowledge by mining associations from video data. The explicit definitions and evaluation measures (e.g., temporal support and confidence) for video associations are proposed by integrating the distinct feature of video data. Our approach uses video processing techniques to find visual and audio cues (e.g., court field, camera motion activities, and applause), introduces multilevel sequential association mining to explore associations among the audio and visual cues, classifies the associations by assigning each of them with a class label, and uses their appearances in the video to construct video indices. Our experimental results demonstrate the performance of the proposed approach.

[1] H. Zhang, A. Kantankanhalli, and S. Smoliar, “Automatic Partitioning of Full-Motion Video,” ACM Multimedia Systems, vol. 1, no. 1, pp. 10-28, 1993.
[2] A. Yoshitaka and T. Ichikawa, “A Survey on Content-Based Retrieval for Multimedia Databases,” IEEE Trans. Knowledge and Data Eng., vol. 11, no. 1, pp. 81-93, Jan./Feb. 1999.
[3] H. Jiang and A.K. Elmagarmid, “WVTDB— A Semantic Content-Based Video Database System on the World Wide Web,” IEEE Trans. Knowledge and Data Eng., vol. 10, no. 6, pp. 947-966, Nov./Dec. 1998.
[4] C. Snoek and M. Worring, “Multimodal Video Indexing: A Review of the State-of-the-Art,” Multimedia Tools and Applications, to be published in 2005.
[5] F. Kokkoras, H. Jiang, I. Vlahavas, A. Elmagarmid, E. Houstis, and W. Aref, “Smart VideoText: A Video Data Model Based on Conceptual Graphs,” ACM/Springer Multimedia Systems, vol. 8, no. 4, pp. 328-338, 2002.
[6] X. Zhu, J. Fan, W.G. Aref, and A.K. Elmagarmid, “ClassMiner: Mining Medical Video Content Structure and Events Towards Efficient Access and Scalable Skimming,” Proc. ACM SIGMOD Workshop, pp. 9-16, 2002.
[7] X. Zhu, W. Aref, J. Fan, A. Catlin, and A. Elmagarmid, “Medical Video Mining for Efficient Database Indexing, Management and Access,” Proc. 19th Int'l Conf. Data Eng., pp. 569-580, 2003.
[8] Y. Matsuo, K. Shirahama, and K. Uehara, “Video Data Mining: Extracting Cinematic Rules from Movie,” Proc. Int'l Workshop Multimedia Data Management (MDM-KDD), 2003.
[9] R.R. Wang and T.S. Huang, “A Framework of Human Motion Tracking and Event Detection for Video Indexing and Mining,” Proc. DIMACS Workshop Video Mining, 2002.
[10] J. Oh and B. Bandi, “Multimedia Data Mining Framework for Raw Video Sequence,” Proc. Int'l Workshop Multimedia Data Management (MDM-KDD), 2002.
[11] J. Pan and C. Faloutsos, “VideoCube: A Novel Tool for Video Mining and Classification,” Proc. Int'l Conf. Asian Digital Libraries (ICADL), pp. 194-205, 2002.
[12] J. Pan and C. Faloutsos, “GeoPlot: Spatial Data Mining on Video Libraries,” Proc. Int'l Conf. Information and Knowledge Management, pp. 405-412, 2002.
[13] X. Zhu and X. Wu, “Mining Video Association for Efficient Database Management,” Proc. Int'l Joint Conf. Artificial Intelligence, pp. 1422-1424, 2003.
[14] X. Zhu and X. Wu, “Sequential Association Mining for Video Summarization,” Proc. IEEE Int'l Conf. Multimedia and Expo, vol. 3, pp. 333-336, 2003.
[15] J. Fan, X. Zhu, and X. Lin, “Mining of Video Database,” Multimedia Data Mining, 2002.
[16] L. Xie, S.-F. Chang, A. Divakaran, and H. Sun, “Unsupervised Mining of Statistical Temporal Structures in Video,” Video Mining, A. Rosenfeld, D. Doremann, and D. Dementhon eds., Kluwer Academic, 2003.
[17] D. Wijesekera and D. Barbara, “Mining Cinematic Knowledge: Work in Progress,” Proc. Int'l Workshop Multimedia Data Management (MDM-KDD), 2000.
[18] M. Windhouwer, R. Zwol, H. Blok, W. Jonker, M. Kersten, and P. Apers, “Content-Based Video Indexing for the Support of Digital Library Search,” Proc. Int'l Conf. Data Eng., pp. 494-495, 2002.
[19] S. Newsam, J. Tesic, L. Wang, and B.S. Manjunath, “Mining Images and Video,” Proc. DIMACS Workshop Video Mining, 2002.
[20] R. Agrawal and R. Srikant, “Fast Algorithms for Mining Association Rules,” Proc. Very Large Data Bases Conf., pp. 487-499, 1994.
[21] R. Agrawal and R. Srikant, “Mining Sequential Patterns,” Proc. 11th Int'l Conf. Data Eng., 1995.
[22] J. Han and M. Kamber, Data Mining: Concepts and Techniques. Morgan Kaufmann, 2000.
[23] B. Thuraisingham, Managing and Mining Multimedia Database. CRC Press, 2001.
[24] O. Zaiane, J. Han, Z. Li, S. Chee, and J. Chiang, “MultimediaMiner: A System Prototype for Multimedia Data Mining,” Proc. ACM SIGMOD, pp. 581-583, 1998.
[25] S. Nepal, U. Srinivasan, and G. Reynolds, “Automatic Detection of ‘Goal’ Segments in Basketball Videos,” Proc. Ninth ACM Multimedia Conf., pp. 261-269, 2001.
[26] L. Duan, M. Xu, T. Chua, Q. Tian, and C. Xu, “A Mid-Level Representation Framework for Semantic Sports Video Analysis,” Proc. of 11th ACM Multimedia Conf., pp. 33-44, 2003.
[27] J. Fan, W.G. Aref, A.K. Elmagarmid, M. Hacid, M. Marzouk, and X. Zhu, “MultiView: Multi-Level Video Content Representation and Retrieval,” J. Electronic Imaging, vol. 10, no. 4, pp. 895-908, 2001.
[28] W. Zhou, A. Vellaikal, and C. Kuo, “Rule-Based Video Classification System for Basketball Video Indexing,” Proc. ACM Multimedia Workshops, pp. 213-216, 2000.
[29] L. Xie, S. Chang, A. Divakaran, and H. Sun, “Structure Analysis of Soccer Video with Hidden Markov Models,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP), 2002.
[30] W. Wolf, “Key Frame Selection by Motion Analysis,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. 1228-1231, 1996.
[31] H. Tamura, S. Mori, and T. Yamawaki, “Texture Features Corresponding to Visual Perception,” IEEE Trans. Systems, Man, and Cybernetics, vol. 8, no. 6, pp. 460-473, 1978.
[32] S. Horowitz and T. Pavlidis, “Picture Segmentation by a Directed Split-and-Merge Procedure,” Proc. Int'l Joint Conf. Pattern Recognition, pp. 424-433, 1974.
[33] WOCAR Engine 2.5,, 2004.
[34] A. Jain and B. Yu, “Automatic Text Location in Images and Video Frames,” Pattern Recognition, vol. 31, no. 12, pp. 2055-2076, 1998.
[35] X. Zhu, A.K. Elmagarmid, X. Xue, L. Wu, and A. Catlin, “InsightVide: Towards Hierarchical Video Content Organization for Efficient Browsing, Summarization, and Retrieval,” IEEE Trans. Multimedia, 2004.
[36] T. Oates and P. Cohen, “Searching for Structure in Multiple Streams of Data,” Proc. 13th Int'l Conf. Machine Learning, pp. 346-354, 1996.
[37] J. Han, G. Dong, and Y. Yin, “Efficient Mining Partial Periodic Patterns in Time Series Database,” Proc. Int'l Conf. Data Eng., pp. 106-115, 1999.
[38] R. Srikant and R. Agrawal, “Mining Generalized Association Rules,” Proc. 21th Very Large Data Bases Conf., 1995.
[39] R. Gwadera, M. Atallah, and W. Szpankowski, “Reliable Detection of Episodes in Event Sequences,” Proc. Third Int'l Conf. Data Mining, pp. 67-74, 2003.
[40] T. Cormen, C. Leiserson, R. Rivest, and C. Stein, Introduction to Algorithms. MIT Press, 2001.
[41] B. Cui, B. Ooi, J. Su, and K. Tan, “Contorting High Dimensional Data for Efficient Main Memory Processing,” Proc. SIGMOD Conf., pp. 479-490, 2003.
[42] A. Guttman, “R-Trees: A Dynamic Index Structure for Spatial Searching,” Proc. SIGMOD Conf., pp. 47-57, 1984.
[43] N. Katayama and S. Satoh, “The SR-Tree: An Index Structure for High-Dimensional Nearest Neighbor Queries,” Proc. SIGMOD Conf., pp. 369-380, 1997.
[44] W. Hsu, J. Dai, and M. Lee, “Mining Viewpoint Patterns in Image Databases,” Proc. SIGKDD, pp. 553-558, 2003.
[45] H. Mannila, H. Toivonen, and A. Verkamo, “Discovery of Frequent Episodes in Event Sequences,” Data Mining and Knowledge Discovery, vol. 1, no. 3, pp. 259-289, 1997.
[46] R. Srikant and R. Agrawal, “Mining Sequential Patterns: Generalizations and Performance Improvements,” Proc. Fifth Int'l Conf. Extending Database Technology (EDBT), 1996.

Index Terms:
Video mining, multimedia systems, database management, knowledge-based systems.
Xingquan Zhu, Xindong Wu, Ahmed K. Elmagarmid, Zhe Feng, Lide Wu, "Video Data Mining: Semantic Indexing and Event Detection from the Association Perspective," IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 5, pp. 665-677, May 2005, doi:10.1109/TKDE.2005.83
Usage of this product signifies your acceptance of the Terms of Use.