This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach
November 2011 (vol. 33 no. 11)
pp. 2287-2301
Brendan Tran Morris, University of California, San Diego, La Jolla
Mohan Manubhai Trivedi, University of California, San Diego, La Jolla
Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.

[1] J.K. Aggarwal and Q. Cai, "Human Motion Analysis: A Review," Computer Vision and Image Understanding, vol. 73, pp. 428-440, 1999.
[2] K. Huang and M. Trivedi, "3D Shape Context Based Gesture Analysis Integrated with Tracking Using Omni Video Array," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, June 2005.
[3] L. Gorelick, M. Blank, E. Shechtman, M. Irani, and R. Basri, "Actions as Space-Time Shapes," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 12, pp. 2247-2253, Dec. 2007.
[4] S. Mitra and T. Acharya, "Gesture Recognition: A Survey," IEEE Trans. Systems, Man, and Cybernetics Part C: Applications and Rev., vol. 37, no. 3, pp. 311-324, May 2007.
[5] M.M. Trivedi, T.L. Gandhi, and K.S. Huang, "Distributed Interactive Video Arrays for Event Capture and Enhanced Situational Awareness," IEEE Intelligent Systems, vol. 20, no. 5, pp. 58-66, Sept./Oct. 2005.
[6] W. Yan and D.A. Forsyth, "Learning the Behavior of Users in a Public Space through Video Tracking," Proc. IEEE Workshop Applications of Computer Vision, pp. 370-377, Jan. 2005.
[7] B.T. Morris and M.M. Trivedi, "Learning, Modeling, and Classification of Vehicle Track Patterns from Live Video," IEEE Trans. Intelligent Transport Systems, vol. 9, no. 3, pp. 425-437, Sept. 2008.
[8] S. Messelodi, C.M. Modena, and M. Zanin, "A Computer Vision System for the Detection and Classification of Vehicles at Urban Road Intersections," Pattern Analysis and Applications, vol. 8, nos. 1/2, pp. 17-31, Sept. 2005.
[9] D. Demirdjian, K. Tollmar, K. Koile, N. Checka, and T. Darrell, "Activity Maps for Location-Aware Computing," Proc. IEEE Workshop Applications of Computer Vision, pp. 70-75, Dec. 2002.
[10] N. Johnson and D. Hogg, "Learning the Distribution of Object Trajectories for Event Recognition," Proc. British Conf. Machine Vision, vol. 2, pp. 583-592, Sept. 1995.
[11] N. Sumpter and A.J. Bulpitt, "Learning Spatio-Temporal Patterns for Predicting Object Behavior," Image and Vision Computing, vol. 18, pp. 697-704, June 2000.
[12] J. Owens and A. Hunter, "Application of the Self-Organising Map to Trajectory Classification," Proc. Third IEEE Int'l Workshop Visual Surveillance, pp. 77-83, July 2000.
[13] W. Hu, D. Xie, T. Tan, and S. Maybank, "Learning Activity Patterns Using Fuzzy Self-Organizing Neural Network," IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 34, no. 3, pp. 1618-1626, June 2004.
[14] I.N. Junejo, O. Javed, and M. Shah, "Multi Feature Path Modeling for Video Surveillance," Proc. 17th Int'l Conf. Pattern Recognition, pp. 716-719, Aug. 2004.
[15] W. Hu, X. Xiao, Z. Fu, D. Xie, T. Tan, and S. Maybank, "A System for Learning Statistical Motion Patterns," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 28, no. 9, pp. 1450-1464, Sept. 2006.
[16] B. Morris and M. Trivedi, "Learning and Classification of Trajectories in Dynamic Scenes: A General Framework for Live Video Analysis," Proc. IEEE Int'l Conf. Advanced Video and Signal Based Surveillance, pp. 154-161, Sept. 2008.
[17] D. Makris and T. Ellis, "Learning Semantic Scene Models from Observing Activity in Visual Surveillance," IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 35, no. 3, pp. 397-408, June 2005.
[18] F.I. Bashir, A.A. Khokhar, and D. Schonfeld, "Object Trajectory-Based Activity Classification and Recognition Using Hidden Markov Models," IEEE Trans. Image Processing, vol. 16, no. 7, pp. 1912-1919, July 2007.
[19] C. Piciarelli and G.L. Foresti, "Online Trajectory Clustering for Anomalous Events Detection," Pattern Recognition Letters, vol. 27, no. 15, pp. 1835-1842, Nov. 2006.
[20] X. Wang, X. Ma, and E. Grimson, "Unsupervised Activity Perception by Hierarchical Bayesian Models," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2007.
[21] X. Wang, K.T. Ma, G.-W. Ng, and W. Grimson, "Trajectory Analysis and Semantic Region Modeling Using a Nonparametric Bayesian Model," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2008.
[22] X. Wang, X. Ma, and W. Grimson, "Unsupervised Activity Perception in Crowded and Complicated Scenes Using Hierarchical Bayesian Models," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 31, no. 3, pp. 539-555, Mar. 2009.
[23] T. Xiang and S. Gong, "Video Behavior Profiling for Anomaly Detection," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 30, no. 5, pp. 893-908, May 2008.
[24] B.T. Morris and M.M. Trivedi, "A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance," IEEE Trans. Circuits and Systems Video Technology, special issue on video surveillance, vol. 18, no. 8, pp. 1114-1127, Aug. 2008.
[25] F. Porikli, "Learning Object Trajectory Patterns by Spectral Clustering," Proc. IEEE Int'l Conf. Multimedia and Expo, vol. 2, pp. 1171-1174, June 2004.
[26] F. Bashir, W. Qu, A. Khokhar, and D. Schonfeld, "HMM-Based Motion Recognition System Using Segmented Pca," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 3, pp. 1288-1291, Sept. 2005.
[27] A. Naftel and S. Khalid, "Classifying Spatiotemporal Object Trajectories Using Unsupervised Learning in the Coefficient Feature Space," Multimedia Systems, vol. 12, no. 3, pp. 227-238, Dec. 2006.
[28] Z. Zhang, K. Huang, and T. Tan, "Comparison of Similarity Measures for Trajectory Clustering in Outdoor Surveillance Scenes," Proc. IEEE Int'l Conf. Pattern Recognition, pp. 1135-1138, 2006.
[29] S. Atev, O. Masoud, and N. Papanikolopoulos, "Learning Traffic Patterns at Intersections by Spectral Clustering of Motion Trajectories," Proc. IEEE Conf. Intelligent Robots and Systems, pp. 4851-4856, Oct. 2006.
[30] P. Dickinson and A. Hunter, "Using Inactivity to Detect Unusual Behavior," Proc. IEEE Workshop Motion and Video Computing, Jan. 2008.
[31] A.P. Dempster, N.M. Laird, and D.B. Rubin, "Maximum Likelihood from Incomplete Data via the EM Algorithm," J. Royal Statistical Soc., vol. 39, pp. 1-38, 1977.
[32] M.A.T. Figueiredo and A.K. Jain, "Unsupervised Learning of Finite Mixture Models," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 24, no. 3, pp. 381-396, Mar. 2002.
[33] A.Y. Ng, M.I. Jordan, and Y. Weiss, "On Spectral Clustering: Analysis and an Algorithm," Proc. Advances in Neural Information Processing Systems, vol. 14, pp. 849-856, Sept. 2002.
[34] B. Morris and M. Trivedi, "Learning Trajectory Patterns by Clustering: Experimental Studies and Comparative Evaluation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 312-319, June 2009.
[35] M. Vlachos, G. Kollios, and D. Gunopulos, "Discovering Similar Multidimensional Trajectories," Proc. IEEE Conf. Data Eng., pp. 673-684, Feb. 2002.
[36] W. Pedrycz, Knowledge-Based Clustering: From Data to Information Granules. John Wiley, 2005.
[37] W. Hu, D. Xie, Z. Fu, W. Zeng, and S. Maybank, "Semantic-Based Surveillance Video Retrieval," IEEE Trans. Image Processing, vol. 16, no. 4, pp. 1168-1181, Apr. 2007.
[38] L. Rabiner and B. Juang, Fundamentals of Speech Recognition. Prentice-Hall, 1993.
[39] E. Swears, A. Hoogs, and A. Perera, "Learning Motion Patterns in Surveillance Video Using HMM Clustering," Proc. IEEE Workshop Motion and Video Computing, pp. 1-8, Jan. 2008.
[40] M. Gales, D. Pye, and P. Woodland, "Variance Compensation within the MLLR Framework for Robust Speech Recognition and Speaker Adaptation," Proc. IEEE Int'l Conf. Spoken Language, pp. 1832-1835, Oct. 1996.
[41] M. Russell and R. Moore, "Explicit Modelling of State Occupancy in Hidden Markov Models for Automatic Speech Recognition," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 5-8, Apr. 1985.
[42] B.T. Morris and M.M. Trivedi, "Contextual Activity Visualization from Long-Term Video Observations," IEEE Intelligent Systems, special issue on intelligent monitoring of complex environments, vol. 25, no. 3, pp. 50-62, May/June 2010.
[43] B. Morris and M. Trivedi, "Unsupervised Learning of Motion Patterns of Rear Surrounding Vehicles," Proc. IEEE Int'l Conf. Vehicular Electronics and Safety, pp. 80-85, Nov. 2009.

Index Terms:
Trajectory clustering, real-time activity analysis, abnormality detection, trajectory learning, activity prediction.
Citation:
Brendan Tran Morris, Mohan Manubhai Trivedi, "Trajectory Learning for Activity Understanding: Unsupervised, Multilevel, and Long-Term Adaptive Approach," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 11, pp. 2287-2301, Nov. 2011, doi:10.1109/TPAMI.2011.64
Usage of this product signifies your acceptance of the Terms of Use.