This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Online Empirical Evaluation of Tracking Algorithms
August 2010 (vol. 32 no. 8)
pp. 1443-1458
ASCII Text
x 
 
Hao Wu, Aswin C. Sankaranarayanan, Rama Chellappa, "Online Empirical Evaluation of Tracking Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1443-1458, August, 2010.
 
 
BibTex
x
 
@article{ 10.1109/TPAMI.2009.135,
author = {Hao Wu and Aswin C. Sankaranarayanan and Rama Chellappa},
title = {Online Empirical Evaluation of Tracking Algorithms},
journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {32},
number = {8},
issn = {0162-8828},
year = {2010},
pages = {1443-1458},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2009.135},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
TI - Online Empirical Evaluation of Tracking Algorithms
IS - 8
SN - 0162-8828
SP1443
EP1458
EPD - 1443-1458
A1 - Hao Wu,
A1 - Aswin C. Sankaranarayanan,
A1 - Rama Chellappa,
PY - 2010
KW - Performance evaluation
KW - tracking
KW - particle filters
KW - model validation.
VL - 32
JA - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -
 
Hao Wu, University of Maryland, College Park
Aswin C. Sankaranarayanan, University of Maryland, College Park
Rama Chellappa, University of Maryland, College Park
Evaluation of tracking algorithms in the absence of ground truth is a challenging problem. There exist a variety of approaches for this problem, ranging from formal model validation techniques to heuristics that look for mismatches between track properties and the observed data. However, few of these methods scale up to the task of visual tracking, where the models are usually nonlinear and complex and typically lie in a high-dimensional space. Further, scenarios that cause track failures and/or poor tracking performance are also quite diverse for the visual tracking problem. In this paper, we propose an online performance evaluation strategy for tracking systems based on particle filters using a time-reversed Markov chain. The key intuition of our proposed methodology relies on the time-reversible nature of physical motion exhibited by most objects, which in turn should be possessed by a good tracker. In the presence of tracking failures due to occlusion, low SNR, or modeling errors, this reversible nature of the tracker is violated. We use this property for detection of track failures. To evaluate the performance of the tracker at time instant t, we use the posterior of the tracking algorithm to initialize a time-reversed Markov chain. We compute the posterior density of track parameters at the starting time t=0 by filtering back in time to the initial time instant. The distance between the posterior density of the time-reversed chain (at t=0) and the prior density used to initialize the tracking algorithm forms the decision statistic for evaluation. It is observed that when the data are generated by the underlying models, the decision statistic takes a low value. We provide a thorough experimental analysis of the evaluation methodology. Specifically, we demonstrate the effectiveness of our approach for tackling common challenges such as occlusion, pose, and illumination changes and provide the Receiver Operating Characteristic (ROC) curves. Finally, we also show the applicability of the core ideas of the paper to other tracking algorithms such as the Kanade-Lucas-Tomasi (KLT) feature tracker and the mean-shift tracker.

[1] J. Black, T. Ellis, and P. Rosin, "A Novel Method for Video Tracking Performance Evaluation," Proc. Joint IEEE Int'l Workshop Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 125-132, 2003.
[2] A.T. Nghiem, F. Bremond, M. Thonnat, and R. Ma, "A New Evaluation Approach for Video Processing Algorithms," Proc. IEEE Workshop Motion and Video Computing, pp. 15-15, Feb. 2007.
[3] T. Schlogl, C. Beleznai, M. Winter, and H. Bischof, "Performance Evaluation Metrics for Motion Detection and Tracking," Proc. Int'l Conf. Pattern Recognition, vol. 4, 2004.
[4] A.T. Nghiem, F. Bremond, M. Thonnat, and V. Valentin, "ETISEO, Performance Evaluation for Video Surveillance Systems," Proc. IEEE Conf. Advanced Video and Signal Based Surveillance, pp. 476-481, Sept. 2007.
[5] R. Kasturi, D. Goldgof, P. Soundararajan, V. Manohar, M. Boonstra, and V. Korzhova, "Performance Evaluation Protocol for Face, Person and Vehicle Detection & Tracking in Video Analysis and Content Extraction (VACE II) CLEAR—Classification of Events, Activities and Relationships," Advanced Research and Development Activity, Jan. 2006.
[6] J.C. Nascimento and J.S. Marques, "Performance Evaluation of Object Detection Algorithms for Video Surveillance," IEEE Trans. Multimedia, vol. 8, no. 4, pp. 761-774, Aug. 2006.
[7] B. Georis, F. Bremond, M. Thonnat, and B. Macq, "Use of an Evaluation and Diagnosis Method to Improve Tracking Performances," Proc. Int'l Conf. Visualization, Imaging, and Image Processing, 2003.
[8] T. List, J. Bins, J. Vazquez, and R.B. Fisher, "Performance Evaluating the Evaluator," Proc. Joint IEEE Int'l Workshop Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 129-136, Oct. 2005.
[9] N. Vaswani, "Additive Change Detection in Nonlinear Systems with Unknown Change Parameters," IEEE Trans. Signal Processing, vol. 55, no. 3, pp. 859-872, Mar. 2007.
[10] C. Andrieu, A. Doucet, S.S. Singh, and V.B. Tadic, "Particle Methods for Change Detection, System Identification, and Control," Proc. IEEE, vol. 92, no. 3, pp. 423-438, Mar. 2004.
[11] R. Gerlach, C. Carter, and R. Kohn, "Diagnostics for Time Series Analysis," J. Time Series Analysis, vol. 20, pp. 309-330, 1999.
[12] J. Vermaak, C. Andrieu, A. Doucet, and S. Godsill, "Particle Methods for Bayesian Modelling and Enhancement of Speech Signals," IEEE Trans. Speech and Audio Processing, vol. 10, no. 3, pp. 173-185, Mar. 2002.
[13] F. van der Heijen, "Consistency Checks for Particle Filters," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 1, pp. 140-145, Jan. 2006.
[14] L. Lu, X. Dai, and G. Hager, "A Particle Filter without Dynamics for Robust 3D Face Tracking," IEEE Proc. Computer Vision and Pattern Recognition Workshops, 2004.
[15] C.E. Erdem, A.M. Tekalp, and B. Sankur, "Metrics for Performance Evaluation of Video Object Segmentation and Tracking without Ground-Truth," Proc. IEEE Int'l Conf. Image Processing, pp. 69-72, Oct. 2001.
[16] C.E. Erdem, B. Sankur, and A.M. Tekalp, "Non-Rigid Object Tracking Using Performance Evaluation Measures as Feedback," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. II-323-II-330, 2001.
[17] H. Wu and Q. Zheng, "Performance Self-Evaluation of Visual Tracking Systems," Proc. Army Science Conf., 2004.
[18] B.D. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 674-679, 1981.
[19] C. Tomasi and T. Kanade, "Detection and Tracking of Point Features," Technical Report CMU-CS-91-132, Carnegie Mellon Univ., Apr. 1991.
[20] J. Shi and C. Tomasi, "Good Features to Track," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 593-600, 1994.
[21] D. Comaniciu, V. Ramesh, and P. Meer, "Real-Time Tracking of Non-Rigid Objects Using Mean Shift," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 142-149, 2000.
[22] D. Comaniciu, V. Ramesh, and P. Meer, "Kernel-Based Object Tracking," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 25, no. 5, pp. 564-575, May 2003.
[23] H. Wu, A.C. Sankaranarayanan, and R. Chellappa, "In Situ Evaluation of Tracking Algorithms Using Time Reversed Chain," Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2007.
[24] M. Isard and A. Blake, "Condensation-Conditional Density Propagation for Visual Tracking," Int'l J. Computer Vision, vol. 29, no. 1, pp. 5-28, 1998.
[25] A. Doucet, N. De Freitas, and N. Gordon, Sequential Monto Carlo Methods in Practice. Springer-Verlag, 2001.
[26] A. Veeraraghavan, R. Chellappa, and M. Srinivasan, "Shape-and-Behavior Encoded Tracking of Bee Dances," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 30, no. 3, pp. 463-476, Mar. 2008.
[27] Q. Zheng and R. Chellappa, "Automatic Feature Point Extraction and Tracking in Image Sequences for Arbitrary Camera Motion," Int'l J. Computer Vision, vol. 15, pp. 31-76, June 1995.
[28] M. Klass, M. Briers, N. De Freitas, A. Doucet, S. Maskell, and D. Lang, "Fast Particle Smoothing: If I Had a Million Particles," Proc. IEEE Int'l Conf. Machine Learning, 2006.
[29] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, second ed. Wiley Interscience, 2000.
[30] J. Goldberger, S. Gordon, and H. Greenspan, "An Efficient Image Similarity Measure Based on Approximations of KL-Divergence between Two Gaussian Mixtures," Proc. IEEE Int'l Conf. Computer Vision, pp. 487-493, Oct. 2003.
[31] H. Wu, R. Chellappa, A.C. Sankaranarayanan, and S.K. Zhou, "Robust Visual Tracking Using the Time-Reversibility Constraint," Proc. IEEE 11th Int'l Conf. Computer Vision, pp. 1-8, 2007.
[32] A. Yilmaz, "Object Tracking by Asymmetric Kernel Mean Shift with Automatic Scale and Orientation Selection," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-6, June 2007.
[33] S.K. Yeung and P. Shi, "Stochastic Inverse Consistency in Medical Image Registration," Proc. Medical Image Computing and Computer-Assisted Intervention, pp. 188-196, 2005.
[34] G.E. Christensen and H.J. Johnson, "Consistent Image Registration," IEEE Trans. Medical Imaging, vol. 20, no. 7, pp. 568-582, July 2001.
[35] S.M. Ross, Stochastic Processes. John Wiley and Sons, 1996.
[36] M. Klaas, M. Briers, N. de Freitas, A. Doucet, S. Maskell, and D. Lang, "Fast Particle Smoothing: If I Had a Million Particles," Proc. 23rd Int'l Conf. Machine Learning, pp. 481-488, 2006.
[37] S.K. Zhou, R. Chellappa, and B. Moghaddam, "Visual Tracking and Recognition Using Appearance-Adaptive Models in Particle Filters," IEEE Trans. Image Processing, vol. 13, no. 11, pp. 1491-1506, Nov. 2004.
[38] A.D. Jepson, D.J. Fleet, and T.F. El-Maraghi, "Robust Online Appearance Models for Visual Tracking," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1296-1311, Oct. 2003.

Index Terms:
Performance evaluation, tracking, particle filters, model validation.
Citation:
Hao Wu, Aswin C. Sankaranarayanan, Rama Chellappa, "Online Empirical Evaluation of Tracking Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1443-1458, Aug. 2010, doi:10.1109/TPAMI.2009.135
Usage of this product signifies your acceptance of the Terms of Use.