Subscribe
Issue No.10 - Oct. (2013 vol.35)
pp: 2553-2560
I. Leichter , Adv. Technol. Labs. Israel- Microsoft Res., Microsoft R&D Center, Haifa, Israel
E. Krupka , Adv. Technol. Labs. Israel- Microsoft Res., Microsoft R&D Center, Haifa, Israel
ABSTRACT
There exists an abundance of systems and algorithms for multiple target detection and tracking in video, and many measures for evaluating the quality of their output have been proposed. The contribution of this paper lies in the following: first, it argues that such performance measures should have two fundamental properties-monotonicity and error type differentiability; second, it shows that the recently proposed measures do not have either of these properties and are, thus, less usable; third, it composes a set of simple measures, partly built on common practice, that does have these properties. The informativeness of the proposed set of performance measures is demonstrated through their application on face detection and tracking results.
INDEX TERMS
Target tracking, Measurement uncertainty, Corporate acquisitions, Indexes, Object detection, Context,multiple targets, Performance evaluation, tracking
CITATION
I. Leichter, E. Krupka, "Monotonicity and Error Type Differentiability in Performance Measures for Target Detection and Tracking in Video", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 10, pp. 2553-2560, Oct. 2013, doi:10.1109/TPAMI.2013.70
REFERENCES
[1] M. Fischer, "Person Re-Identification in TV Series," https://cvhci.anthropomatik.kit.edu/~mfischer/ researchperson-reidentification/, 2013.
[2] ViPER: The Video Performance Evaluation Resource, http:/viper- toolkit.sourceforge.net/, 2013.
[3] Classification of Events, Activities and Relationships (CLEAR) Evaluation and Workshop, 2006-2007, http:/www.clear-evaluation.org/, 2013.
[4] K. Bernardin and R. Stiefelhagen, "Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics," EURASIP J. Image and Video Processing, vol. 2008, article 246309, 2008.
[5] R. Collins, X. Zhou, and S. Teh, "An Open Source Tracking Testbed and Evaluation Web Site," Proc. IEEE Int'l Workshop Performance Evaluation of Tracking and Surveillance, 2005.
[6] E. Kao, M. Daggett, and M. Hurley, "An Information Theoretic Approach for Tracker Performance Evaluation," Proc. IEEE Int'l Conf. Computer Vision, pp 1523-1529, 2009.
[7] R. Kasturi, D. Goldgof, P. Soundararajan, V. Manohar, J. Garofolo, R. Bowers, M. Boonstra, V. Korzhova, and J. Zhang, "Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 2 pp. 319-336, Feb. 2009.
[8] I. Leichter and E. Krupka, Technical Report MSR-TR-2012-23, Microsoft Research (the code is in http://research.microsoft.com/pubs/160662 Performance_Measures.zip), 2012.
[9] I. Leichter, M. Lindenbaum, and E. Rivlin, "Tracking by Affine Kernel Transformations Using Color and Boundary Cues," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1 pp. 164-171, Jan. 2009.
[10] P. Viola and M. Jones, "Robust Real-Time Face Detection," Int'l J. Computer Vision, vol. 57, no. 2 pp. 137-154, 2004.
[11] F. Yin, D. Makris, and S. Velastin, "Performance Evaluation of Object Tracking Algorithms," Proc. IEEE 10th Int'l Workshop Performance Evaluation of Tracking and Surveillance, pp. 17-24, 2007.
[12] J. Yuen, B. Russell, C. Liu, and A. Torralba, "LabelMe Video: Building a Video Database with Human Annotations," Proc. 12th IEEE Int'l Conf. Computer Vision, pp. 1451-1458, 2009.