The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.09 - September (2010 vol.32)
pp: 1582-1596
Koen E.A. van de Sande , University of Amsterdam, Amsterdam
Theo Gevers , University of Amsterdam, Amsterdam
Cees G.M. Snoek , University of Amsterdam, Amsterdam
ABSTRACT
Image category recognition is important to access visual information on the level of objects and scene types. So far, intensity-based descriptors have been widely used for feature extraction at salient points. To increase illumination invariance and discriminative power, color descriptors have been proposed. Because many different descriptors exist, a structured overview is required of color invariant descriptors in the context of image category recognition. Therefore, this paper studies the invariance properties and the distinctiveness of color descriptors (software to compute the color descriptors from this paper is available from http://www.colordescriptors.com) in a structured way. The analytical invariance properties of color descriptors are explored, using a taxonomy based on invariance properties with respect to photometric transformations, and tested experimentally using a data set with known illumination conditions. In addition, the distinctiveness of color descriptors is assessed experimentally using two benchmarks, one from the image domain and one from the video domain. From the theoretical and experimental results, it can be derived that invariance to light intensity changes and light color changes affects category recognition. The results further reveal that, for light intensity shifts, the usefulness of invariance is category-specific. Overall, when choosing a single descriptor and no prior knowledge about the data set and object and scene categories is available, the OpponentSIFT is recommended. Furthermore, a combined set of color descriptors outperforms intensity-based SIFT and improves category recognition by 8 percent on the PASCAL VOC 2007 and by 7 percent on the Mediamill Challenge.
INDEX TERMS
Image/video retrieval, evaluation/methodology, color, invariants, pattern recognition.
CITATION
Koen E.A. van de Sande, Theo Gevers, Cees G.M. Snoek, "Evaluating Color Descriptors for Object and Scene Recognition", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 9, pp. 1582-1596, September 2010, doi:10.1109/TPAMI.2009.154
REFERENCES
[1] R. Datta, D. Joshi, J. Li, and J.Z. Wang, "Image Retrieval: Ideas, Influences, and Trends of the New Age," ACM Computing Surveys, vol. 40, no. 2, pp. 1-60, 2008.
[2] R. Fergus, F.-F. Li, P. Perona, and A. Zisserman, "Learning Object Categories from Google's Image Search," Proc. IEEE Int'l Conf. Computer Vision, pp. 1816-1823, 2005.
[3] S. Lazebnik, C. Schmid, and J. Ponce, "Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 2169-2178, 2006.
[4] J. Vogel and B. Schiele, "Semantic Modeling of Natural Scenes for Content-Based Image Retrieval," Int'l J. Computer Vision, vol. 72, no. 2, pp. 133-157, 2007.
[5] J. Zhang, M. Marszałek, S. Lazebnik, and C. Schmid, "Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study," Int'l J. Computer Vision, vol. 73, no. 2, pp. 213-238, 2007.
[6] S.-F. Chang, D. Ellis, W. Jiang, K. Lee, A. Yanagawa, A.C. Loui, and J. Luo, "Large-Scale Multimodal Semantic Concept Detection for Consumer Video," Proc. ACM Int'l Workshop Multimedia Information Retrieval, pp. 255-264, 2007.
[7] A.F. Smeaton, P. Over, and W. Kraaij, "Evaluation Campaigns and TRECVid," Proc. ACM Int'l Workshop Multimedia Information Retrieval, pp. 321-330, 2006.
[8] Y.-G. Jiang, C.-W. Ngo, and J. Yang, "Towards Optimal Bag-of-Features for Object Categorization and Semantic Video Retrieval," Proc. ACM Int'l Conf. Image and Video Retrieval, pp. 494-501, 2007.
[9] D.G. Lowe, "Distinctive Image Features from Scale Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[10] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool, "A Comparison of Affine Region Detectors," Int'l J. Computer Vision, vol. 65, nos. 1/2, pp. 43-72, 2005.
[11] T. Tuytelaars and K. Mikolajczyk, "Local Invariant Feature Detectors: A Survey," Foundations and Trends in Computer Graphics and Vision, vol. 3, no. 3, pp. 177-280, 2008.
[12] A.E. Abdel-Hakim and A.A. Farag, "CSIFT: A SIFT Descriptor with Color Invariant Characteristics," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1978-1983, 2006.
[13] J.M. Geusebroek, R. van den Boomgaard, A.W.M. Smeulders, and H. Geerts, "Color Invariance," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 12, pp. 1338-1350, Dec. 2001.
[14] J. van de Weijer, T. Gevers, and A. Bagdanov, "Boosting Color Saliency in Image Feature Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 1, pp. 150-156, Jan. 2006.
[15] G.J. Burghouts and J.M. Geusebroek, "Performance Evaluation of Local Color Invariants," Computer Vision and Image Understanding, vol. 113, pp. 48-62, 2009.
[16] A. Bosch, A. Zisserman, and X. Muñoz, "Scene Classification Using a Hybrid Generative/Discriminative Approach," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 4, pp. 712-727, Apr. 2008.
[17] G.D. Finlayson, M.S. Drew, and B.V. Funt, "Spectral Sharpening: Sensor Transformations for Improved Color Constancy," J. Optical Soc. Am. A, vol. 11, no. 5, p. 1553, 1994.
[18] J. von Kries, "Influence of Adaptation on the Effects Produced by Luminous Stimuli," Sources of Color Vision, D.L. MacAdam, ed., MIT Press, 1970.
[19] K.E.A. van de Sande, T. Gevers, and C.G.M. Snoek, "Evaluation of Color Descriptors for Object and Scene Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2008.
[20] J.M. Geusebroek, G.J. Burghouts, and A.W.M. Smeulders, "The Amsterdam Library of Object Images," Int'l J. Computer Vision, vol. 61, no. 1, pp. 103-112, 2005.
[21] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results," http://www.pascal-network.org/ challenges/ VOC voc2007/, 2010.
[22] C.G.M. Snoek, M. Worring, J.C. van Gemert, J.-M. Geusebroek, and A.W.M. Smeulders, "The Challenge Problem for Automated Detection of 101 Semantic Concepts in Multimedia," Proc. ACM Int'l Conf. Multimedia, pp. 421-430, 2006.
[23] M. Shafer, "Using Color to Seperate Reflection Components," Color Research and Applications, vol. 10, no. 4, pp. 210-218, 1985.
[24] G.D. Finlayson, S.D. Hordley, and R. Xu, "Convex Programming Colour Constancy with a Diagonal-Offset Model," Proc. IEEE Int'l Conf. Image Processing, pp. 948-951, 2005.
[25] T. Gevers, J. van de Weijer, and H. Stokman, Color Image Processing: Methods and Applications: Color Feature Detection: An Overview, chapter 9, pp. 203-226. CRC Press, 2006.
[26] F. Mindru, T. Tuytelaars, L. Van Gool, and T. Moons, "Moment Invariants for Recognition under Changing Viewpoint and Illumination," Computer Vision and Image Understanding, vol. 94, nos. 1-3, pp. 3-27, 2004.
[27] J. Matas, O. Chum, M. Urban, and T. Pajdla, "Robust Wide Baseline Stereo from Maximally Stable Extremal Regions," Image and Vision Computing, vol. 22, no. 10, pp. 761-767, 2004.
[28] P.-E. Forssén, "Maximally Stable Colour Regions for Recognition and Matching," Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2007.
[29] J. Sivic and A. Zisserman, "Video Google: A Text Retrieval Approach to Object Matching in Videos," Proc. IEEE Int'l Conf. Computer Vision, pp. 1470-1477, 2003.
[30] T.K. Leung and J. Malik, "Representing and Recognizing the Visual Appearance of Materials Using Three-Dimensional Textons," Int'l J. Computer Vision, vol. 43, no. 1, pp. 29-44, 2001.
[31] R. Fergus, P. Perona, and A. Zisserman, "Object Class Recognition by Unsupervised Scale Invariant Learning," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 264-271, 2003.
[32] F. Jurie and B. Triggs, "Creating Efficient Codebooks for Visual Recognition," Proc. IEEE Int'l Conf. Computer Vision, pp. 604-610, 2005.
[33] B. Leibe and B. Schiele, "Interleaved Object Categorization and Segmentation," Proc. British Machine Vision Conf., pp. 759-768, 2003.
[34] C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, http://www.csie.ntu.edu.tw/cjlinlibsvm, 2001.
[35] M. Naphade, J.R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis, "Large-Scale Concept Ontology for Multimedia," IEEE Multimedia, vol. 13, no. 3, pp. 86-91, July-Sept. 2006.
[36] C.M. Bishop, Pattern Recognition and Machine Learning. Springer, Aug. 2006.
[37] B. Efron, "Bootstrap Methods: Another Look at the Jackknife," Annals of Statistics, vol. 7, pp. 1-26, 1979.
[38] M. Marszałek, C. Schmid, H. Harzallah, and J. van de Weijer, "Learning Object Representations for Visual Object Class Recognition," Proc. Visual Recognition Challenge Workshop, in Conjunction with IEEE Int'l Conf. Computer Vision, http://lear.inrialpes.fr/pubs/2007MSHV07 , 2007.
[39] J.C. van Gemert, J.-M. Geusebroek, C.J. Veenman, C.G.M. Snoek, and A.W.M. Smeulders, "Robust Scene Categorization by Learning Image Statistics in Context," Proc. IEEE CVPR Workshop Semantic Learning Applications in Multimedia, 2006.
[40] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results," http://www.pascal-network.org/ challenges/ VOCvoc2008/, 2010.
[41] M.A. Tahir et al., "University of Amsterdam and University of Surrey at PASCAL VOC 2008," Proc. PASCAL Visual Object Classes Challenge Workshop, in Conjunction with IEEE European Conf. Computer Vision, http://staff.science.uva.nl/~ksande/pub vandesandepascalvoc2008.pdf , 2008.
[42] J.C. van Gemert, C.J. Veenman, A.W.M. Smeulders, and J.-M. Geusebroek, "Visual Word Ambiguity," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 7, pp. 1271-1283, July 2010.
[43] C.G.M. Snoek et al., "The MediaMill TRECVID 2008 Semantic Video Search Engine," Proc. Sixth TRECVID Workshop, Nov. 2008.
17 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool