The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - October (2011 vol.33)
pp: 1962-1977
S. Nayar , Comput. Sci. Dept., Columbia Univ., New York, NY, USA
ABSTRACT
We introduce the use of describable visual attributes for face verification and image search. Describable visual attributes are labels that can be given to an image to describe its appearance. This paper focuses on images of faces and the attributes used to describe them, although the concepts also apply to other domains. Examples of face attributes include gender, age, jaw shape, nose size, etc. The advantages of an attribute-based representation for vision tasks are manifold: They can be composed to create descriptions at various levels of specificity; they are generalizable, as they can be learned once and then applied to recognize new objects or categories without any further training; and they are efficient, possibly requiring exponentially fewer attributes (and training data) than explicitly naming each category. We show how one can create and label large data sets of real-world images to train classifiers which measure the presence, absence, or degree to which an attribute is expressed in images. These classifiers can then automatically label new images. We demonstrate the current effectiveness-and explore the future potential-of using attributes for face verification and image search via human and computational experiments. Finally, we introduce two new face data sets, named FaceTracer and PubFig, with labeled attributes and identities, respectively.
INDEX TERMS
image retrieval, content-based retrieval, face recognition, image classification, image representation, face recognition, describable visual attributes, face verification, image search, attribute-based representation, FaceTracer, PubFig, attribute classification, feature selection, content-based image retrieval, Face, Visualization, Lighting, Search engines, Databases, Face recognition, Accuracy, image search., Face recognition, attribute classification, feature selection, classifier training, content-based image retrieval
CITATION
S. Nayar, "Describable Visual Attributes for Face Verification and Image Search", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 10, pp. 1962-1977, October 2011, doi:10.1109/TPAMI.2011.48
REFERENCES
[1] S. Baluja and H. Rowley, "Boosting Sex Identification Performance," Int'l J. Computer Vision, vol. 71, pp. 111-119, 2007.
[2] P. Belhumeur, J. Hespanha, and D. Kriegman, "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection," Proc. European Conf. Computer Vision, pp. 45-58. 1996.
[3] T.L. Berg, A.C. Berg, J. Edwards, M. Maire, R. White, Y.-W. Teh, E. Learned-Miller, and D. Forsyth, "Names and Faces in the News," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2004.
[4] V. Blanz, S. Romdhani, and T. Vetter, "Face Identification across Different Poses and Illuminations with a 3D Morphable Model," Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, 2002.
[5] V. Bruce, Z. Henderson, K. Greenwood, P.J.B. Hancock, A.M. Burton, and P.I. Miller, "Verification of Face Identities from Images Captured on Video," J. Experimental Psychology: Applied, vol. 5, pp. 339-360, 1999.
[6] A.M. Burton, S. Wilson, M. Cowan, and V. Bruce, "Face Recognition in Poor-Quality Video: Evidence from Security Surveillance," Psychological Science, vol. 10, no. 3, pp. 243-248, 1999.
[7] C.D. Castillo and D.W. Jacobs, "Using Stereo Matching for 2-D Face Recognition across Pose," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[8] C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, http://www.csie.ntu.edu.tw/~cjlinlibsvm, 2001.
[9] H. Chen, P. Belhumeur, and D. Jacobs, "In Search of Illumination Invariants," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[10] T. Cootes, K. Walker, and C. Taylor, "View-Based Active Appearance Models," Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, 2000.
[11] C. Cortes and V. Vapnik, "Support-Vector Networks," Machine Learning, vol. 20, no. 3, pp. 273-297, 1995.
[12] G.W. Cottrell and J. Metcalfe, "Empath: Face, Emotion, and Gender Recognition Using Holons," Proc. Conf. Advances in Neural Information Processing Systems, pp. 564-571, 1990.
[13] N. Dalal and B. Triggs, "Histograms of Oriented Gradients for Human Detection," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 886-893, 2005.
[14] R. Datta, J. Li, and J.Z. Wang, "Content-Based Image Retrieval: Approaches and Trends of the New Age," Proc. ACM SIGMM Int'l Workshop Multimedia Information Retrieval, pp. 253-262, 2005.
[15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A Large-Scale Hierarchical Image Database," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[16] M. Everingham, J. Sivic, and A. Zisserman, "Hello! My Name Is... Buffy—Automatic Naming of Characters in TV Video," Proc. British Machine Vision Conf., 2006.
[17] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, and A. Zisserman, "The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results," http://www.pascal-network.org/ challenges/ VOC/voc2008workshop, 2011.
[18] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, "Describing Objects by Their Attributes," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[19] A. Ferencz, E. Learned-Miller, and J. Malik, "Learning to Locate Informative Features for Visual Identification," Int'l J. Computer Vision, special issue on learning and vision, vol. 77, pp. 3-24, 2007.
[20] V. Ferrari and A. Zisserman, "Learning Visual Attributes," Proc. Advances in Neural Information Processing Systems, 2007.
[21] Y. Freund and R. Shapire, "Experiments with a New Boosting Algorithm," Proc. Int'l Conf. Machine Learning, 1996.
[22] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, "From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp. 643-660, June 2001.
[23] B.A. Golomb, D.T. Lawrence, and T.J. Sejnowski, "SexNet: A Neural Network Identifies Sex from Human Faces," Proc. Conf. Advances in Neural Information Processing Systems, pp. 572-577, 1990.
[24] R. Gross, J. Shi, and J. Cohn, "Quo Vadis Face Recognition?" Proc. Workshop Empirical Evaluation Methods in Computer Vision, Dec. 2001.
[25] G. Hua and A. Akbarzadeh, "A Robust Elastic and Partial Matching Metric for Face Recognition," Proc. IEEE Int'l Conf. Computer Vision, pp. 2082-2089. 2009.
[26] G. Huang, V. Jain, and E. Learned-Miller, "Unsupervised Joint Alignment of Complex Images," Proc. IEEE Int'l Conf. Computer Vision, 2007.
[27] G. Huang, M. Jones, and E. Learned-Miller, "LFW Results Using a Combined Nowak Plus MERL Recognizer," Proc. Faces in Real-Life Images Workshop European Conf. Computer Vision, 2008.
[28] G. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments," Technical Report 07-49, Univ. of Massachuestts Amherst, Oct. 2007.
[29] M. Kirby and L. Sirovich, "Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 1, pp. 103-108, Jan. 1990.
[30] N. Kumar, P.N. Belhumeur, and S.K. Nayar, "FaceTracer: A Search Engine for Large Collections of Images with Faces," Proc. European Conf. Computer Vision, 2008.
[31] N. Kumar, A.C. Berg, P.N. Belhumeur, and S.K. Nayar, "Attribute and Simile Classifiers for Face Verification," Proc. IEEE Int'l Conf. Computer Vision, 2009.
[32] C. Lampert, H. Nickisch, and S. Harmeling, "Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[33] M.S. Lew, N. Sebe, C. Djeraba, and R. Jain, "Content-Based Multimedia Information Retrieval: State of the Art and Challenges," ACM Trans. Multimedia Computing, Comm., and Applications, vol. 2, no. 1, pp. 1-19, 2006.
[34] H. Ling, S. Soatto, N. Ramanathan, and D. Jacobs, "A Study of Face Recognition as People Age," Proc. IEEE Int'l Conf. Computer Vision, 2007.
[35] D. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Intl. J. Computer Vision, vol. 60, pp. 91-110, 2004.
[36] B. Moghaddam and M.-H. Yang, "Learning Gender with Support Faces," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 707-711, May 2002.
[37] D. Nister and H. Stewenius, "Scalable Recognition with a Vocabulary Tree," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 2161-2168. 2006,
[38] E. Nowak and F. Jurie, "Learning Visual Similarity Measures for Comparing Never Seen Objects," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[39] E. Nowak, F. Jurie, and B. Triggs, "Sampling Strategies for Bag-of-Features Image Classification," Proc. European Conf. Computer Vision, pp. 490-503, 2006.
[40] Omron, "OKAO Vision," http://www.omron.com/r_d/coretech/vision okao.html, 2010.
[41] A. O'Toole, P. Phillips, F. Jiang, J. Ayyad, N. Penard, and H. Abdi, "Face Recognition Algorithms Surpass Humans Matching Faces over Changes in Illumination," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 9, pp. 1642-1646, Sept. 2007.
[42] M. Palatucci, D. Pomerleau, G. Hinton, and T. Mitchell, "Zero-Shot Learning with Semantic Output Codes," Proc. Conf. Advances in Neural Information Processing Systems, 2009.
[43] A. Pentland, B. Moghaddam, and T. Starner, "View-Based and Modular Eigenspaces for Face Recognition," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 1994.
[44] A. Pentland, R. Picard, and S. Sclaroff, "Photobook: Content-Based Manipulation of Image Databases," Int'l J. Computer Vision, vol. 18, pp. 233-254, 1996.
[45] P. Phillips, H. Moon, S. Rizvi, and P. Rauss, "The FERET Evaluation Methodology for Face-Recognition Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000.
[46] P.J. Phillips, P.J. Flynn, T. Scruggs, K.W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek, "Overview of the Face Recognition Grand Challenge," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2005.
[47] P. Phillips, P. Flynn, T. Scruggs, K. Bowyer, and W. Worek, "Preliminary Face Recognition Grand Challenge Results," Proc. IEEE Conf. Automatic Face and Gesture Recognition, 2006.
[48] N. Pinto, J.J. DiCarlo, and D.D. Cox, "How Far Can You Get with a Modern Face Recognition Test Set Using Only Simple Features?" Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[49] B. Russell, A. Torralba, and K. Murphy, "LabelMe: A Database and Web-Based Tool for Image Annotation," Intl. J. Computer Vision, vol. 77, nos. 1-3, pp. 157-173, 2008.
[50] F. Samaria and A. Harter, "Parameterisation of a Stochastic Model for Human Face Identification," Proc. IEEE Workshop Applications of Computer Vision, 1994.
[51] G. Shakhnarovich, P. Viola, and B. Moghaddam, "A Unified Learning Framework for Real Time Face Detection and Classification," Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, 2002.
[52] T. Sim, S. Baker, and M. Bsat, "The CMU Pose, Illumination, and Expression (PIE) Database," Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 46-51, 2002.
[53] P. Sinha, B. Balas, Y. Ostrovsky, and R. Russell, "Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About," Proc. IEEE, vol. 94, no. 11, pp. 1948-1962, Nov. 2006.
[54] P. Sinha and T. Poggio, "I Think I Know That Face...," Nature, vol. 384, no. 6608, p. 404, 1996.
[55] J. Sivic and A. Zisserman, "Video Google: A Text Retrieval Approach to Object Matching in Videos," Proc. IEEE Int'l Conf. Computer Vision, vol. 2, pp. 1470-1477, 2003.
[56] Y. Taigman, L. Wolf, and T. Hassner, "Multiple One-Shots for Utilizing Class Label Information," Proc. British Machine Vision Conf., 2009.
[57] M. Turk and A. Pentland, "Face Recognition Using Eigenfaces," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1991.
[58] P. Viola and M. Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2001.
[59] L. Wiskott, J.-M. Fellous, N. Krüger, and C. von der Malsburg, "Face Recognition by Elastic Bunch Graph Matching," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 775-779, July 1997.
[60] L. Wolf, T. Hassner, and Y. Taigman, "Descriptor Based Methods in the Wild," Proc. Faces in Real-Life Images Workshop European Conf. Computer Vision, 2008.
[61] L. Wolf, T. Hassner, and Y. Taigman, "Similarity Scores Based on Background Samples," Proc. Asian Conf. Computer Vision, 2009.
[62] W. Zhao, R. Chellappa, P.J. Phillips, and A. Rosenfeld, "Face Recognition: A Literature Survey," ACM Computing Surveys, vol. 35, no. 4, pp. 399-458, 2003.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool