The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan. (2014 vol.26)
pp: 166-179
Dayong Wang , Nanyang Technological University, Singapore
Steven C.H. Hoi , Nanyang Technological University, Singapore
Ying He , Nanyang Technological University, Singapore
Jianke Zhu , Zhejiang University, Hangzhou
ABSTRACT
This paper investigates a framework of search-based face annotation (SBFA) by mining weakly labeled facial images that are freely available on the World Wide Web (WWW). One challenging problem for search-based face annotation scheme is how to effectively perform annotation by exploiting the list of most similar facial images and their weak labels that are often noisy and incomplete. To tackle this problem, we propose an effective unsupervised label refinement (ULR) approach for refining the labels of web facial images using machine learning techniques. We formulate the learning problem as a convex optimization and develop effective optimization algorithms to solve the large-scale learning task efficiently. To further speed up the proposed scheme, we also propose a clustering-based approximation algorithm which can improve the scalability considerably. We have conducted an extensive set of empirical studies on a large-scale web facial image testbed, in which encouraging results showed that the proposed ULR algorithms can significantly boost the performance of the promising SBFA scheme.
INDEX TERMS
Face, Optimization, Feature extraction, Noise measurement, Machine learning, Approximation algorithms, Humans,weak label, Face annotation, content-based image retrieval, machine learning, label refinement, web facial images
CITATION
Dayong Wang, Steven C.H. Hoi, Ying He, Jianke Zhu, "Mining Weakly Labeled Web Facial Images for Search-Based Face Annotation", IEEE Transactions on Knowledge & Data Engineering, vol.26, no. 1, pp. 166-179, Jan. 2014, doi:10.1109/TKDE.2012.240
REFERENCES
[1] Social Media Modeling and Computing, S.C.H. Hoi, J. Luo, S. Boll, D. Xu, and R. Jin, eds. Springer, 2011.
[2] S. Satoh, Y. Nakamura, and T. Kanade, "Name-It: Naming and Detecting Faces in News Videos," IEEE MultiMedia, vol. 6, no. 1, pp. 22-35, Jan.-Mar. 1999.
[3] P.T. Pham, T. Tuytelaars, and M.-F. Moens, "Naming People in News Videos with Label Propagation," IEEE Multimedia, vol. 18, no. 3, pp. 44-55, Mar. 2011.
[4] L. Zhang, L. Chen, M. Li, and H. Zhang, "Automated Annotation of Human Faces in Family Albums," Proc. 11th ACM Int'l Conf. Multimedia (Multimedia), 2003.
[5] T.L. Berg, A.C. Berg, J. Edwards, M. Maire, R. White, Y.W. Teh, E.G. Learned-Miller, and D.A. Forsyth, "Names and Faces in the News," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), pp. 848-854, 2004.
[6] J. Yang and A.G. Hauptmann, "Naming Every Individual in News Video Monologues," Proc. 12th Ann. ACM Int'l Conf. Multimedia (Multimedia), pp. 580-587. 2004.
[7] J. Zhu, S.C.H. Hoi, and M.R. Lyu, "Face Annotation Using Transductive Kernel Fisher Discriminant," IEEE Trans. Multimedia, vol. 10, no. 1, pp. 86-96, Jan. 2008.
[8] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, "Content-Based Image Retrieval at the End of the Early Years," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
[9] S.C.H. Hoi, R. Jin, J. Zhu, and M.R. Lyu, "Semi-Supervised SVM Batch Mode Active Learning with Applications to Image Retrieval," ACM Trans. Information Systems, vol. 27, pp. 1-29, 2009.
[10] X.-J. Wang, L. Zhang, F. Jing, and W.-Y. Ma, "AnnoSearch: Image Auto-Annotation by Search," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1483-1490, 2006.
[11] L. Wu, S.C.H. Hoi, R. Jin, J. Zhu, and N. Yu, "Distance Metric Learning from Uncertain Side Information for Automated Photo Tagging," ACM Trans. Intelligent Systems and Technology, vol. 2, no. 2, p. 13, 2011.
[12] P. Wu, S.C.H. Hoi, P. Zhao, and Y. He, "Mining Social Images with Distance Metric Learning for Automated Image Tagging," Proc. Fourth ACM Int'l Conf. Web Search and Data Mining (WSDM '11), pp. 197-206, 2011.
[13] D. Wang, S.C.H. Hoi, and Y. He, "Mining Weakly Labeled Web Facial Images for Search-Based Face Annotation," Proc. 34th Int'l ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR), 2011.
[14] P. Belhumeur, J. Hespanha, and D. Kriegman, "Eigenfaces versus Fisherfaces: Recognition Using Class Specific Linear Projection," IEEE Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[15] W. Zhao, R. Chellappa, P.J. Phillips, and A. Rosenfeld, "Face Recognition: A Literature Survey," ACM Computing Survey, vol. 35, pp. 399-458, 2003.
[16] G.B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, "Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments," technical report 07-49, 2007.
[17] H.V. Nguyen and L. Bai, "Cosine Similarity Metric Learning for Face Verification," Proc. 10th Asian Conf. Computer Vision (ACCV '10), 2008.
[18] M. Guillaumin, J. Verbeek, and C. Schmid, "Is that You? Metric Learning Approaches for Face Identification," Proc. IEEE 12th Int'l Conf. Computer Vision (ICCV), 2009.
[19] Z. Cao, Q. Yin, X. Tang, and J. Sun, "Face Recognition with Learning-Based Descriptor," IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 2707-2714, 2010.
[20] E. Hjelmås and B.K. Low, "Face Detection: A Survey," Computer Vision and Image Understanding, vol. 83, no. 3, pp. 236-274, 2001.
[21] R. Jafri and H.R. Arabnia, "A Survey of Face Recognition Techniques," J. Information Processing Systems, vol. 5, pp. 41-68, 2009.
[22] K. Delac and M. Grgic, Face Recognition, IN-TECH, 2007.
[23] M.G. Kresimir Delac and M.S. Bartlett, Recent Advances in Face Recognition. I-Tech Education and Publishing, 2008.
[24] A. Hanbury, "A Survey of Methods for Image Annotation," J. Visual Languages and Computing, vol. 19, pp. 617-627, Oct. 2008.
[25] Y. Yang, Y. Yang, Z. Huang, H.T. Shen, and F. Nie, "Tag Localization with Spatial Correlations and Joint Group Sparsity," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 881-888, 2011.
[26] J. Fan, Y. Gao, and H. Luo, "Hierarchical Classification for Automatic Image Annotation," Proc. 30th Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR), pp. 111-118, 2007.
[27] Z. Lin, G. Ding, and J. Wang, "Image Annotation Based on Recommendation Model," Proc. 34th Int'l ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR), pp. 1097-1098, 2011.
[28] P. Duygulu, K. Barnard, J. de Freitas, and D.A. Forsyth, "Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary," Proc. Seventh European Conf. Computer Vision (ECCV), pp. 97-112, 2002.
[29] J. Fan, Y. Gao, and H. Luo, "Multi-Level Annotation of Natural Scenes Using Dominant Image Components and Semantic Concepts," Proc. 12th Ann. ACM Int'l Conf. Multimedia (Multimedia), pp. 540-547, 2004.
[30] G. Carneiro, A.B. Chan, P. Moreno, and N. Vasconcelos, "Supervised Learning of Semantic Classes for Image Annotation and Retrieval," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 3, pp. 394-410, Mar. 2007.
[31] C. Wang, F. Jing, L. Zhang, and H.-J. Zhang, "Image Annotation Refinement Using Random Walk with Restarts," 14th Ann. ACM Int'l Conf. Multimedia, pp. 647-650, 2006.
[32] P. Pham, M.-F. Moens, and T. Tuytelaars, "Naming Persons in News Video with Label Propagation," Proc. VCIDS, pp. 1528-1533, 2010.
[33] J. Tang, R. Hong, S. Yan, T.-S. Chua, G.-J. Qi, and R. Jain, "Image Annotation by KNN-Sparse Graph-Based Label Propagation over Noisily Tagged Web Images," ACM Trans. Intelligent Systems and Technology, vol. 2, pp. 14:1-14:15, 2011.
[34] L. Page, S. Brin, R. Motwani, and T. Winograd, "The Pagerank Citation Ranking: Bringing Order to the Web," Technical Report 1999-66, Stanford InfoLab, Nov. 1999.
[35] X. Rui, M. Li, Z. Li, W.-Y. Ma, and N. Yu, "Bipartite Graph Reinforcement Model for Web Image Annotation," Proc. 15th ACM Int'l Conf. Multimedia, pp. 585-594, 2007.
[36] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman, "LabelMe: A Database and Web-Based Tool for Image Annotation," Int'l J. Computer Vision, vol. 77, nos. 1-3, pp. 157-173, 2008.
[37] Y. Tian, W. Liu, R. Xiao, F. Wen, and X. Tang, "A Face Annotation Framework with Partial Clustering and Interactive Labeling," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2007.
[38] J. Cui, F. Wen, R. Xiao, Y. Tian, and X. Tang, "EasyAlbum: An Interactive Photo Annotation System Based on Face Clustering and Re-Ranking," Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI), pp. 367-376, 2007.
[39] D. Anguelov, K. Chih Lee, S.B. Göktürk, and B. Sumengen, "Contextual Identity Recognition in Personal Photo Albums," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR '07), 2007.
[40] J.Y. Choi, W.D. Neve, K.N. Plataniotis, and Y.M. Ro, "Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks," IEEE Trans. Multimedia, vol. 13, no. 1, pp. 14-28, Feb. 2011.
[41] D. Ozkan and P. Duygulu, "A Graph Based Approach for Naming Faces in News Photos," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1477-1482, 2006.
[42] D.-D. Le and S. Satoh, "Unsupervised Face Annotation by Mining the Web," Proc. IEEE Eighth Int'l Conf. Data Mining (ICDM), pp. 383-392, 2008.
[43] M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid, "Automatic Face Naming with Caption-Based Supervision," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[44] M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid, "Face Recognition from Caption-Based Supervision," Int'l J. Computer Vision, vol. 96, pp. 64-82, 2011.
[45] T. Mensink and J.J. Verbeek, "Improving People Search Using Query Expansions," Proc. 10th European Conf. Computer Vision (ECCV), vol. 2, pp. 86-99, 2008.
[46] T.L. Berg, A.C. Berg, J. Edwards, and D. Forsyth, "Who's in the Picture," Proc. Neural Information Processing Systems Conf. (NIPS), 2005.
[47] Z. Wu, Q. Ke, J. Sun, and H.-Y. Shum, "Scalable Face Image Retrieval with Identity-Based Quantization and Multi-Reference Re-Ranking," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 3469-3476, 2010.
[48] M. Zhao, J. Yagnik, H. Adam, and D. Bau, "Large Scale Learning and Recognition of Faces in Web Videos," Proc. IEEE Eighth Int'l Conf. Automatic Face and Gesture Recognition (FG), pp. 1-7, 2008.
[49] D. Wang, S.C.H. Hoi, Y. He, and J. Zhu, "Retrieval-Based Face Annotation by Weak Label Regularized Local Coordinate Coding," Proc. 19th ACM Int'l Conf. Multimedia (Multimedia), pp. 353-362, 2011.
[50] D. Wang, S.C.H. Hoi, and Y. He, "A Unified Learning Framework for Auto Face Annotation by Mining Web Facial Images," Proc. 21st ACM Int'l Conf. Information and Knowledge Management (CIKM), pp. 1392-1401, 2012.
[51] X. Zhu, Z. Ghahramani, and J.D. Lafferty, "Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions," Proc. 20th Int'l Conf. Machine Learning (ICML), pp. 912-919, 2003.
[52] Y.-Y. Sun, Y. Zhang, and Z.-H. Zhou, "Multi-Label Learning with Weak Label," Proc. 24th AAAI Conf. Artificial Intelligence (AAAI), 2010.
[53] Semi-Supervised Learning, O. Chapelle. B. Schölkopf, and A, Zien, eds. MIT Press, 2006.
[54] J. Zhu, S.C.H. Hoi, and L.V. Gool, "Unsupervised Face Alignment by Robust Nonrigid Mapping," Proc. 12th Int'l Conf. Computer Vision (ICCV), 2009.
[55] C. Siagian and L. Itti, "Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 300-312, Feb. 2007.
[56] W. Dong, Z. Wang, W. Josephson, M. Charikar, and K. Li, "Modeling LSH for Performance Tuning," Proc. 17th ACM Conf. Information and Knowledge Management (CIKM), pp. 669-678, 2008.
[57] Y. Zhou, R. Jin, and S.C.-H. Hoi, "Exclusive Lasso for Multi-Task Feature Selection," Proc. AISTATS, pp. 988-995, 2010.
[58] J. Liu and J. Ye, "Efficient Euclidean Projections in Linear Time," Proc. 26th Ann. Int'l Conf. Machine Learning (ICML), pp. 657-664, 2009.
[59] F. Zang and J.-S. Zhang, "Label Propagation Through Sparse Neighborhood and Its Applications," Neurocomputing, vol. 97, pp. 267-277, 2012.
[60] T. Ahonen, A. Hadid, and M. Pietikainen, "Face Recognition with Local Binary Patterns," Proc. European Conf. Computer Vision (ECCV), vol. 1, pp. 469-481, 2004.
32 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool