Subscribe
Issue No.11 - Nov. (2012 vol.34)
pp: 2259-2273
A. J. Joshi , Google, Inc., Mountain View, CA, USA
F. Porikli , Mitsubishi Electr. Res. Labs., Cambridge, MA, USA
N. P. Papanikolopoulos , Dept. of Comput. Sci. & Eng., Univ. of Minnesota, Twin Cities, Minneapolis, MN, USA
ABSTRACT
Machine learning techniques for computer vision applications like object recognition, scene classification, etc., require a large number of training samples for satisfactory performance. Especially when classification is to be performed over many categories, providing enough training samples for each category is infeasible. This paper describes new ideas in multiclass active learning to deal with the training bottleneck, making it easier to train large multiclass image classification systems. First, we propose a new interaction modality for training which requires only yes-no type binary feedback instead of a precise category label. The modality is especially powerful in the presence of hundreds of categories. For the proposed modality, we develop a Value-of-Information (VOI) algorithm that chooses informative queries while also considering user annotation cost. Second, we propose an active selection measure that works with many categories and is extremely fast to compute. This measure is employed to perform a fast seed search before computing VOI, resulting in an algorithm that scales linearly with dataset size. Third, we use locality sensitive hashing to provide a very fast approximation to active learning, which gives sublinear time scaling, allowing application to very large datasets. The approximation provides up to two orders of magnitude speedups with little loss in accuracy. Thorough empirical evaluation of classification accuracy, noise sensitivity, imbalanced data, and computational performance on a diverse set of image datasets demonstrates the strengths of the proposed algorithms.
INDEX TERMS
Training, Support vector machines, Training data, Noise, Accuracy, Learning systems, Couplings, object recognition, Active learning, scalable machine learning, multiclass classification
CITATION
A. J. Joshi, F. Porikli, N. P. Papanikolopoulos, "Scalable Active Learning for Multiclass Image Classification", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.34, no. 11, pp. 2259-2273, Nov. 2012, doi:10.1109/TPAMI.2012.21
REFERENCES
 [1] "Amazon Mechanical Turk," http:/www.mturk.com, 2012. [2] A. Asuncion and D.J. Newman, "UCI Machine Learning Repository," Univ. of California, Irvine, School of Information and Computer Sciences, http://archive.ics.uci.edu/mldatasets.html , 2007. [3] S. Basu, A. Banerjee, and R. Mooney, "Semi-Supervised Clustering by Seeding," Proc. 19th Int'l Conf. Machine Learning, 2002. [4] C. Campbell, N. Cristianini, and A.J. Smola, "Query Learning with Large Margin Classifiers," Proc. Seventh Int'l Conf. Machine Learning, 2000. [5] C.-C. Chang and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, software available at http://www.csie.ntu.edu.tw/cjlinlibsvm, 2001. [6] M. Datar, N. Immorlica, P. Indyk, and V. Mirrokni, "Locality-Sensitive Hashing Scheme Based on $p$ -Stable Distributions," Proc. 20th Ann. Symp. Computational Geometry, 2004. [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A Large-Scale Hierarchical Image Database," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009. [8] C.H.Q. Ding, X. He, H. Zha, M. Gu, and H.D. Simon, "A Min-Max Cut Algorithm for Graph Partitioning and Data Clustering," Proc. IEEE Int'l Conf. Data Mining, 2001. [9] K.-B. Duan and S.S. Keerthi, "Which Is the Best Multi-Class SVM Method? An Empirical Study," Proc. Sixth Int'l Workshop Multiple Classifier Systems, 2005. [10] S. Ertekin, J. Huang, L. Bottou, and L. Giles, "Learning on the Border: Active Learning in Imbalanced Data Classification," Proc. 16th ACM Conf. Information and Knowledge Management, 2007. [11] L. Fei-Fei and P. Perona, "A Bayesian Hierarchical Model for Learning Natural Scene Categories," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005. [12] L. Fei-Fei, P. Perona, and R. Fergus, "One-Shot Learning of Object Categories," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594-611, Apr. 2006. [13] Y. Freund, H.S. Seung, E. Shamir, and N. Tishby, "Selective Sampling Using the Query by Committee Algorithm," Machine Learning, vol. 28, pp. 133-168, 1997. [14] K. Grauman and T. Darrell, "The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features," Proc. 10th IEEE Int'l Conf. Computer Vision, 2005. [15] Y. Guo and R. Greiner, "Optimistic Active Learning Using Mutual Information," Proc. 20th Int'l Joint Conf. Artifical Intelligence, 2007. [16] S.C. Hoi, R. Jin, J. Zhu, and M.R. Lyu, "Semi-Supervised SVM Batch Mode Active Learning for Image Retrieval," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008. [17] A. Holub, P. Perona, and M. Burl, "Entropy-Based Active Learning for Object Recognition," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition Workshops, 2008. [18] C.-W. Hsu and C.-J. Lin, "A Comparison of Methods for Multi-Class Support Vector Machines," IEEE Trans. Neural Networks, vol. 13, no. 2, pp. 415-425, Mar. 2002. [19] P. Indyk and R. Motwani, "Approximate Nearest Neighbor: Towards Removing the Curse of Dimensionality." Proc. Symp. Theory of Computing, 1998. [20] P. Jain and A. Kapoor, "Active Learning for Large Multi-Class Problems," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009. [21] A.J. Joshi, F. Porikli, and N. Papanikolopoulos, "Multi-Class Active Learning for Image Classification," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009. [22] A.J. Joshi, F. Porikli, and N. Papanikolopoulos, "Breaking the Interactive Bottleneck in Multi-Class Classification with Active Selection and Binary Feedback," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010. [23] A. Kapoor, K. Grauman, R. Urtasun, and T. Darrell, "Active Learning with Gaussian Processes for Object Categorization," Proc. 11th IEEE Int'l Conf. Computer Vision, 2007. [24] A. Kapoor, E. Horvitz, and S. Basu, "Selective Supervision: Guiding Supervised Learning with Decision-Theoretic Active Learning," Proc. IEEE Eighth Int'l Conf. Data Mining, 2007. [25] A. Krause and C. Guestrin, "Near-Optimal Nonmyopic Value of Information in Graphical Models," Proc. 21st Ann. Conf. Uncertainty in Artificial Intelligence, 2005. [26] A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images," technical report, Univ. of Toronto, 2009. [27] B. Kulis and K. Grauman, "Kernelized Locality-Sensitive Hashing for Scalable Image Search," Proc. 12th IEEE Int'l Conf. Computer Vision, 2009. [28] M. Li and I. Sethi, "Confidence-Based Active Learning," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 8, pp. 1251-1261, Aug. 2006. [29] H.-T. Lin, C.-J. Lin, and R.C. Weng, "A Note on Platt's Probabilistic Outputs for Support Vector Machines," Machine Learning, vol. 68, pp. 267-276, 2007. [30] T. Mitchell, Machine Learning. McGraw-Hill, 1997. [31] A. Oliva and A. Torralba, "Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope." Int'l J. Computer Vision, vol. 42, no. 3, pp. 145-175, 2001. [32] N. Panda, K. Goh, and E. Chang, "Active Learning in Very Large Image Databases," J. Multimedia Tools and Applications, special issue on computer vision meets databases, vol. 31, no. 3, pp. 249-267, 2006. [33] J. Platt, "Probabilistic Outputs for Support Vector Machines and Comparison to Regularized Likelihood Methods," Advances in Large Margin Classifiers, MIT Press, 2000. [34] G.-J. Qi, X.-S. Hua, Y. Rui, J. Tang, and H.-J. Zhang, "Two-Dimensional Active Learning for Image Classification," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008. [35] R. Segal, T. Markowitz, and W. Arnold, "Fast Uncertainty Sampling for Labeling Large E-Mail Corpora," Proc. Conf. Email and Anti-Spam, 2006. [36] B. Settles, "Active Learning Literature Survey," Computer Sciences Technical Report 1648, Univ. of Wisconsin–Madison, 2009. [37] A. Sorokin and D. Forsyth, "Utility Data Annotation with Amazon Mechanical Turk," Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, 2008. [38] S. Tong and E. Chang, "Support Vector Machine Active Learning for Image Retrieval," Proc. Ninth ACM Int'l Conf. Multimedia, 2001. [39] S. Tong and D. Koller, "Support Vector Machine Active Learning with Applications to Text Classification," J. Machine Learning Research, vol. 2, pp. 45-66, 2001. [40] A. Torralba, R. Fergus, and W.T. Freeman, "80 Million Tiny Images: A Large Database for Non-Parametric Object and Scene Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 11, pp. 1958-1970, Nov. 2008. [41] S. Vijayanarasimhan and K. Grauman, "What's It Going to Cost You?: Predicting Effort vs. Informativeness for Multi-Label Image Annotations," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009. [42] T.-F. Wu, C.-J. Lin, and R.C. Weng, "Probability Estimates for Multi-Class Classification by Pairwise Coupling," J. Machine Learning Research, vol. 5, pp. 975-1005, 2004. [43] R. Yan, J. Yang, and A. Hauptmann, "Automatically Labeling Video Data Using Multi-Class Active Learning," Proc. Ninth IEEE Int'l Conf. Computer Vision, pp. 516-523, 2003. [44] W. Zhao, J. Long, E. Zhu, and Y. Liu, "A Scalable Algorithm for Graph-Based Active Learning," Proc. Second Ann. Int'l Workshop Frontiers in Algorithmics, 2008. [45] J. Shaw-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge Univ. Press, 2004.