The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2010 vol.32)
pp: 2039-2053
Shiming Xiang , Chinese Academy of Sciences, Beijing
Feiping Nie , Tsinghua University, Beijing
Changshui Zhang , Tsinghua University, Beijing
ABSTRACT
This paper presents local spline regression for semi-supervised classification. The core idea in our approach is to introduce splines developed in Sobolev space to map the data points directly to be class labels. The spline is composed of polynomials and Green's functions. It is smooth, nonlinear, and able to interpolate the scattered data points with high accuracy. Specifically, in each neighborhood, an optimal spline is estimated via regularized least squares regression. With this spline, each of the neighboring data points is mapped to be a class label. Then, the regularized loss is evaluated and further formulated in terms of class label vector. Finally, all of the losses evaluated in local neighborhoods are accumulated together to measure the global consistency on the labeled and unlabeled data. To achieve the goal of semi-supervised classification, an objective function is constructed by combining together the global loss of the local spline regressions and the squared errors of the class labels of the labeled data. In this way, a transductive classification algorithm is developed in which a globally optimal classification can be finally obtained. In the semi-supervised learning setting, the proposed algorithm is analyzed and addressed into the Laplacian regularization framework. Comparative classification experiments on many public data sets and applications to interactive image segmentation and image matting illustrate the validity of our method.
INDEX TERMS
Semi-supervised classification, local spline regression, interactive image segmentation.
CITATION
Shiming Xiang, Feiping Nie, Changshui Zhang, "Semi-Supervised Classification via Local Spline Regression", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 11, pp. 2039-2053, November 2010, doi:10.1109/TPAMI.2010.35
REFERENCES
[1] R.A. Adams, Sobolev Spaces. Academic Press, 1975.
[2] R.K. Ando and T. Zhang, "Two-View Feature Generation Model for Semisupervised Learning," Proc. Int'l Conf. Machine Learning, pp. 25-32, 2007.
[3] M.F. Balcan, A. Blum, and K. Yang, "Co-Training and Expansion: Towards Bridging Theory and Practice," Proc. Advances in Neural Information Processing Systems, pp. 89-96, 2004.
[4] M. Belkin, I. Matveeva, and P. Niyogi, "Regularization and Semi-Supervised Learning on Large Graphs," Proc. Int'l Conf. Learning Theory, pp. 624-638, 2004.
[5] M. Belkin and P. Niyogi, "Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering," Advances in Neural Information Processing Systems 14, pp. 585-591, MIT Press, 2002.
[6] M. Belkin, P. Niyogi, and V. Sindhwani, "Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples," J. Machine Learning Research, vol. 7, pp. 2399-2434, 2006.
[7] M. Belkin and P. Niyogi, "Laplacian Eigenmaps for Dimensionality Reduction and Data Representation," Neural Computation., vol. 15, no. 6, pp. 1373-1396, 2003.
[8] A. Blake, C. Rother, M. Brown, P. Perez, and P. Torr, "Interactive Image Segmentation Using an Adaptive GMMRF Model," Proc. European Conf. Computer Vision, pp. 428-441, 2004.
[9] A. Blum and S. Chawla, "Learning from Labeled and Unlabeled Data Using Graph Mincuts," Proc. Int'l Conf. Machine Learning, pp. 19-26, 2001.
[10] A. Blum and T. Mitchell, "Combining Labeled and Unlabeled Data with Co-Training," Proc. Ann. Conf. Computational Learning Theory, pp. 92-100, 1998.
[11] F. Bookstein, "Principal Warps: Thin-Plate Splines and the Decomposition of Deformations," IEEE Trans. Pattern Analysis and Machine Learning, vol. 11, no. 6, pp. 567-585, June 1989.
[12] Y.Y. Boykov and M.P. Jolly, "Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in n-d Images," Proc. IEEE Int'l Conf. Computer Vision, pp. 105-112, 2001.
[13] O. Chapelle, B. Schölkopf, and A. Zien, Semi-Supervised Learning. MIT Press, 2006.
[14] N.V. Chawla and G. Karakoulas, "Learning from Labeled and Unlabeled Data: An Empirical Study across Techniques and Domains," J. Artificial Intelligence Research, vol. 23, pp. 331-366, 2005.
[15] F.G. Cozman, I. Cohen, and M.C. Cirelo, "Semi-Supervised Learning of Mixture Models," Proc. Int'l Conf. Machine Learning, pp. 99-106, 2003.
[16] S. Dasgupta, M.L. Littman, and D. McAllester, "Pac Generalization Bounds for Co-Training," Advances in Neural Information Processing Systems 14, MIT Press, 2001.
[17] J. Duchon, "Splines Minimizing Rotation-Invariant Semi-Norms in Sobolev Spaces," Constructive Theory of Functions of Several Variables, A. Dold and B. Eckmann, eds., pp. 85-100, Springer-Verlag, 1977.
[18] A. Fujino, N. Ueda, and K. Saito, "A Hybrid Generative/Discriminative Approach Semi-Supervised Classifier Design," Proc. Conf. Artificial Intelligence, pp. 764-769, 2005.
[19] G. Getz, N. Shental, and E. Domany, "Semi-Supervised Learning—A Statistical Physics Approach," Proc. ICML Workshop Learning with Partially Classified Training Data, 2005.
[20] S. Goldman and Y. Zhou, "Enhancing Supervised Learning with Unlabeled Data," Proc. Int'l Conf. Machine Learning, pp. 327-334, 2000.
[21] G.H. Golub and C.F. van Loan, Matrix Computations, third ed. The Johns Hopkins Univ. Press, 1996.
[22] L. Grady, T. Schiwietz, S. Aharon, and R. Westermann, "Random Walks for Interactive Alpha-Matting," Proc. Fifth IASTED Int'l Conf. Visualization, Imaging and Image Processing, pp. 423-429, 2005.
[23] X. He and P. Niyogi, "Locality Preserving Projections," Proc. Ann. Conf. Neural Information Processing Systems, 2003.
[24] T. Joachims, "Transductive Learning via Spectral Graph Partitioning," Proc. Int'l Conf. Machine Learning, pp. 290-297, 2003.
[25] T. Joachims, "Transductive Inference for Text Classification Using Support Vector Machines," Proc. Int'l Conf. Machine Learning, pp. 200-209, 1999.
[26] I.T. Jolliffe, Principal Component Analysis, second ed. Springer, 2002.
[27] R. Jones, "Learning to Extract Entities from Labeled and Unlabeled Text," Technical Report CMU-LTI-05-191, Carnegie Mellon Univ., 2005.
[28] N.D. Lawrence and M.I. Jordan, "Semi-Supervised Learning via Gaussian Processes," Advances in Neural Information Processing Systems 14, pp. 753-760, MIT Press, 2004.
[29] A. Levin, D. Lischinski, and Y. Weiss, "A Closed-Form Solution to Natural Image Matting," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 228-242, Feb. 2008.
[30] A. Levin, A. Rav-Acha, and D. Lischinski, "Spectral Matting," Proc. Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[31] Y. Li, J. Sun, C. Tang, and H. Shum, "Lazy Snapping," Proc. ACM SIGGRAPH, pp. 303-308, 2004.
[32] D.R. Martin, C. Fowlkes, D. Tal, and J. Malik, "A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics," Proc. IEEE Int'l Conf. Computer Vision, pp. 416-425, 2001.
[33] D.J. Miller and H.S. Uyar, "A Mixture of Experts Classifier with Learning Based on Both Labelled and Unlabelled Data," Advances in Neural Information Processing Systems 9, pp. 571-577, MIT Press, 1996.
[34] K. Nigam and R. Ghani, "Analyzing the Effectiveness and Applicability of Co-Training," Proc. Int'l Conf. Information and Knowledge Management, pp. 86-93, 2000.
[35] K. Nigam, A.K. McCallum, S. Thrun, and T.M. Mitchell, "Text Classification from Labeled and Unlabeled Documents Using EM," Machine Learning, vol. 39, nos. 2/3, pp. 103-134, 2000.
[36] C. Rother, V. Kolmogorov, and A. Blake, "Grabcut—Interactive Foreground Extraction Using Iterated Graph Cuts," Proc. ACM SIGGRAPH, pp. 309-314, 2004.
[37] L.K. Saul and S.T. Roweis, "Thinking Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds," J. Machine Learning Research, vol. 4, pp. 119-155, 2003.
[38] B. Shahshahani and D. Landgrebe, "The Effect of Unlabeled Samples in Reducing the Small Sample Size Problem and Mitigating the Hughes Phenomenon," IEEE Trans. Geoscience and Remote Sensing, vol. 32, no. 5, pp. 1087-1095, Sept. 1994.
[39] J.B. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[40] J. Sun, J. Jia, C.K. Tang, and H.-Y. Shum, "Poisson Matting," ACM Trans. Graphics, vol. 23, no. 3, pp. 315-321, 2004.
[41] M. Szummer and T. Jaakkola, "Partially Labeled Classification with Markov Random Walks," Advances in Neural Information Processing Systems 14, pp. 945-952, MIT Press, 2001.
[42] G. Wahba, Spline Models for Observational Data. SIAM Press, 1990.
[43] C. Walder and O. Chapelle, "Learning with Transformation Invariant Kernels," Advances in Neural Information Processing Systems, pp. 1561-1568, MIT Press, 2007.
[44] F. Wang, S.J. Wang, C.S. Zhang, and O. Winther, "Semi-Supervised Mean Fields," Proc. Int'l Conf. Artificial Intelligence and Statistics, pp. 596-603, 2007.
[45] F. Wang, T. Li, G. Wang, and C. Zhang, "Semi-Supervised Classification Using Local and Global Regularization," Proc. 23rd AAAI Conf. Artificial Intelligence, pp. 726-731, 2008.
[46] F. Wang and C. Zhang, "Label Propagation through Linear Neighborhoods," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 1, pp. 55-67, Jan. 2008.
[47] F. Wang and C. Zhang, "On Discriminative Semi-Supervised Classification," Proc. The 23rd AAAI Conf. Artificial Intelligence, pp. 720-725, 2008.
[48] J. Wang and M.F. Cohen, "An Iterative Optimization Approach for Unified Image Segmentation and Matting," Proc. Int'l Conf. Computer Vision, pp. 936-943, 2005.
[49] H. Wendland, Scattered Data Approximation. Cambridge Univ. Press, 2005.
[50] M. Wu, K. Yu, S. Yu, and B. Schölkopf, "Local Learning Projections," Proc. Int'l Conf. Machine Learning, pp. 1039-1046, 2007.
[51] M.R. Wu and B. Schölkopf, "A Local Learning Approach for Clustering," Advances in Neural Information Processing Systems 19, pp. 1529-1536, MIT Press, 2007.
[52] M.R. Wu and B. Schölkopf, "Transductive Classification via Local Learning Regularization," Proc. Int'l Conf. Artificial Intelligence and Statistics, pp. 628-635, 2007.
[53] S.M. Xiang, F.P. Nie, C.S. Zhang, and C.X. Zhang, "Spline Embedding for Nonlinear Dimensionality Reduction," Proc. European Conf. Machine Learning, pp. 825-832, 2006.
[54] J. Yoon, "Spectral Approximation Orders of Radial Basis Function Interpolation on the Sobolev Space," SIAM J. Math. Analysis, vol. 33, no. 4, pp. 946-958, 2001.
[55] Z. Zhang and H. Zha, "Principal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment," SIAM J. Scientific Computing, vol. 26, no. 1, pp. 313-338, 2004.
[56] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf, "Learning with Local and Global Consistency," Advances in Neural Information Processing Systems 16, pp. 321-328, MIT Press, 2003.
[57] Z.H. Zhou and M. Li, "Tri-Training: Exploiting Unlabeled Data Using Three Classifiers," IEEE Trans. Knowledge and Data Eng., vol. 17, no. 11, pp. 1529-1541, Nov. 2005.
[58] Z.H. Zhou, D.C. Zhan, and Q. Yang, "Semi-Supervised Learning with Very Few Labeled Training Examples," Proc. Conf. Artificial Intelligence, pp. 675-680, 2007.
[59] X.J. Zhu, Z. Ghahramani, and J. Lafferty, "Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions," Proc. Int'l Conf. Machine Learning, pp. 912-919, 2003.
[60] X.J. Zhu, "Semi-Supervised Learning Literature Survey," Technical Report Computer Sciences TR 1530, Univ. of Wisconsin-Madison, 2007.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool