The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - February (2010 vol.32)
pp: 335-347
Carlos Alzate , Katholieke Universiteit Leuven, Leuven
Johan A.K. Suykens , Katholieke Universiteit Leuven, Leuven
ABSTRACT
A new formulation for multiway spectral clustering is proposed. This method corresponds to a weighted kernel principal component analysis (PCA) approach based on primal-dual least-squares support vector machine (LS-SVM) formulations. The formulation allows the extension to out-of-sample points. In this way, the proposed clustering model can be trained, validated, and tested. The clustering information is contained on the eigendecomposition of a modified similarity matrix derived from the data. This eigenvalue problem corresponds to the dual solution of a primal optimization problem formulated in a high-dimensional feature space. A model selection criterion called the Balanced Line Fit (BLF) is also proposed. This criterion is based on the out-of-sample extension and exploits the structure of the eigenvectors and the corresponding projections when the clusters are well formed. The BLF criterion can be used to obtain clustering parameters in a learning framework. Experimental results with difficult toy problems and image segmentation show improved performance in terms of generalization to new samples and computation times.
INDEX TERMS
Spectral clustering, kernel principal component analysis, out-of-sample extensions, model selection.
CITATION
Carlos Alzate, Johan A.K. Suykens, "Multiway Spectral Clustering with Out-of-Sample Extensions through Weighted Kernel PCA", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 2, pp. 335-347, February 2010, doi:10.1109/TPAMI.2008.292
REFERENCES
[1] J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[2] F.R.K. Chung, Spectral Graph Theory. Am. Math. Soc., 1997.
[3] A.Y. Ng, M.I. Jordan, and Y. Weiss, “On Spectral Clustering: Analysis and an Algorithm,” Advances in Neural Information Processing Systems, vol. 14, pp. 849-856, MIT Press, 2002.
[4] L. Hagen and A. Kahng, “New Spectral Methods for Ratio Cut Partitioning Algorithms,” IEEE Trans. Computer-Aided Design, vol. 11, no. 9, pp. 1074-1085, Sept. 1992.
[5] P.K. Chan, M.D.F. Schlag, and J.Y. Zien, “Spectral k-Way Ratio-Cut Partitioning and Clustering,” IEEE Trans. Computer-Aided Design, vol. 13, no. 9, pp. 1088-1096, Sept. 1994.
[6] M. Meila and J. Shi, “A Random Walks View of Spectral Segmentation,” Proc. Int'l Conf. Artificial Intelligence and Statistics, 2001.
[7] F.R. Bach and M.I. Jordan, “Learning Spectral Clustering, with Application to Speech Separation,” J. Machine Learning Research, vol. 7, pp. 1963-2001, 2006.
[8] L. Zelnik-Manor and P. Perona, “Self-Tuning Spectral Clustering,” Advances in Neural Information Processing Systems, vol. 17, pp. 1601-1608, MIT Press, 2005.
[9] C. Ding and X. He, “Linearized Cluster Assignment via Spectral Ordering,” Proc. 21st Int'l Conf. Machine Learning, p. 30, 2004.
[10] Y. Bengio, J. Paiement, P. Vincent, O. Delalleau, N. Le Roux, and M. Ouimet, “Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering,” Advances in Neural Information Processing Systems, vol. 16, MIT Press, 2004.
[11] C. Fowlkes, S. Belongie, F. Chung, and J. Malik, “Spectral Grouping Using the Nyström Method,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 214-225, Feb. 2004.
[12] C. Baker, The Numerical Treatment of Integral Equations. Clarendon Press, 1977.
[13] C. Williams and M. Seeger, “Using the Nyström Method to Speed Up Kernel Machines,” Advances in Neural Information Processing Systems, vol. 13, MIT Press, 2001.
[14] B. Schölkopf, A.J. Smola, and K.R. Müller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, pp. 1299-1319, 1998.
[15] J. Ham, D.D. Lee, S. Mika, and B. Schölkopf, “A Kernel View of the Dimensionality Reduction of Manifolds,” Proc. 21st Int'l Conf. Machine Learning, pp. 369-376, 2004.
[16] C. Alzate and J.A.K. Suykens, “A Weighted Kernel PCA Formulation with Out-of-Sample Extensions for Spectral Clustering Methods,” Proc. Int'l Joint Conf. Neural Networks, pp. 138-144, 2006.
[17] C. Alzate and J.A.K. Suykens, “Kernel Component Analysis Using an Epsilon Insensitive Robust Loss Function,” IEEE Trans. Neural Networks, vol. 19, no. 9, pp. 1583-1598, Sept. 2008.
[18] J.A.K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle, Least Squares Support Vector Machines. World Scientific, 2002.
[19] J.A.K. Suykens, T. Van Gestel, J. Vandewalle, and B. De Moor, “A Support Vector Machine Formulation to PCA Analysis and Its Kernel Version,” IEEE Trans. Neural Networks, vol. 14, no. 2, pp. 447-450, Mar. 2003.
[20] M. Fiedler, “A Property of Eigenvectors of Nonnegative Symmetric Matrices and Its Applications to Graph Theory,” Czechoslovak Math. J., vol. 25, no. 100, pp. 619-633, 1975.
[21] A. Pothen, H.D. Simon, and K.P. Liou, “Partitioning Sparse Matrices with Eigenvectors of Graphs,” SIAM J. Matrix Analysis and Applications, vol. 11, pp. 430-452, 1990.
[22] M. Hein, J.-Y. Audibert, and U. von Luxburg, “Graph Laplacians and Their Convergence on Random Neighborhood Graphs,” J.Machine Learning Research, vol. 8, pp. 1325-1370, 2007.
[23] M. Gu, H. Zha, C. Ding, X. He, and H. Simon, “Spectral Relaxation Models and Structure Analysis for k-Way Graph Clustering and Bi-Clustering,” Technical Report CSE-01-007, Computer Science and Eng., Pennsylvania State Univ., 2001.
[24] K. Fan, “On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations,” Proc. Nat'l Academy of Science USA, vol. 35, no. 11, pp. 652-655, 1949.
[25] M. Meila, “Multiway Cuts and Spectral Clustering,” technical report, Univ. of Washington, 2003.
[26] C. Davis and W.M. Kahan, “The Rotation of Eigenvectors by a Perturbation. III,” SIAM J. Numerical Analysis, vol. 7, no. 1, pp. 1-46, 1970.
[27] U. von Luxburg, “A Tutorial on Spectral Clustering,” Statistics and Computing, vol. 17, no. 4, pp. 395-416, 2007.
[28] F.R. Bach and M.I. Jordan, “Learning Spectral Clustering,” Advances in Neural Information Processing Systems, vol. 16, MIT Press, 2004.
[29] H. Zha, C. Ding, M. Gu, X. He, and H. Simon, “Spectral Relaxation for k-Means Clustering,” Advances in Neural Information Processing Systems, vol. 14, MIT Press, 2002.
[30] N. Cristianini, J. Shawe-Taylor, and J. Kandola, “Spectral Kernel Methods for Clustering,” Advances in Neural Information Processing Systems, vol. 14, pp. 649-655, MIT Press, 2002.
[31] N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola, “On Kernel-Target Alignment,” Advances in Neural Information Processing Systems, vol. 14, MIT Press, 2002.
[32] T.G. Dietterich and G. Bakiri, “Solving Multiclass Learning Problems via Error-Correcting Output Codes,” J. Machine Learning Research, vol. 2, pp. 263-286, 1995.
[33] S.T. Roweis and L.K. Saul, “Nonlinear Dimensionality Reduction by Locally Linear Embedding,” Science, vol. 290, pp. 2323-2326, 2000.
[34] J.B. Tenenbaum, V. de Silva, and J.C. Langford, “A Global Geometric Framework for Nonlinear Dimensionality Reduction,” Science, vol. 290, pp. 2319-2323, 2000.
[35] M. Belkin and P. Niyogi, “Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering,” Advances in Neural Information Processing Systems, vol. 14, pp. 585-591, MIT Press, 2002.
[36] X. He and P. Niyogi, “Locality Preserving Projections,” Advances in Neural Information Processing Systems, vol. 16, MIT Press, 2004.
[37] R. Tibshirani, G. Walther, and T. Hastie, “Estimating the Number of Clusters in a Dataset via the Gap Statistic,” J. Royal Statistical Soc. B, vol. 63, pp. 411-423, 2001.
[38] P.J. Rousseeuw, “Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis,” J. Computational and Applied Math., vol. 20, no. 1, pp. 53-65, 1987.
[39] L. Hubert and P. Arabie, “Comparing Partitions,” J. Classification, vol. 2, pp. 193-218, 1985.
[40] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics,” Proc. Eighth Int'l Conf. Computer Vision, vol. 2, pp. 416-423, 2001.
[41] J. Puzicha, T. Hofmann, and J. Buhmann, “Non-Parametric Similarity Measures for Unsupervised Texture Segmentation and Image Retrieval,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 267-272, 1997.
[42] C. Van Rijsbergen, Information Retrieval. Dept. of Computer Science, Univ. of Glasgow, 1979.
[43] D. Martin, C. Fowlkes, and J. Malik, “Learning to Detect Natural Image Boundaries Using Local Brightness, Color and Texture Cues,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 1-20, May 2004.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool