Subscribe
Issue No.05 - May (2013 vol.25)
pp: 1148-1161
Zhao Zhang , City University of Hong Kong, Hong Kong
Tommy W.S. Chow , City University of Hong Kong, Hong Kong
Mingbo Zhao , City University of Hong Kong, Hong Kong
ABSTRACT
Visualizing similarity data of different objects by exhibiting more separate organizations with local and multimodal characteristics preserved is important in multivariate data analysis. Laplacian Eigenmaps (LAE) and Locally Linear Embedding (LLE) aim at preserving the embeddings of all similarity pairs in the close vicinity of the reduced output space, but they are unable to identify and separate interclass neighbors. This paper considers the semi-supervised manifold learning problems. We apply the pairwise Cannot-Link and Must-Link constraints induced by the neighborhood graph to specify the types of neighboring pairs. More flexible regulation on supervised information is provided. Two novel multimodal nonlinear techniques, which we call trace ratio (TR) criterion-based semi-supervised LAE ($({\rm S}^2{\rm LAE})$) and LLE ($({\rm S}^2{\rm LLE})$), are then proposed for marginal manifold visualization. We also present the kernelized $({\rm S}^2{\rm LAE})$ and $({\rm S}^2{\rm LLE})$. We verify the feasibility of $({\rm S}^2{\rm LAE})$ and $({\rm S}^2{\rm LLE})$ through extensive simulations over benchmark real-world MIT CBCL, CMU PIE, MNIST, and USPS data sets. Manifold visualizations show that $({\rm S}^2{\rm LAE})$ and $({\rm S}^2{\rm LLE})$ are able to deliver large margins between different clusters or classes with multimodal distributions preserved. Clustering evaluations show they can achieve comparable to or even better results than some widely used methods.
INDEX TERMS
Manifolds, Data visualization, Kernel, Optimization, Laplace equations, Symmetric matrices, Vectors, marginal manifold visualization, Semi-supervised manifold learning, trace ratio optimization, nonlinear dimensionality reduction, multimodality preservation, pairwise constraints
CITATION
Zhao Zhang, Tommy W.S. Chow, Mingbo Zhao, "Trace Ratio Optimization-Based Semi-Supervised Nonlinear Dimensionality Reduction for Marginal Manifold Visualization", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 5, pp. 1148-1161, May 2013, doi:10.1109/TKDE.2012.47
REFERENCES
 [1] S. Roweis and L. Saul, "Nonlinear Dimensionality Reduction by Locally Linear Embedding," Science, vol. 290, pp. 2323-2326, 2000. [2] G.E. Hinton and R.R. Salakhutdinov, "Reducing the Dimensionality of Data with Neural Networks," Science, vol. 313, no. 5786, pp. 504-507, 2006. [3] A.P. Dempster, "An Overview of Multivariate Data Analysis," J. Multivariate Analysis, vol. 1, no. 3, pp. 316-346, 1971. [4] A.M. Martinez and A.C. Kak, "PCA versus LDA," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 228-233, Feb. 2001. [5] J.P. Maaten, E.O. Postma, and H.J. Herik, "Dimensionality Reduction: A Comparative Review," Technical Report TiCC-TR 2009-005, Tilburg Univ., 2009. [6] Y. Yang, F.P. Nie, S.M. Xiang, Y.T. Zhuang, and W.H. Wang, "Local and Global Regressive Mapping for Manifold Learning with Out-of-Sample Extrapolation," Proc. Am. Assoc. for Artificial Intelligence Conf., pp. 649-654, 2010. [7] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf, "Learning with Local and Global Consistency," Proc. Advances in Neural Information Processing Systems, pp. 321-328, 2004. [8] Y. Liu and J. Rong, "Distance Metric Learning: A Comprehensive Survey," technical report, Michigan State Univ., 2006. [9] D. Donoho and C. Grimes, "Hessian Eigenmaps: Locally Linear Embedding Techniques for High-Dimensional Data," Proc. Nat'l Academy Sciences USA, vol. 100, pp. 5591-5596, 2003. [10] M. Belkin and P. Niyogi, "Laplacian Eigenmaps for Dimensionality Reduction and Data Representation," Neural Computation, vol. 15, no. 6, pp. 1373-1396, 2003. [11] Z.Y. Zhang and H.Y. Zha, "Principal Manifolds and Nonlinear Dimension Reduction by Local Tangent Space Alignment," SIAM J. Scientific Computing, vol. 26, no. 1, pp. 313-338, 2005. [12] B. Schölkopf and A. Smola, Learning with Kernels. MIT Press, 2002. [13] J.J. Hull, "A Database for Handwritten Text Recognition Research," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 5, pp. 550-554, May 1994. [14] T. Sim, S. Baker, and M. Bsat, "The CMU Pose, Illumination, and Expression Database," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec. 2003. [15] D.Q. Zhang, Z.H. Zhou, and S.C. Chen, "Semi-Supervised Dimensionality Reduction," Proc. SIAM Int'l Conf. Data Mining, pp. 629-634, 2007. [16] X. Yang, H.Y. Fu, H.Y. Zha, and L. Jesse, "Semi-Supervised Nonlinear Dimensionality Reduction," Proc. Int'l Conf. Machine Learning, pp. 1065-1072, 2006. [17] Z.Y. Zhang, H.Y. Zha, and M. Zhang, "Spectral Methods for Semi-Supervised Manifold Learning," Proc. Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-6, 2008. [18] A. Bar-Hillel, T. Hertz, N. Shental, and D. Weinshall, "Learning a Mahalanobis Metric from Equivalence Constraints," J. Machine Learning Research, vol. 6, pp. 937-965, 2005. [19] M.S. Baghshah and S.B. Shouraki, "Semi-Supervised Metric Learning Using Pairwise Constraints," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 1217-1222, 2009. [20] E. Kokiopoulou1, J. Chen, and Y. Saad, "Trace Optimization and Eigenproblems in Dimension Reduction Methods," Numerical Linear Algebra with Applications, vol. 18, pp. 565-602, 2011. [21] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1991. [22] D.Q. Zhang, S.C. Chen, and Z.H. Zhou, "Constraint Score: A New Filter Method for Feature Selection with Pairwise Constraints," Pattern Recognition, vol. 41, no. 5, pp. 1440-1451, 2008. [23] L. Zelnik-Manor and P. Perona, "Self-Tuning Spectral Clustering," Proc. Advances in Neural Information Processing Systems, pp. 1601-1608, 2005. [24] S.C. Yan, D. Xu, B.Y. Zhang, H.J. Zhang, Q. Yang, and S. Lin, "Graph Embedding and Extensions: A General Framework for Dimensionality Reduction," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 1, pp. 40-51, Jan. 2007. [25] Y. Guo, J. Gao, and P.W. Kwan, "Kernel Laplacian Eigenmaps for Visualization of Non-Vectorial Data," Proc. Australian Joint Conf. Artificial Intelligence: Advances in Artificial Intelligence, pp. 1179-1183, 2006. [26] X. Geng, D.C. Zhan, and Z.H. Zhou, "Supervised Nonlinear Dimensionality Reduction for Visualization and Classification," IEEE Trans. Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 35, no. 6, pp. 1098-1107, Dec. 2005. [27] D. Ridder, O. Kouropteva, O. Okun, M. Pietikäinen, and R.P.W. Duin, "Supervised Locally Linear Embedding," Proc. Int'l Conf. Artificial Neural Networks, pp. 333-341, 2003. [28] Y. Guo, S. Li, J. Yang, T. Shu, and L. Wu, "A Generalized Foley-Sammon Transform Based on Generalized Fisher Discriminant Criterion and Its Application to Face Recognition," Pattern Recognition Letter, vol. 24, nos. 1-3, pp. 147-158, 2003. [29] H. Wang, S. Yan, D. Xu, X. Tang, and T. Huang, "Trace Ratio vs. Ratio Trace for Dimensionality Reduction," Proc. Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007. [30] Y. Jia, F. Nie, and C. Zhang, "Trace Ratio Problem Revisited," IEEE Trans. Neural Network, vol. 20, no. 4, pp. 729-735, Apr. 2009. [31] L. Lovász and M.D. Plummer, Matching Theory. North Holland, 1986. [32] D. Cai, X. He, and J.W. Han, "Document Clustering Using Locality Preserving Indexing," IEEE Trans. Knowledge and Data Eng., vol. 17, no. 12, pp. 1624-1637, Dec. 2005. [33] Y. Yang, F. Nie, D. Xu, J. Luo, Y. Zhuang, and Y. Pan, "A Multimedia Retrieval Framework Based on Semi-Supervised Ranking and Relevance Feedback," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 34, no. 4, pp. 723-742, Apr. 2012. [34] O. Kayo, "Locally Linear Embedding Algorithm. Extensions and Applications," PhD thesis, Univ. of Oulu, Finland, 2006. [35] X.F. He, D. Cai, and J. Han, "Learning a Maximum Margin Subspace for Image Retrieval," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 2, pp. 189-201, Feb. 2008. [36] J.B. Tenenbaum, V. Silva, and J.C. Langford, "A Global Geometric Framework for Nonlinear Dimensionality Reduction," Science, vol. 290, no. 5500, pp. 2319-2323, 2000.