The Community for Technology Leaders
Subscribe
Issue No.08 - August (2010 vol.32)
pp: 1377-1391
Dan Kushnir , Yale University, New Haven
Meirav Galun , Weizmann Institute of Science, Rehovot
Achi Brandt , Weizmann Institute of Science, Rehovot
ABSTRACT
Multigrid solvers proved very efficient for solving massive systems of equations in various fields. These solvers are based on iterative relaxation schemes together with the approximation of the “smooth” error function on a coarser level (grid). We present two efficient multilevel eigensolvers for solving massive eigenvalue problems that emerge in data analysis tasks. The first solver, a version of classical algebraic multigrid (AMG), is applied to eigenproblems arising in clustering, image segmentation, and dimensionality reduction, demonstrating an order of magnitude speedup compared to the popular Lanczos algorithm. The second solver is based on a new, much more accurate interpolation scheme. It enables calculating a large number of eigenvectors very inexpensively.
INDEX TERMS
Eigenvalues and eigenvectors, multigrid and multilevel methods, graph algorithms, segmentation, clustering.
CITATION
Dan Kushnir, Meirav Galun, Achi Brandt, "Efficient Multilevel Eigensolvers with Applications to Data Analysis Tasks", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 8, pp. 1377-1391, August 2010, doi:10.1109/TPAMI.2009.147
REFERENCES
 [1] D. Kushnir, M. Galun, and A. Brandt, "Fast Multiscale Clustering and Manifold Identification," Pattern Recognition, vol. 39, no. 10, pp. 1876-1891, Oct. 2006. [2] I.S. Dhillon, Y. Guan, and B. Kulis, "Weighted Graph Cuts without Eigenvectors: A Multilevel Approach," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 11, pp. 1944-1957, Nov. 2007. [3] G.H. Golub and C.F. Van Loan, Matrix Computations. John Hopkins Univ. Press, 1989. [4] A. Brandt, S. McCormick, and J. Ruge, "Algebraic Multigrid (AMG) for Automatic Multigrid Solution with Application to Geodetic Computations," report, Inst. for Computational Studies, 1982. [5] U. Trottenberg, A. Schuller, and C. Oosterlee, Multigrid. Acdemic Press, 2001. [6] A. Brandt, "Algebraic Multigrid Theory: The Symmetric Case," Applied Math. and Computation, vol. 19, no. 23, pp. 23-56, July 1986. [7] A. Brandt, "Multiscale Scientific Computation: Review 2001," Multiscale and Multiresolution Methods: Theory and Applications, Springer-Verlag, 2002. [8] A. Brandt, "Multilevel Adaptive Solutions to Boundary Value Problems," Math. of Computation, vol. 31, no. 138, pp. 333-390, Apr. 1977. [9] A. Brandt and D. Ron, "Multigrid Solvers and Multiscale Optimization Strategies," Multilevel Optimization and VLSICAD, Kluwer, 2003. [10] S. Kaczmarz, "Angenäherte Auflösung von Systemen Linearer Gleichungen," Bull. Acad. Polon. Sci. et Lett. A, pp. 355-357, 1937. [11] C. Fowlkes, S. Belongie, F. Chung, and J. Malik, "Spectral Grouping Using the Nystrom Method," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 214-225, Feb. 2004. [12] T. Cour, F. Benezit, and J. Shim, "Spectral Segmentation with Multiscale Graph Decomposition," Proc. IEEE Int'l. Conf. Computer Vision and Pattern Recognition, 2005. [13] Y. Ng Andrew, M. Jordan, and Y. Weiss, "On Spectral Clustering: Analysis and an Algorithm," Advances in Neural Information Processing Systems, vol. 14, pp. 849-856, MIT Press, Dec. 2001. [14] J. Shi and J. Malik, "Normalized Cuts and Image Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000. [15] Code obtained from the URL http://www.cis.upenn.edu/jshisoftware, 2008. [16] F. Chung, "Spectral Graph Theory," Am. Math. Soc., 92, 1997. [17] M. Belkin and P. Niyogi, "Laplacian Eigenmaps for Dimensionality Reduction and Data Representation," Neural Computation, vol. 15, no. 6, pp. 1373-1396, June 2003. [18] C. Chennubhotla and A.D. Jepson, "Hierarchical Eigensolver for Transition Matrices in Spectral Methods," Advances in Neural Information Processing Systems, vol. 17, p. 273-280, MIT Press, Dec. 2005. [19] A. Brandt, S. McCormick, and J. Ruge, "Multigrid Methods for Differential Eigenproblems," SIAM J. Scientific and Statistical Computing, vol. 4, no. 2, pp. 244-260, June 1983. [20] S. McCormick, "A Mesh Refinement Method for $Ax=\lambda Ax$ ," Math. of Computation, vol. 36, no. 154, pp. 485-498, Apr. 1981. [21] S. Costiner and S. Taasan, "Adaptive Multigrid Techniques for Large-Scale Eigenvalue Problems: Solutions of the Schrödinger Problem in Two and Three Dimensions," Physical Rev. E, vol. 51, no. 4, pp. 3704-3717, Apr. 1995. [22] O. Livne and A. Brandt, "O(NlogN) Multilevel Calculation of N Eigenfunctions," Multiscale Computational Methods in Chemistry and Physics, IOS Press, 2001. [23] I. Livshits, "An Algebraic Multigrid Wave-Ray Algorithm to Solve Eigenvalue Problems for The Helmholtz Operator," Numerical Linear Algebra with Applications, vol. 11, nos. 2-3, pp. 229-239, Mar. 2004. [24] I. Livshits, "One-Dimensional Algorithm for Finding Eigenbasis of the Schrödinger Operator," SIAM J. Scientific Computing, vol. 30, no. 1, pp. 416-440, Nov. 2007. [25] Y.A. Erlangga, C.W. Oosterlee, and C. Vuik, "A Novel Multigrid Based Preconditioner for Heterogeneous Helmholtz Problems," SIAM J. Scientific Computing, vol. 27, No. 4, pp. 1471-1492, 2006. [26] H.C. Elman, O.G. Ernst, and D.P. Oleary, "A Multigrid Method Enhanced by Krylov Subspace Iteration for Discrete Helmholtz Equations," SIAM J. Scientific Computing, vol. 23, no. 4, pp. 1291-1315, 2001. [27] A. Borzi and G. Borzi, "Algebraic Multigrid Methods for Solving Generalized Eigenvalue Problems," Int'l J. Numerical Methods in Eng., vol. 65, no. 8, pp. 1186-1196, Sept. 2005. [28] U. Hetmaniuk, "A Rayleigh Quotient Minimization Algorithm Based on Algebraic Multigrid," Numerical Linear Algebra with Applications, vol. 14, pp. 563-580, 2007. [29] J. Mandel and S. McCormick, "A Multilevel Variational Method for $Au =\lambda Bu$ on Composite Grids," J. Computational Physics, vol. 80, pp. 442-452, 1989. [30] M. Brezina, T. Manfeuffel, S. McCormick, J. Ruge, G. Sanders, and P. Vassilevski, "Smoothed Aggregation-Based Eigensolvers," Numerical Linear Algebra with Applications, vol. 15, pp. 249-269, 2008. [31] P. Vanek, J. Mandel, and M. Brezina, "Algebraic Multigrid by Smoothed Aggregation for Second and Fourth Order Elliptic Problems," Computing, vol. 56, pp. 179-196, 1996. [32] M. Brezina, R. Falgout, S. MacLachlan, T. Manfeuffel, S. McCormick, and J. Ruge, "Adaptive Smoothed Aggregation ($\alpha SA$ )," SIAM J. Scientific Computing, vol. 25, pp. 1896-1920, 2004. [33] M. Brezina, R. Falgout, S. MacLachlan, T. Manteuffel, S. McCormick, and J. Ruge, "Adaptive Smoothed Aggregation ($\alpha SA$ ) Multigrid," SIAM Rev., vol. 47, no.2, pp. 317-346, 2005. [34] M. Brezina, R. Falgout, S. MacLachlan, T. Manfeuffel, S. McCormick, and J. Ruge, "Adaptive Algebraic Multigrid," SIAM J. Scientific Computing, Vol. 27, no. 4, pp. 1261-1286, 2006. [35] H. de Sterck, T. Manteuffel, S. McCormick, Q. Nguyen, and J. Ruge, "Multilevel Adaptive Aggregation for Markov Chains, with Application to Web Ranking," SIAM J. Scientific Computing, vol. 30, pp. 2235-2262, 2008. [36] Y. Takahashi, "A Lumping Method for Numerical Calculations of Stationary Distributions of Markov Chains," Research Report B-18, Dept. of Information Sciences, Tokyo 23 Inst. of Tech nology, 1975. [37] G. Horton and S.T. Leutenegger, "A Multi-Level Solution Algorithm for Steady-State Markov Chains," Proc. ACM SIGMETRICS, pp. 191-200, 1994. [38] S.T. Leutenegger and G. Horton, "On the Utility of the Multi-Level Algorithm for the Solution of Nearly Completely Decomposable Markov Chains," Numerical Solution of Markov Chains, W. Stewart, ed., pp. 425-443, Kluwer Publishers, 1995. [39] U.R. Krieger, "Numerical Solution of Large Finite Markov Chains by Algebraic Multigrid Techniques," Numerical Solution of Markov Chains, W. Stewart, ed., pp. 403-424, Kluwer Publishers, 1995. [40] C. Isensee and G. Horton, "A Multi-Level Method for the Steady State Solution of Markov Chains," Simulation und Visualisierung, SCS European Publishing House, 2004. [41] S. Yu and J. Shi, "Multiclass Spectral Clustering," Proc. Ninth IEEE Int'l Conf. Computer Vision, 2003. [42] R.B. Lehoucq, D.C. Sorensen, and C. Yang, ARPACK User Guide, SIAM Publications, 1998. [43] M. de Berg, M. van Kreveld, M. Overmars, and O. Schwarzkopf, Computational Geometry: Algorithms and Applications. Springer-Verlag, 1997. [44] Code obtained from the URL http://www.autonlab.org/ autonweb2.html , 2009. [45] J.A. Hartigan and M.A. Wong, "A k-Means Clustering Algorithm," Applied Statistics, vol. 28, no. 1, pp. 100-108, 1979. [46] A. Brandt, "Principles of Systematic Upscaling," Bridging the Scales in Science and Eng., J. Fish, ed., Oxford Univ. Press, 2008.