Subscribe
Issue No.03 - March (2013 vol.24)
pp: 428-438
Chi-Yuan Yeh , National Sun Yat-Sen University, Kaohsiung
Yu-Ting Peng , National Sun Yat-Sen University, Kaohsiung
Shie-Jue Lee , National Sun Yat-Sen University, Kaohsiung
ABSTRACT
Singular value decomposition (SVD) is a popular decomposition method for solving least squares estimation (LSE) problems. However, for large data sets, applying SVD directly on the coefficient matrix is very time consuming and memory demanding in obtaining least squares solutions. In this paper, we propose an iterative divide-and-merge-based estimator for solving large-scale LSE problems. Iteratively, the LSE problem to be solved is processed and transformed to equivalent but smaller LSE problems. In each iteration, the input matrices are subdivided into a set of small submatrices. The submatrices are decomposed by SVD, respectively, and the results are merged, and the resulting matrices become the input of the next iteration. The process is iterated until the resulting matrices are small enough which can then be solved directly and efficiently by SVD. The number of iterations required is determined dynamically according to the size of the input data set. As a result, the requirements in time and space for finding least squares solutions are greatly improved. Furthermore, the decomposition and merging of the submatrices in each iteration can be independently done in parallel. The idea can be easily implemented in MapReduce and experimental results show that the proposed approach can solve large-scale LSE problems effectively.
INDEX TERMS
Matrix decomposition, Least squares approximation, Complexity theory, Iterative methods, Approximation algorithms, Equations, Educational institutions, MapReduce, Linear system, matrix decomposition, error minimization, least squares solution, large-scale data set, batch SVD
CITATION
Chi-Yuan Yeh, Yu-Ting Peng, Shie-Jue Lee, "An Iterative Divide-and-Merge-Based Approach for Solving Large-Scale Least Squares Problems", IEEE Transactions on Parallel & Distributed Systems, vol.24, no. 3, pp. 428-438, March 2013, doi:10.1109/TPDS.2012.161
REFERENCES
 [1] G.H. Golub and C. Reinsch, "Singular Value Decomposition and Least Squares Solutions," Numerische Mathematik, vol. 14, no. 5, pp. 403-420, Apr. 1970. [2] G.H. Golub and C.F.V. Loan, Matrix Computations, third ed. The Johns Hopkins Univ. Press, Oct. 1996. [3] D.C. Montgomery, E.A. Peck, and G.G. Vining, Introduction to Linear Regression Analysis, fourth ed. Wiley-Interscience, July 2006. [4] R.H. Myers, D.C. Montgomery, G.G. Vining, and T.J. Robinson, Generalized Linear Models: With Applications in Engineering and the Sciences, second ed. Wiley-Interscience, Mar. 2010. [5] O. Bretscher, Linear Algebra with Applications, third ed. Prentice Hall, July 2004. [6] Å. Björck, Numerical Methods for Least Squares Problems, first ed. SIAM, Dec. 1996. [7] S.S. Niu, L. Ljung, and Å. Björck, "Decomposition Methods for Solving Least-Squares Parameter Estimation," IEEE Trans. Signal Processing, vol. 44, no. 1, pp. 2847-2852, Nov. 1996. [8] Å. Björck and J.Y. Yuan, "Preconditioners for Least Squares Problems by LU Factorization," Electronic Trans. Numerical Analysis, vol. 8, pp. 26-35, Nov. 1999. [9] S.J. Lee and C.S. Ouyang, "A Neuro-Fuzzy System Modeling with Self-Constructing Rule Generation and Hybrid SVD-Based Learning," IEEE Trans. Fuzzy Systems, vol. 11, no. 3, pp. 341-353, June 2003. [10] L.V. Foster, "Solving Rank-Deficient and Ill-Posed Problems Using UTV and QR Factorizations," SIAM J. Matrix Analysis and Applications, vol. 25, no. 2, pp. 582-600, Feb. 2003. [11] C.B. Moler, Numerical Computing with Matlab. Soc. for Industrial Math., Jan. 2004. [12] W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Numerical Recipes in C: The Art of Scientific Computing, second ed. Cambridge Univ. Press, Oct. 1992. [13] L. Giraud, S. Gratton, and J. Langou, "A Rank-$k$ Update Procedure for Reorthogonalizing the Orthogonal Factor from Modified Gram-Schmidt," SIAM J. Matrix Analysis and Applications, vol. 25, no. 4, pp. 1163-1177, Apr. 2004. [14] R.P. Brent and F.T. Luk, "The Solution of Singular-Value and Symmetric Eigenvalue Problems on Multiprocessor Arrays," SIAM J. Scientific and Statistical Computation, vol. 16, no. 1, pp. 69-84, 1985. [15] C.R. Vogel and J.G. Wade, "Iterative SVD-Based Methods for Ill-Posed Problems," SIAM J. Scientific Computing, vol. 15, no. 3, pp. 736-754, May 1994. [16] G.O.M. Bečka and M. Vajteršic, "Dynamic Ordering for a Parallel Block-Jacobi SVD Algorithm," Parallel Computing, vol. 28, pp. 243-262, Feb. 2002. [17] V. Hari, "Accelerating the SVD Block-Jacobi Method," Computing, vol. 75, no. 1, pp. 27-53, Mar. 2005. [18] H. Zamiri-Jafarian and G. Gulak, "Iterative MIMO Channel SVD Estimation," Proc. IEEE Int'l Conf. Comm., pp. 1157-1161, May 2005. [19] Y. Yamamoto, T. Fukaya, T. Uneyama, M. Takata, K. Kimura, M. Iwasaki, and Y. Nakamura, "Accelerating the Singular Value Decomposition of Rectangular Matrices with the CSX600 and the Integrable SVD," Proc. Int'l Conf. Parallel Computing Technologies, pp. 340-345, 2007. [20] S. Lahabar and P.J. Narayanan, "Singular Value Decomposition on GPU Using CUDA," Proc. IEEE Int'l Symp. Parallel and Distributed Processing, pp. 1-10, 2009. [21] T. Kondaa and Y. Nakamura, "A New Algorithm for Singular Value Decomposition and Its Parallelization," Parallel Computing, vol. 35, no. 6, pp. 331-344, June 2009. [22] H. Ltaief, J. Kurzak, and J. Dongarra, "Parallel Two-Sided Matrix Reduction to Band Bidiagonal form on Multicore Architectures," IEEE Trans. Parallel and Distributed Systems, vol. 21, no. 4, pp. 417-423, Apr. 2010. [23] S. Wei and Z. Lin, "Accelerating Iterations Involving Eigenvalue or Singular Value Decomposition by Block Lanczos with Warm Start," technical report, Microsoft Corp., Dec. 2010. [24] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Comm. ACM, vol. 51, no. 1, pp. 107-113, Jan. 2008. [25] W. Zhao, H. Ma, and Q. He, "Parallel $k$ -Means Clustering Based on MapReduce," Proc. Int'l Conf. Cloud Computing, pp. 674-679, 2009. [26] J. Cohen, "Graph Twiddling in a Mapreduce World," Computing in Science & Eng., vol. 11, no. 4, pp. 29-41, Jan. 2009. [27] S.J. Matthews and T.L. Williams, "MrsRF: An Efficient MapReduce Algorithm for Analyzing Large Collections of Evolutionary Trees," BMC Bioinformatics, vol. 11, no. Suppl. 1, Jan. 2010. [28] W. Fang, B. He, Q. Luo, and N.K. Govindaraju, "Mars: Accelerating Mapreduce with Graphics Processors," IEEE Trans. Parallel and Distributed Systems, vol. 22, no. 4, pp. 608-620, Apr. 2011. [29] L.G. Valiant, "A Bridging Model for Parallel Computation," Comm. ACM, vol. 33, no. 8, pp. 103-111, Aug. 1990. [30] http://en.wikipedia.org/wikiBulk_synchronous_parallel , 2012. [31] R.A. van de Geijn, Using PLAPACK Parallel Linear Algebra Package. MIT Press, June 1997. [32] G. Baker, J. Gunnels, G. Morrow, B. Riviere, and R.A. van de Geijn, "PLAPACK: High Performance through High Level Abstraction" Proc. Int'l Conf. Parallel Processing, 1998.