This Article 
 Bibliographic References 
 Add to: 
Dynamic Load Balancing and Efficient Load Estimators for Asynchronous Iterative Algorithms
April 2005 (vol. 16 no. 4)
pp. 289-299

Abstract—In a previous paper [CHECK END OF SENTENCE], we have shown the very high power of asynchronism for parallel iterative algorithms in a global context of grid computing. In this article, we study the interest of coupling load balancing with asynchronism in such algorithms. After proposing a noncentralized version of dynamic load balancing which is best suited to asynchronism, we verify its efficiency by some experiments on a general Partial Differential Equation (PDE) problem. Finally, we give some general conditions for the use of load balancing to obtain good results with this kind of algorithm and discuss the choice of the residual as an efficient load estimator.

[1] J.M. Bahi, S. Contassot-Vivier, and R. Couturier, “Asynchronism for Iterative Algorithms in a Global Computing Environment,” Proc. 16th Ann. Int'l Symp. High Performance Computing Systems and Applications (HPCS 2002), pp. 90-97, June 2002.
[2] J.M. Bahi, K. Rhofir, and J.-C. Miellou, “Parallel Solution of Linear DAES by Multisplitting Waveform Relaxation Methods,” Linear Algebra and Its Applications, vol. 3, nos. 332-334, pp. 181-196, 2001.
[3] D.E. Baz, P. Spiteri, J.C. Miellou, and D. Gazen, “Asynchronous Iterative Algorithms with Flexible Communication for Nonlinear Network Flow Problems,” J. Parallel and Distributed Computing, vol. 38, no. 1, pp. 1-15, Oct. 1996.
[4] K. Burrage, Parallel and Sequential Methods for Ordinary Differential Equations. New York: Oxford Univ. Press Inc., 1995.
[5] E. Hairer and G. Wanner, Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems, Springer series in computational math., Springer-Verlag, vol. 14, pp. 5-8, 1991.
[6] Y. Saad, Iterative Methods for Sparse Linear Systems. New York: PWS Publishing, 1996.
[7] D. Szyld, “Perspectives on Asynchronous Computations for Fluid Flow Problems,” technical report, Dept. of Math., Temple Univ.,, 2000.
[8] J. Bahi, S. Contassot-Vivier, and R. Couturier, “On the Interest of Load Balancing Asynchronous Parallel Iterative Algorithms,” technical report, LIFC, AND Team, 2003.
[9] A. Frommer and D. Szyld, “On Asynchronous Iterations,” J. Computational and Applied Math., vol. 23, pp. 201-216, 2000.
[10] R.S. Varga, Matrix Iterative Analysis. Prentice-Hall, 1962.
[11] A. Herz and C. Marcus, “Distributed Dynamics in Neural Networks,” Physical Rev. E, vol. 47, no. 3, pp. 2155-2161, 1993.
[12] M.E. Tarazi, “Some Convergence Results for Asynchronous Algorithms,” Numerical Math., vol. 39, pp. 325-340, 1982.
[13] D.P. Bertsekas and J.N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. Englewood Cliffs, N.J.: Prentice Hall, 1989.
[14] J.C. Miellou, D.E. Baz, and P. Spitéri, “A New Class of Asynchronous Iterative Algorithms with Order Intervals,” Math. of Computation, vol. 221, no. 67, pp. 237-255, 1998.
[15] J.M. Bahi, “Asynchronous Iterative Algorithms for Nonexpansive Linear Systems,” J. Parallel and Distributed Computing, vol. 60, no. 1, pp. 92-112, Jan. 2000.
[16] C. Xu and F. Lau, Load Balancing in Parallel Computers: Theory and Practice. Kluwer Academic Publishers, 1996.
[17] S.H. Hosseini, B. Litow, M. Malkawi, J. McPherson, and K. Vairavan, “Analysis of a Graph Coloring Based Distributed Load Balancing Algorithm,” J. Parallel and Distributed Computing, vol. 10, no. 2, pp. 160-166, Oct. 1990.
[18] G. Cybenko, “Dynamic Load Balancing for Distributed Memory Multiprocessors,” J. Parallel and Distributed Computing, vol. 7, no. 2, pp. 279-301, Oct. 1989.
[19] B. Ghosh, F.T. Leighton, B.M. Maggs, S. Muthukrishnan, C.G. Plaxton, R. Rajaraman, A.W. Richa, R.E. Tarjan, and D. Zuckerman, “Tight Analyses of Two Local Load Balancing Algorithms,” Proc. 27th Ann. ACM Symp. Theory of Computing, pp. 548-558, June 1995.
[20] G. Karagiorgos and N.M. Missirlis, “Accelerated Diffusion Algorithms for Dynamic Load Balancing,” Information Processing Letters, vol. 84, no. 2, pp. 61-67, Oct. 2002.
[21] H. Rim, J.-W. Jang, and S. Kim, “An Efficient Dynamic Load Balancing Using the Dimension Exchange Method for Balancing of Quantized Loads on Hypercube Multiprocessors,” Proc. Int'l Parallel Processing Symp. and Symp. Parallel and Distributed Processing IPPS/SPDP 1999, pp. 708-713, 1999.
[22] R. Elsasser, B. Monien, and R. Preis, “Diffusion Schemes for Load Balancing on Heterogeneous Networks,” Theory of Computing Systems, vol. 35, pp. 305-320, 2002.
[23] C.W. Gear, “Massive Parallelism across Space in ODEs,” Applied Numerical Math.: Trans. IMACS, vol. 11, nos. 1-3, pp. 27-43, Jan. 1993.
[24] R.E. White, “Multisplitting of a Symmetric Positive Definite Matrix,” SIAM J. Matrix Analysis and Applications, vol. 11, pp. 69-82, 1990.
[25] R. Namyst and J.-F. Méhaut, “$PM^2$ : Parallel Multithreaded Machine. A Computing Environment for Distributed Architectures,” Proc. Int'l Conf. Parallel Computing: State-of-the-Art and Perspectives (ParCo '95), vol. 11, pp. 279-285, 1996.

Index Terms:
Parallel iterative algorithms, asynchronism, load-balancing.
Jacques M. Bahi, Sylvain Contassot-Vivier, Raphaël Couturier, "Dynamic Load Balancing and Efficient Load Estimators for Asynchronous Iterative Algorithms," IEEE Transactions on Parallel and Distributed Systems, vol. 16, no. 4, pp. 289-299, April 2005, doi:10.1109/TPDS.2005.45
Usage of this product signifies your acceptance of the Terms of Use.