This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Optimal Algorithms for Well-Conditioned Nonlinear Systems of Equations
July 2001 (vol. 50 no. 7)
pp. 689-698

Abstract—We propose solving nonlinear systems of equations by function optimization and we give an optimal algorithm which relies on a special canonical form of gradient descent. The algorithm can be applied under certain assumptions on the function to be optimized, that is, an upper bound must exist for the norm of the Hessian, whereas the norm of the gradient must be lower bounded. Due to its intrinsic structure, the algorithm looks particularly appealing for a parallel implementation. As a particular case, more specific results are given for linear systems. We prove that reaching a solution with a degree of precision $\varepsilon$ takes $\Theta(n^2 k^2 \log {\frac{k}{\varepsilon}})$, $k$ being the condition number of ${{\schmi A}}$ and $n$ the problem dimension. Related results hold for systems of quadratic equations for which an estimation for the requested bounds can be devised. Finally, we report numerical results in order to establish the actual computational burden of the proposed method and to assess its performances with respect to classical algorithms for solving linear and quadratic equations.

[1] J.J. Moré and M.Y. Cosnard, “Numerical Solution of NonlinearEquations,” ACM Trans. Math. Software, vol. 5, no. 1, pp. 64-85, 1979.
[2] J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. SIAM, 1996.
[3] A. Eiger, K. Sikorski, and F. Stenger, “A Bisection Method for Systems of Nonlinear Equations,” ACM Trans. Math. Software, vol. 10, no. 4, pp. 367-377, 1984.
[4] A.A.M. Cuyt, “Computational Implementation of the Multivariate Halley Method for Solving Nonlinear Systems of Equations,” ACM Trans. Math. Software, vol. 11, no. 1, pp. 20-36, 1985.
[5] P. Deuflhard, Newton Techniques for Highly Nonlinear Problems—Theory and Algorithms. Academic Press, 1996.
[6] J.J. Moré, B.S. Garbow, and K.E. Hillstrom, “User's Guide for MINPACK-1,” Technical Report ANL-80-74, Argonne Nat'l Laboratory, Argonne, Ill., 1980.
[7] L.T. Watson, S.C. Billups, and A.P. Morgan, “Algorithm 652: HOMPACK: A Suite of Codes for Globally Convergent Homotopy Algorithms,” ACM Trans. Math. Software, vol. 13, pp. 281-310, 1987.
[8] F. Zirilli, “The Solutions of Nonlinear Systems of Equations by Second Order Systems of Ordinary Differential Equations and Linearly Implicit A–Stable Techniques,” SIAM J. Numerical Analysis, vol. 19, no. 4, pp. 800-815, 1982.
[9] H. Jarausch and W. Mackens, “Solving Large Nonlinear Systems of Equations by an Adaptive Condensation Process,” Numerical Math., vol. 50, no. 6, pp. 633-653, 1987.
[10] J. Abaffy, C.G. Broyden, and E. Spedicato, “A Class of Direct Methods for Linear Systems,” Numerical Math., vol. 45, no. 3, pp. 361-376, 1984.
[11] J. Abaffy, A. Galántai, and E. Spedicato, “The Local Convergence of ABS Methods for Nonlinear Algebraic Equations,” Numerical Math., vol. 51, no. 4, pp. 429-439, 1987.
[12] R.P. Brent, “Some Efficient Algorithms for Solving Systems of Nonlinear Equations,” SIAM J. Numerical Analysis, vol. 10, no. 2, pp. 327-344, 1973.
[13] K.M. Brown, “A Quadratically Convergent Newton-Like Method Based upon Gaussian-Elimination,” SIAM J. Numerical Analysis, vol. 6, no. 4, pp. 560-569, 1969.
[14] J.M. Ortega and W.C. Rheinboldt,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, Inc., San Diego, CA, 1970.
[15] O. Axelsson and A.T. Chronopoulos, “On Nonlinear Generalized Conjugate Gradient Methods,” Numerical Math., vol. 69, no. 1, pp. 1-15, 1994.
[16] Y. Saad and M.H. Schultz, "GMRES: A Generalized Minimal Residual Algorithm for Solving Nnonsymmetric Linear Systems," SIAM J. Scientific and Statistical Computing, Vol. 7, No. 3, 1986, pp. 856-869.
[17] D.R. Fokkema, G.L.G. Sleijpen, and H.A. Van der Vorst, “Accelerated Inexact Newton Schemes for Large Systems of Nonlinear Equations,” SIAM J. Scientific Computing, vol. 19, no. 2, pp. 657-674, 1998.
[18] S.Y. Berkovich, “An Overlaying Technique for Solving Linear Equations in Real-Time Computing,” IEEE Trans. Computers, vol. 42, no. 5, pp. 513-517, 1993.
[19] R.W. Freund and N.M. Nachtigal, “QMR: A Quasi-Minimal Residual Method for Non-Hermitian Linear Systems,” Numerical Math., vol. 60, pp. 315-339, 1991.
[20] R.W. Freund, “A Transpose-Free Quasi-Minimal Residual Algorithm for Non-Hermitian Linear Systems,” SIAM J. Scientific Computing, vol. 14, no. 2, pp. 470-482, 1993.
[21] M. Zak, “Terminal Attractors for Addressable Memory in Neural Networks,” Physics Letters A, vol. 133, no. 1.2, pp. 18-22, 1988.
[22] M. Zak, “Terminal Attractors in Neural Networks,” Neural Networks, vol. 2, no. 4, pp. 259-274, 1989.
[23] S. Wang and C.H. Hsu, “Terminal Attractor Learning Algorithms for Backpropagation Neural Networks,” Proc. Int'l Joint Conf. Neural Networks, pp. 183-189, Nov. 1991.
[24] J.J. Moré, B.S. Garbow, and K.E. Hillstrom, “Testing Unconstrained Optimization Software,” ACM Trans. Math. Software, vol. 7, no. 1, pp. 17-41, 1981.
[25] V. Kumar, A. Grama, A. Gupta, and G. Karypis, Introduction to Parallel Computing: Design and Analysis of Algorithms. Benjamin Cummings, 1994.
[26] C. Aykanat, F. Ozguner, F. Ercal, and P. Sadayappan, “Iterative Algorithms for Solution of Large Sparse Systems of Linear Equations on Hypercubes,” IEEE Trans. Computers, vol. 37, no. 12, pp. 1,554–1,567, Dec. 1988.
[27] D.H. Lawrie and A.H. Sameh, “The Computation and Communication Complexity of a Parallel Banded System Solver,” ACM Trans. Math. Software, vol. 10, pp. 185-195, 1984.
[28] E. Dekker and L. Dekker, “Parallel Minimal Norm Method for Tridiagonal Linear Systems,” IEEE Trans. Computers, vol. 44, no. 7, pp. 942-946, July 1995.
[29] A. Roy-Chowdhury, N. Bellas, and P. Banerjee, Algorithm-Based Error Detection Schemes for Iterative Solution of Partial Differential Equations IEEE Trans. Computers, vol. 45, no. 4, pp. 394-407, Apr. 1996.
[30] A.K. Cline, C.B. Moler, G.W. Stewart, and J.H. Wilkinson, “An Estimate for the Condition Number of a Matrix,” SIAM J. Numerical Analysis, vol. 16, no. 2, pp. 368-375, 1979.
[31] D.P. O'Leary, “Estimating Matrix Condition Numbers,” SIAM J. Scientific and Statistical Computing, vol. 1, pp. 205-209, 1980.
[32] A.K. Cline and R.K. Rew, “A Set of Counter-Examples to Three Condition Number Estimators,” SIAM J. Scientific and Statistical Computing, vol. 4, no. 4, pp. 602-611, 1983.
[33] W.W. Hager, “Condition Estimates,” SIAM J. Scientific and Statistical Computing, vol. 5, no. 2, pp. 311-316, 1984.
[34] “MINPACK Documentation,” Applied Math. Division, Argonne Nat'l Laboratory, Argonne III.,http://www.nctlib.org/minpackreadme, 2001.
[35] F.P. Preparata and M.I. Shamos, Computational Geometry. Springer-Verlag, 1985.
[36] G.H. Golub and J.M. Ortega, Scientific Computing and Differential Equations: An Introduction to Numerical Methods. Academic Press, 1992.

Index Terms:
Computational complexity, nonlinear systems of equations, parallel processing, terminal attraction dynamics.
Citation:
Monica Bianchini, Stefano Fanelli, Marco Gori, "Optimal Algorithms for Well-Conditioned Nonlinear Systems of Equations," IEEE Transactions on Computers, vol. 50, no. 7, pp. 689-698, July 2001, doi:10.1109/12.936235
Usage of this product signifies your acceptance of the Terms of Use.