This Article 
 Bibliographic References 
 Add to: 
Optimization Using Neural Networks
December 1991 (vol. 40 no. 12)
pp. 1347-1358

The design of feedback (or recurrent) neural networks to produce good solutions to complex optimization problems is discussed. The theoretical basis for applying neural networks to optimization problems is reviewed, and a design rule that serves as a primitive for constructing a wide class of constraints is introduced. The use of the design rule is illustrated by developing a neural network for producing high-quality solutions to a probabilistic resource allocation task. The resulting neural network has been simulated on a high-performance parallel processor that has been optimized for neural network simulation.

[1] S. Amari, "A theory of adaptive pattern classifiers,"IEEE Trans. Electron. Comput., vol. EC-16, pp. 279-307, 1967.
[2] C. S. Beightler, D. T. Phillips, and D. J. Wilde,Foundations of Optimization, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, 1979.
[3] M. A. Cohen and S. Grossberg, "Absolute stability of global pattern formation and parallel memory storage by competitive neural networks,"IEEE Trans. Syst., Man, Cybern., vol. SMC-13, pp. 815-826, Sept./Oct. 1983.
[4] G. W. Cottrell, P. Munro, and D. Zipser, "Image compression by backpropagation: An example of extensional programming," ICS Rep. 8702, Univ. of California at San Diego, Feb. 1987.
[5] E. D. Dahl, "Neural network algorithm for an NP-complete problem: Map and graph coloring," inProc. 1st. Int. Conf. Neural Networks, vol. III, San Diego, CA, 1987, pp. 113-120.
[6] K. Fukushima, "Cognitron: A self-organizing multilayered neural network,"Biol. Cybern., vol. 20, no. 3/4, pp. 121-136.
[7] R. P. Gorman and T. J. Sejnowski, "Analysis of hidden units in a layered network trained to classify sonar targets,"Neural Networks, vol. 1, no. 1, pp. 75-89, 1988.
[8] S. Grossberg,Studies of Mind and Brain: Neural Principles of Learning, Perception, Development, Cognition, and Motor Control. Boston, MA: Reidel, 1982.
[9] S. Grossberg, "Nonlinear neural networks: Principles, mechanisms, and architectures,"Neural Networks, vol. 1, no. 1, pp. 17-61, 1988.
[10] D. Hammerstrom, "A VLSI architecture for high-performance, low-cost, on-chip learning," inProc. IJCNN'90, vol. II, San Diego, CA, June 17-21, 1990, pp. 537-544.
[11] J. B. Hampshire, II and A. H. Waibel, "A novel objective function for improved phoneme recognition using time-delay neural networks,"IEEE Trans. Neural Networks, vol. 1, no. 2, pp. 216-228, June 1990.
[12] R. Hecht-Nielson,Neurocomputing. Reading, MA: Addison-Wesley, 1990.
[13] M. Holler, S. Tam, H. Castro, and R. Benson, "An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses," inProc. Int. Joint Conf Neural Networks, vol. II, Washington, DC, June 1989, pp. 191-196.
[14] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities, "Proc. Nat. Acad. Sci. U.S.A., vol. 79, pp. 2554-2558, 1982.
[15] J. J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons,"Proc. Nat. Acad. Sci. U.S.A., vol. 81, pp. 3088-3092, 1984.
[16] J. J. Hopfleld and D. W. Tank, "'Neural' composition of decisions optimization problems,"Biol. Cybern., vol. 52, pp. 141-152, 1985.
[17] J. J. Hopfield, "Computing with neural circuits: A model,"Science, vol. 233, pp. 625-633, Aug. 1986.
[18] T. Kimoto, K. Asakawa, M. Yoda, and M. Takeoka, "Stock market prediction system with modular neural networks," inProc. IJCNN'90, vol. I, 1990, pp. 1-6.
[19] T. Kohonen, "An introduction to neural computing,"Neural Networks, vol. 1, no. 1, pp. 3-16, 1988.
[20] Y. Le Cun, "Learning process in an asymmetric threshold network," inDisordered Systems and Biological Organization, E. Bienenstock, Ed. New York: Springer-Verlag, 1986, pp. 233-240.
[21] W-C. Lin, F-Y. Liao, C-K. Tsao, and T. Lingutla, "A hierarchical multiple-view approach to three-dimensional object recognition,"IEEE Trans. Neural Networks, vol. 2, pp. 84-92, Jan. 1991.
[22] S. P. Lloyd and H. S. Witsenhausen, "Weapon allocation is NP-complete," inProc. 1986 Summer Comput. Simulation Conf., Reno, NV, 1986.
[23] C. A. Mead,Analog VLSI and Neural Systems. Reading, MA: Addison-Wesley, 1989.
[24] C. A. Mead, "Neuromorphic electronic systems,"Proc. IEEE, vol. 78, pp. 1629-1636, Oct. 1990.
[25] W. A. Metler and F. L. Preston, "Solutions to a probabilistic resource allocation problem," inProc. 28th IEEE Conf. Decision and Contr., Tampa, FL, Dec. 1989, pp. 1606-1611.
[26] K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks,"IEEE Trans. Neural Networks, vol. 1, pp. 4-27, Mar. 1990.
[27] E. W. Page and G. A. Tagliarini, "Algorithm development for neural networks," inProc. SPIE Symp. Innovative Sci. and Technol., vol. 880, Los Angeles, CA, 1988, pp. 11-19.
[28] D. B. Parker, "Learning logic: Casting the cortex of the human brain in silicon," TR-47, M.I.T. Center for Computational Research in Economics and Management Science, Cambridge, MA, Feb. 1985.
[29] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representation by error propagation,"Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vols. 1 and 2. Cambridge, MA: MIT Press, 1986.
[30] G. A. Tagliarini and E. W. Page, "A neural-network solution to the concentrator assignment problem," inProc. IEEE Conf. Neural Information Processing Systems--Natural and Synthetic, Denver, CO, IEEE/AIP, 1987, pp. 775-782.
[31] G. A. Tagliarini and E. W. Page, "Solving constraint satisfaction problems with neural networks," inProc. 1st. Int. Conf. Neural Networks, vol. III, San Diego, CA, 1987, pp. 741-748.
[32] G. A. Tagliarini, "Undesirable equilibria in systematically designed neural networks," inProc. IEEE SOUTHEASTCON'89, vol. I, Columbia, SC, Apr. 1989, pp. 63-67.
[33] Y. Takefuji and K-C. Lee, "A near optimum parallel planarization algorithm,"Science, vol. 245, pp. 1221-1223, 1989.
[34] D. W. Tank and J. J. Hopfield, "Simple 'neural' organization networks: An A/D converter, signal decision circuit and a linear programming circuit,"IEEE Trans. Circuits Syst., vol. CAS-33, pp. 533-541, May 1986.
[35] P. Werbos, "Beyond regression: New tools for predicting and analysis in the behavioral sciences," Ph.D. dissertation, Harvard Univ., 1974.
[36] B. Widrow and M. A. Lehr, "30 years of adaptive neural networks: Perceptron, madaline, and backpropagation,"Proc. IEEE, vol. 78, no. 9, pp. 1415-1442, 1990.

Index Terms:
optimisation; neural networks; feedback; design rule; probabilistic resource allocation task; high-performance parallel processor; simulation; feedback; neural nets; optimisation; simulation.
G.A. Tagliarini, J.F. Christ, E.W. Page, "Optimization Using Neural Networks," IEEE Transactions on Computers, vol. 40, no. 12, pp. 1347-1358, Dec. 1991, doi:10.1109/12.106220
Usage of this product signifies your acceptance of the Terms of Use.