On Derivation of MLP Backpropagation from the Kelley-Bryson Optimal-Control Gradient Formula and Its Application
Neural Networks, IEEE - INNS - ENNS International Joint Conference on (2000)
July 24, 2000 to July 27, 2000
Eiji Mizutani , University of California at Berkeley and Sony US Research Laboratories
Stuart E. Dreyfus , University of California at Berkeley
Kenichi Nishio , Sony Corporation Personal IT Network Company
The well-known backpropagation (BP) derivative computation process for multilayer perceptrons (MLP) learning can be viewed as a simplified version of the Kelley-Bryson gradient formula in the classical discrete-time optimal control theory . We detail the derivation in the spirit of dynamic programming, showing how they can serve to implement more elaborate learning whereby teacher signals can be presented to any nodes at any hidden layers, as well as at the terminal output layer. We illustrate such an elaborate training scheme using a small-scale industrial problem as a concrete example, in which some hidden nodes are taught to produce specified target values. In this context, part of the hidden layer is no longer “hidden”.
E. Mizutani, S. E. Dreyfus and K. Nishio, "On Derivation of MLP Backpropagation from the Kelley-Bryson Optimal-Control Gradient Formula and Its Application," Neural Networks, IEEE - INNS - ENNS International Joint Conference on(IJCNN), Como, Italy, 2000, pp. 2167.