This Article 
 Bibliographic References 
 Add to: 
Algorithmic mapping of neural network Models onto Parallel SIMD Machines
December 1991 (vol. 40 no. 12)
pp. 1390-1401

Implementations of neural networks on programmable massively parallel computers are addressed. The methods are based on a graph theoretic approach and are applicable to a large class of networks in which the computations can be described by means of matrix and vector operations. A detailed characterization of the target machine is provided. Two mappings are presented. The first is designed for a processor array consisting of a very large number of small processing units. The neurons and the nonzero synaptic weights are assigned to the processors in a predetermined order, one per processor. The data transfers between processors containing neurons and weights are implemented using a novel routing algorithm. The second mapping is designed for the data array of size N*N and a smaller processor array of size P*P, P>>N, i.e., it addresses the partitioned case. These mappings are applicable to most of the mesh-connected single-instruction-multiple-data (SIMD) machines.

[1] G. Belloch and C. Rosenberg, "Network learning on the Connection Machine," inProc. 10th Int. Joint Conf. Artif. Intell., Milan, Italy, 1987.
[2] V. E. Benes, "On rearrangeable three-stage connecting networks,"Bell Syst. Tech. J., vol. 41, pp. 117-125, Sept. 1962.
[3] C. Clos, "A study of non-blocking switching networks,"Bell Syst. Tech. J., vol. 32, pp. 406-424, 1953.
[4] Y. L. Cun, J. Denker, S. Solla, R. Howard, and L. Jockel, "Optimal brain damage," inProc. 1989 IEEE Conf. Neural Inform. Processing Syst.--Natural and Synthetic, Denver, CO, Nov. 1989.
[5] DAP Series Technical Overview, Active Memory Technology, Inc., Oct. 1989.
[6] S. E. Fahlman and G. E. Hinton, "Connectionist architectures for artificial intelligence,"IEEE Comput. Mag., vol. 20, no. 1, pp. 100-109, 1987.
[7] D. Gale, "A theorem on flow in networks,"Pacific J. Mathemat., pp. 1073-1082, 1957.
[8] H. M. Hastings and S. Waner, "Neural nets on the MPP," inFrontiers of Massively Parallel Scientific Computation, J. R. Fisher, Ed., NASA, July 1987.
[9] D. Hillis,The Connection Machine. Cambridge, MA: M.I.T. Press, 1985.
[10] S. Y. Kung, and J. N. Hwang, "Systolic architectures for artificial neural nets," inProc. Int. Conf. Neural Networks, San Diego, CA, 1988.
[11] S. Y. Kung, and J. N. Hwang, "A unified modeling of connectionist neural networks,"J. Parallel Distributed Comput., vol. 6, pp. 358-387, 1989.
[12] W.-M. Lin, "Mapping image algorithms onto fixed size window architectures," Ph.D. dissertation, EE-Systems Dep., U.S.C., Aug. 1991.
[13] M. J. Little and J. Grinberg, "The 3-D computer: An integrated stack of WSI wafers," inWafer Scale Integration, E. Swartzlander, Ed. Norwell, MA: Kluwer Academic.
[14] J. L. McClelland, D. E. Rumelhart, and the PDP Research Group,Parallel Distributed Computing, Vol.I. Cambridge, MA: M.I.T. Press, 1986.
[15] D. Nassimi and S. Sahni, "Data broadcasting in SIMD computers,"IEEE Trans. Comput., 1981.
[16] D. A. Pomerleau, G. L. Gusciora, D. S. Touretzky, and H. T. Kung, "Neural network simulation at warp speed: How we got 17 million connections per second," inProc. Int. Conf. Neural Networks, San Diego, CA, June 1988.
[17] J. L. Potter, Ed.,The Massively Parallel Processor. Cambridge, MA: M.I.T. Press, 1985.
[18] V. K. Prasanna Kumar, Ed.,Parallel Architectures and Algorithms for Image Understanding. Academic, 1991.
[19] K. W. Przytula and G. Nash, "A special purpose coprocessor for signal processing," inProc. 21st Asilomar Conf. Signals, Syst. and Comput., Monterey, CA, 1987.
[20] V. K. Prasanna Kumar and K. Wojtek Przytula, "Algorithmic mapping of neural network models onto parallel SIMD machines," inProc. Int. Conf. Appl.-Specific Array Processor, 1990.
[21] K. Wojtek Przytula, W.-M. Lin, and V. K. Prasanna Kumar, "Partitioned implementation of neural networks on mesh connected array processors," inProc. I990 IEEE Workshop VLSI Signal Processing.
[22] K. Wojtek Przytula and V. K. Prasanna Kumar,Parallel Digital Implementations of Neural Networks. Englewood Cliffs, NJ: Prentice-Hall, to be published.
[23] U. Schweigelsohn, "A shortperiodic two-dimensional systolic sorting algorithm," inProc. Int. Conf. Systolic Arrays, 1988, pp. 257-264.
[24] S. Shams and K. Wojtek Przytula, "Mapping of neural networks onto programmable parallel machines," inProc. IEEE Int. Symp. Circuits Syst., New Orleans, LA, May 1990.
[25] S. Tomboulian, "Overview and extensions of a system for routing directed graphs on SIMD architectures," inFrontiers of Massively Parallel Processing, 1988.

Index Terms:
algorithmic mapping; matrix operations; neural network models; parallel SIMD machines; programmable massively parallel computers; graph theoretic approach; vector operations; processor array; nonzero synaptic weights; data transfers; neurons; routing algorithm; graph theory; neural nets; parallel processing.
W.-M. Lin, V.K. Prasanna, K.W. Przytula, "Algorithmic mapping of neural network Models onto Parallel SIMD Machines," IEEE Transactions on Computers, vol. 40, no. 12, pp. 1390-1401, Dec. 1991, doi:10.1109/12.106224
Usage of this product signifies your acceptance of the Terms of Use.