This Article 
 Bibliographic References 
 Add to: 
Depth-Size Tradeoffs for Neural Computation
December 1991 (vol. 40 no. 12)
pp. 1402-1412

The tradeoffs between the depth (i.e., the time for parallel computation) and the size (i.e., the number of threshold gates) in neural networks are studied. The authors focus the study on the neural computations of symmetric Boolean functions and some arithmetic functions. It is shown that a significant reduction in the size is possible for symmetric functions and some arithmetic functions, at the expense of a small constant increase in depth. In the process, several neural networks which have the minimum size among all the known constructions have been developed. Results on implementing symmetric functions can be used to improve results about arbitrary Boolean functions. In particular, it is shown that any Boolean function can be computed in a depth-3 neural network with O(2/sup n/ /sup 2/) threshold gates; it is also proven that at least Omega (2/sup n/ /sup 3/) threshold gates are required.

[1] P. W. Beame, S. A. Cook, and H. J. Hoover, "Log depth circuits for division and related problems,"SIAM J. Comput., vol. 15, pp. 994-1003, 1986.
[2] J. Bruck, "Harmonic analysis of polynomial threshold functions,"SIAM J. Discrete Math., May 1990.
[3] J. Bruck and R. Smolensky, "Polynomial threshold functions,AC0functions and spectral norms," Tech. Rep. RJ 7140, IBM Research, Nov. 1989. Appeared inProc. IEEE Symp. Foundations Comput. Sci. Oct. 1990.
[4] A. K. Chandra, L. Stockmeyer, and R. Lipton, "Unbounded fan-in circuits and associative functions," inProc. 15th ACM Symp. Theory Comput., 1983, pp. 52-60.
[5] A. K. Chandra, L. Stockmeyer, and U. Vishkin, "Constant depth reducibility,"SIAM J. Comput., vol. 13, pp. 423-439, 1984.
[6] F. Fich, "Two problems in concrete complexity: Cycle detection and parallel prefix computation," Ph.D. dissertation, Univ. of California, Berkeley, 1982.
[7] M. Furst, J. B. Saxe, and M. Sipser, "Parity, circuits and the polynomial-time hierarchy," inProc. IEEE Symp. Foundations Comput. Sci., vol. 22, 1981, pp. 260-270.
[8] A. Hajnal, W. Maass, P. Pudlak, M. Szegedy, and G. Turan, "Threshold circuits of bounded depth,"IEEE Symp. Foundations Comput. Sci., vol. 28, 1987, pp. 99-110.
[9] T. Hofmeister, W. Hohberg, and S. Kohling, "Some notes on threshold circuits, and multiplication in depth 4," unpublished manuscript, 1990.
[10] W. Kautz, "The realization of symmetric switching functions with linearinput logical elements,"IRE Trans. Electron. Comput., vol. EC-10, 1961.
[11] R. E. Ladner and M. J. Fischer, "Parallel prefix computation,"J. ACM, vol. 27, no. 4, pp. 831-838, Oct. 1980.
[12] N. Linial, Y. Mansour, and N. Nisan, "Constant depth circuits, Fourier transforms, and learnability," inProc. 30th IEEE Symp. Foundations Comput. Sci., 1989.
[13] R. Minnick, "Linear-input logic,"IEEE Trans. Electron. Comput., vol. EC-10, 1961.
[14] D. E. Muller and F. P. Preparata, "Bounds to complexities of networks for sorting and switching,"J. ACM, vol. 22, no. 2, Apr. 1975.
[15] S. Muroga, "The principle of majority decision logic elements and the complexity of their circuits," inProc. Int. Conf. Inform. Processing, Paris, France, June 1959.
[16] R. Paturi and M. Saks, "On threshold circuits for parity," inProc. IEEE Symp. Foundations Comput. Sci., Oct. 1990.
[17] N. Pippenger, "The complexity of computations by networks,"IBM J. Res. Develop., vol. 31, no. 2, Mar. 1987.
[18] N. P. Redkin, "Synthesis of threshold circuits for certain classes of Boolean functions,"Kibernetika, no. 5, pp. 6-9, 1970.
[19] J. Reif, "On threshold circuits and polynomial computation," inProc. 2nd Structure in Complexity Theory Conf., 1987, pp. 118-123.
[20] V. P. Roychowdhury, K. Y. Siu, A. Orlitsky, and T. Kailath, "On the circuit complexity of neural networks," inAdvances in Neural Information Processing Systems 3, D. S. Touretzky and R. Lippman Eds., Apr. 1991.
[21] V. Roychowdhury, K. Y. Siu, A. Orlitsky, and T. Kailath, "A geometric approach to threshold circuit complexity," inProc. 4th Annu. Workshop Computational Learning Theory (COLT), Aug. 1991.
[22] C. Shannon, "The synthesis of two-terminal switching circuits,"Bell Syst. Tech. J.,vol. 28, pp. 59-98, 1949.
[23] K. Y. Siu and J. Bruck, "On the power of threshold circuits with small weights,"SIAM J. Disc. Math., Aug. 1991.
[24] K. Y. Siu and J. Bruck, "Neural computation of arithmetic functions,"Proc. IEEE, vol. 78, pp. 1669-1675, Oct. 1990.
[25] K. Y. Siu and J. Bruck, "On the dynamic range of linear threshold elements," Tech. Rep. RJ 7237, IBM Research, Jan. 1990.
[26] K. Y. Siu, J. Bruck, and T. Kailath, "Depth-efficient neural networks for division and related problems,"IEEE Trans. Inform. Theory, submitted for publication.
[27] K. Y. Siu, V. P. Roychowdhury, and T. Kailath, "Computing with almost optimal size threshold circuits," inProc. Int. Symp. Inform. Theory, Budapest, Hungary, June 1991.
[28] L. Stockmeyer, "On the combinational complexity of certain symmetric Boolean functions,"Math. Syst. Theory, vol. 10, pp. 323-336, 1977.
[29] I. Wegener,The Complexity of Boolean Functions. New York: Wiley, 1987.

Index Terms:
neural computation; parallel computation; size; threshold gates; symmetric Boolean functions; arithmetic functions; depth-3; Boolean functions; neural nets; threshold logic.
K.Y. Siu, V.P. Roychowdhury, T. Kailath, "Depth-Size Tradeoffs for Neural Computation," IEEE Transactions on Computers, vol. 40, no. 12, pp. 1402-1412, Dec. 1991, doi:10.1109/12.106225
Usage of this product signifies your acceptance of the Terms of Use.