This Article 
 Bibliographic References 
 Add to: 
Simple Strategies to Encode Tree Automata in Sigmoid Recursive Neural Networks
March/April 2001 (vol. 13 no. 2)
pp. 148-156

Abstract—Recently, a number of authors have explored the use of recursive neural nets (RNN) for the adaptive processing of trees or tree-like structures. One of the most important language-theoretical formalizations of the processing of tree-structured data is that of deterministic finite-state tree automata (DFSTA). DFSTA may easily be realized as RNN using discrete-state units, such as the threshold linear unit. A recent result by Síima (Neural Network World7 (1997), pp. 679–686) shows that any threshold linear unit operating on binary inputs can be implemented in an analog unit using a continuous activation function and bounded real inputs. The constructive proof finds a scaling factor for the weights and reestimates the bias accordingly. In this paper, we explore the application of this result to simulate DFSTA in sigmoid RNN (that is, analog RNN using monotonically growing activation functions) and also present an alternative scheme for one-hot encoding of the input that yields smaller weight values and, therefore, works at a lower saturation level.

[1] R. Alquézar and A. Sanfeliu, “An Algebraic Framework to Represent Finite State Automata in Single-Layer Recurrent Neural Networks,” Neural Computation, vol. 7, no. 5, pp. 931–949, 1995.
[2] R.C. Carrasco, M.L. Forcada, M. Ángeles Valdés-Muñoz, and R.P. Ñeco, “Stable Encoding of Finite-State Machines in Discrete-Time Recurrent Neural Nets with sigmoid Units,” Neural Computation, vol. 12, no. 9, pp. 2129–2174, Sept. 2000.
[3] J.L. Elman, “Finding Structure in Time,” Cognitive Science, vol. 14, pp. 179–211, 1990.
[4] M.L. Forcada and R.C. Carrasco, “Learning the Initial State of a Second-Order Recurrent Neural Network During Regular-Language Inference,” Neural Computation, vol. 7, no. 5, pp. 923–930, 1995.
[5] P. Frasconi, M. Gori, and A. Sperduti, “A General Framework for Adaptive Processing of Data Structures,” IEEE Trans. Neural Networks, vol. 9, no. 5, pp. 768–786, 1998.
[6] C.L. Giles, C.B. Miller, D. Chen, H.H. Chen, G.Z. Sun, and Y.C. Lee, “Learning and Extracted Finite State Automata with Second-Order Recurrent Neural Networks,” Neural Computation, vol. 4, no. 3, pp. 393–405, 1992.
[7] R.C. Gonzalez and M.G. Thomason, Syntactical Pattern Recognition. Menlo Park, Calif.: Addison-Wesley, 1978.
[8] M.W. Goudreau, C.L. Giles, and S.T. Chakradhar, D. Chen, “First-Order vs. Second-Order Single Layer Recurrent Neural Networks,” IEEE Trans. Neural Networks, vol. 5, no. 3, pp. 511–513, 1994.
[9] J.E. Hopcroft and J.D. Ullman, Introduction to Automata Theory, Languages and Computation. Addison-Wesley, Apr. 1979.
[10] S.C. Kremer, “On the Computational Power of Elman-Style Recurrent Networks,” IEEE Trans. Neural Networks, vol. 6, no. 4, pp. 1000–1004, 1995.
[11] S.C. Kremer, R.P. Ñeco, and M.L. Forcada, “Constrained Second-Order Recurrent Networks for Finite-State Automata Induction,” Proc. Eighth Int'l Conf. Artificial Neural Networks ICANN '98, L. Niklasson, M. Bodín, and T. Ziemke, eds., vol. 2, pp. 529–534, 1998.
[12] R.P. Ñeco, M.L. Forcada, R.C. Carrasco, and M.A. Valdés-Muñoz, “Encoding of Sequential Translators in Discrete-Time Recurrent Neural Nets,” Proc. European Symp. Artificial Neural Networks ESANN '99, pp. 375–380, 1999.
[13] C.W. Omlin and C.L. Giles, “Constructing Deterministic Finite-State Automata in Recurrent Neural Networks,” J. ACM, vol. 43, no. 6, pp. 937–972, 1996.
[14] C.W. Omlin and C.L. Giles, “Stable Encoding of Large Finite-State Automata in Recurrent Neural Networks with sigmoid Discriminants,” Neural Computation, vol. 8, pp. 675–696, 1996.
[15] A. Salomaa, Formal Languages. New York: Academic Press, 1973.
[16] J. Síma, “Analog Stable Simulation of Discrete Neural Networks,” Neural Network World, vol. 7, pp. 679–686, 1997.
[17] A. Sperduti, “On the Computational Power ofNeural Networks for Structures,” Neural Networks, vol. 10, no. 3, pp. 395–400, 1997.
[18] A. Sperduti and A. Starita, “Supervised Neural Networks for the Classification of Structures,” IEEE Trans. Neural Networks, vol. 8, no. 3, pp. 714–735, 1997.
[19] J.W. Thatcher, “Tree Automata: An Informal Survey,” Currents in the Theory of Computing, A.V. Aho, ed., 1973.

Index Terms:
Tree automata, recursive neural networks, neural computation, analog neural networks.
Rafael C. Carrasco, Mikel L. Forcada, "Simple Strategies to Encode Tree Automata in Sigmoid Recursive Neural Networks," IEEE Transactions on Knowledge and Data Engineering, vol. 13, no. 2, pp. 148-156, March-April 2001, doi:10.1109/69.917555
Usage of this product signifies your acceptance of the Terms of Use.