This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Learning Distributed Representations of Concepts Using Linear Relational Embedding
March/April 2001 (vol. 13 no. 2)
pp. 232-244

Abstract—In this paper, we introduce Linear Relational Embedding as a means of learning a distributed representation of concepts from data consisting of binary relations between these concepts. The key idea is to represent concepts as vectors, binary relations as matrices, and the operation of applying a relation to a concept as a matrix-vector multiplication that produces an approximation to the related concept. A representation for concepts and relations is learned by maximizing an appropriate discriminative goodness function using gradient ascent. On a task involving family relationships, learning is fast and leads to good generalization.

[1] J. Bridle, “Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition,” Neurocomputing: Algorithms, Architectures and Applications, F.F. Souliéand J. Hérault, eds., pp. 227–236, 1990.
[2] A. Cleeremans, D. Servan-Schreiber, and J. McClelland, “Finite State Automata and Simple Recurrent Neural Networks,” Neural Computation, vol. 1, no. 3, pp. 372–381, 1989.
[3] S. Deerwester, S.T. Dumais, G. Furnas, T.K. Landauer, and R. Harshman, “Indexing by Latent Semantic Analysis,” J. Am. Soc. for Information Science, vol. 41, pp. 391–407, 1990.
[4] J. Elman, “Finding Structure in Time,” Cognitive Science, vol. 14, pp. 179–211, 1990.
[5] C.L. Giles, C.B. Miller, D. Chen, H.H. Chen, G.Z. Sun, and Y.C. Lee, “Learning and Extracted Finite State Automata with Second-Order Recurrent Neural Networks,” Neural Computation, vol. 4, no. 3, pp. 393–405, 1992.
[6] G.E. Hinton, “Learning Distributed Representations of Concepts,” Proc. Eighth Ann. Conf. Cognitive Science Soc., pp. 1–12, 1986.
[7] J.B. Kruskal, “Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis,” Psychometrika, vol. 29, no. 1, pp. 1–27, 1964.
[8] T.K. Landauer and S.T. Dumais, “A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge,” Psychological Rev., vol. 104, no. 2, pp. 211–240, 1997.
[9] T.K. Landauer, D. Laham, and P. Foltz, “Learning Human-Like Knowledge by Singular Value Decomposition: A Progress Report,” Advances in Neural Processing Information Systems 10, M.I. Jordan, M.J. Kearns, and S.A. Solla, eds., pp. 45–51, 1998.
[10] M. Møller, “A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning,” Neural Networks, vol. 6, pp. 525–533, 1993.
[11] R.C. O'Reilly, “The LEABRA Model of Neural Interactions and Learning in the Neocortex,” PhD thesis, Dept. of Psychology, Carnegie Mellon Univ., 1996.
[12] J.B. Pollack, “Recursive Distributed Representations,” Artificial Intelligence, vol. 46, nos. 1-2, pp. 77–106, 1990.
[13] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning Internal Representations by Error Propagation," Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1: Foundations, D.E. Rumelhart and J.L. McClelland et al., eds., chapter 8, pp. 318-362.Cambridge, Mass.: MIT Press, 1986.
[14] A. Sperduti, “Labeling RAAM,” Connection Science, vol. 6, pp. 429–459, 1994.
[15] J. Tenenbaum and W.T. Freeman, “Separating Style and Content,” Advances in Neural Processing Information Systems 9, M.C. Mozer, M.I. Jordan, and T. Petsche, eds., pp. 662–668, 1996.
[16] F.W. Young and R.M. Hamer, Multidimensional Scaling: History, Theory and Applications. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1987.

Index Terms:
Distributed representations, feature learning, concept learning, learning structured data, generalization on relational data, Linear Relational Embedding.
Citation:
Alberto Paccanaro, Geoffrey E. Hinton, "Learning Distributed Representations of Concepts Using Linear Relational Embedding," IEEE Transactions on Knowledge and Data Engineering, vol. 13, no. 2, pp. 232-244, March-April 2001, doi:10.1109/69.917563
Usage of this product signifies your acceptance of the Terms of Use.