This Article 
 Bibliographic References 
 Add to: 
Integrating Linguistic Primitives in Learning Context-Dependent Representation
March/April 2001 (vol. 13 no. 2)
pp. 157-175

Abstract—This paper presents an explicit connectionist-inspired, language learning model in which the process of settling on a particular interpretation for a sentence emerges from the interaction of a set of “soft” lexical, semantic, and syntactic primitives. We address how these distinct linguistic primitives can be encoded from different modular knowledge sources but strongly involved in an interactive processing in such a way as to make implicit linguistic information explicit. The learning of a quasi-logical form, called context-dependent representation, is inherently incremental and dynamical in such a way that every semantic interpretation will be related to what has already been presented in the context created by prior utterances. With the aid of the context-dependent representation, the capability of the language learning model in text understanding is strengthened. This approach also shows how the recursive and compositional role of a sentence as conveyed in the syntactic structure can be modeled in a neurobiologically motivated linguistics based on dynamical systems rather on combinatorial symbolic architecture. Experiments with more than 2,000 sentences in different languages illustrating the influences of the context-dependent representation on semantic interpretation, among other issues, are included.

[1] J.R. Anderson and C. Lebiere, The Atomic Components of Thought. Lawrence Erlbaum, 1998.
[2] M. Ariel, “Interpreting Anaphoric Expressions: A Cognitive Versus A Pragmatic Approach,” J. Linguistics, vol. 30, pp. 3–42, 1994.
[3] S. Azzam, “Anaphors, PPs, and Disambiguation Process for Conceptual Analysis,” Proc. Int'l Joint Conf. Artificial Intelligence, pp. 1354–1359, 1995.
[4] D.S. Blank, L.A. Meeden, and J.B. Marshall, “Exploring the Symbolic/Subsymbolic Continuum: A Case Study of RAAM,” Closing the Gap: Symbolism vs. Connectionism, J. Dinsmore, ed., 1992.
[5] L.A. Bookman, Trajectories through Knowledge Space: A Dynamic Framework for Machine Comprehension. Kluwer Academic, 1994.
[6] P. Brézillon, “Using Context in Applications,” Int'l J. Human-Machine Studies, introduction the the special issue, vol. 48, pp. 303–305,
[7] J.G. Carbonell and R.D. Brown, “Anaphora Resolution: A Multi-Strategy Approach,” Proc. Int'l Conf. Computational Linguistics (COLING '88), pp. 96–101, 1988.
[8] T.D.C. Little and D. Venkatesh, “Popularity-Based Assignment of Movies to Storage Devices in a Video-on-Demand System,” Proc. Fourth Int'l Workshop Network and Operating System Support for Digital Audio and Video, Nov. 1993.
[9] E. Charniak and E. Santos, “A Connectionist Context-Free Parser Which Is Not Context-Free, but then It Is Not Really Connectionist Either,” Proc. Ninth Ann. Conf. Cognitive Science Soc., pp. 70–77, 1987.
[10] Chinese Knowledge Information Processing Group, Academia Sinica Balanced Corpus, Version 3.0. Inst. of Information Science, Academia Sinica, 1997.
[11] G.W. Cottrell and S.L. Small, “A Connectionist Scheme for Modelling Word Sense Disambiguation,” Cognition and Brain Theory, vol. 6, pp. 89–120, 1983.
[12] M.G. Dyer, M. Flowers, and Y.J.A. Wang, “Distributed Symbol Discovery through Symbol Recirculation: Toward Natural Language Processing in a Distributed Connectionist Network,” Connectionist Approaches to Natural Language Processing, R.G. Reilly and N.E. Sharkey, eds., pp. 21–48, 1992.
[13] J.L. Elman, “Distributed Representations, Simple Recurrent Networks, and Grammatical Structure,” Machine Learning, vol. 7, pp. 195–225, 1991.
[14] J.A. Feldman, “Structured Connectionist Models and Language Processing,” Artificial Intelligence Review, vol. 7, no. 5, pp. 301–312, 1993.
[15] S. Finch, N. Chater, and M. Redington, “Acquiring Syntactic Information from Distributional Statistics,” Connectionist Models of Memory and Language, J.P. Levy, D. Bairaktaris, J.A. Bullinaria, and P. Cairns, eds., pp. 229–242, 1995.
[16] J.A. Fodor and Z.W. Pylyshyn, “Connectionism and Cognitive Architecture: A Critical Analysis,” Connections and Symbols, S. Pinker and J. Mehler, eds., pp. 3–71, 1988.
[17] P. Frasconi, M. Gori, and A. Sperduti, “A General Framework for Adaptive Processing of Data Structures,” IEEE Trans. Neural Networks, vol. 9, no. 5, pp. 768–786, 1998.
[18] A. Garnham, “Context-Dependent Models as Representations of Text,” Memory and Cognition, vol. 9, pp. 560–565, 1981.
[19] A. Garnham, Context-Dependent Models as Representations of Discourse and Text. Ellis Horwood, 1987.
[20] T.V. Geetha and R.K. Subramanian, “Representing Natural Language with Prolog,” IEEE Software, vol. 7, no. 2, pp. 85–92, 1990.
[21] P. Gupta and D.S. Touretzky, “Connectionist Models and Linguistic Theory: Investigations of Stress Systems in Language,” Cognitive Science, vol. 18, pp. 1–50, 1994.
[22] R.F. Hadley and V.C. Cardei, “Language Acquisition from Sparse Input Without Error Feedback,” Neural Networks, vol. 12, pp. 217–235, 1999.
[23] G.E. Hinton, “Representing Part-Whole Hierarchies in Connectionist Networks,” Technical Report CRG TR882, Connectionist Research Group, Univ. of Toronto, 1988.
[24] C.-R. Huang and K.-J. Chen, “Issues and Topics in Chinese Natural Language Processing,” J. Chinese Linguistics, C.-R. Huang, K.-J. Chen, and B.K. T'sou, eds., Monograph no. 9, pp. 1–22, 1996.
[25] P.N. Johnson-Laird, Context-Dependent Models. Harvard Univ. Press, 1983.
[26] W. Kintsch, “The Role of Knowledge in Discourse Processing: A Construction-Integration Model,” Psychological Review, vol. 95, pp. 163–182, 1988.
[27] W. Kintsch, “How Readers Construct Situation Models for Stories: The Role of Syntactic Cues and Causal Inferences,” From Learning Processes to Cognitive Processes: Essays in Honor of William K. Estes, A.F. Healy, S.M. Kosslyn, and R.M. Shiffrin, eds., vol. 2, pp. 261–278, 1992.
[28] W. Kintsch and T.A. van Dijk, “Toward a Model of Text Comprehension and Production,” Psychological Review, vol. 85, no. 5, pp. 363–394, 1978.
[29] T.E. Lange, “A Structured Connectionist Approach to Inferencing and Retrieval,” Computational Architectures Integrating Neural and Symbolic Processes: A Perspective on the State of the Art, R. Sun and L.A. Bookman, eds., pp. 69–115, 1995.
[30] C.N. Li and S.A. Thompson, Mandarin Chinese: A Functional Reference Grammar. Univ. of California Press, 1981.
[31] T. McArthur, Longman Lexicon of Contemporary English (English-Chinese Edition). Longman, 1992.
[32] J.L. McClelland and A.H. Kawamoto, “Mechanisms of Sentence Processing: Assigning Roles to Constituents of Sentences,” Parallel and Distributed Processing, D.E. Rumelhart and J.L. McClelland, eds., vol. 2, pp. 272–325, 1986.
[33] R. Miikkulainen, Subsymbolic Natural Language Processing: An Integrated Model of Scripts, Lexicon, and Memory. MIT Press, 1993.
[34] J. Pearl, Probabilistic Inference in Intelligent Systems. Morgan Kaufmann, 1988.
[35] A. Pirkola and K. Jarvelin, “The Effect of Anaphora and Ellipsis Resolution on Proximity Searching in a Text Database,” Information Processing and Management, vol. 32, no. 2, pp. 199–216, 1996.
[36] T.A. Plate, “Holographic Reduced Representations,” IEEE Trans. Neural Networks, vol. 6, no. 3, pp. 623–641, 1995.
[37] J.B. Pollack, “Implications of Recursive Distributed Representations,” Advances in Neural Information Processing Systems I, D.S. Touretzky, ed., pp. 527–636, 1989.
[38] J.B. Pollack, “Recursive Distributed Representations,” Artificial Intelligence, vol. 46, nos. 1-2, pp. 77–106, 1990.
[39] R.G. Reilly, “A Connectionist Model of Some Aspects of Anaphor Resolution,” Proc. 10th Int'l Conf. Computational Linguistics and 22nd Ann. Meeting Assoc. for Computational Linguistics, pp. 144–149, 1984.
[40] E. Rich and S. Luperfoy, “An Architecture for Anaphora Resolution,” Proc. 26th Annual Meeting Assoc. for Computational Linguistics, pp. 18–24, 1988.
[41] D.EE. Rumelhart and J.L. McClelland, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, Cambridge, Mass., 1986.
[42] G. Salton et al., "Automatic Text Structuring and Summarization," Information Processing&Management, Vol. 33, No. 2, 1997, pp. 193-207.
[43] B. Selman and G. Hirst, “Parsing as an Energy Minimization Problem,” Parallel Natural Language Processing, G. Adriaens and U. Hahn, eds., pp. 238–254, 1994.
[44] D. Servan-Schreiber, A. Cleeremans, and J.L. McClelland, “Graded State Machines: The Representation of Temporal Contingencies in Simple Recurrent Networks,” Machine Learning, vol. 7, pp. 161–194, 1991.
[45] N. Sharkey, Natural Language Processing. Kluwer Academic, 1992.
[46] L. Shastri and V. Ajjanagadde, “From Simple Associations to Systematic Reasoning: A Connectionist Representation of Rules, Variables, and Dynamic Bindings Using Temporal Synchrony,” Behavioral and Brain Sciences, vol. 16, no. 4, pp. 417–494, 1993.
[47] L. Shastri, V. Ajjanagadde, L. Bonatti, T.E. Lange, and M.G. Dyer, “From Simple Associations to Systematic Reasoning: A Connectionist Representation of Rules, Variables, and Dynamic Bindings Using Temporal Synchrony,” Behavioral and Brain Sciences, vol. 19, no. 2, pp. 326–337, 1996.
[48] R.F. Simmons, “Semantic Networks: Their Computation and Use for Understanding English Sentences,” R.C. Schank and K.M. Colby, eds., Computer Models of Thought and Language, pp. 63–113, 1973.
[49] P. Smolensky, “Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems,” Artificial Intelligence, vol. 46, pp. 159–216, 1990.
[50] A. Sperduti, “Stability Properties of Labeling Recursive Auto-Associative Memory,” IEEE Trans. Neural Networks, vol. 6, pp. 1452-1460, 1995.
[51] M.F. St. John and J.L. McClelland, “Learning and Applying Contextual Constraints in Sentence Comprehension,” Artificial Intelligence, vol. 46, pp. 217–257, 1990.
[52] R. Sun, “Structuring Knowledge in Vague Domains,” IEEE Trans. Knowledge and Data Eng., vol. 7, no. 1, pp. 120–136, Feb. 1995.
[53] K. van Hoek, Anaphora and Conceptual Structure. Univ. of Chicago Press, 1997.
[54] C.J. van Rijsbergen, Information Retrieval. London: Butterworths, 1979.
[55] D.L. Waltz and J.B. Pollack, “Massively Parallel Parsing: A Strongly Interactive Model of Natural Language Interpretation,” Cognitive Science, vol. 9, pp. 51–74, 1985.
[56] S. Wermter and V. Weber, “SCREEN: Learning a Flat Syntactic and Semantic Spoken Language Analysis Using Artificial Neural Networks,” J. Artificial Intelligence Research, vol. 6, pp. 35–85, 1997.
[57] Y. Wilks, D. Fass, C.-M. Guo, J. McDonald, T. Plate, and B. Slator, “Providing Machine Tractable Dictionary Tools,” Semantics and the Lexicon, J. Pustejovsky, ed., pp. 341–401, 1993.

Index Terms:
Language learning, connectionism, text understanding, computational linguistics.
Samuel W.K. Chan, "Integrating Linguistic Primitives in Learning Context-Dependent Representation," IEEE Transactions on Knowledge and Data Engineering, vol. 13, no. 2, pp. 157-175, March-April 2001, doi:10.1109/69.917558
Usage of this product signifies your acceptance of the Terms of Use.