Issue No. 08 - August (2007 vol. 29)
Giorgio Satta , IEEE Computer Society
In this paper, we consider probabilistic context-free grammars, a class of generative devices that has been successfully exploited in several applications of syntactic pattern matching, especially in statistical natural language parsing. We investigate the problem of training probabilistic context-free grammars on the basis of distributions defined over an infinite set of trees or an infinite set of sentences by minimizing the cross-entropy. This problem has applications in cases of context-free approximation of distributions generated by more expressive statistical models. We show several interesting theoretical properties of probabilistic context-free grammars that are estimated in this way, including the previously unknown equivalence between the grammar cross-entropy with the input distribution and the so-called derivational entropy of the grammar itself. We discuss important consequences of these results involving the standard application of the maximum-likelihood estimator on finite tree and sentence samples, as well as other finite-state models such as Hidden Markov Models and probabilistic finite automata.
Probabilistic context-free grammars, maximum-likelihood estimation, derivational entropy, cross-entropy, expectation-maximization methods, Hidden Markov Models.
Giorgio Satta, Anna Corazza, "Probabilistic Context-Free Grammars Estimated from Infinite Distributions", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 29, no. , pp. 1379-1393, August 2007, doi:10.1109/TPAMI.2007.1065