The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.06 - November (1988 vol.10)
pp: 960-970
ABSTRACT
<p>The inherent statistical characteristics, including the economy, entropy, and redundancy, of a very large set containing 93681 words from the Shorter Oxford English Dictionary is investigated. Analytical n-gram statistics are also presented for applications in natural language understanding, text processing, test compression, error detection and correction, and speech synthesis and recognition. Experimental results show how the distribution of n-grams in the dictionary varies from the ideal as n increases from 2 to 5, that is, from bigrams to pentagrams; it is shown that the corresponding redundancy increases from 0.1067 to 0.3409. The results are of interest because, (1) the dictionary provides a finite list for deterministic analyses, (2) each entry (word) appears once, compared to free-running text where words are repeated, and (3) all entries, even rarely occurring ones, have equal weight.</p>
INDEX TERMS
entropy; redundancy; inherent statistical characteristics; Shorter Oxford English Dictionary; natural language understanding; text processing; test compression; error detection; glossaries; information analysis; natural languages; redundancy; statistical analysis
CITATION
E.J. Yannakoudakis, G. Angelidakis, "An Insight into the Entropy and Redundancy of the English Dictionary", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.10, no. 6, pp. 960-970, November 1988, doi:10.1109/34.9119
16 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool