loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
International Conference on Semantic Computing (ICSC 2007)
N-gram Statistics in English and Chinese: Similarities and Differences
Irvine, California
September 17-September 19
ISBN: 0-7695-2997-6
ASCII Text
x 
 
Stewart Yang, Hongjun Zhu, Ariel Apostoli, Pei Cao, "N-gram Statistics in English and Chinese: Similarities and Differences," International Conference on Semantic Computing, pp. 454-460, International Conference on Semantic Computing (ICSC 2007), 2007.
 
 
BibTex
x
 
@article{ 10.1109/ICSC.2007.46,
author = {Stewart Yang and Hongjun Zhu and Ariel Apostoli and Pei Cao},
title = {N-gram Statistics in English and Chinese: Similarities and Differences},
journal ={International Conference on Semantic Computing},
volume = {0},
year = {2007},
isbn = {0-7695-2997-6},
pages = {454-460},
doi = {http://doi.ieeecomputersociety.org/10.1109/ICSC.2007.46},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - CONF
JO - International Conference on Semantic Computing
TI - N-gram Statistics in English and Chinese: Similarities and Differences
SN - 0-7695-2997-6
SP454
EP460
A1 - Stewart Yang,
A1 - Hongjun Zhu,
A1 - Ariel Apostoli,
A1 - Pei Cao,
PY - 2007
KW - null
VL - 0
JA - International Conference on Semantic Computing
ER -
 
Stewart Yang, Google, Inc., USA
Hongjun Zhu, Google, Inc., USA
Ariel Apostoli, Google, Inc., USA
Pei Cao, Google, Inc., USA
Chinese and English belong to two very different families of human languages. Yet, since the underlying human concepts are universal, one can expect that there are many statistical similarities between Chinese texts and English texts. In this paper, we present results of analyzing the quantity and frequency of N-grams in 200 million randomly-sampled English and Chinese web pages.

The similarities and differences in N-gram frequency distributions yield important insights about the two languages. First, the distribution of the unique number of N-grams is similar between English and Chinese, yet the Chinese distribution is "shifted" to larger N. The distribution indicates that on average, 1.5 Chinese characters correspond to 1 English word. Second, while frequency distributions of uni-grams and bi-grams are very different between Chinese and English, the frequency distribution for 3-grams and 4- grams are strikingly similar between Chinese and English. This leads to the conjecture that in both languages, frequent 3-grams and 4-grams represent the same set of concepts and patterns.

Citation:
Stewart Yang, Hongjun Zhu, Ariel Apostoli, Pei Cao, "N-gram Statistics in English and Chinese: Similarities and Differences," icsc, pp.454-460, International Conference on Semantic Computing (ICSC 2007), 2007
Usage of this product signifies your acceptance of the Terms of Use.