The Community for Technology Leaders
Green Image
Issue No. 03 - March (2017 vol. 29)
ISSN: 1041-4347
pp: 499-512
Wen Hua , School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, QLD, Australia
Zhongyuan Wang , Microsoft Research Asia, Beijing, China
Haixun Wang , Google Research, Mountain View, CA
Kai Zheng , School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, QLD, Australia
Xiaofang Zhou , School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, QLD, Australia
ABSTRACT
Understanding short texts is crucial to many applications, but challenges abound. First, short texts do not always observe the syntax of a written language. As a result, traditional natural language processing tools, ranging from part-of-speech tagging to dependency parsing, cannot be easily applied. Second, short texts usually do not contain sufficient statistical signals to support many state-of-the-art approaches for text mining such as topic modeling. Third, short texts are more ambiguous and noisy, and are generated in an enormous volume, which further increases the difficulty to handle them. We argue that semantic knowledge is required in order to better understand short texts. In this work, we build a prototype system for short text understanding which exploits semantic knowledge provided by a well-known knowledgebase and automatically harvested from a web corpus. Our knowledge-intensive approaches disrupt traditional methods for tasks such as text segmentation, part-of-speech tagging, and concept labeling, in the sense that we focus on semantics in all these tasks. We conduct a comprehensive performance evaluation on real-life data. The results show that semantic knowledge is indispensable for short text understanding, and our knowledge-intensive approaches are both effective and efficient in discovering semantics of short texts.
INDEX TERMS
Semantics, Tagging, Labeling, Vocabulary, Hidden Markov models, Coherence, Context
CITATION

W. Hua, Z. Wang, H. Wang, K. Zheng and X. Zhou, "Understand Short Texts by Harvesting and Analyzing Semantic Knowledge," in IEEE Transactions on Knowledge & Data Engineering, vol. 29, no. 3, pp. 499-512, 2017.
doi:10.1109/TKDE.2016.2571687
190 ms
(Ver 3.3 (11022016))