2011 IEEE 27th International Conference on Data Engineering (2011)
Apr. 11, 2011 to Apr. 16, 2011
Jiannan Wang , Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
Guoliang Li , Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
Jianhua Fe , Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
String similarity join that finds similar string pairs between two string sets is an essential operation in many applications, and has attracted significant attention recently in the database community. A significant challenge in similarity join is to implement an effective fuzzy match operation to find all similar string pairs which may not match exactly. In this paper, we propose a new similarity metrics, called "fuzzy token matching based similarity", which extends token-based similarity functions (e.g., Jaccard similarity and Cosine similarity) by allowing fuzzy match between two tokens. We study the problem of similarity join using this new similarity metrics and present a signature-based method to address this problem. We propose new signature schemes and develop effective pruning techniques to improve the performance. Experimental results show that our approach achieves high efficiency and result quality, and significantly outperforms state-of-the-art methods.
G. Li, J. Fe and J. Wang, "Fast-join: An efficient method for fuzzy token matching based string similarity join," 2011 IEEE 27th International Conference on Data Engineering(ICDE), Hannover, Germany, 2011, pp. 458-469.