Issue No. 10 - Oct. (2013 vol. 25)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TKDE.2012.195
Chuitian Rong , Renmin University of China, Bejing
Wei Lu , National University of Singapore, Singapore
Xiaoli Wang , National University of Singapore, Singapore
Xiaoyong Du , Renmin University of China, Beijing
Yueguo Chen , Renmin University of China, Beijing
Anthony K.H. Tung , National University of Singapore, Singapore
The string similarity join is a basic operation of many applications that need to find all string pairs from a collection given a similarity function and a user-specified threshold. Recently, there has been considerable interest in designing new algorithms with the assistant of an inverted index to support efficient string similarity joins. These algorithms typically adopt a two-step filter-and-refine approach in identifying similar string pairs: 1) generating candidate pairs by traversing the inverted index; and 2) verifying the candidate pairs by computing the similarity. However, these algorithms either suffer from poor filtering power (which results in high verification cost), or incur too much computational cost to guarantee the filtering power. In this paper, we propose a multiple prefix filtering method based on different global orderings such that the number of candidate pairs can be reduced significantly. We also propose a parallel extension of the algorithm that is efficient and scalable in a MapReduce framework. We conduct extensive experiments on both centralized and Hadoop systems using both real and synthetic data sets, and the results show that our proposed approach outperforms existing approaches in both efficiency and scalability.
Indexes, Pipeline processing, Algorithm design and analysis, Filtering, Educational institutions, XML, Transforms, multiple filtering, Indexes, Pipeline processing, Algorithm design and analysis, Filtering, Educational institutions, XML, Transforms, MapReduce, Similarity join
C. Rong, W. Lu, X. Wang, X. Du, Y. Chen and A. K. Tung, "Efficient and Scalable Processing of String Similarity Join," in IEEE Transactions on Knowledge & Data Engineering, vol. 25, no. , pp. 2217-2230, 2013.