The Community for Technology Leaders
RSS Icon
Issue No.11 - Nov. (2012 vol.23)
pp: 2033-2044
Zheng Wei , University of Maryland, College Park
Joseph JaJa , University of Maryland, College Park
Current high-throughput algorithms for constructing inverted files all follow the MapReduce framework, which presents a high-level programming model that hides the complexities of parallel programming. In this paper, we take an alternative approach and develop a novel strategy that exploits the current and emerging architectures of multicore processors. Our algorithm is based on a high-throughput pipelined strategy that produces parallel parsed streams, which are immediately consumed at the same rate by parallel indexers. We have performed extensive tests of our algorithm on a cluster of 32 nodes, and were able to achieve a throughput close to the peak throughput of the I/O system: a throughput of 280 MB/s on a single node and a throughput that ranges between 5.15 GB/s (1 Gb/s Ethernet interconnect) and 6.12 GB/s (10 Gb/s InfiniBand interconnect) on a cluster with 32 nodes for processing the ClueWeb09 data set. Such a performance represents a substantial gain over the best known MapReduce algorithms even when comparing the single node performance of our algorithm to MapReduce algorithms running on large clusters. Our results shed a light on the extent of the performance cost that may be incurred by using the simpler, higher level MapReduce programming model for large scale applications.
Indexing, Clustering algorithms, Program processors, Multicore processing, Dictionaries, Throughput, pipeline, Inverted files, MapReduce, multicore processors, cluster, I/O throughput, parallel algorithms, parallel parsing and indexing
Zheng Wei, Joseph JaJa, "An Optimized High-Throughput Strategy for Constructing Inverted Files", IEEE Transactions on Parallel & Distributed Systems, vol.23, no. 11, pp. 2033-2044, Nov. 2012, doi:10.1109/TPDS.2012.43
[1] Z. Wei and J. JaJa, "A Fast Algorithm for Constructing Inverted Files on Heterogeneous Platforms," Proc. IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS '11), May 2011.
[2] A. Moffat and T.A.H. Bell, "In Situ Generation of Compressed Inverted Files," J. Am. Soc. Information Science vol. 46, no. 7, pp. 537-550, Aug. 1995.
[3] S. Heinz and J. Zobel, "Efficient Single-pass Index Construction for Text Databases," J. Am. Soc. for Information Science and Technology, vol. 54, no. 8, pp. 713-729, June 2003.
[4] S. Melink, S. Raghavan, B. Yang, and H. Garcia-Molina, "Building a Distributed Full-Text Index for the Web," ACM Trans. Information Systems, vol. 19, no. 3, pp. 217-241, July 2001.
[5] B. Ribeiro-Neto, E.S. Moura, M.S. Neubert, and N. Ziviani, "Efficient Distributed Algorithms to Build Inverted Files," SIGIR '99: Proc. 22nd Ann. Int'l ACM SIGIR Conf. Research and Development in Information Retrieval, pp. 105-112, 1999.
[6] J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters," Proc. Sixth Conf. Symp. Opearting Systems Design & Implementation (OSDI '04), Dec. 2004.
[7] R. McCreadie, C. Mcdonald, and I. Ounis, "Proc. Comparing Distributed Indexing: To MapReduce or Not?," Proc. Seventh Workshop Large-Scale Distributed Systems for Information Retrieval, 2009.
[8] R. McCreadie, C. Mcdonald, and I. Ounis, "MapReduce Indexing Strategies: Studying Scalability and Efficiency," Information Processing and Management, In Press, 2011.
[9] J. Lin, D. Metzler, T. Elsayed, and L. Wang, "Of Ivory and Smurfs: Loxodontan MapReduce Experiments for Web Search," Proc. 18th Text Retrieval Conf. (TREC '09), Nov. 2009.
[10] T. Elsayed, F. Ture, and J. Lin, "Brute-Force Approaches to Batch Retrieval: Scalable Indexing with MapReduce, or Why Bother?," Technical Report HCIL-2010-23, Univ. of Maryland, College Park, Oct. 2010.
[11] J. Lin Personal Comm., Apr. 2011.
45 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool