The Community for Technology Leaders
RSS Icon
Issue No.09 - September (2011 vol.23)
pp: 1282-1298
Foto N. Afrati , National Techincal University Athens, Athens
Implementations of map-reduce are being used to perform many operations on very large data. We examine strategies for joining several relations in the map-reduce environment. Our new approach begins by identifying the “map-key,” the set of attributes that identify the Reduce process to which a Map process must send a particular tuple. Each attribute of the map-key gets a “share,” which is the number of buckets into which its values are hashed, to form a component of the identifier of a Reduce process. Relations have their tuples replicated in limited fashion, the degree of replication depending on the shares for those map-key attributes that are missing from their schema. We study the problem of optimizing the shares, given a fixed number of Reduce processes. An algorithm for detecting and fixing problems where a variable is mistakenly included in the map-key is given. Then, we consider two important special cases: chain joins and star joins. In each case, we are able to determine the map-key and determine the shares that yield the least replication. While the method we propose is not always superior to the conventional way of using map-reduce to implement joins, there are some important cases involving large-scale data where our method wins, including: 1) analytic queries in which a very large fact table is joined with smaller dimension tables, and 2) queries involving paths through graphs with high out-degree, such as the Web or a social network.
Map-reduce, joins, parallel computing, query optimization.
Foto N. Afrati, "Optimizing Multiway Joins in a Map-Reduce Environment", IEEE Transactions on Knowledge & Data Engineering, vol.23, no. 9, pp. 1282-1298, September 2011, doi:10.1109/TKDE.2011.47
[1] F.N. Afrati and J.D. Ullman, "Optimizing Joins in a Map-Reduce Environment," Proc. 13th Int'l Conf. Extending Database Technology (EDBT '10), 2010.
[2] Apache, "Hadoop," http:/, 2006.
[3] Apache, "Hive,", 2008.
[4] R. Avnur and J.M. Hellerstein, "Eddies: Continuously Adaptive Query Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '00), pp. 261-272, 2000.
[5] S. Babu, K. Munagala, J. Widom, and R. Motwani, "Adaptive Caching for Continuous Queries," Proc. 21st Int'l Conf. Data Eng. (ICDE '05), pp. 118-129, 2005.
[6] S. Babu and J. Widom, "Streamon: An Adaptive Engine for Stream Query Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '04), pp. 931-932, 2004.
[7] D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke, "Nephele/Pacts: A Programming Model and Execution Framework for Web-Scale Analytical Processing," Proc. First ACM Symp. Cloud Computing (SoCC '10), pp. 119-130, 2010.
[8] V. Borkar, M. Carey, R. Grover, N. Onose, and R. Vernica, "Hyracks: A Flexible and Extensible Foundation for Data-Intensive Computing," Proc. IEEE 27th Int'l Conf. Data Eng. (ICDE), 2011.
[9] S. Brin and L. Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," Computer Networks and ISDN Systems, vol. 30, nos. 1-7, pp. 107-117, 1998.
[10] Y. Bu, B. Howe, M. Balazinska, and M. Ernst, "Haloop: Efficient Iterative Data Processing on Large Clusters," Proc. VLDB Endowment, vol. 3, no. 1/2, pp. 285-296, 2010.
[11] R. Chaiken, B. Jenkins, P. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "Scope: Easy and Efficient Parallel Processing of Massive Data Sets," Proc. VLDB Endowment, vol. 1, no. 2, pp. 1265-1276, 2008.
[12] F. Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D.A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R.E. Gruber, "Bigtable: A Distributed Storage System for Structured Data," ACM Trans. Computer Systems, vol. 26, no. 2, pp. 1-26, 2008.
[13] B.F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon, H.-A. Jacobsen, N. Puz, D. Weaver, and R. Yerneni, "Pnuts: Yahoo!'s Hosted Data Serving Platform," Proc. VLDB Endowment, vol. 1, no. 2, pp. 1277-1288, 2008.
[14] J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters," Comm. ACM, vol. 51, no. 1, pp. 107-113, 2008.
[15] A. Deshpande and L. Hellerstein, "Flow Algorithms for Parallel Query Optimization," Proc. IEEE 24th Int'l Conf. Data Eng. (ICDE), pp. 754-763, 2008.
[16] D.J. DeWitt, J.F. Naughton, D.A. Schneider, and S. Seshadri, "Practical Skew Handling in Parallel Joins," Proc. 18th Int'l Conf. Very Large Data Bases (VLDB '92), pp. 27-40, 1992.
[17] D.J. DeWitt, E. Paulson, E. Robinson, J.F. Naughton, J. Royalty, S. Shankar, and A. Krioukov, "Clustera: An Integrated Computation and Data Management System," Proc. VLDB Endowment, vol. 1, no. 1, pp. 28-41, 2008.
[18] S. Ganguly, A. Silberschatz, and S. Tsur, "A Framework for the Parallel Processing of Datalog Queries," ACM SIGMOD Record, vol. 19, pp. 143-152, June 1990.
[19] H. Garcia-Molina, J.D. Ullman, and J. Widom, Database Systems: The Complete Book. Pearson Prentice Hall, 2008.
[20] S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google File System," Proc. 19th ACM Symp. Operating Systems Principles, 2003.
[21] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," ACM SIGOPS Operating Systems Rev., vol. 41, no. 3, pp. 59-72, 2007.
[22] Z.G. Ives, D. Florescu, M. Friedman, A.Y. Levy, and D.S. Weld, "An Adaptive Query Execution System for Data Integration," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '99), pp. 299-310, 1999.
[23] H. Jakobsson, "Mixed-Approach Algorithms for Transitive Closure," Proc. 10th ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems (PODS '91), pp. 199-205, 1991.
[24] H. Jakobsson, "On Tree-Based Techniques for Query Evaluation," Proc. 11th ACM SIGACT-SIGMOD-SIGART Symp. Principles of Database Systems (PODS '92), pp. 380-392, 1992.
[25] H. Jakobsson, "Tree-Based Techniques for Query Evaluation," PhD thesis, Dept. of CS, Stanford Univ., STAN-CS-93-1492, 1993.
[26] J.M. Kleinberg, "Authoritative Sources in a Hyperlinked Environment," J. ACM, vol. 46, pp. 668-677, 1999.
[27] S. Madden, M.A. Shah, J.M. Hellerstein, and V. Raman, "Continuously Adaptive Continuous Queries over Streams," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '02), pp. 49-60, 2002.
[28] G. Malewicz, M. Austern, A. Bik, J. Dehnert, I. horn, N. Leiser, and G. Czajkowski, "Pregel: A System for Large-Scale Graph Processing," Proc. Int'l Conf. Management of Data (SIGMOD '10), 2010.
[29] C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A Not-so-Foreign Language for Data Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data (SIGMOD '08), pp. 1099-1110, 2008.
[30] K.A. Ross and J. Cieslewicz, "Optimal Splitters for Database Partitioning with Size Bounds," Proc. 12th Int'l Conf. Database Theory (ICDT '09), pp. 98-110, 2009.
[31] U. Srivastava, K. Munagala, J. Widom, and R. Motwani, "Query Optimization over Web Services," Proc. 32nd Int'l Conf. Very Large Data Bases (VLDB '06), pp. 355-366, 2006.
[32] K.-L. Tan and H. Lu, "A Note on the Strategy Space of Multiway Join Query Optimization Problem in Parallel Systems," ACM SIGMOD Record, vol. 20, no. 4, pp. 81-82, 1991.
[33] S.D. Viglas, J.F. Naughton, and J. Burger, "Maximizing the Output Rate of Multi-Way Join Queries over Streaming Information Sources," Proc. 29th Int'l Conf. Very Large Data Bases (VLDB '03), pp. 285-296, 2003.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool