This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
2011 International Green Computing Conference and Workshops
Predictive data and energy management in GreenHDFS
Orlando, FL
July 25-July 28
ISBN: 978-1-4577-1222-7
Rini T. Kaushik, Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA
Tarek Abdelzaher, Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA
Klara Nahrstedt, Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA
The sheer scale and rapid rise of Big Data mandates highly scalable, self-adaptive, and energy-conserving data-intensive compute clusters. Based on our analysis of the traces from a production Hadoop cluster at Yahoo!, we observe that file size, file lifespan, and file heat are statistically correlated and very strongly associated with the hierarchical directory structure (i.e., absolute file path) in which the files are organized. Leveraging that observation, we present predictive GreenHDFS; an energy-conserving variant of the Hadoop distributed file system that uses a supervised machine learning technique to learn the correlation between the directory hierarchy and the file attributes to guide novel predictive file zone placement, migration, and replication policies that significantly outperform the current state-of-the-art reactive approaches. Using real-world traces from a large-scale (2600 servers, 5 Petabytes) production Hadoop cluster at Yahoo! in our GreenHDFS simulations, we show how predictive GreenHDFS results in a much better trade-off between performance and energy consumption.
Index Terms:
energy consumption, predictive data, energy management, data-intensive compute clusters, Yahoo, file size, file lifespan, file heat, hierarchical directory structure, absolute file path, predictive GreenHDFS, Hadoop distributed file system, supervised machine learning technique, directory hierarchy, file attributes, predictive file zone placement, file migration, replication policy, large-scale production Hadoop cluster, GreenHDFS simulations
Citation:
Rini T. Kaushik, Tarek Abdelzaher, Ryota Egashira, Klara Nahrstedt, "Predictive data and energy management in GreenHDFS," igcc, pp.1-9, 2011 International Green Computing Conference and Workshops, 2011
Usage of this product signifies your acceptance of the Terms of Use.