CORE: Augmenting regenerating-coding-based recovery for single and concurrent failures in distributed storage systems
Mass Storage Systems and Technologies, IEEE / NASA Goddard Conference on (2013)
Long Beach, CA, USA USA
May 6, 2013 to May 10, 2013
Runhui Li , Department of Computer Science and Engineering, The Chinese University of Hong Kong
Jian Lin , Department of Computer Science and Engineering, The Chinese University of Hong Kong
Patrick P. C. Lee , Department of Computer Science and Engineering, The Chinese University of Hong Kong
Data availability is critical in distributed storage systems, especially when node failures are prevalent in real life. A key requirement is to minimize the amount of data transferred among nodes when recovering the lost or unavailable data of failed nodes. This paper explores recovery solutions based on regenerating codes, which are shown to provide fault-tolerant storage and minimum recovery bandwidth. Existing optimal regenerating codes are designed for single node failures. We build a system called CORE, which augments existing optimal regenerating codes to support a general number of failures including single and concurrent failures. We theoretically show that CORE achieves the minimum possible recovery bandwidth for most cases. We implement CORE and evaluate our prototype atop a Hadoop HDFS cluster testbed with up to 20 storage nodes. We demonstrate that our CORE prototype conforms to our theoretical findings and achieves recovery bandwidth saving when compared to the conventional recovery approach based on erasure codes.
Bandwidth, Throughput, Nickel, Equations, Strips, Encoding, Availability, experiments and implementation, regenerating codes, failure recovery, distributed storage systems, coding theory
R. Li, J. Lin and P. P. Lee, "CORE: Augmenting regenerating-coding-based recovery for single and concurrent failures in distributed storage systems," 2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST 2013)(MSST), Long Beach, CA, 2013, pp. 1-6.