This Article 
 Bibliographic References 
 Add to: 
27th International Conference on Distributed Computing Systems (ICDCS '07)
Defragmenting DHT-based Distributed File Systems
Toronto, Canada
June 25-June 27
ISBN: 0-7695-2837-3
Jeffrey Pang, Carnegie Mellon University
Phillip B. Gibbons, Intel Research Pittsburgh
Michael Kaminsky, Intel Research Pittsburgh
Srinivasan Seshan, Carnegie Mellon University
Haifeng Yu, National University of Singapore
Existing DHT-based file systems use consistent hashing to assign file blocks to random machines. As a result, a user task accessing an entire file or multiple files needs to retrieve blocks from many different machines. This paper demonstrates that significant availability and performance gains can be achieved if, instead, users are able to retrieve all the data needed for a given task from only a few DHT nodes. We explore the design and implications of such a "defragmented" DHT-based distributed file system, called D2, that also maintains important DHT properties like storage load balance. We show using real-world file system traces that a simple key encoding scheme is sufficient to maintain good defragmentation for most user tasks. Using both simulation and an actual 1,000 node deployment, we show that D2 increases availability by over an order of magnitude and improves user-perceived latency by 30" 100% compared to a traditional design.
Jeffrey Pang, Phillip B. Gibbons, Michael Kaminsky, Srinivasan Seshan, Haifeng Yu, "Defragmenting DHT-based Distributed File Systems," icdcs, pp.14, 27th International Conference on Distributed Computing Systems (ICDCS '07), 2007
Usage of this product signifies your acceptance of the Terms of Use.