June 25, 2007 to June 27, 2007
Phillip B. Gibbons , Intel Research Pittsburgh
Jeffrey Pang , Carnegie Mellon University
Srinivasan Seshan , Carnegie Mellon University
Haifeng Yu , National University of Singapore
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ICDCS.2007.97
Existing DHT-based file systems use consistent hashing to assign file blocks to random machines. As a result, a user task accessing an entire file or multiple files needs to retrieve blocks from many different machines. This paper demonstrates that significant availability and performance gains can be achieved if, instead, users are able to retrieve all the data needed for a given task from only a few DHT nodes. We explore the design and implications of such a "defragmented" DHT-based distributed file system, called D2, that also maintains important DHT properties like storage load balance. We show using real-world file system traces that a simple key encoding scheme is sufficient to maintain good defragmentation for most user tasks. Using both simulation and an actual 1,000 node deployment, we show that D2 increases availability by over an order of magnitude and improves user-perceived latency by 30" 100% compared to a traditional design.
Phillip B. Gibbons, Jeffrey Pang, Srinivasan Seshan, Haifeng Yu, "Defragmenting DHT-based Distributed File Systems", ICDCS, 2007, 27th International Conference on Distributed Computing Systems (ICDCS '07), 27th International Conference on Distributed Computing Systems (ICDCS '07) 2007, pp. 14, doi:10.1109/ICDCS.2007.97