loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC '09)
Adaptive and scalable metadata management to support a trillion files
Portland, Oregon
November 14-November 20
ISBN: 978-1-60558-744-8
Jing Xing, Chinese Academy of Sciences and Graduate University of Chinese Academy of Sciences
Jin Xiong, Chinese Academy of Sciences
Ninghui Sun, Chinese Academy of Sciences
Jie Ma, Chinese Academy of Sciences
Nowadays more and more applications require file systems to efficiently maintain million or more files. How to provide high access performance with such a huge number of files and such large directories is a big challenge for cluster file systems. Limited by static directory structures, existing file systems will be prohibitively inefficient for this use. To address this problem, we present a scalable and adaptive metadata management system which aims to maintain a trillion files efficiently. Firstly, our system exploits an adaptive two-level directory partitioning based on extendible hashing to manage very large directories. Secondly, our system utilizes fine-grained parallel processing within a directory and greatly improves performance of file creation or deletion. Thirdly, our system uses multiple-layered metadata cache management which improves memory utilization on the servers. And finally, our system uses a dynamic loadbalance mechanism based on consistent hashing which enables our system to scale up and down easily. Our performance results on 32 metadata servers show that our user-level prototype implementation can create more than 74 thousand files per second and can get more than 270 thousand files' attributes per second in a single directory with 100 million files. Moreover, it delivers a peak throughput of more than 60 thousand file creates/second in a single directory with 1 billion files.
Citation:
Jing Xing, Jin Xiong, Ninghui Sun, Jie Ma, "Adaptive and scalable metadata management to support a trillion files," sc, pp.1-11, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC '09), 2009
Usage of this product signifies your acceptance of the Terms of Use.