Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload
2013 IEEE International Conference on Cluster Computing (CLUSTER) (2013)
Indianapolis, IN, USA
Sept. 23, 2013 to Sept. 27, 2013
Seung Woo Son , Northwestern University, USA
Saba Sehrish , Fermi National Accelerator Laboratory, USA
Wei-keng Liao , Northwestern University, USA
Ron Oldfield , Sandia National Laboratories, USA
Alok Choudhary , Northwestern University, USA
As the number of compute cores on modern parallel machines increases to more than hundreds of thousands, scalable and consistent I/O performance is becoming hard to obtain due to fluctuating file system performance. This fluctuation is often caused by rebuilding RAID disk from hardware failures or concurrent jobs competing for I/O. We present a mechanism that stripes across a dynamically-selected subset of I/O servers with the lightest workload to achieve the best I/O bandwidth available from the system. We implement this mechanism into an I/O software layer that enables memory-to-file data layout transformation and allows transparent file partitioning. File partitioning is a technique that divides data among a set of files and manages file access, making data appear as a single file to users. Experimental results on NERSC's Hopper indicate that our approach effectively isolates I/O variation on shared systems and improves overall I/O performance significantly.
File partitioning, Collective I/O, Parallel NetCDF
S. W. Son, S. Sehrish, W. Liao, R. Oldfield and A. Choudhary, "Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload," 2013 IEEE International Conference on Cluster Computing (CLUSTER), Indianapolis, IN, USA USA, 2014, pp. 1-8.