High Performance Computing, Networking Storage and Analysis, SC Companion: (2012)
Salt Lake City, Utah, USA
June 24, 2012 to June 29, 2012
HDF5 is a data model, library and file format for storing and managing data. It is designed for flexible and efficient I/O for high volume and complex data. Natively, it uses a single-file format where multiple HDF5 objects are stored in a single file. In a parallel HDF5 application, multiple processes access a single file, thereby resulting in a performance bottleneck in I/O. Additionally, a single-file format does not allow semantic post processing on individual objects outside the scope of the HDF5 application. We have developed a new plugin for HDF5 using its Virtual Object Layer that serves two purposes: 1) it uses PLFS to convert the single-file layout into a data layout that is optimized for the underlying file system, and 2) it stores data in a unique way that enables semantic post-processing on data. We measure the performance of the plugin and discuss work leveraging the new semantic post-processing functionality enabled. We further discuss the applicability of this approach for exascale burst buffer storage systems.
Semantic Analysis, HDF5, PLFS, Parallel I/O
Gary Grider, Edgar Gabriel, Aaron Torres, John Bent, Kshitij Mehta, "A Plugin for HDF5 Using PLFS for Improved I/O Performance and Semantic Analysis", High Performance Computing, Networking Storage and Analysis, SC Companion:, vol. 00, no. , pp. 746-752, 2012, doi:10.1109/SC.Companion.2012.102