loading...
 This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing
Improving Collective I/O Performance Using Threads
San Juan, Puerto Rico
April 12-April 16
ISBN: 0-7695-0143-5
Phillip M. Dickens, Illinois Institute of Technology
Rajeev Thakur, Argonne National Laboratory
Massively parallel computers are increasingly being used to solve large, I/O intensive applications in many different fields. For such applications, the I/O requirements quite often present a significant obstacle in the way of achieving good performance, and an important area of current research is the development of techniques by which these costs can be reduced. One such approach is collective I/O, where the processors cooperatively develop an I/O strategy that reduces the number, and increases the size, of I/O requests, making a much better use of the I/O subsystem. Collective I/O has been shown to significantly reduce the cost of performing I/O in many large, parallel applications, and for this reason serves as an important base upon which we can explore other mechanisms which can further reduce these costs. One promising approach is to use threads to perform the collective I/O in the background while the main thread continues with other computation in the foreground. In this paper, we explore the issues associated with implementing collective I/O in the background using threads.The most natural approach is to simply spawn off an I/O thread to perform the collective I/O in the background while the main thread continues with other computation. However, our research demonstrates that this approach is frequently the worst implementation option, often performing much more poorly than just executing collective I/O completely in the foreground. To improve the performance of thread-based collective I/O, we developed an alternate approach where part of the collective I/O operation is performed in the background, and part is performed in the foreground. We demonstrate that this new technique can significantly improve the performance of thread-based collective I/O, providing up to an 80% improvement over sequential collective I/O (where there is no attempt to overlap computation with I/O). Also, we discuss one very important application of this research which is the implementation of the split-collective parallel I/O operations defined in MPI 2.0.
Citation:
Phillip M. Dickens, Rajeev Thakur, "Improving Collective I/O Performance Using Threads," ipps, pp.38, 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing, 1999
Usage of this product signifies your acceptance of the Terms of Use.