18th International Parallel and Distributed Processing Symposium (IPDPS'04) - Papers
High Performance Implementation of MPI Derived Datatype Communication over InfiniBand
Santa Fe, New Mexico
April 26-April 30
ISBN: 0-7695-2132-0
In this paper, a systematic study of two main types of approach for MPI datatype communication (Pack/Unpack-based approaches and Copy-Reduced approaches) is carried out on the InfiniBand network. We focus on overlapping packing, network communication, and unpacking in the Pack/Unpack-based approaches. We use RDMA operations to avoid packing and/or unpacking in the Copy-Reduced approaches. Four schemes (Buffer-Centric Segment Pack/Unpack, RDMA Write Gather With Unpack, Pack with RDMA Read Scatter, and Multiple RDMA Writes have been proposed. Three of them have been implemented and evaluated based on one MPI implementation over InfiniBand. Performance results of a vector micro-benchmark demonstrate that latency is improved by a factor of up to 3.4 and bandwidth by a factor of up to 3.6 compared to the current datatype communication implementation. Collective operations like MPI_Alltoall are demonstrated to benefit. A factor of up to 2.0 improvement has been seen in our measurements of those collective operations on an 8-node system.
Citation:
Jiesheng Wu, Pete Wyckoff, Dhabaleswar Panda, "High Performance Implementation of MPI Derived Datatype Communication over InfiniBand," ipdps, vol. 1, pp.14a, 18th International Parallel and Distributed Processing Symposium (IPDPS'04) - Papers, 2004