International Parallel and Distributed Processing Symposium (IPDPS'03)
A Performance Analysis of 4X InfiniBand Data Transfer Operations
Nice, France
April 22-April 26
ISBN: 0-7695-1926-1
The performance of 4X InfiniBand Send/Receive and RDMA operations is studied by running tests to measure latency, data rate, number of operations per second, and CPU load. The measurements performed are for application-to-application data transfers using user-level InfiniBand (IB) verbs. It is shown that IB is capable of low latencies (10µs for small messages) and very high data rates at low CPU loads (over 6Gbs with 64KB messages at under 20% CPU load). A very large number of operations per second (over 400,000) is obtained for small messages. Some comparisons are made with the performance of TCP/IP on Gigabit Ethernet. In addition, the paper studies the impact of varying the number of outstanding requests on the obtained throughput, and shows when the peak throughput can be obtained for messages of varying sizes. Finally, an approach for handling completions in user space without a busy wait and without the use of signals is introduced and CPU load results based on this approach are presented.
Index Terms:
InfiniBand, RDMA, Gigabit Ethernet, TCP, performance
Citation:
Ariel Cohen, "A Performance Analysis of 4X InfiniBand Data Transfer Operations," ipdps, pp.202b, International Parallel and Distributed Processing Symposium (IPDPS'03), 2003