The Community for Technology Leaders
SC Conference (2012)
Salt Lake City, UT, USA USA
Nov. 10, 2012 to Nov. 16, 2012
ISSN: 2167-4329
ISBN: 978-1-4673-0805-2
pp: 1-12
ABSTRACT
Hadoop Distributed File System (HDFS) acts as the primary storage of Hadoop and has been adopted by reputed organizations (Facebook, Yahoo! etc.) due to its portability and fault-tolerance. The existing implementation of HDFS uses Javasocket interface for communication which delivers suboptimal performance in terms of latency and throughput. For dataintensive applications, network performance becomes key component as the amount of data being stored and replicated to HDFS increases. In this paper, we present a novel design of HDFS using Remote Direct Memory Access (RDMA) over InfiniBand via JNI interfaces. Experimental results show that, for 5GB HDFS file writes, the new design reduces the communication time by 87% and 30% over 1Gigabit Ethernet (1GigE) and IP-over-InfiniBand (IPoIB), respectively, on QDR platform (32Gbps). For HBase, the Put operation performance is improved by 26% with our design. To the best of our knowledge, this is the first design of HDFS over InfiniBand networks.
INDEX TERMS
Sockets, Java, IP networks, Libraries, Switches, Software, Servers
CITATION

N. S. Islam et al., "High performance RDMA-based design of HDFS over InfiniBand," 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis(SC), Salt Lake City, UT, 2012, pp. 1-12.
doi:10.1109/SC.2012.65
94 ms
(Ver 3.3 (11022016))