2017 IEEE 42nd Conference on Local Computer Networks (LCN) (2017)
Oct. 9, 2017 to Oct. 12, 2017
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/LCN.2017.64
Bulk data transfer is facing significant challenges in the coming era of big data. There are multiple performance bottlenecks along the end-to-end path from the source to destination storage system. The limitations of current generation data transfer tools themselves can have a significant impact on end-to-end data transfer rates. In this paper, we identify the issues that lead to underperformance of these tools, and present a new data transfer tool with an innovative I/O scheduler called MDTM. The MDTM scheduler exploits underlying multicore layouts to optimize throughput by reducing delay and contention for I/O reading and writing operations. With our evaluations, we show how MDTM successfully avoids NUMA-based congestion and significantly improves end-to-end data transfer rates across high-speed wide area networks.
Big Data, multiprocessing systems, optimisation, parallel processing, processor scheduling, telecommunication congestion control, wide area networks
L. Zhang, P. Demar, B. Kim and W. Wu, "MDTM: Optimizing Data Transfer Using Multicore-Aware I/O Scheduling," 2017 IEEE 42nd Conference on Local Computer Networks (LCN), Singapore, Singapore, 2018, pp. 104-111.