This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Martini: A Network Interface Controller Chip for High Performance Computing with Distributed PCs
September 2007 (vol. 18 no. 9)
pp. 1282-1295
In this paper, “Martini,” a network interface controller chip for our original network called RHiNET is described. Martini is designed to provide high-bandwidth and low-latency communication with small overhead. To obtain high performance communication, protected user-level zero-copy RDMA communication functions are completely implemented by a hardwired logic. Also, to reduce the communication latency efficiently, we have proposed PIO-based communication mechanisms called “On-the-fly (OTF)” and have implemented them on Martini. The evaluation results show that Martini connected to a 64bit/66MHz PCI-bus achieves 470MByte/s maximum bidirectional bandwidth and 1.74 μsec minimum latency on host-to-host memory copying.

[1] I. Foster, C. Kesselman, and S. Tuecke, “The Anatomy of the Grid: Enabling Scalable Virtual Organizations,” Proc. First IEEE/ACM Int'l Symp. Cluster Computing and the Grid, pp. 6-7, 2001.
[2] D.P. Anderson, J. Cobb, E. Korpela, M. Lebofsky, and D. Werthimer, “SETI@home: An Experiment in Public-Resource Computing,” Comm. ACM, vol. 45, no. 11, pp. 56-61, 2002.
[3] N.J. Boden, D. Cohen, R.E. Felderman, A.E. Kulawik, C.L. Seitz, J.N. Seizovic, and W. Su, “Myrinet—A Gigabit per Second Local Area Network,” IEEE Micro, vol. 15, no. 1, pp. 29-36, 1995.
[4] T.L. Sterling, D. Savarese, D.J. Becker, J.E. Dorband, U.A. Ranawake, and C.V. Packer, “BEOWULF: A Parallel Workstation for Scientific Computation,” Proc. Int'l Conf. Parallel Processing (ICPP '95), pp. 11-14, Aug. 1995.
[5] T. Takahashi, S. Sumimoto, A. Hori, H. Harada, and Y. Ishikawa, “PM2: High Performance Communication Middleware for Heterogeneous Network Environment,” Proc. Supercomputing Conf., pp. 52-53, Nov. 2000.
[6] G. Ciaccio, “Messaging on Gigabit Ethernet: Some Experiments with GAMMA and Other Systems,” Proc. IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS '01), pp. 1624-1631, Apr. 2001.
[7] R.P. Martin, A.M. Vahdat, D.E. Culler, and T.E. Anderson, “Effects of Communication Latency, Overhead, and Bandwidth in a Cluster Architecture,” Proc. 24th Ann. Int'l Symp. Computer Architecture (ISCA '97), pp. 85-97, June 1997.
[8] Myricom Inc., Myrinet Performance, http://www.myri.com/ myrinetperformance /, 2005.
[9] W. Feng, J.G. Hurwitz, H. Newman, S. Ravot, R. Cottrell, O. Martin, F. Coccetti, C. Jin, X.D. Wei, and S. Low, “Optimizing 10-Gigabit Ethernet for Networks of Workstations, Clusters and Grids: A Case Study,” Proc. Supercomputing Conf., Nov. 2003.
[10] RDMA Consortium, “Architectural Specifications for RDMA over TCP/IP,” http:/www.rdmaconsortium.org, 2005.
[11] W. Feng, P. Balaji, C. Baron, L.N. Bhuyan, and D.K. Panda, “Performance Characterization of a 10-Gigabit Ethernet TOE,” Proc. 13th Symp. High Performance Interconnects (HOTI '05), pp. 58-63, Aug. 2005.
[12] T. Kudoh, S. Nishimura, J. Yamamoto, H. Nishi, O. Tatebe, and H. Amano, “RHiNET: A Network for High Performance Parallel Processing Using Locally Distributed Computers,” Proc. Int'l Workshop Innovative Architecture (IWIA '99), pp. 69-73, Nov. 1999.
[13] N. Tanabe, A. Kitamura, T. Miyashiro, Y. Miyabe, T. Izawa, Y. Hamada, H. Nakajo, and H. Amano, “Preliminary Evaluation of a FPGA-Based-Prototype of DIMMnet-2 Network Interface,” Proc. Int'l Workshop Innovative Architecture on Future Generation High-Performance Processors and Systems (IWIA '05), pp. 119-127, Jan. 2005.
[14] D. Pham, S. Asano, M. Bolliger, M.N. Day, H.P. Hofstee, C. Johns, J. Kahle, A. Kameyama, J. Keaty, Y. Masubuchi, M. Riley, D. Shippy, D. Stasiak, M. Suzuoki, M. Wang, J. Warnock, S. Weitzel, D. Wendel, T. Yamazaki, and K. Yazawa, “The Design and Implementation of a First-Generation CELL Processor,” Proc. IEEE Int'l Solid-State Circuits Conf. Digest of Technical Papers, pp. 184-185, Feb. 2005.
[15] K. Watanabe, T. Otsuka, J. Tsuchiya, T. Kudoh, and H. Amano, “Performance Evaluation of RHiNET-2/NI: A Network Interface for Distributed Parallel Computing Systems,” Proc. Third IEEE/ACM Int'l Symp. Cluster Computing and the Grid (CCGrid '03), pp.318-325, May 2003.
[16] M. Koibuchi, K. Watanabe, T. Otsuka, and H. Amano, “Performance Evaluation of Deterministic Routings, Multicasts, and Topologies on RHiNET-2 Cluster,” IEEE Trans. Parallel and Distributed Systems, vol. 16, no. 8, pp. 747-759, Aug. 2005.
[17] K. Watanabe, J. Yamamoto, J. Tsuchiya, N. Tanabe, H. Nishi, T. Kudoh, and H. Amano, “Preliminary Evaluation of Martini: A Novel Network Interface Controller Chip for Cluster-Based Parallel Processing,” Proc. IASTED Int'l Conf. Applied Informatics, pp. 390-395, Feb. 2002.
[18] M.P. Merlin and J.P. Schweitzer, “Deadlock Avoidance in Store-and-Forward Networks,” IEEE Trans. Comm., vol. 28, no. 3, pp.345-354, Mar. 1980.
[19] S. Nishimura, T. Kudoh, H. Nishi, J. Yamamoto, K. Harasawa, N. Matsudaira, S. Akutsu, K. Tasho, and H. Amano, “High-Speed Network Switch RHiNET-2/SW and Its Implementation with Optical Interconnections,” Technical Digest of Hot Interconnects, vol. 8, pp. 31-38, Aug. 2000.
[20] S. Nishimura, T. Kudoh, H. Nishi, J. Yamamoto, K. Harasawa, N. Matsudaira, S. Akutsu, and H. Amano, “64-Gbit/s Highly Reliable Network Switch (RHiNET-2/SW) Using Parallel Optical Interconnection,” IEEE J. Lightwave Technology, special issue on optical networks, vol. 18, no. 12, pp. 1620-1627, Dec. 2000.
[21] S. Nishimura, K. Harasawa, N. Matsudaira, S. Akutsu, T. Kudoh, H. Nishi, and H. Amano, “RHiNET-2/SW: A Large-Throughput, Compact Network-Switch Using 8.8-Gbit/s Optical Interconnection,” New Generation Computing, vol. 18, no. 2, pp. 188-197, Jan. 2000.
[22] S. Nishimura, T. Kudoh, H. Nishi, J. Yamamoto, K. Harasawa, N. Matsudaira, S. Akutsu, K. Tasho, and H. Amano, “RHiNET-3/SW: An 80-Gbit/s High-Speed Network Switch for Distributed Parallel Computing,” Hot Interconnects, vol. 9, pp. 119-123, Aug. 2001.
[23] H. Nishi, J. Yamamoto, K. Ohsugi, K. Harasawa, and S. Nishimura, “Deskew-LSI for 10-Gbit/s Parallel Optical Links in RHiNET-3 System,” Proc. COOL Chips V: An Int'l Symp. Low-Power and High-Speed Chips, vol. 1 , pp. 37-46, Apr. 2002.
[24] T. Yokoyama, N. Izu, J. Tsuchiya, K. Watanabe, H. Amano, and T. Kudoh, “Design and Implementation of RHiNET-2/NI0: A Reconfigurable Network Interface for Cluster Computing,” IEICE Trans. Information and Systems, no. 5, pp. 789-795, 2003.
[25] T. von Eicken, D.E. Culler, S.C. Goldstein, and K.E. Schauser, “Active Messages: A Mechanism for Integrated Communication and Computation,” Proc. 19th Int'l Symp. Computer Architecture, pp. 256-266, May 1992.
[26] T. von Eicken, A. Basu, V. Buch, and W. Vogels, “U-Net: A User-Level Network Interface for Parallel and Distributed Computing,” Proc. 15th ACM Symp. Operating Systems Principles (SOSP' 95), pp.40-53, Dec. 1995.
[27] S. Pakin, M. Lauria, and A. Chien, “High Performance Messaging on Workstations: Illinois Fast Messages (FM) for Myrinet,” Proc. Conf. Supercomputing, Dec. 1995.
[28] J.M. Smith and C.B.S. Traw, “Giving Applications Access to Gb/s Networking,” IEEE Network, vol. 7, no. 4, pp. 44-52, July 1993.
[29] D. Anderson, J. Chase, S. Gadde, A. Gallatin, K. Yocum, and M. Feeley, “Cheating the I/O Bottleneck: Network Storage with Trapeze/Myrinet,” Proc. Usenix Ann. Technical Conf., pp. 143-154, June 1998.
[30] N. Tanabe, J. Yamamoto, H. Nishi, T. Kudoh, Y. Hamada, H. Nakajo, and H. Amano, “On-the-Fly Sending: A Low Latency High Bandwidth Message Transfer Mechanism,” Proc. Int'l Symp. Parallel Architectures, Algorithms and Networks (ISPAN '00), pp. 186-193, Dec. 2000.
[31] H. Tezuka, F. O'Carroll, A. Hori, and Y. Ishikawa, “Pin-Down Cache: A Virtual Memory Management Technique for Zero-Copy Communication,” Proc. 12th Int'l Parallel Processing Symp./Ninth Symp. Parallel and Distributed Processing (IPPS/SPDP '98), pp. 308-314, Mar.-Apr. 1998.
[32] PCI-SIG, http:/www.pcisig.com/, 2005.
[33] N. Tanabe, J. Yamamoto, H. Nishi, T. Kudoh, Y. Hamada, H. Nakajo, and H. Amano, “MEMOnet: Network Interface Plugged into a Memory Slot,” Proc. IEEE Int'l Conf. Cluster Computing (Cluster '00), pp. 17-26, Nov. 2000.
[34] K. Watanabe, H. Amano, J. Yamamoto, J. Tsuchiya, and T. Kudoh, “Taking over Mechanism: A Cooperation Methodology of Hardware and Software in Network Controllers,” Proc. Int'l Workshop Synthesis and System Integration of Mixed Information Technologies (SASIMI '03), pp. 386-393, Apr. 2003.
[35] T. Yoshikawa, I. Hatakeyama, K. Miyoshi, and K. Kurata, “Optical Interconnection as an Intellectual Property of a CMOS Library,” Hot Interconnects, vol. 9, pp. 31-35, Aug. 2001.
[36] Hitachi, “Micro Device Division,” http://www.hitachi.co.jp/Div/mdd/english index.html, 2005.
[37] TOP500.org, “Top 500 Supercomputer Sites,” http:/www. top500.org, 2005.
[38] F. Petrini, W. Feng, A. Hoisie, S. Coll, and E. Frachtenberg, “The Quadrics Network: High Performance Clustering Technology,” IEEE Micro, vol. 22, no. 1, pp. 46-57, Jan.-Feb. 2002.
[39] F. Petrini, W. Feng, A. Hoisie, S. Coll, and E. Frachtenberg, “The Quadrics Network (QsNet): High-Performance Clustering Technology,” Hot Interconnects, pp. 125-130, Aug. 2001.
[40] InfiniBand Trade Association, InfiniBand Architecture Specification, Release 1.0, Oct. 2000.
[41] Myricom Inc., http:/www.myri.com/, 2005.
[42] Myricom Inc., “Myricom Custom-VLSI Chips,” http://www. myri.comvlsi/, 2005.
[43] F. Petrini, E. Frachtenberg, A. Hoisie, and S. Coll, “Performance Evaluation of the Quadrics Interconnection Network,” J. Cluster Computing, vol. 6, no. 2, pp. 125-142, Apr. 2003.
[44] Quadrics Inc., http:/www.quadrics.com/, 2005.
[45] J. Beecroft, D. Addison, D. Hewson, M. McLaren, D. Roweth, F. Petrini, and J. Nieplocha, “QsNetII: Defining High-Performance Network Design,” IEEE Micro, vol. 25, no. 4, pp. 34-47, July-Aug. 2005.
[46] Mellanox Technologies Inc., http:/www.mellanox.com/, 2005.
[47] Chelsio Communications, http:/www.chelsio.com/, 2005.
[48] J. Liu, A. Mamidala, A. Vishnu, and D.K. Panda, “Evaluating InfiniBand Performance with PCI Express,” IEEE Micro, vol. 25, no. 1, pp. 20-29, Jan.-Feb. 2005.
[49] T. Otsuka, K. Watanabe, J. Tsuchiya, H. Harada, J. Yamamoto, H. Nishi, T. Kudoh, and H. Amano, “Performance Evaluation of a Prototype of RHiNET-2: A Network-Based Distributed Parallel Computing System,” Proc. IASTED Int'l Multi-Conf. Applied Informatics (AI '03), pp. 738-743, Feb. 2003.
[50] P. Balaji, W. Feng, and D.K. Panda, “Bridging the Ethernet-Ethernot Performance Gap,” IEEE Micro, vol. 26, no. 3, pp. 24-40, May-Jun. 2006.

Index Terms:
Network Interface Controller, System Area Network, RHiNET, PC Clusters
Citation:
Konosuke Watanabe, Tomohiro Otsuka, Junichiro Tsuchiya, Hiroaki Nishi, Junji Yamamoto, Noboru Tanabe, Tomohiro Kudoh, Hideharu Amano, "Martini: A Network Interface Controller Chip for High Performance Computing with Distributed PCs," IEEE Transactions on Parallel and Distributed Systems, vol. 18, no. 9, pp. 1282-1295, Sept. 2007, doi:10.1109/TPDS.2007.1064
Usage of this product signifies your acceptance of the Terms of Use.