This Article 
 Bibliographic References 
 Add to: 
A Holistic Approach to Designing Energy-Efficient Cluster Interconnects
June 2005 (vol. 54 no. 6)
pp. 660-671
Designing energy-efficient clusters has recently become an important concern to make these systems economically attractive for many applications. Since the cluster interconnect is a major part of the system, the focus of this paper is to characterize and optimize the energy consumption in the entire interconnect. Using a cycle-accurate simulator of an InfiniBand Architecture (IBA) compliant interconnect fabric and actual designs of its components, we investigate the energy behavior on regular and irregular interconnects. The energy profile of the three major components (switches, network interface cards (NICs), and links) reveals that the links and switch buffers consume the major portion of the power budget. Hence, we focus on energy optimization of these two components. To minimize power in the links, first we investigate the dynamic voltage scaling (DVS) algorithm and then propose a novel dynamic link shutdown (DLS) technique. The DLS technique makes use of an appropriate adaptive routing algorithm to shut down the links intelligently. We also present an optimized buffer design for reducing leakage energy in 70nm technology. Our analysis on different networks reveals that, while DVS is an effective energy conservation technique, it incurs significant performance penalty at low to medium workload. Moreover, energy saving with DVS reduces as the buffer leakage current becomes significant with 70nm design. On the other hand, the proposed DLS technique can provide optimized performance-energy behavior (up to 40 percent energy savings with less than 5 percent performance degradation in the best case) for the cluster interconnects.

[1] The New York Times, “There's Money in Housing Internet Servers,” Apr. 2001, .
[2] R. Bryce, “Power Struggle,” Dec. 2000, au/newstech/enterprise/ story0,2000025001,20107749,00.htm.
[3] Mellanox Technologies Inc., “Mellanox Performance, Price, Power, Volumn Metric (PPPV),” , 2004.
[4] Digital Equipment Corp., Alpha Architecture Technical Summary, 1992.
[5] W. Dally, P. Carvey, and L. Dennison, “The Avici Terabit Switch/Router,” Proc. Hot Interconnects 6, pp. 41-50, Aug. 1998.
[6] T.T. Ye, L. Benini, and G.D. Micheli, “Analysis of Power Consumption on Switch Fabrics in Network Routers,” Proc. 39th Conf. Design Automation (DAC), pp. 524-529, June 2002.
[7] G.-Y. Wei, M. Horowitz, and A. Kim, “Energy-Efficient Design of High-Speed Links,” Power Aware Design Methodologies, M. Pedram and J. Rabaey, eds., chapter 8, Kluwer Academic, 2002.
[8] G.-Y. Wei, J. Kim, D. Liu, S. Sidiropoulos, and M. Horowitz, “A Variable-Frequency Parallel I/O Interface with Adaptive Power-Supply Regulation,” IEEE J. Solid-State Circuits, vol. 35, pp. 1600-1610, Nov. 2000.
[9] C.S. Patel, S.M. Chai, S. Yalamanchili, and D.E. Schimmel, “Power Constrained Design of Multiprocessor Interconnection Networks,” Proc. Int'l Conf. Computer Design (ICCD), pp. 408-416, Oct. 1997.
[10] A.G. Wassal and M.A. Hasan, “Low-Power System-Level Design of VLSI Packet Switching Fabrics,” IEEE Trans. Computer-Aided Design of Integrated Circuits and Systems, vol. 20, pp. 723-738, June 2001.
[11] H.-S. Wang, L.-S. Peh, and S. Malik, “A Power Model for Routers: Modeling Alpha 21364 and InfiniBand Routers,” Proc. Hot Interconnects 10, pp. 21-27, Aug. 2002.
[12] H.-S. Wang, X. Zhu, L.-S. Peh, and S. Malik, “Orion: A Power-Performance Simulator for Interconnection Networks,” Proc. 35th Int'l Symp. Microarchitecture (MICRO), pp. 294-305, Nov. 2002.
[13] L. Shang, L.-S. Peh, and N.K. Jha, “Dynamic Voltage Scaling with Links for Power Optimization of Interconnection Networks,” Proc. IEEE Int'l Symp. High-Performance Computer Architecture (HPCA), pp. 91-102, Feb. 2003.
[14] V. Soteriou and L.-S. Peh, “Dynamic Power Management for Power Optimization of Interconnection Networks Using On/Off Links,” Proc. 11th Symp. High Performance Interconnects (Hot Interconnects), Aug. 2003.
[15] InfiniBand Trade Assoc., “InfiniBand Architecture Specification, Volume 1, Release 1.0,” Oct. 2000, http:/
[16] E.J. Kim, K.H. Yum, C.R. Das, M. Yousif, and J. Duato, “Performance Enhancement Techniques for InfiniBand™ Architecture,” Proc. IEEE Int'l Symp. High-Performance Computer Architecture (HPCA), pp. 253-262. Feb. 2003.
[17] J. Duato, S. Yalamanchili, and L. Ni, Interconnection Networks: An Engineering Approach. San Francisco: Morgan Kaufmann, July 2002.
[18] W.J. Dally, “Virtual-Channel Flow Control,” IEEE Trans. Parallel and Distributed Systems, vol. 3, pp. 194-205, May 1992.
[19] M.B. Kamble and K. Ghose, “Analytical Energy Dissipation Models for Low-Power Caches,” Proc. 1997 Int'l Symp. Low Power Electronics and Design (ISLPED), pp. 143-148, Aug. 1997.
[20] M.D. Powell, S. Yang, B. Falsafi, K. Roy, and T.N. Vijaykumar, “Reducing Leakage in a High-Performance Deep-Submicron Instruction Cache,” IEEE Trans. Very Large Scale Integration (VLSI) Systems, vol. 9, pp. 77-90, Feb. 2001.
[21] E. Geethanjali, V. Narayanan, and M.J. Irwin, “An Analytical Power Estimation Model for Crossbar Interconnects,” Proc. IEEE Int'l ASIC/SOC Conf., pp. 119-123, Sept. 2002.
[22] T.G. Tip, “RDRAM Power Estimation and Thermal Considerations,” Oct. 2001, .
[23] T. Simunic, L. Benini, and G.D. Micheli, “Cycle-Accurate Simulation of Energy Consumption in Embedded Systems,” Proc. 36th Conf. Design Automation (DAC), pp. 867-872, June 1999.
[24] J. Kim and M. Horowitz, “Adaptive Supply Serial Links with Sub-1V Operation and Per-Pin Clock Recovery,” Proc. Int'l Solid-State Circuits Conf. (ISSCC), pp. 268-269, Feb. 2002.
[25] J. Kim and M. Horowitz, “An Efficient Digital Sliding Controller for Adaptive Power Supply Regulation,” IEEE J. Solid-State Circuits, vol. 37, pp. 639-647, May 2002.
[26] S. Borkar, “Design Challenges of Technology Scaling,” IEEE Micro, vol. 19, pp. 23-29, July-Aug. 1999.
[27] D. Duarte, Y.-F. Tsai, N. Vijaykrishnan, and M.J. Irwin, “Evaluating Run-Time Techniques for Leakage Power Reduction,” Proc. ASP-DAC/VLSI Design, pp. 31-38, Jan. 2002.
[28] J. Qiu and E. Knightly, “QoS Control via Robust Envelop-Based MBAC,” Proc. Sixth Int'l Workshop Quality of Service (IWQoS '98), pp. 62-64, May 1998.
[29] M.B. Caminero, F.J. Quiles, J. Duato, D.S. Love, and S. Yalamanchili, “Performance Evaluation of the Multimedia Router with MPEG-2 Video Traffic,” Proc. Third Int'l Workshop Comm., Architecture, and Applications on Network Based Parallel Computing (CANPC '99), pp. 62-76, Jan. 1999.

Index Terms:
Buffer design, cluster interconnect, dynamic voltage scaling, dynamic link shutdown, energy optimization, link design, switch design.
Eun Jung Kim, Greg M. Link, Ki Hwan Yum, N. Vijaykrishnan, Mahmut Kandemir, Mary J. Irwin, Chita R. Das, "A Holistic Approach to Designing Energy-Efficient Cluster Interconnects," IEEE Transactions on Computers, vol. 54, no. 6, pp. 660-671, June 2005, doi:10.1109/TC.2005.86
Usage of this product signifies your acceptance of the Terms of Use.