This Article 
 Bibliographic References 
 Add to: 
On the Use of Virtual Channels in Networks of Workstations with Irregular Topology
August 2000 (vol. 11 no. 8)
pp. 813-828

Abstract—Networks of workstations are becoming increasingly popular as a cost-effective alternative to parallel computers. Typically, these networks connect workstations using irregular topologies, providing the wiring flexibility, scalability, and incremental expansion capability required in this environment. Recently, we proposed two methodologies for the design of adaptive routing algorithms for networks with irregular topology, as well as fully adaptive routing algorithms for these networks. These algorithms increase throughput considerably with respect to previously existing ones, but require the use of at least two virtual channels. In this paper, we propose a very efficient flow control protocol to support virtual channels when link wires are very long and/or have different lengths. This flow control protocol relies on the use of channel pipelining and control flits. Control traffic is minimized by assigning physical bandwidth to virtual channels until the corresponding message blocks or it is completely transmitted. Simulation results show that this flow control protocol performs as efficiently as an ideal network with short wires and flit-by-flit multiplexing. The effect of additional virtual channels per physical channel has also been studied, revealing that the optimal number of virtual channels varies with network size. The use of virtual channel priorities is also analyzed. The proposed flow control protocol may increase short message latency, due to long messages monopolizing channels and hindering the progress of short messages. Therefore, we have analyzed the impact of limiting the number of flits (block size) that a virtual channel may forward once it gets the link. Simulation results show that limiting the maximum block size causes the overall network performance to decrease.

[1] K. Aoyama and A.A. Chien, “The Cost of Adaptivity and Virtual Lanes in a Wormhole Router,” technical report, Dept. of Electrical Eng., Univ. of Illinois at Urbana-Champaign, 1994.
[2] N. Boden et al., "Myrinet: A Gigabit-per-Second Local Area Network," IEEE Micro, Feb. 1995, pp. 29-36.
[3] R. Boppana and S. Chalasani, "A Comparison of Adaptive Wormhole Routing Algorithms," Proc. 20th Ann. Int'l Symp. Computer Architecture," pp. 351-360, 1993.
[4] J. Carbonaro and F. Verhoorn, “Cavallino: The Teraflops Router and NIC,” Proc. Hot Interconnects Symp. IV, Aug. 1996.
[5] A.A. Chien, “A Cost and Speed Model for k-Ary n-Cube Wormhole Routers,” Proc. Hot Interconnects '93, Aug. 1993.
[6] W.J. Dally and C.L. Seitz, “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks,” IEEE Trans. Computers, Vol. C-36, No. 5, May 1987, pp. 547-553.
[7] W.J. Dally, "Virtual-Channel Flow Control," IEEE Trans. Parallel and Distributed Systems, vol. 3, no. 2, pp. 194-205, Mar. 1992.
[8] W.J. Dally, L.R. Dennison, D. Harris, K. Kan, and T. Xanthopoulus, “The Reliable Router: A Reliable and High-Performance Communication Substrate for Parallel Computers,” Proc. Workshop Parallel Computer Routing and Comm., pp. 241–255, May 1994.
[9] W.J. Dally, L.R. Dennison, D. Harris, K. Kan, and T. Xanthopoulos, “Architecture and implementation of the Reliable Router,” Proc. Hot Interconnects II, Aug. 1994.
[10] B.V. Dao, S. Yalamanchili, and J. Duato, “Architectural Support for Reducing Communication Overhead in Pipelined Networks,” Proc. Third Int'l Symp. High Performance Computer Architecture, Feb. 1997.
[11] J. Duato, "A New Theory of Deadlock-Free Adaptive Routing in Wormhole Networks," IEEE Trans. Parallel and Distributed Systems, vol. 4, no. 12, pp. 1,320-1,331, Dec. 1993.
[12] T. von Eicken et al., "Active Messages: a Mechanism for Integrated Communications and Computation," Computer Architecture News, Vol. 20, No. 2, May 1992, pp. 256-266.
[13] M. Galles, “Spider: A High Speed Network Interconnect” IEEE Micro, vol. 17, no. 1, pp. 34–39 Jan.-Feb. 1997.
[14] D. Garcia, “ServerNet II,” Proc. 1997 Parallel Computing, Routing, and Comm. Workshop, June 1997.
[15] R. Horst, “ServerNet Deadlock Avoidance and Fractahedral Topologies,” Proc. Int'l Parallel Processing Symp., pp. 274–280, Apr. 1996.
[16] J.-M. Hsu and P. Banerjee, “Performance Measurement and Trace Driven Simulation of Parallel CAD and Numeric Applications on a Hypercube Multicomputer,” IEEE Trans. Parallel and Distributed Systems, vol. 3, no. 4, pp. 451–464, July 1992.
[17] V. Karamcheti and A.A. Chien, “Do Faster Routers Imply Faster Communication?,” Parallel Computer Routing and Communications, K. Bolding and L. Snyder, eds., Springer-Verlag, New York, 1994, pp. 1-15.
[18] R.J. Littlefield, “Characterizing and Tuning Communications Performance for Real Applications,” Proc. First Intel DELTA Applications Workshop, Feb. 1992.
[19] W. Qiao and L.M. Ni, “Adaptive Routing in Irregular Networks Using Cut-Through Switches,” Proc. 1996 Int'l Conf. Parallel Processing, Aug. 1996.
[20] M.D. Schroeder, A.D. Birrell, M. Burrows, H. Murray, R.M. Needham, T.L. Rodeheffer, E.H. Satterthwaite, and C.P. Thacker, “Autonet: A High-Speed, Self-Configuring Local Area Network Using Point-to-Point Links,” SRC Research Report 59, DEC, Apr. 1990.
[21] S. L. Scott, J.R. Goodman, The Impact of Pipelined Channels on K-Ary N-Cube Networks IEEE Trans. Parallel and Distributed Systems, vol. 5, no. 1, pp. 2-16, Jan. 1994.
[22] S.L. Scott and G. Thorson, “Optimized Routing in the Cray T3D,” Proc. Workshop Parallel Computer Routing and Comm., pp. 281–294, May 1994.
[23] S.L. Scott and G. Thorson, “The Cray T3E Networks: Adaptive Routing in a High Performance 3D torus,” Proc. Hot Interconnects IV, Aug. 1996.
[24] F. Silla and J. Duato, “Improving the Efficiency of Adaptive Routing in Networks with Irregular Topology,” Proc. 1997 Int'l Conf. High Performance Computing, Dec. 1997.
[25] F. Silla and J. Duato, “High-Performance Routing in Networks of Workstations with Irregular Topology,” IEEE Trans. Parallel and Distributed Systems, vol. 11, no. 7, pp. 699-719, July 2000.
[26] F. Silla, M.P. Malumbres, A. Robles, P. López, and J. Duato, Efficient Adaptive Routing in Networks of Workstations with Irregular Topology Proc. Workshop Comm. and Architectural Support for Network-Based Parallel Computing, Feb. 1997.
[27] F. Silla, “Routing and Flow Control in Networks of Workstations,” PhD. thesis, Dept. of Information Systems and Computer Architecture, Universidad Politecnica de Valencia, Spain, 1999. Available athttp:/
[28] C. Stunkel, D. Shea, B. Abali, M. Atkins, C. Bender, D. Grice, P. Hochshild, D. Joseph, B. Nathanson, R. Swetz, R. Stucke, M. Tsao, and P. Varker, “The SP2 High-Performance Switch,” IBM Systems J., vol. 34, no. 2,pp. 185–204, 1995.

Index Terms:
Networks of workstations, irregular topology, wormhole switching, virtual channels, flow control, block multiplexing, channel pipelining.
Federico Silla, José Duato, "On the Use of Virtual Channels in Networks of Workstations with Irregular Topology," IEEE Transactions on Parallel and Distributed Systems, vol. 11, no. 8, pp. 813-828, Aug. 2000, doi:10.1109/71.877939
Usage of this product signifies your acceptance of the Terms of Use.