This Article 
 Bibliographic References 
 Add to: 
A Router Architecture for Real-Time Communication in Multicomputer Networks
October 1998 (vol. 47 no. 10)
pp. 1088-1101

Abstract—Parallel machines have the potential to satisfy the large computational demands of real-time applications. These applications require a predictable communication network, where time-constrained traffic requires bounds on throughput and latency, while good average performance suffices for best-effort packets. This paper presents a new router architecture that tailors low-level routing, switching, arbitration, flow-control, and deadlock-avoidance policies to the conflicting demands of each traffic class. The router implements bandwidth regulation and deadline-based scheduling, with packet switching and table-driven multicast routing, to bound end-to-end delay and buffer requirements for time-constrained traffic while allowing best-effort traffic to capitalize on the low-latency routing and switching schemes common in modern parallel machines. To limit the cost of servicing time-constrained traffic, the router includes a novel packet scheduler that shares link-scheduling logic across the multiple output ports, while masking the effects of clock rollover on the represention of packet eligibility times and deadlines. Using the Verilog hardware description language and the Epoch silicon compiler, we demonstrate that the router design meets the performance goals of both traffic classes in a single-chip solution. Verilog simulation experiments on a detailed timing model of the chip show how the implementation and performance properties of the packet scheduler scale over a range of architectural parameters.

[1] D. Ferrari, Client Requirements for Real-Time Communication Services IEEE Comm. Magazine, vol. 28, no. 11, pp. 76-90, Nov. 1990.
[2] L.R. Welch and K. Toda, "Architectural Support for Real-Time Systems: Issues and Trade-Offs," Proc. Int'l Workshop Real-Time Computing Systems and Applications, Dec. 1994.
[3] M. Mutka, "Using Rate Monotonic Scheduling Technology for Real-Time Communications in a Wormhole-Routed Network," Proc. Second Workshop Distributed and Parallel Real-Time Systems, pp. 194-199, 1994.
[4] J.-P. Li and M. Mutka, "Priority Based Real-Time Communication for Large Scale Wormhole Networks," Proc. Eighth Int'l Parallel Processing Symp., pp. 433-438, 1994.
[5] A. Saha, "Simulator for Real-Time Parallel Processing Architectures," Proc. IEEE Ann. Simulation Symp., pp. 74-83, Apr. 1995.
[6] K. Toda, K. Nishida, E. Takahashi, N. Michell, and Y. Yamaguchi, "Design and Implementation of a Priority Forwarding Router Chip for Real-Time Interconnection Networks," Int'l J. Mini and Microcomputers, vol. 17, no. 1, pp. 42-51, 1995.
[7] R. Games, A. Kanevsky, P. Krupp, and L. Monk, "Real-Time Communications Scheduling for Massively Parallel Processors," Proc. Real-Time Technology and Applications Symp., pp. 76-85, May 1995.
[8] S. Balakrishnan and F. Özgüner, "Providing Message Delivery Guarantees in Pipelined Flit-Buffered Multiprocessor Networks," Proc. IEEE Real-Time Technology and Applications Symp., pp. 120-129, June 1996.
[9] R.S. Raji, “Smart Networks for Control,” IEEE Spectrum, vol. 31, no. 6, pp. 49–55, June 1994.
[10] C.M. Aras, J.F. Kurose, D.S. Reeves, and H. Schulzrinne, “Real-Time Communication in Packet-Switched Networks,” Proc. IEEE, vol. 82, no. 1, pp. 122-139, Jan. 1994.
[11] D.D. Kandlur, K.G. Shin, and D. Ferrari, “Real-Time Communication in Multi-Hop Networks,” IEEE Trans. Parallel and Distributed Systems, vol. 5, no. 10, pp. 1,044-1,056, Oct. 1994.
[12] D. Verma, H. Zhang, and D. Ferrari, “Delay Jitter Control for Real-Time Communication in a Packet Switching Network,” Proc. TriComm '91, pp. 35-43, 1991.
[13] D. Ferrari and D.C. Verma,“A scheme for real-time channel establishment in wide-area networks, IEEE J. Selected Areas in Comm., vol. 8, no. 3, pp. 368-379, Apr. 1990.
[14] H. Zhang and D. Ferrari, "Rate-Controlled Service Disciplines," J. High Speed Networks, vol. 3, no. 4, pp. 389-412, 1994.
[15] H. Zhang, "Providing End-to-End Performance Guarantees Using Non-Work-Conserving Disciplines," Computer Comm., vol. 18, pp. 769-781, Oct. 1995.
[16] L. Georgiadis, R. Guérin, V. Peris, and K.N. Sivarajan, Efficient Network QoS Provisioning Based on Per Node Traffic Shaping IEEE/ACM Trans. Networking, vol. 4, no. 4, pp. 482-501, 1996.
[17] Y. Ofek and M. Yung, "The Integrated MetaNet Architecture: A Switch-Based Multimedia LAN for Parallel Computing and Real-Time Traffic," Proc. IEEE INFOCOM, pp. 802-811, 1994.
[18] W.J. Dally and C.L. Seitz, "The Torus Routing Chip," J. Distributed Computing, vol. 1, no. 3, pp. 187-196, 1986.
[19] R.L. Cruz, A Calculus for Network Delay, Part I: Network Elements in Isolation IEEE Trans. Information Theory, vol. 37, pp. 114-131, Jan. 1991.
[20] Q. Zheng and K.G. Shin,“On the ability of establishing real-time channelsin point-to-point packet-switched networks,” IEEE Trans. Comm. pp. 1,096-1,105, Feb./Mar./Apr. 1994.
[21] C.L. Liu and J.W. Layland, “Scheduling Algorithms for Multiprogramming in a Hard Real-Time Environment,” J. ACM, vol. 20, no. 1, pp. 40-61, 1973.
[22] P. Kermani and L. Kleinrock, "Virtual Cut-Through: A New Computer Communication Switching Technique," Computer Networks, vol. 3, pp. 267-286, Sept. 1979.
[23] W.J. Dally, "Virtual-Channel Flow Control," IEEE Trans. Parallel and Distributed Systems, vol. 3, no. 2, pp. 194-205, Mar. 1992.
[24] J. Rexford, J. Dolter, and K. Shin, "Hardware Support for Controlled Interaction of Guaranteed and Best-Effort Communication," Proc. Third IPPS Workshop Parallel and Distributed Real-Time Systems, pp. 188-193, Apr. 1994.
[25] J. Rexford and K.G. Shin, "Support for Multiple Classes of Traffic in Multicomputer Routers," Proc. Parallel Computer Routing and Communication Workshop, May 1994.
[26] J. Rexford, W. Feng, J. Dolter, and K.G. Shin, "PP-MESS-SIM: A Flexible and Extensible Simulator for Evaluating Multicomputer Networks," IEEE Trans. Parallel and Distributed Systems, vol. 8, no. 1, pp. 25-40, Jan. 1997.
[27] J. Duato and P. Lopez, "Bandwidth Requirements for Wormhole Switches: A Simple and Efficient Design," Proc. Euromicro Workshop Parallel and Distributed Processing, pp. 377-384, 1994.
[28] C. Stunkel, D. Shea, B. Abali, M. Atkins, C. Bender, D. Grice, P. Hochshild, D. Joseph, B. Nathanson, R. Swetz, R. Stucke, M. Tsao, and P. Varker, “The SP2 High-Performance Switch,” IBM Systems J., vol. 34, no. 2,pp. 185–204, 1995.
[29] F.A. Tobagi, "Fast Packet Switch Architectures for Broadband Integrated Services Digital Networks," Proc. IEEE, vol. 78, pp. 133-167, Jan. 1990.
[30] W.J. Dally and C.L. Seitz, “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks,” IEEE Trans. Computers, Vol. C-36, No. 5, May 1987, pp. 547-553.
[31] L.M. Ni and P.K. McKinley, "A Survey of Wormhole Routing Techniques in Direct Networks," Computer, vol. 26, no. 2, pp. 62-76, Feb. 1993.
[32] K. Aoyama and A. Chien, "Cost of Adaptivity and Virtual Lanes in a Wormhole Router," J. VLSI Design, vol. 2, no. 4, pp. 315-333, 1995.
[33] W.C. Lee, M.G. Hluchyj, and P.A. Humblet, "Routing Subject to Quality of Service Constraints in Integrated Communication Networks," IEEE Network, pp. 46-55, July/Aug. 1995.
[34] Q. Zheng, K.G. Shin, and C. Shen, "Real-Time Communication in ATM," Proc. Ann. Conf. Local Computer Networks, pp. 156-164, Oct. 1994.
[35] H.J. Chao, "A Novel Architecture for Queue Management in the ATM Network," IEEE J. Selected Areas in Comm., vol. 9, pp. 1,110-1,118, Sept. 1991.
[36] D. Picker and R.D. Fellman, "VLSI Priority Packeet Queue with Inheritance and Overwrite," IEEE Trans. VLSI, vol. 3, pp. 245-253, June 1995.
[37] J. Liebeherr, D.E. Wrege, and D. Ferrari, "Exact Admission Control for Networks with Bounded Delay Services," IEEE/ACM Trans. Networking, vol. 4, pp. 885-901, Dec. 1996.
[38] J.L. Rexford, A.G. Greenberg, and F.G. Bonomi, Hardware-Efficient Fair Queuing Architectures for High-Speed Networks Proc. INFOCOMM, pp. 638-646, Mar. 1996.
[39] S.-W. Moon, K. Shin, and J. Rexford, "Scalable Hardware Priority Queue Architectures for High-Speed Packet Switches," Proc. Real-Time Technology and Applications Symp., pp. 203-212, June 1997.
[40] H.J. Chao and N. Uzun, "A VLSI Sequencer Chip for ATM Traffic Shaper and Queue Manager," IEEE J. Solid-State Circuits, vol. 27, pp. 1,634-1,643, Nov. 1992.
[41] C.E. Leiserson, "Systolic Priority Queues," Proc. Caltech Conf. VLSI, pp. 200-214, Jan. 1979.
[42] J. Rexford, F. Bonomi, A. Greenberg, and A. Wong, "Scalable Architectures for Integrated Traffic Shaping and Link Scheduling in High-Speed ATM Switches," IEEE J. Selected Areas in Comm., vol. 15, pp. 938-950, June 1997.

Index Terms:
Multicomputer router, real-time communication, link scheduling, wormhole switching, packet switching.
Jennifer Rexford, John Hall, Kang G. Shin, "A Router Architecture for Real-Time Communication in Multicomputer Networks," IEEE Transactions on Computers, vol. 47, no. 10, pp. 1088-1101, Oct. 1998, doi:10.1109/12.729792
Usage of this product signifies your acceptance of the Terms of Use.