This Article 
 Bibliographic References 
 Add to: 
Feedback-Based Synchronization in System Area Networks for Cluster Computing
October 2005 (vol. 16 no. 10)
pp. 908-920
Andrew A. Chien, IEEE Computer Society

Abstract—Many applications in cluster computing require QoS (Quality of Service) services. Since performance predictability is essential to provide QoS service, underlying systems must provide predictable performance guarantees. One way to ensure such guarantees from network subsystems is to generate global schedules from applications' network requests and to execute the local portion of the schedules at each network interface. To ensure accurate execution of the schedules, it is essential that a global time base must be maintained by local clocks at each network interface. The task of providing a single time base is called a synchronization problem and this paper addresses the problem for system area networks. To solve the synchronization problem, FM-QoS [CHECK END OF SENTENCE] proposed a simple synchronization mechanism called FBS (Feedback-Based Synchronization) which uses built-in flow control signals. This paper extends the basic notion of FM-QoS to a theoretical framework and generalizes it: 1) to identify a set of built-in network flow control signals for synchrony and to formalize it as a synchronizing schedule and 2) to analyze the synchronization precision of FBS in terms of flow control parameters. Based on generalization, two application classes are studied for a single switch network and a multiple switch network. For each class, a synchronizing schedule is proposed and its bounded skew is analyzed. Unlike FM-QoS, the synchronizing schedule is proven to minimize the bounded skew value for a single switch network. To understand the analysis results in practical networks, skew values are obtained with flow control parameters of Myrinet-2000 [CHECK END OF SENTENCE]. We observed that the maximum bounded skew of FBS is 5.79\musec or less over all our experiments. Based on this result, we came to a conclusion that FBS was a feasible synchronization mechanism in system area networks.

[1] Infiniband Trade Assoc., Infiniband Architecture Specification Release 1.0, Oct. 2000.
[2] A. Chien et al. HPVM Software Distributions, index.html, 1999.
[3] B. Chun, A. Mainwaring, and D. Culler, “Virtual Network Transport Protocols for Myrinet,” Proc. Hot Interconnects V, 1997.
[4] K. Connelly, “FM-QoS: A Quality of Service Messaging Substrate for Asynchronous Local-Area Networks with Hardware-Level Network Feedback,” master's thesis, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, 1999.
[5] A. Chien et al., “Design and Evaluation of an HPVM-Based Windows NT Supercomputer,” The Int'l J. High Performance Computing Applications, vol. 13, no. 3, pp. 201-219, Fall 1999.
[6] F. Petrini et al., “The Quadrics Network: High-Performance Clustering Technology,” IEEE Micro, pp. 46-57, Jan.-Feb. 2002.
[7] I. Foster et al., “A Distributed Resource Management Architecture that Supports Advance Reservations and Co-Allocation,” Proc. Int'l Workshop Quality of Service, 1999.
[8] J. Liu et al., “MPI over InfiniBand: Early Experiences,” Technical Report OSU-CISRC-10/02-TR25, Ohio State Univ., 2002.
[9] N.J. Boden et al., “Myrinet— A Gigabit-per-Second Local-Area Network,” IEEE Micro, vol. 15, no. 1, pp. 29-36, Feb. 1995.
[10] R. Krishnamurthy et al., “Architecture and Hardware for Scheduling Gigabit Packet Stream,” Proc. 10th Symp. High Performance Interconnects, 2002.
[11] R. Sivaram, “A Reliable Hardware Barrier Synchronization Scheme,” Proc. Int'l Parallel Processing Symp., pp. 274-280, 1997.
[12] S. Chakravarthi et al., “A Fine-Grain Clock Synchronization Mechanism for Myrinet Clusters,” Proc. 27th IEEE Conf. Local Computer Networks, 2002.
[13] J. Sanchez, F. Alfaro, and J. Duato, “QoS in Infiniband Subnetworks,” IEEE Trans. Parallel and Distributed Systems, vol. 15, no. 9, pp. 810-823, Sept. 2004.
[14] E. Horowitz and S. Sahni, Fundamental of Computer Algorithms. Computer Science Press, 1978.
[15] J.H. Kim, “Bandwidth and Latency Guarantees in Low-Cost, High Performance Networks,” PhD thesis, Univ. of Illinois at Urbana-Champaign, Jan. 1997.
[16] J.H. Kim and A.A. Chien, “Compressionless Routing: A Framework for Adaptive and Fault-Tolerant Routing,” IEEE Trans. Computer, vol. 8, no. 3, pp. 229-244, Mar. 1997.
[17] H. Kopetz and W. Ochsenreiter, “Clock Synchronization in Distributed Real-Time Systems,” IEEE Trans. Computer, vol. 36, no. 8, pp. 933-941, Aug. 1987.
[18] J. Rexford, A. Greenberg, and F. Bonomi, “Hardware-Efficient Fair Queueing Architectures for High-Speed Networks,” Proc. IEEE INFOCOM, 1996.
[19] P.F. Reynolds, C. Williams Jr., and R.R. Wagner Jr., “Isotach Networks,” IEEE Trans. Parallel and Distributed Systems, vol. 8, no. 4, 1997.
[20] F. Schmuck and F. Cristian, “Continuous Clock Amortization Need Not Affect the Precision of a Clock Synchronization Algorithm,” Proc. ACM Symp. Principles of Distributed Computing, pp. 133-143, 1990.
[21] S.L. Scott, “Synchronization and Communication in the T3E Multiprocessor,” Proc. ASPLOS VII, 1996.
[22] M. Shreedhar and G. Varghese, “Efficient Fair Queueing Using Deficit Round Robin,” Proc. ACM SIGCOMM, 1995.
[23] M. Snir, P. Hochschild, D.D. Frye, and K.J. Gildea, “The Communication Software and Parallel Environment of the IBM SP2,” IBM System J., vol. 34, no. 2, pp. 205-221, 1995.
[24] “A New Approach to Clustering,” Technical Report 2270WP, Mellanox Tech nology, 2002.
[25] N. Vasanthavada and P.N. Marinos, “Synchronization of Fault-Tolerant Clocks in the Presence of Malicious Faults,” IEEE Trans. Computers, vol. 37, no. 4, pp. 440-448, Apr. 1998.
[26] R.J. Vetter and D. Du, “Issues and Challenges in ATM Networks,” Comm. ACM, vol. 38, no. 2, pp. 28-29, Feb. 1995.
[27] Myrinet-on-VME Protocol Specification Standard, VITA 26-199x Draft 1.1, VITA Standards Organization, Aug. 1998.
[28] L. Zhang, “Virtual Clock: A New Traffic Control Algorithm for Packet Switching Networks,” Proc. ACM SIGCOMM, pp. 19-29, 1990.
[29] L. Zhang, S. Deering, D. Estrin, S. Shenker, and D. Zappala, “RSVP: A New Resource Reservation Protocol,” IEEE Network, pp. 8-18, Sept. 1993.

Index Terms:
Synchronization, link level flow control, system area networks, cluster computing.
Hyo Jung Song, Andrew A. Chien, "Feedback-Based Synchronization in System Area Networks for Cluster Computing," IEEE Transactions on Parallel and Distributed Systems, vol. 16, no. 10, pp. 908-920, Oct. 2005, doi:10.1109/TPDS.2005.122
Usage of this product signifies your acceptance of the Terms of Use.