This Article 
 Bibliographic References 
 Add to: 
Isotach Networks
April 1997 (vol. 8 no. 4)
pp. 337-348

Abstract—We introduce a class of networks called isotach networks designed to reduce the cost of synchronization in parallel computations. Isotach networks maintain an invariant that allows each process to control the logical times at which its messages are received and consequently executed. This control allows processes to pipeline operations without sacrificing sequential consistency and to send isochrons, groups of operations that appear to be received and executed as an indivisible unit. Isochrons allow processes to execute atomic actions without locks. Other uses of isotach networks include ensuring causal message delivery and consistency among replicated data. Isotach networks are characterized by this invariant, not by their topology. They can be implemented in a wide variety of configurations, including NUMA (nonuniform memory access) multiprocessors. Empirical and analytic studies of isotach synchronization techniques show that they outperform conventional techniques, in some cases by an order of magnitude or more. Results presented here assume fault-free systems; we are exploring extension to selected failure models.

[1] Y. Amir, L.E. Moser, M. Melliar-Smith, D.A. Agarwal, and P. Ciarfella, “The Totem Single-Ring Ordering and Membership Protocol,” ACM Trans. Computer Systems, vol. 13, no. 4, pp. 311–342, 1995.
[2] T.E. Anderson,“The performance of spin lock alternatives for shared memory multiprocessors,” IEEE Trans. Parallel and Distributed Systems, vol. 1, no. 1, pp. 6-16, 1990.
[3] B. Awerbuch, “Complexity of Network Synchronization,” J. ACM, vol. 32, pp. 804–823, 1985.
[4] H. Bal and A. Tanenbaum, "Distributed Programming with Shared Data," Proc. Int'l Conf. Computer Languages, IEEE Computer Society Press, Los Alamitos, Calif., 1988, pp. 82-91.
[5] Y. Birk, P.B. Gibbons, J.L.C. Sanz, and D. Soroker, "A Simple Mechanism for Efficient Barrier Synchronization in MIMD Machines," Technical Report RJ 7078, IBM, Oct. 1989.
[6] K. Birman and T. Joseph, "Reliable Communications in Presence of Failures," ACM Trans. Computing Systems, vol. 5, no. 1, pp. 47-76, 1987.
[7] K. Birman, A. Schiper, and P. Stephenson, “Lightweight Causal and Atomic Group Multicast,” ACM Trans. Computer Systems, vol. 9, no. 3, pp. 272-314, Aug. 1991.
[8] K. Birman, "A Response to Cheriton and Skeen's Criticism of Causal and Totally Ordered Communication," OS Review, vol. 28, no. 1, pp. 11-21, Jan. 1994.
[9] N. Boden et al., "Myrinet: A Gigabit-per-Second Local Area Network," IEEE Micro, Feb. 1995, pp. 29-36.
[10] K. Chandy and J. Misra, "Distributed Simulation: A Case Study in Design and Verification of Distributed Programs," IEEE Trans. Software Eng., vol. 5, no. 5, pp. 440-452, Sept. 1979.
[11] J. Chang and N. Maxemchuk,“Reliable broadcast protocols,”ACM Trans. Comput. Syst., vol. 2, pp. 251–273, Aug. 1984.
[12] D.R. Cheriton and D. Skeen, "Understanding the Limitations of Causally and Totally Ordered Communications," Operating Systems Rev., Dec. 1993, pp. 44-57.
[13] F. Cristian, "Synchronous Atomic Broadcast for Redundant Broadcast Channels," Technical Report RJ 7203 (67682), IBM, Dec. 1989.
[14] M. Dasser, "TOMP: A Total Ordering Multicast Protocol," Operating Systems Review, vol. 26, no. 1, Jan. 1992.
[15] H. Garcia-Molina and A. Spauster,"Ordered and Reliable Multicast Communication," ACM Trans. Computer Systems, vol. 6, no. 3, Aug. 1991, pp. 242-271.
[16] K. Gharachorloo, A. Gupta, and J. Hennessy, "Performance Evaluation of Memory Consistency Models for Shared-Memory Multiprocessors," Proc. ASPLOS IV, pp. 245-257, 1991.
[17] P.B. Gibbons, "The Asynchronous PRAM: A Semi-Synchronous Model for Shared Memory MIMD Machines," 89-062, ICSI, Berkeley, Calif., December, 1989.
[18] K.J. Goldman, "Highly Concurrent Logically Synchronous Multicast," Distributed Computing, pp. 94-108,Berlin-Heidelburg-New York: Springer-Verlag, 1989.
[19] Y. Jenq, "Performance Analysis of a Packet Switch Based on Single-Buffered Banyan Network," IEEE J. Selected Areas Comm., vol. 1, no. 6, pp. 1,014-1,021, Dec. 1983.
[20] M.F. Kaashoek, A.S. Tanenbaum, S. Hummel, and H.E. Bal, “An Efficient Reliable Broadcast Protocol,” Operating Systems Review, vol. 23, no. 4, pp. 5–19, Oct. 1989.
[21] M.J. Karol, M.G. Hluchyj, and S.P. Morgan, “Input versus Output Queueing on a Space-Division Packet Switch,” IEEE Trans. Comm., vol. 35, no. 12, pp. 1347-1356, Dec. 1987.
[22] D.E. Knuth, "Sorting and Searching," Fundamental Algorithms, vol. 3, p. 397. Addison-Wesley, 1973.
[23] C.P. Kruskal, L. Rudolph, and M. Snir, "Efficient Synchronization on Multiprocessors with Shared Memory," ACM Trans. Programming Languages and Systems, vol. 10, no. 4, pp. 579-601, Oct. 1988.
[24] M. Kumar and J.R. Jump, "Performance Enhancement of Buffered Delta Networks Using Crossbar Switches and Multiple Links," J. Parallel and Distributed Computing, vol. 1, pp. 81-103, 1984.
[25] L. Lamport, "Time, clocks and the ordering of events in a distributed system," Comm. ACM, vol. 21, no. 7, pp. 558-565, July 1978.
[26] L. Lamport, "How to Make a Multiprocessor Computer That Correctly Executes Multiprocessor Programs," IEEE Trans. Computers, vol. 28, pp. 690-691, 1979.
[27] L. Lamport, "On Interprocess Communication," Distributed Computing, vol. 1, no. 2, pp. 77-101, Apr. 1986.
[28] D.B. Lomet, "Process Structuring, Synchronisation and Recovery Using Atomic Actions," Proc. ACM Trans. Language Design for Reliable Software, SIGPLAN, pp. 128-137, 1977.
[29] J. M. Mellor-Crummey and M. L. Scott,“Algorithms for scalable synchronization on shared-memory multiprocessors,”ACM Trans. Comput. Syst., vol, 9, no. 1, pp. 21–65, Feb. 1991.
[30] M.M. Michael and M.L. Scott, "Scalability of Atomic Primitives on Distributed Shared Memory Multprocessors," Technical Report 528, Univ. of Rochester, Computer Science Dept., July 1994.
[31] S. Owicki and D. Gries, "An Axiomatic Proof Technique for Parallel Programs I," Acta Informatica, vol. 6, pp. 319-340, 1976.
[32] L. Lamport and S. Owicki, "Proving Liveness Properties of Concurrent Programs," ACM Trans. Programming Languages and Systems, vol. 4, pp. 455-495, July 1982.
[33] C. Papadimitriou, The Theory of Database Concurrency Control. Computer Science Press, 1986.
[34] L. Peterson, N. Bucholz, and R. Schlichting, “Preserving and Using Context Information in Interprocess Communication,” ACM Trans. Computer Systems, Aug. 1989, pp. 217‐246.
[35] A.G. Ranade, S.N. Bhatt, and S.L. Johnsson, "The Fluent Abstract Machine," Technical Report 573, Yale Univ., Dept. of Computer Science, Jan. 1988.
[36] A. Ranade and A.G. Ranade, "How to Emulate Shared Memory," IEEE Ann. Symp. Foundations of Computer Science, pp. 185-194,Los Angeles, 1987.
[37] M. Raynal and M. Singhal, "Logical Clocks: A Way to Capture Causality in Distributed Systems," Computer, vol. 29, no. 2, pp. 49-56, Feb. 1996.
[38] D Reed, "Implementing Atomic Actions on Decentralized Data," ACM Trans. Computer Systems, vol. 1, no. 1, Feb. 1983.
[39] R. Renesse, "Why Bother With CATOCS?" OS Review, vol. 28, no. 1, pp. 22-27, Jan. 1994.
[40] P.F. Reynolds Jr., C. Williams, and R.R. Wagner Jr., "Empirical Analysis of Isotach Networks," Technical Report 92-19, Univ. of Virginia, Dept. of Computer Science, June 1992.
[41] P.F. Reynolds Jr. and C. Williams, "Asynchronous Rule-Based Systems in CGF," Proc. Fifth Conf. Computer Generated Forces and Behavioral Representation,Orlando, Fla., May 1995.
[42] J.H. Saltzer, D.P. Reed, and D.D. Clark, "End-to-End Arguments in System Design," ACM Trans. Computer Systems, vol. 2, no. 4, Nov. 1984, pp. 277-288.
[43] C. Scheurich and M. Dubois, “Correct Memory Operation of Cache-Based Multiprocessors,” Proc. 14th Ann. Int'l Symp. Computer Architecture, pp. 234-243, 1987.
[44] B.R. Supinski, C. Williams, and P.F. Reynolds Jr., "Performance Evaluation of the Late Delta Cache Coherence Protocol," Technical Report CS-96-05, Univ. of Virginia, Dept. of Computer Science, Mar. 1996.
[45] K. Verstoep, K. Langendoen, and H.E. Bal, "Efficient Reliable Multicast on Myrinet," Technical Report IR-399, Vrije Universiteit, Amsterdam, Jan. 1996.
[46] R.R. Wagner Jr., "On the Implementation of Local Synchrony," CS-93-33, Univ. of Virginia, Dept. of Computer Science, June 1993.
[47] C. Williams and P.F. Reynolds Jr., "Delta-Cache Protocols: A New Class of Cache Coherence Protocols," Technical Report 90-34, Univ. of Virginia, Dept. of Computer Science, Dec. 1990.
[48] C. Williams, Concurrency Control in Asynchronous Computations, doctoral dissertation, Dept. Computer Science, Univ. of Virginia, 1993.
[49] C.C. Williams and P.F. Reynolds Jr., "Combining Atomic Actions," J. Parallel and Distributed Computing, pp. 152-163, Feb. 1995.

Index Terms:
Logical time, interprocess coordination, concurrency control, isochronicity, atomicity, sequential consistency, interconnection networks, multiprocessor systems.
Paul F. Reynolds, Jr., Craig Williams, Raymond R. Wagner, Jr., "Isotach Networks," IEEE Transactions on Parallel and Distributed Systems, vol. 8, no. 4, pp. 337-348, April 1997, doi:10.1109/71.588601
Usage of this product signifies your acceptance of the Terms of Use.