This Article 
 Bibliographic References 
 Add to: 
Distributed Synchronous Clocking
March 1995 (vol. 6 no. 3)
pp. 314-328

Abstract—It has historically been difficult to distribute a well-aligned hardware clock throughout the physical extent of a synchronous processor. Traditionally, this task has been accomplished by distributing the output of a central oscillator over a tree-like network, with repeaters at necessary intervals. While straightforward in concept, this method suffers from poor reliability, poor scalability and high skew.

In this paper, we present an alternative approach—Distributed Synchronous Clocking—that maintains the simplicity of synchronous operation without suffering the drawbacks of centralized clocking. A network of independent oscillators takes the place of the centralized clock source, providing separate clock signals to the physically distant parts of a computing system. A distributed error correction algorithm effects global phase alignment by utilizing local comparisons of neighboring oscillator phase.

In contrast to centralized clock distribution, distributed clocking has the inherent potential for complete scalability and graceful degradation. However, because oscillator phase is a modular quantity, a naive implementation of distributed synchronous clocking can suffer from mode-lock—the trapping of local oscillator phase in undesirable stable equilibria where global phase is not aligned [26]. We present a simple method for eliminating this problem in $k$-ary Cartesian meshes and give a proof of its correctness for two-dimensional networks. An electronic implementation is also presented and several engineering issues relating to error tolerance are discussed.

Index Terms—Computer hardware clocking, clock synchronization, distributed clocking, oscillator synchronization, phase-locked loops, mode-locking.

[1] K. Alnes, M. Schanke, and E. Kristiansen,“SCI clock synchronization,”IEEE P1596 Std. Committee Doc. 123, Sept. 1989.
[2] AT&T,STUGLA VCXO Specification, 1991.
[3] R. P. Christfield, D. T. Doan, K. R. Williams, and R. H. Wrange,“Phase-locked clocking,”IBM Tech. Discl. Bull., vol. 23, no. 7A, pp. 2924–2926, Dec. 1980.
[4] B. Dally, personal communication.
[5] W.J. Dally and C.L. Seitz, “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks,” IEEE Trans. Computers, Vol. C-36, No. 5, May 1987, pp. 547-553.
[6] W. M. Daly, A. L. Hopkins, Jr., and J. F. McKenna,“A fault-tolerant digital clocking system,”inProc. 3rd Int. Symp. Fault-Tolerant Comput., 1973, pp. 17–21.
[7] A. El-Amawy,“Branch-and-combine clocking of arbitrarily large computing networks,”in1991 Int. Conf. Parallel Processing, 1991, pp. I409–I417.
[8] F. Gardner,Phaselock Techniques. New York: Wiley, 1979.
[9] F. M. Gardner,“Charge-pump phase-lock loops,”IEEE Trans. Commun., vol. COM-28, pp. 1849–1858, Nov. 1980.
[10] ——,“Phase accuracy of charge-pump PPL's,”IEEE Trans. Commun., vol. COM-30, pp. 2363–2364, Oct. 1982.
[11] D.-K. Jeong, G. Borriello, D. A. Hodges, and R. H. Katz,“Design of PPL-based clock generation circuits,”IEEE J. Solid State Circuits, vol. SC-22, pp. 255–261, Apr. 1987.
[12] M. G. Johnson and E. L. Hudson,“A variable delay line PLL for CPU-coprocessor synchronization,”IEEE J. Solid State Circuits, vol. 23, pp. 1218–1223, Oct. 1988.
[13] J. L. W. Kessels,“Two designs of a fault-tolerant clocking system,”IEEE Trans. Comput., vol. C-33, pp. 912–919, Oct. 1984.
[14] R. K. Kostuk, L. W., and Y-T. Huang,“Optical clock distribution with holographic optical elements,”inProc. SPIE, vol. 977, Aug. 1988, pp. 24–36.
[15] C. M. Krishna, K. G. Shin, and R. W. Butler,“Ensuring fault tolerance of phase-locked clocks,”IEEE Trans. Comput., vol. C-34, pp. 752–756, Aug. 1985.
[16] W. C. Lindsey, A. V. Kantak, and A. Dobrogowski,“Network synchronization by means of a returnable timing system,”IEEE Trans. Commun., vol. COM-26, pp. 892–896, June 1978.
[17] M. R. Miller,“Feasibility studies of synchronised-oscillator systems for P.C.M. telephone networks,”Proc. IEE, vol. 116, pp. 1135–1143, July 1969.
[18] A. Nowatzyk,“A communication architecture for multiprocessor networks,”CMU Ph.D. dissertartion, Carnegie-Mellon CS-89-181, Dec. 1989.
[19] T. Saito,“Application of phase-locked oscillator for PCM network synchronization,”IEEE Trans. Commun., vol. COM-30, pp. 2344–2354, Oct. 1982.
[20] M. Schanke,“Proposal for clock distribution in SCI,”IEEE P1596 Std. Committee Doc. 77, May 1989.
[21] C. L. Seitz,“System timing,”inIntroduction to VLSI Systems, C. A. Mead and L. A. Conway, Eds. Reading, MA: Addison-Wesley, 1980, ch. 7.
[22] K. G. Shin and P. Ramanathan,“Transmission delays in hardware clock synchronization,”IEEE Trans. Comput., vol. 37, pp. 1465–1467, Nov. 1988.
[23] T. B. Smith,“Fault-tolerant clocking system,”inProc. 11th Int. Symp. Fault-Tolerant Comput., 1981, pp. 262–264.
[24] N. Vasanthavada, P. N. Marinos, and G. S. Mersten,“Design and performance evaluation of mutually synchronized fault-tolerant clock systems,”inProc. 16th Int. Symp. Fault-Tolerant Comput., 1986, pp. 206–210.
[25] S. Ward,“The NuMesh: A scalable, modular, 3D interconnect,”inNuMesh Industrial Partners Meeting,Cambridge, MA, Feb. 1991.
[26] M. W. Willard,“Analysis of a system of mutually synchronized oscillators,”IEEE Trans. Commun. Technol., vol. COM-18, pp. 467–483, Oct. 1970.
[27] R. Woudsma and J. M. Noteboom,“The modular design of clock-generator circuits in a CMOS building-block system,”IEEE J. Solid State Circuits, vol. SC-20, pp. 770–774, June 1985.

Gill A. Pratt, John Nguyen, "Distributed Synchronous Clocking," IEEE Transactions on Parallel and Distributed Systems, vol. 6, no. 3, pp. 314-328, March 1995, doi:10.1109/71.372779
Usage of this product signifies your acceptance of the Terms of Use.