This Article 
 Bibliographic References 
 Add to: 
An Optical Interconnection Network and a Modified Snooping Protocol for the Design of Large-Scale Symmetric Multiprocessors (SMPs)
December 2004 (vol. 15 no. 12)
pp. 1093-1104

Abstract—In Symmetric Multiprocessors (SMPs), the cache coherence overhead and the speed of the shared buses limit the address/snoop bandwidth needed to broadcast transactions to all processors. As a solution, a scalable address subnetwork called Symmetric Multiprocessor Network (SYMNET) is proposed in which address requests and snoop responses of SMPs are implemented optically. SYMNET not only uses passive optical interconnects that increases the speed of the proposed network, but also pipelines address requests at a much faster rate than electronics. This increases the address bandwidth for snooping, but the preservation of cache coherence can no longer be maintained with the usual snooping protocols. A modified coherence protocol, Coherence in SYMNET (COSYM), is introduced to solve the coherence problem. COSYM was evaluated with a subset of Splash-2 benchmarks and compared with the electrical bus-based MOESI protocol. The simulation studies have shown a 5-66 percent improvement in execution time for COSYM as compared to MOESI for various applications. Simulations have also shown that the average latency for a transaction to complete using COSYM protocol was 5-78 percent better than the MOESI protocol. It is also seen that SYMNET can scale up to hundreds of processors while still using fast snooping-based cache coherence protocols, and additional performance gains may be attained with further improvement in optical device technology.

[1] D.E. Culler, J.P. Singh, and A. Gupta, Parallel Computer Architecture: A Hardware/Software Approach. San Francisco, Calif.: Morgan Kaufmann, 1999.
[2] A. Charlesworth, “The Sun Fireplane SMP Interconnect in the Sunfire 3800-6800,” Hot Interconnects 9, pp. 37-42, Aug. 2001.
[3] F. Pong, M. Dubois, and K. Lee, “Design and Performance of SMPS with Asynchronous Caches,” Technical Report HPL-1999-149, Hewlett Packard, HP Laboratories Palo Alto, Nov. 1999.
[4] D.A.B. Miller, “Rationale and Challenges for Optical Interconnects to Electronic Chips,” Proc. IEEE, vol. 88, pp. 728-749, June 2000.
[5] J.H. Collet, W. Hlayhel, and D. Litaize, “Parallel Optical Interconnects May Reduce the Communication Bottleneck in Symmetric Multiprocessors,” Applied Optics, vol. 40, pp. 3371-3378, 2001.
[6] A. Charlesworth, “Starfire: Extending the SMP Envelope,” IEEE Micro, vol. 18, no. 1, pp. 39-49, Jan.-Feb. 1998.
[7] M. Galles and E. Williams, “Performance Optimizations, Implementation and Verification of the SGI Challenge Multiprocessor,” Proc. 27th Ann. Hawaii Int'l Conf. Systems Sciences, pp. 134-143, 1996.
[8] M.M. Martin, D.J. Sorin, A. Ailamaki, A.R. Alameldeen, R.M. Dickson, C.J. Mauer, M. Plakal, M.D. Hill, and D.A. Wood, “Timestamp Snooping: An Approach for Extending SMPS,” Proc. Ninth Int'l Conf. Architectural Support for Programming Languages and Operating Systems, pp. 25-36, Nov. 2000.
[9] J.H. Collet, D. Litaize, J.V. Campenhut, C. Jesshope, M. Desmulliez, H. Thienpont, J. Goodman, and A. Louri, “Architectural Approaches to the Role of Optics in Mono and Multiprocessor Machines,” Applied Optics, special issue on optics in computing, vol. 39, pp. 671-682, 2000.
[10] Y.S. Liu, R.J. Wojnarowski, W.A. Hennessy, P.A. Piacente, J. Rowlette, J. Stack, M. Kader-Kallen, Y. Liu, A. Peczalski, A. Nahata, and J. Yardley, “Plastic VCSEL Array Packaging and High Density Polymer Waveguides for Board and Backplane Optical Interconnect,” Proc. Electronic Components and Technology Conf., pp. 999-1005, 1998.
[11] F. Mederer, F. Eberhard, I. Ecker, R. Jager, J. Joos, C. Jung, M. Kicherer, R. King, P. Schnitzer, H. Unold, D. Wiedenmann, and K.J. Ebeling, “High Performance Selectively Oxidized VCSELs and Arrays for Parallel High-Speed Optical Interconnects,” Proc. 50th Electronic Components and Technology Conf., pp. 1242-1251, May 2000.
[12] R. Pu, E.M. Hayes, C.W. Wilmsen, K.D. Ohoquette, H.Q. Hou, and K.M. Geib, “Comparison of Techniques for Bonding VCSELs Directly to ICS,” J. Optical Soc. Am., vol. 1, pp. 324-329, 1999.
[13] C.S. Woo, M. Ohara, E. Torrie, J.P. Singh, and A. Gupta, “The Splash-2 Programs: Characterization and Methodological Considerations,” Proc. 22nd Ann. Int'l Symp. Computer Architecture, pp. 24-37, June 1995.
[14] D.M. Chiarulli, S.P. Levitan, R.G. Melhem, M. Bidnurkar, R. Ditmore, G. Gravenstreter, Z. Guo, C. Qiao, M.F. Sakr, and J.P. Teza, “Optoelectronic Buses for High-Performance Computing,” Proc. IEEE, pp. 1701-1710, 1994.
[15] P. Lukowicz, “The Photobus Smart Pixel Interconnection System for Symmetric Multiprocessing Using Workstation Clusters,” Proc. Sixth Int'l Conf. Parallel Interconnects, pp. 106-113, 1999.
[16] J.-H. Ha and T.M. Pinkston, “A New Token-Based Channel Access Protocol for Wavelength Division Multiplexed Multiprocessor Interconnects,” J. Parallel and Distributed Computing, vol. 60, no. 2, pp. 169-188, Feb. 2000.
[17] B. Webb and A. Louri, “A Class of Highly Scalable Optical Crossbar-Connected Interconnection Networks (SOCNS) for Parallel Computing Systems,” IEEE Trans. Parallel and Distributed Systems, vol. 11, no. 1, pp. 444-458, May 2000.
[18] P. Dowd, J. Perreault, J. Chu, D.C. Hoffmeister, R. Minnich, D. Burns, F. Hady, Y.J. Chen, and M. Dagenais, “Lighnting Network and Systems Architecture,” J. Lightwave Technology, vol. 14, pp. 1371-1387, 1996.
[19] A. Lindstrom, “Parallel Links Transform Networking Equipment,” FiberSystems Int'l, pp. 29-32, Feb. 2002.
[20] A.V. Krishnamoorthy and K.W. Goossen, “Optoelectronic-VLSI: Photonic Integrated with VLSI Circuits,” IEEE J. Selected Topics in Quantum Electronics, vol. 4, pp. 899-912, 1998.
[21] A. Louri and A.K. Kodi, “Parallel Optical Interconnection Network for Address Transactions in Large-Scale Cache Coherent Symmetric Multiprocessors,” IEEE J. Selected Topics in Quantum Electronics, special issue on optical interconnects, vol. 9, pp. 667-676, 2003.
[22] D.J. Sorin, M. Plakal, A.E. Condon, M.D. Hill, M.M. Martin, and D.A. Wood, “Specifying and Verifying a Broadcast and a Multicast Snooping Cache Coherence Protocol,” IEEE Trans. Parallel and Distributed Systems, vol. 13, 2002.
[23] I. Ikodinovic, A. Milenkovic, V. Milutinovic, and D. Magdic, “Limes: A Multiprocessor Simulation Environment for PC Platforms,” Proc. Int'l Conf. Parallel Processing and Applied Math., Sept. 1999.

Index Terms:
SMPs, parallel optical interconnects, cache coherence, scalable optical networks.
Ahmed Louri, Avinash Karanth Kodi, "An Optical Interconnection Network and a Modified Snooping Protocol for the Design of Large-Scale Symmetric Multiprocessors (SMPs)," IEEE Transactions on Parallel and Distributed Systems, vol. 15, no. 12, pp. 1093-1104, Dec. 2004, doi:10.1109/TPDS.2004.75
Usage of this product signifies your acceptance of the Terms of Use.