This Article 
 Bibliographic References 
 Add to: 
The Offset Cube: A Three-Dimensional Multicomputer Network Topology Using Through-Wafer Optics
September 1998 (vol. 9 no. 9)
pp. 893-908

Abstract—Three-dimensional packaging technologies are critical for enabling ultra-compact, massively parallel processors (MPPs) for embedded applications. Through-wafer optical interconnect has been proposed as a useful technology for building ultra-compact MPPs since it provides a simplified mechanism for interconnecting stacked multichip substrates. This paper presents the offset cube, a new network topology designed to exploit the packaging benefits of through-wafer optical interconnect in ultra-compact MPP systems. We validate the offset cube's topological efficiency by developing deadlock-free adaptive routing protocols with modest virtual channel requirements (only two virtual channels per link needed for full adaptivity). A preliminary analysis of router complexity suggests these protocols can be efficiently implemented in hardware. We also present a 3D mesh embedding for the offset cube. Network simulations show the offset cube performs comparably to a bidirectional 3D mesh of equal size under uniform, hot-spot, and trace-driven traffic loads. While the offset cube is not proposed as a general replacement for the mesh topology, it leverages the benefits of through-wafer optical interconnect more effectively than a mesh by completely eliminating chip-to-chip wires for data signals. Hence, the offset cube is an effective topology for interconnecting ultra-compact MCM-level MPP systems.

[1] S. Abraham and K. Padmanabhan, "Performance of Multicomputer Networks under Pin-Out Constraints," J. Parallel and Distributed Computing, July 1991, pp. 237-248.
[2] A. Agarwal, "Limits on Interconnection Network Performance," IEEE Trans. Parallel and Distributed Systems, vol. 2, no. 4, pp. 398-412, Oct. 1991.
[3] K.V. Anjan and T.M. Pinkston, “An Efficient, Fully Adaptive Deadlock Recovery Scheme: Disha,” Proc. 22nd Ann. Int'l Symp. Computer Architecture, pp. 201-210, June 1995.
[4] S. Bandypadhyay, "Coupling and Cross-Talk Between High Speed Interconnects in Ultralarge Scale Integrated Circuits," IEEE J. Quantum Electronics, vol. 28, pp. 1,554-1,561, June 1992.
[5] J. Bristow, J. Lehman, M. Hibbs-Brenner, and Y. Liu, "Applying Optical Interconnects to Electronic Systems: Promise vs. Practicality," Proc. Fourth Int'l Conf. Massively Parallel Processing Using Optical Interconnections, pp. 54-60, 1997.
[6] K.H. Calhoun, C.B. Camperi-Ginestet, and N.M. Jokerst, "Vertical Optical Communication Through Stacked Silicon Wafers Using Hybrid Monolithic Thin Film InGaAsP Emitters and Detectors," IEEE Photonics Technology Letters, vol. 5, pp. 254-257, Feb. 1993.
[7] L.J. Camp, R. Sharma, and M.R. Feldman, "Guided-Wave and Free-Space Optical Interconnects for Parallel Processing Systems: A Comparison," Applied Optics., vol. 33, pp. 6,168-6,180, Sept.10 1994.
[8] L. Cheng and A.A. Sawchuck, "Three-Dimensional Omega Networks for Optical Implementation," Applied Optics., vol. 31, pp. 5,468-5,479, Sept.10 1992.
[9] A. Chien, "A Cost and Speed Model for k-ary n-Cube Wormhole Routers," Proc. Hot Interconnects, 1993.
[10] P. Cinato, K.C. Young, "Optical Interconnections Within Multichip Modules," Optical Eng., vol. 32, pp. 852-860, Apr. 1993.
[11] J.L. Cruz-Rivera, D.S. Wills, T.K. Gaylord, and E.N. Glytsis, "Optimal Usage of the Available Wiring Resources in Diffractive-Reflective Optoelectronic MCMs," Applied Optics., special issue on optical interconnections for massively parallel systems, Jan.10 1998.
[12] J.L. Cruz-Rivera, E.V.R. DiBella, D.S. Wills, T.K. Gaylord, and E.N. Glytsis, "Parallelized Formulation of the Maximum Likelihood Expectation Maximization Algorithm for Fine-Grain Message-passing Architectures," IEEE Trans. Medical Imaging, pp. 758-762, Dec. 1995.
[13] W.J. Dally, "Virtual-Channel Flow Control," IEEE Trans. Parallel and Distributed Systems, vol. 3, no. 2, pp. 194-205, Mar. 1992.
[14] W.J. Dally, "Performance Analysis of k-ary n-Cube Interconnection Networks," IEEE Trans. Computers, vol. 39, no. 6, pp. 775-785, June 1992.
[15] W.J. Dally and C.L. Seitz, “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks,” IEEE Trans. Computers, Vol. C-36, No. 5, May 1987, pp. 547-553.
[16] P. Dowd, K. Bogineni, K.A. Aly, and J. Perreault, "Hierarchical Scalable Photonic Architectures for High Performance Processor Interconnection," IEEE Trans. Computers, vol. 42, no. 9, pp. 1,105-1,120, Sept. 1993.
[17] J. Duato, "A New Theory of Deadlock-Free Adaptive Routing in Wormhole Networks," IEEE Trans. Parallel and Distributed Systems, vol. 4, no. 12, pp. 1,320-1,331, Dec. 1993.
[18] J. Duato, “A Necessary and Sufficient Condition for Deadlock-Free Adaptive Routing in Wormhole Networks,” IEEE Trans. Parallel and Distributed Systems, vol. 6, no. 10, pp. 1,055–1,067, Oct. 1995.
[19] R.C. Eden, "Capabilities of Normal Metal Electrical Interconnects for 3D MCM Electronic Packaging," Proc. SPIE, vol. 2,153, pp. 132-145, 1994.
[20] M.R. Feldman, S.C. Esener, C.C. Guest, and S.H. Lee, "Comparison Between Optical and Electrical Interconnects Based on Power and Speed Considerations," Applied Optics., vol. 27, pp. 1,742-1,751, May1 1988.
[21] M.R. Feldman, C.C. Guest, T.J. Drabik, and S.C. Esener, "Comparison Between Electrical and Free-Space Optical Interconnections for Fine-Grain Processor Arrays Based on Interconnect Density Capabilities," Applied Optics., vol. 28, pp. 3,820-3,829, Sept.15 1989.
[22] G.C. Fox, "Matrix Algorithms on a Hypercube I: Matrix Multiplication," Parallel Computing, vol. 4, 1987.
[23] C.J. Glass and L.M. Ni, "The Turn Model for Adaptive Routing," Proc. 19th Int'l Symp. Computer Architecture, vol. 20, no. 2, pp. 278-287, May 1992.
[24] L.A. Hornak and S.K. Tewksbury, "On the Feasibility of Through-Wafer Optical Interconnects for Hybrid Wafer-Scale-Integrated Architectures," IEEE Trans. Electronic Devices, vol. 34, pp. 1,557-1,563, July 1987.
[25] F. Kiamilev, P. Marchand, A. Krishnamoorthy, S. Esener, and S. Lee, “Performance Comparison between Optoelectronic and VLSI Multistage Interconnection Networks,” J. Lightwave Technology, vol. 9, no. 12, pp. 1,674–1,692, Dec. 1991.
[26] J.H. Kim, Z. Liu, and A.A. Chien, "Compressionless Routing: A Framework for Adaptive and Fault Tolerant Routing," Proc. 21st Ann. Int'l Symp. Computer Architecture, pp. 289-300, Apr. 1994.
[27] W.S. Lacy, "Preliminary Analysis of Offset Cube Router Complexity," technical report, Dept. of Computer Systems Eng., Univ. of Arkansas, Fayetteville, Dec. 1997.
[28] R.A. Lee, W.A. Moreno, R.A. Gassman, and D. Miller, "Excimer vs. Nd:YAG Laser Creation Of Silicon Vias for 3D Interconnects," Proc. 13th IEEE/CHMT Int'l Electronics Manufacturing Technology Symp., pp. 358-360, 1992.
[29] C.S. Li, C.M. Olsen, and D.G. Messerschmitt, "Analysis of Crosstalk Penalty in Dense Optical Chip Interconnects Using Single-Mode Waveguides," J. Lightwave Technology, vol. 9, pp. 1,693-1,701, Dec. 1991.
[30] Y. Li, B. Ha, T. Wang, S. Wang, A. Katz, X.J. Lu, and E. Kanterakis, "Ring-Array Processor Distribution Topology for Optical Interconnects," Applied Optics., vol. 31, pp. 5,548-5,558, Sept.10 1992.
[31] Y. Li, T. Wang, and J. Sharony, "Free-Space Optical Interconnects Using Connectivity-Enhanced, Mesh-Based Networks," Optical Eng., vol. 33, pp. 1,532-1,542, May 1994.
[32] S. Linder, H. Baltes, F. Gnaedinger, and E. Doering, "Fabrication Technology for Wafer Through-Hole Interconnections and Three-Dimensional Stacks of Chips and Wafers," Proc. IEEE Micro Electro Mechanical Systems, pp. 349-54, 1994.
[33] M.J. Little, R.D. Etchells, J. Grinberg, S.P. Laub, J.G. Nash, and M.W. Yung, "The 3-D Computer," Proc. Int'l Conf. Wafer Scale Integration, pp. 55-64, 1989.
[34] A. Louri and H. Sung, "An Optical Multi-Mesh Hypercube: A Scalable Optical Interconnection Network for Massively Parallel Computing," IEEE J. Lightwave Technology, vol. 12, pp. 704-716, Apr. 1994.
[35] P. May, M. Lee, S.T. Wilkinson, O. Vendier, Z. Ho, S.W. Bond, D.S. Wills, M. Brooke, N.M. Jokerst, and A. Brown, “A 100 Mbps, LED Through-Wafer Optoelectronic Link for Multicomputer Interconnection Networks,” J. Parallel and Distributed Computing, vol. 41, no. 1 pp. 3-19, Feb. 1997.
[36] D. Miyazaki, S. Masuda, and K. Matsushita, "Self-Alignment With Optical Microconnectors for Free-Space Optical Interconnections," Applied Optics, vol. 37, pp. 228-232, Jan.10 1998.
[37] J. Nguyen, J. Pezaris, G. Pratt, and S. Ward, “Three-Dimensional Network Topologies,” Proc. Int'l Workshop Parallel Computer Routing and Comm., pp. 101-115, May 1994.
[38] S.T. Obenaus et al., "Embedding Star Graphs into Optical Meshes Without Bends," J. Parallel and Distributed Computing, Aug. 1997.
[39] T.M. Pinkston and J.W. Goodman, "Design of an Optical Reconfigurable Shared-Bus-Hypercube Interconnect," Applied Optics, vol. 33, pp. 1,434-1,443, Mar.10 1994.
[40] M. Raksapatcharawong and T.M. Pinkston, "An Optical Interconnect Model for k-ary n-Cube Wormhole Networks," Proc. 10th Int'l Parallel Processing Symp., pp. 666-672, Apr. 1996.
[41] B. Robertson, "Design of a Compact Alignment Tolerant Optical Interconnect for Photonic Backplane Applications," Proc. Fourth Int'l Conf. Massively Parallel Processing Using Optical Interconnections, pp. 68-77, 1997.
[42] T. Sakano, T. Matsumoto, and K. Noguchi, "Three-Dimensional Board-To-Board Free-Space Optical Interconnects and Their Application to the Prototype Processor System:COSINE III," Applied Optics, vol. 34, pp. 1,815-1,822, 1995.
[43] S. L. Scott, J.R. Goodman, The Impact of Pipelined Channels on K-Ary N-Cube Networks IEEE Trans. Parallel and Distributed Systems, vol. 5, no. 1, pp. 2-16, Jan. 1994.
[44] J.M. Segelken, L.J. Wu, M.Y. Lau, K.L. Tai, R.R. Shively, and T.G. Grau, "Ultra-Dense: An MCM-Based 3-D Digital Signal Processor," IEEE Trans. Components, Hybrids, and Manufacturing Technology, vol. 15, no. 4, pp. 438-443, Aug. 1992.
[45] S.E. Schacham, H. Merkelo, L.-T. Hwang, B.D. McCredie, M.S. Veatch, and I. Turlik, "Comparative Evaluation of Optical Waveguides as Alternative Interconnection for High-Performance Packaging," IEEE Trans. Components, Hybrids, and Manufacturing Technology, vol. 15, pp. 63-72, Feb. 1992.
[46] T. Szymanski and H.S. Hinton, "Design of a Terabit Free-Space Photonic Backplane for Parallel Computing," Proc. Second Int'l Conf. Massively Parallel Processing Using Optical Interconnections, pp. 16-27, 1995.
[47] S. Tang, R.T. Chen, and M.A. Peskin, "Packaging Density and Interconnection Length of a Highly Parallel Optical Interconnect Using Polymer-Based, Single-Mode Bus Arrays," Optical Eng., vol. 33, pp. 1,581-1,586, May 1994.
[48] S.K. Tewksbury, Wafer-Level Integrated Systems: Implementation Issues, chapter 12. Boston: Kluwer Academic, 1989.
[49] D.Z. Tsang and T.J. Goblick, "Free-Space Optical Interconnection Technology in Parallel Processing Systems," Optical Eng., vol. 33, pp. 1,524-1,531, May 1994.
[50] G.K. Wallace, "Overview of the JPEG (ISO/CCITT) Still Image Compression Standard," SPIE Advent Technology Series, vol. 1, pp. 358-371, 1990.
[51] D.S. Wills, H.H. Cat, J.C. Cruz-Rivera, W.S. Lacy, J.M. Baker, J.C. Eble, A.L. Lopez-Lagunas, and M. Hopper, “High-Throughput, Low-Memory Applications on the Pica Architecture,” IEEE Trans. Parallel and Distributed Systems, vol. 8, no. 10, pp. 1055-1067, Oct. 1997.
[52] D.S. Wills, W.S. Lacy, C. Camperi-Ginestet, B. Buchanan, H.H. Cat, S. Wilkinson, M. Lee, N.M. Jokerst, and M.A. Brooke, "A Three-Dimensional High-Throughput Architecture Using Through-Wafer Optical Interconnect," IEEE J. Lightwave Technology, vol. 13, no. 6, pp. 1,085-1,092, June 1995.
[53] D.S. Wills, W.S. Lacy, and J.L. Cruz-Rivera, "The Offset Cube: An Optoelectronic Interconnection Network," Parallel Computer Routing and Communication, K. Bolding and L. Synder, eds., vol. 853, pp. 86-100, 1994.
[54] S.-M. Moon,S. Carson,, and A. Agrawala,“Hardware implementation of a general multi-way jump mechanism,” Proc. 23rd Ann. Symp. Microarchitecture (Micro-23), pp. 38-45, Dec. 1990.
[55] G.I. Yayla, P.J. Marchand, and S.C. Esener, "Speed and Energy Analysis of Digital Interconnections: Comparison of On-Chip, Off-Chip, and Free-Space Technologies," Applied Optics, vol. 37, pp. 205-227,10 Jan. 1998.

Index Terms:
MPP networks, ultra-compact systems, offset cube, 3D mesh, adaptive routing, deadlock freedom, 3D packaging, optical interconnect, through-wafer signaling.
W. Stephen Lacy, José L. Cruz-Rivera, D. Scott Wills, "The Offset Cube: A Three-Dimensional Multicomputer Network Topology Using Through-Wafer Optics," IEEE Transactions on Parallel and Distributed Systems, vol. 9, no. 9, pp. 893-908, Sept. 1998, doi:10.1109/71.722222
Usage of this product signifies your acceptance of the Terms of Use.