Viable opto-electronic HPC interconnect fabrics

ABSTRACT

We address the problem of how to exploit optics for ultrascale High Performance Computing interconnect fabrics. We show that for high port counts these fabrics require multistage topologies regardless of whether electronic or optical switch components are used. Also, per stage electronic buffers remain indispensable for maintaining throughput, lossless-ness and packet sequence. Although the notion of true all-optical packet switching is not yet viable, we show that appropriate use of optical switching technology offers power and scaling advantages that can be leveraged economically, and propose a hybrid opto-electronic HPC interconnect fabric architecture that combines the strength of electronics in processing and storing information with the strength of optics in switching and transporting high bandwidths. Using Semiconductor Optical Amplifier technology, we are building a prototype demonstrator switch that we believe solves all the technical challenges. Having reached this threshold now enables commercialization of this technology, which we are currently pursuing.

REFERENCES

• 1: [1] David A. B. Miller, "Rationale and challenges for optical interconnects to electronic chips," Proc. IEEE 88 (6), 728-749 (2000).
• 2: [2] A. Neukermans and R. Ramaswami, "MEMS technology for optical networking applications," IEEE Commun. Mag. 39 (1), 2001.
• 3: [3] T. Goh, M. Yasu, K. Hattori, A. Himeno, M. Okuno and Y. Ohmori, "Low loss and high extension ratio strictly nonblocking 16×16 thermooptic matrix switch on 6-in wafer using silica based planer lightwave circuit technology," J. Lightwave Technol., Vol. 19, no. 3, Mar. 2001.
• 4: [4] E. Shekel, A. Feingold, Z. Fradkin, A. Geron, J. Levy, G. Matmon, D. Majer, E. Rafaely, M. Rudman, G. Tidhar, J. Vecht and S. Ruschin, "64 × 64 fast optical switching module," Proc. Optical Fiber Communication Conference and Exhibit, 2002. OFC 2002, pp. 27-29.
• 5: [5] F. Masetti, D. Chiaroni, R. Dragnea, R. Robotham and D. Zriny, "High-speed high-capacity packet-switching fabric: a key system for required flexibility and capcity," J. Opt. Networking 2 (7), July 2003.
• 6: [6] S. Araki, Y. Suemura, N. Henmi, Y. Maeno, A. Tajima and S. Takahashi, "Highly scalable optoelectronic packet-switching fabric based on wavelength-division and space-division optical switch architecture fof use in the photonic core node," J. Opt. Networking 2 (7), July 2003.
• 7: [7] J. Gripp, M. Duelk, J.E. Simsarian, A. Bhardwaj, P. Bernasconi, O. Laznicka and M. Zirngibl, "Optical switch fabrics for ultra-high-capacity IP routers," J. Lightwave Techn. 21 (11), 2839-2850, Nov. 2003.
• 8: [8] A. Von Lehmen, G. Clapp, J.W. Gannett, H. Kobrinski and R.A. Skoog, "Data in the optical domain technology: A network-level perspective," Proc. 17th Annual Meeting of the IEEE Lasers and Electro-Optics Society, LEOS 2004., Vol. 1, Nov. 2004.
• 9: [9] A. Willner, "Physical impairements and network limitations when interconnecting multiple DOD-N routers," Proc. 17th Annual Meeting of the IEEE Lasers and Electro-Optics Society, LEOS 2004. Vol. 1, Nov. 2004.
• 10: [10] Q. Yang and K. Bergman, "Performances of the Data Vortex switch architecture under non-uniform and bursty traffic," IEEE/OSA J. Lightwave Technol. 20 (8), 1242-1247, Aug. 2002.
• 11: [11] C. Minkenberg, "Work-conservingness of CIOQ packet switches with limited output buffers," IEEE Commun. Lett. 6 (10), 452-454, Oct. 2002.
• 12: [12] The 24th TOP500 List, released November 8, 2004, during SC2004 in Pittsburgh, PA, http://www.top500.org
• 13: François Abel , Cyriel Minkenberg , Ronald P. Luijten , Mitchell Gusat , Ilias Iliadis, 10 A Four-Terabit Packet Switch Supporting Long Round-Trip Times, IEEE Micro, v.23 n.1, p.10-24, January 2003 [doi>10.1109/MM.2003.1179894 ]
• 14: [14] S. Noda, "Photonic nanostructures and devices based on photonic crystals," Proc. 7th Annual Meeting of the IEEE Lasers and Electro-Optics Society, LEOS 2004, Vol. 1, Nov. 2004.
• 15: [15] M.T. Hill, H.J.S. Dorren, T. de Vries, X.J.M. Leijtens, J.H. den Besten, B. Smalbrugge, Y.-S. Oei, H. Binsma, G.-D. Khoe and M.K. Smit, "A fast low-power optical memory based on coupled micro-ring lasers," Nature 432, 2004.
• 16: [16] C.B. Stunkel and P.H. Hochschild, "SP2 high-performance switch architecture," Proc. Hot Interconnects II, pp. 190-195, Aug. 1994.
• 17: [17] N. McKeown, V.T. Anantharam and J. Walrand, "Achieving 100% throughput in an input-queued switch," in Proc. IEEE Infocom '96, vol. 1, pp. 296-302, San Francisco, Mar. 1996.
• 18: [18] C. Minkenberg, "Performance of i-SLIP scheduling with large round-trip latency," Proc. Workshop on High Performance Switching and Routing HPSR 2003, pp. 49-54, Turin, Italy, June 2003.
• 19: [19] C. Minkenberg, F. Abel, M. Gusat, "Reliable control protocol for crossbar arbitration," IEEE Commun. Lett. 9 (2), 178-180, Feb. 2005.
• 20: [20] P. Mueller, R. Luijten and U. Bapst, "Hierarchical system synchronization and signaling for high-performance - low-latency interconnects," IEEE Electro/Information Technology Conf. "EIT 2005," Lincoln, NE, May 22-25, 2005.
• 21: [21] R. Hemenway, R. Grzybowski, C. Minkenberg, and R. Luijten, "Optical-packet-switched interconnect for supercomputer applications," OSA J. Opt. Network. 3 (12), 900-913, Dec. 2004.
• 22: [22] C. Minkenberg, I. Iliadis, F. Abel, "Low-Latency Pipelined Crossbar Arbitration," Proc. IEEE Globecom 2004, paper GE15-2, Dallas, TX, Nov. 2004.
• 23: Cyriel Minkenberg , Francois Abel , Peter Muller , Raj Krishnamurthy , Mitchell Gusat , B. Roe Hemenway, Control Path Implementation for a Low-Latency Optical HPC Switch, Proceedings of the 13th Symposium on High Performance Interconnects, p.29-35, August 17-19, 2005 [doi>10.1109/CONECT.2005.15]
• 24: [24] C. Chang; D. Lee; C. Yue, "Providing guaranteed rate services in the load balanced Birkhoff-von Neumann switches", INFOCOM 2003, Volume 3, 30 March-3 April 2003.
• 25: [25] C.K. Yow, Y.J. Chai, C.W. Tee, R. McDougall, R.V. Penty and I.H. White, "All-optical multiwavelength bypass-exchange switching using a hybrid-integrated Mach-Zehnder switch," Proc. ECOC 2004, Paper WE4P118, Stockholm, Sweden, Sept. 2004.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.2 Multiple Data Stream Architectures (Multiprocessors)
          Subjects: Interconnection architectures (e.g., common bus, multiport memory, crossbar switch)

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Circuit-switching networks
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Experimentation, Measurement, Performance

Keywords:
HPC, Interconnect, Switching, Optical Switching