High-radix crossbar switches enabled by proximity communication

ABSTRACT

We describe a novel way to implement high-radix crossbar switches. Our work is enabled by a new chip interconnect technology called Proximity Communication (PxC) that offers unparalleled chip IO density. First, we show how a crossbar architecture is topologically mapped onto a PxC-enabled multi-chip module (MCM). Then, we describe a first prototype implementation of a small-scale switch based on a PxC MCM. Finally, we present a performance analysis of two large-scale switch configurations with 288 ports and 1,728 ports, respectively, contrasting a 1-stage PxC-enabled switch and a multi-stage switch using conventional technology. Our simulation results show that (a) arbitration delays in a large 1-stage switch can be considerable, (b) multi-stage switches are extremely susceptible to saturation under non-uniform traffic, a problem that becomes worse for higher radices (1-stage switches, in contrast, are not affected by this problem).

REFERENCES

• 1: François Abel , Cyriel Minkenberg , Ronald P. Luijten , Mitchell Gusat , Ilias Iliadis, 10 A Four-Terabit Packet Switch Supporting Long Round-Trip Times, IEEE Micro, v.23 n.1, p.10-24, January 2003 [doi>10.1109/MM.2003.1179894]
• 2: ATCA Web Site, www.advancedtca.org.
• 3: V. Benes, Mathematical Theory of Connecting Networks and Telephone Traffic, New York, Academic Press, 1965.
• 4: C. Clos, "A Study of Non-blocking Switching Networks," Bell System Technical Journal, vol. 32, no. 5, pp. 406--424, March 1953.
• 5: R. Drost, R. Hopkins, I. Sutherland, "Proximity Communication," IEEE Custom Integrated Circuits Conference, pp. 469--472, Sep. 2003.
• 6: R. Drost, R. Hopkins, I. Sutherland, "Electronic Alignment for Proximity Communication," IEEE Solid-State Circuits Conference, pp. 144--518, Feb. 2004.
• 7: J. Duato, J. Flich, T. Nachiondo, "Cost-Effective Technique to Reduce HOL-blocking in Single-Stage and Multistage Switch Fabrics," Euromicro Conference on Parallel, Distributed and Network-based Processing, pp. 48--53, Feb. 2004.
• 8: Fulcrum Web Site, http://www.fulcrummicro.com/products/focalpoint/fm2224.htm.
• 9: C. Gomez, F. Gilabert, M. E. Gomez, P. Lopez and J. Duato, "Deterministic versus Adaptive Routing in Fat-Trees," Workshop on Communication Architecture for Clusters, pp. 1--8, March 2007.
• 10: D. Hopkins, A. Chow, R. Bosnyak, B. Coates, J. Ebergen, S. Fairbanks, J. Gainsley, R. Ho, J. Lexau, F. Liu, T. Ono, J. Schauer, I. Sutherland, R. Drost, "Circuit Techniques to Enable 430Gb/s/mm² Proximity Communication," IEEE International Solid-State Circuits Conference, pp. 368--369, Feb. 2007.
• 11: John Kim , William J. Dally , Brian Towles , Amit K. Gupta, Microarchitecture of a High-Radix Router, Proceedings of the 32nd annual international symposium on Computer Architecture, p.420-431, June 04-08, 2005
• 12: Nick McKeown , Martin Izzard , Adisak Mekkittikul , William Ellersick , Mark Horowitz, Tiny Tera: A Packet Switch Core, IEEE Micro, v.17 n.1, p.26-33, January 1997 [doi>10.1109/40.566194]
• 13: Mellanox Web Site, http://www.mellanox.com/products/switch\_silicon.php.
• 14: G. Mora , J. Flich , J. Duato , P. López , E. Baydal , O. Lysne, Towards an efficient switch architecture for high-radix switches, Proceedings of the 2006 ACM/IEEE symposium on Architecture for networking and communications systems, December 03-05, 2006, San Jose, California, USA [doi>10.1145/1185347.1185350]
• 15: W. Olesinski, H. Eberle, N. Gura, "OBIG: the Architecture of an Output Buffered Switch with Input Groups for Large Switches," IEEE GLOBECOM 2007, Nov. 2007.
• 16: W. Olesinski, H. Eberle, N. Gura: "PWWFA: The Parallel Wrapped Wave Front Arbiter for Large Switches," IEEE Workshop on High Performance Switching and Routing, May/June 2007.
• 17: Wladek Olesinski , Nils Gura , Hans Eberle , Andres Mejia, Low-latency scheduling in large switches, Proceedings of the 3rd ACM/IEEE Symposium on Architecture for networking and communications systems, December 03-04, 2007, Orlando, Florida, USA [doi>10.1145/1323548.1323566]
• 18: Steve Scott , Dennis Abts , John Kim , William J. Dally, The BlackWidow High-Radix Clos Network, Proceedings of the 33rd annual international symposium on Computer Architecture, p.16-28, June 17-21, 2006
• 19: Y. Tamir , H. C. Chi, Symmetric Crossbar Arbiters for VLSI Communication Switches, IEEE Transactions on Parallel and Distributed Systems, v.4 n.1, p.13-27, January 1993 [doi>10.1109/71.205650]
• 20: Sun Microsystems, Inc., "The Path to Massive Scale High Performance Computing," www.sun.com/products/networking/datacenter/ds3456.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.2 Multiple Data Stream Architectures (Multiprocessors)
          Subjects: Interconnection architectures (e.g., common bus, multiport memory, crossbar switch)

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.3 Interconnections (subsystems)

  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Experimentation, Measurement, Performance