The Cray BlackWidow: a highly scalable vector multiprocessor

ABSTRACT

This paper describes the system architecture of the Cray BlackWidow scalable vector multiprocessor. The BlackWidow system is a distributed shared memory (DSM) architecture that is scalable to 32K processors, each with a 4-way dispatch scalar execution unit and an 8-pipe vector unit capable of 20.8 Gflops for 64-bit operations and 41.6 Gflops for 32-bit operations at the prototype operating frequency of 1.3 GHz. Global memory is directly accessible with processor loads and stores and is globally coherent. The system supports thousands of outstanding references to hide remote memory latencies, and provides a rich suite of built-in synchronization primitives. Each BlackWidow node is implemented as a 4-way SMP with up to 128 Gbytes of DDR2 main memory capacity. The system supports common programming models such as MPI and OpenMP, as well as global address space languages such as UPC and CAF. We describe the system architecture and microarchitecture of the processor, memory controller, and router chips. We give preliminary performance results and discuss design tradeoffs.

REFERENCES

• 1: Dennis Abts , Steve Scott , David J. Lilja, So Many States, So Little Time: Verifying Memory Coherence in the Cray X1, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.11.2, April 22-26, 2003
• 2: Christopher Batten , Ronny Krashinsky , Steve Gerding , Krste Asanovic, Cache Refill/Access Decoupling for Vector Machines, Proceedings of the 37th annual IEEE/ACM International Symposium on Microarchitecture, p.331-342, December 04-08, 2004, Portland, Oregon [doi>10.1109/MICRO.2004.9]
• 3: Christian Bell , Wei-Yu Chen , Dan Bonachea , Katherine Yelick, Evaluating support for global address space languages on the Cray X1, Proceedings of the 18th annual international conference on Supercomputing, June 26-July 01, 2004, Malo, France [doi>10.1145/1006209.1006236]
• 4: C. Clos. A Study of Non-Blocking Switching Networks. The Bell System technical Journal, 32(2):406--424, March 1953.
• 5: Condensed results for HPCC Challenge Benchmarks. http://icl.cs.utk.edu/hpcc/hpcc_results.cgi
• 6: Cray XI. http://www.cray.com/products/xl/.
• 7: Cray XT3. http://www.cray.com/products/xt3/.
• 8: Cray XT4. http://www.cray.com/products/xt4/.
• 9: Roger Espasa , Federico Ardanaz , Joel Emer , Stephen Felix , Julio Gago , Roger Gramunt , Isaac Hernandez , Toni Juan , Geoff Lowney , Matthew Mattina , André Seznec, Tarantula: a vector extension to the alpha architecture, Proceedings of the 29th annual international symposium on Computer architecture, May 25-29, 2002, Anchorage, Alaska
• 10: Roger Espasa , Mateo Valero, A Simulation Study of Decoupled Vector Architectures, The Journal of Supercomputing, v.14 n.2, p.124-152, Sept. 1999 [doi>10.1023/A:1008158808410]
• 11: HPCC Challenge Benchmarks. http://icl.cs.utk.edu/hpcc/.
• 12: Intel Core2 Duo. http://www.cray.com/products/xdl/.
• 13: A. Johnston. Scaling and Technology Issues for Soft Error Rates. In Proceedings of the 4th Annual Research Conference on Reliability, Stanford, CA, October 2000.
• 14: John Kim , William J. Dally , Brian Towles , Amit K. Gupta, Microarchitecture of a High-Radix Router, Proceedings of the 32nd annual international symposium on Computer Architecture, p.420-431, June 04-08, 2005
• 15: Shubhendu S. Mukherjee , Joel Emer , Steven K. Reinhardt, The Soft Error Problem: An Architectural Perspective, Proceedings of the 11th International Symposium on High-Performance Computer Architecture, p.243-247, February 12-16, 2005 [doi>10.1109/HPCA.2005.37]
• 16: NEC SX-8 Vector supercomputer. http://www.nec.co.jp/press/en/0410/2001.html.
• 17: Leonid Oliker , Jonathan Carter , Michael Wehner , Andrew Canning , Stephane Ethier , Art Mirin , David Parks , Patrick Worley , Shigemune Kitawaki , Yoshinori Tsuda, Leading Computational Methods on Scalar and Vector HEC Platforms, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.62, November 12-18, 2005 [doi>10.1109/SC.2005.41]
• 18: Steve Scott , Dennis Abts , John Kim , William J. Dally, The BlackWidow High-Radix Clos Network, Proceedings of the 33rd annual international symposium on Computer Architecture, p.16-28, June 17-21, 2006
• 19: S. Scott and A. Bataineh. U.S. Patent: Optimized high-bandwidth cache coherence mechanism, http://www.patentstorm.us/patents/7082500.html. 2006.
• 20: Hongzhang Shan , Erich Strohmaier, Performance characteristics of the Cray X1 and their implications for application performance tuning, Proceedings of the 18th annual international conference on Supercomputing, June 26-July 01, 2004, Malo, France [doi>10.1145/1006209.1006235]
• 21: James E. Smith, Decoupled access/execute computer architectures, Proceedings of the 9th annual symposium on Computer Architecture, p.112-119, April 26-29, 1982, Austin, Texas, United States
• 22: T. H. Dunigan, Jr. , M. R. Fahey , J. B. White III , P. H. Worley, Early Evaluation of the Cray X1, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.18, November 15-21, 2003

INDEX TERMS

General Terms:
Design, Performance

Keywords:
MPP, architecture, distributed shared memory, fat-tree, high-radix, multiprocessor, shared memory, vector