Effects of architectural and technological advances on the HP/Convex Exemplar's memory and communication performance

ABSTRACT

Advances in microarchitecture, packaging, and manufacturing processes enable designers to build new systems with higher performance and scalability. Using microbenchmark techniques, we contrast the memory and communication performance of two generations of the HP/Convex Exemplar scalable parallel processing system. The SPP1000 and SPP2000 have significant architectural and implementation differences, but maintain upward binary compatibility. The SPP2000 employs manufacturing and packaging advances to obtain shorter system interconnects with wider data paths and improved functionality, thereby reducing the latency and increasing the bandwidth of remote communication. Although the memory latency is not significantly improved, newer out-of-order execution processors coupled with nonblocking caches achieve much higher memory bandwidth. The SPP2000 has a richer system interconnect topology that allows scalability to a larger number of processors. The SPP2000 also employs innovations in its coherence protocols to improve synchronization and communication performance. This paper characterizes the performance effects of these changes, and identifies some remaining inefficiencies, in the cache coherence protocol and the node configuration, that future systems should address.

REFERENCES

• 1: G. Abandah and E. Davidson. A Comparative Study of Cache-Coherent Nonunitbrm Memory Access Systems. In High Performance Computing Systems and Applications. Kluwer Academic Publishers, May 1998. 12th Ann. lnt'l Symp. High Performance Computing Systems and Applications (HPCS'98).
• 2: Gheith A. Abandah , Edward S. Davidson, Characterizing Distributed Shared Memory Performance: A Case Study of the Convex SPP1000, IEEE Transactions on Parallel and Distributed Systems, v.9 n.2, p.206-216, February 1998 [doi>10.1109/71.663946]
• 3: Tom Asprey , Gregory S. Averill , Eric DeLano , Russ Mason , Bill Weiner , Jeff Yetter, Performance Features of the PA7100 Microprocessor, IEEE Micro, v.13 n.3, p.22-35, May 1993 [doi>10.1109/40.216746]
• 4: G. Astfalk and T. Brewer. An Overview of the HP/Convex Exemplar Hardware. Tech. paper, Hewlett-Packard Co., June 1997. http://www.hp.com/wsg/tech/technical.html.
• 5: G. Astfalk, T. Brewer, and G. Palmer. Cache Coherence in the Convex MPP. Tech. paper, Hewlett-Packard Co., Feb. 1994. http://www.hp.com/wsg/tech/technical.html.
• 6: T. Brewer, A highly scalable system utilizing up to 128 PA-RISC processors, Proceedings of the 40th IEEE Computer Society International Conference, p.133, March 05-09, 1995
• 7: T. Brewer , G. Astfalk, The evolution of the HP/Convex Exemplar, Proceedings of the 42nd IEEE International Computer Conference, p.81, February 23-26, 1997
• 8: W. Bryg, K. Chan, and N. Fiduccia. A High- Performance, Low-Cost Multiprocessor Bus for Workstations and Midrange Servers. HewIett-Packard J., 47(1 ):18- 24, Feb. 1996.
• 9: David Chaiken , Craig Fields , Kiyoshi Kurihara , Anant Agarwal, Directory-Based Cache Coherence in Large-Scale Multiprocessors, Computer, v.23 n.6, p.49-58, June 1990 [doi>10.1109/2.55500]
• 10: David E. Culler , Anoop Gupta , Jaswinder Pal Singh, Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 11: K. Gallivan , D. Gannon , W. Jalby , A. Malony , H. Wijshoff, Experimentally Characterizing the Behavior of Multiprocessor Memory Systems: A Case Study, IEEE Transactions on Software Engineering, v.16 n.2, p.216-223, February 1990 [doi>10.1109/32.44384]
• 12: Cristina Hristea , Daniel Lenoski , John Keen, Measuring memory hierarchy performance of cache-coherent multiprocessors using micro benchmarks, Proceedings of the 1997 ACM/IEEE conference on Supercomputing (CDROM), p.1-12, November 15-21, 1997, San Jose, CA [doi>10.1145/509593.509638]
• 13: D. Hunt, Advanced performance features of the 64-bit PA-8000, Proceedings of the 40th IEEE Computer Society International Conference, p.123, March 05-09, 1995
• 14: IEEE Computer Society. IEEE Standard for Scalable Coherent Interface (SCI), Aug. 1993. IEEE Std 1596-1992.
• 15: Stefanos Kaxiras , James R. Goodman, The GLOW cache coherence protocol extensions for widely shared data, Proceedings of the 10th international conference on Supercomputing, p.35-43, May 25-28, 1996, Philadelphia, Pennsylvania, United States [doi>10.1145/237578.237583]
• 16: James Laudon , Daniel Lenoski, The SGI Origin: a ccNUMA highly scalable server, Proceedings of the 24th annual international symposium on Computer architecture, p.241-251, June 01-04, 1997, Denver, Colorado, United States
• 17: Daniel E. Lenoski , Wolf-Dietrich Weber, Scalable Shared-Memory Multiprocessing, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1994
• 18: L. McVoy and C. Staelin. lmbench: Portable Tools for Per.. formance Analysis. In Proc. USENIX'96 Ann. Technical Conf., pages 279-294, Jan. 1996.
• 19: R. H. Saavedra , R. S. Gaines , M. J. Carlton, Micro benchmark analysis of the KSR1, Proceedings of the 1993 ACM/IEEE conference on Supercomputing, p.202-213, December 1993, Portland, Oregon, United States [doi>10.1145/169627.169695]
• 20: K. Shaw and G. Astfalk. Four-State Cache- Coherence in the Convex Exemplar System. Internal memo, Convex Computer Corp., Oct. 1995. http://www'hp'c~m/wsg/tech/technical'html"
• 21: SPEC CPU95 Benchmarks Results. See the Standard Performance Evaluation Corp., web page http://www.spec.org/.
• 22: Zheng Zhang , Josep Torrellas, Reducing Remote Conflict Misses: NUMA with Remote Cache versus COMA, Proceedings of the 3rd IEEE Symposium on High-Performance Computer Architecture, p.272, February 01-05, 1997

INDEX TERMS

Primary Classification:
  B. Hardware
  B.3 MEMORY STRUCTURES
      B.3.2 Design Styles
          Subjects: Shared memory

Additional Classification:
  B. Hardware
  B.8 Performance and Reliability

  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES