The rise and fall of High Performance Fortran: an historical object lesson

ABSTRACT

High Performance Fortran (HPF) is a high-level data-parallel programming system based on Fortran. The effort to standardize HPF began in 1991, at the Supercomputing Conference in Albuquerque, where a group of industry leaders asked Ken Kennedy to lead an effort to produce a common programming language for the emerging class of distributed-memory parallel computers. The proposed language would focus on data-parallel operations in a single thread of control, a strategy which was pioneered by some earlier commercial and research systems, including Thinking Machines' CM Fortran, Fortran D, and Vienna Fortran.

The standardization group, called the High Performance Fortran Forum (HPFF), took a little over a year to produce a language definition that was published in January 1993 as a Rice technical report [50] and, later that same year, as an article in Scientific Programming [49].

The HPF project had created a great deal of excitement while it was underway and the release was initially well received in the community. However, over a period of several years, enthusiasm for the language waned in the United States, although it has continued to be used in Japan.

This paper traces the origins of HPF through the programming languages on which it was based, leading up to the standardization effort. It reviews the motivation underlying technical decisions that led to the set of features incorporated into the original language and its two follow-ons: HPF 2 (extensions defined by a new series of HPFF meetings) and HPF/JA (the dialect that was used by Japanese manufacturers and runs on the Earth Simulator).

A unique feature of this paper is its discussion and analysis of the technical and sociological mistakes made by both the language designers and the user community:, mistakes that led to the premature abandonment of the very promising approach employed in HPF. It concludes with some lessons for the future and an exploration of the influence of ideas from HPF on new languages emerging from the High Productivity Computing Systems program sponsored by DARPA.

REFERENCES

• 1: Jeanne C. Adams , Brian T. Smith , Jeanne T. Martin , Walter S. Brainerd , Jerrold L. Wagener, FORTRAN 95 Handbook, MIT Press, Cambridge, MA, 1997
• 2: Vikram S. Adve , John Mellor-Crummey , Mark Anderson , Jhy-Chun Wang , Daniel A. Reed , Ken Kennedy, An integrated compilation and performance analysis environment for data parallel programs, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.50-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224340]
• 3: Randy Allen , Ken Kennedy, Automatic translation of FORTRAN programs to vector form, ACM Transactions on Programming Languages and Systems (TOPLAS), v.9 n.4, p.491-542, Oct. 1987 [doi>10.1145/29873.29875]
• 4: Ken Kennedy , John R. Allen, Optimizing compilers for modern architectures: a dependence-based approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001
• 5: Alliant Computer Systems. http://en.wikipedia.org/wiki/Alliant-Computer-Systems.
• 6: Jennifer M. Anderson , Monica S. Lam, Global optimizations for parallelism and locality on scalable parallel machines, Proceedings of the ACM SIGPLAN 1993 conference on Programming language design and implementation, p.112-125, June 21-25, 1993, Albuquerque, New Mexico, United States
• 7: Françoise André , Jean-Louis Pazat , Henry Thomas, Pandore: a system to manage data distribution, Proceedings of the 4th international conference on Supercomputing, p.380-388, June 11-15, 1990, Amsterdam, The Netherlands
• 8: Applied Parallel Research, Sacramento, California. Forge High Performance Fortran xhpf User's Guide, Version 2.1, 1995.
• 9: G. H. Barnes, R. M. Brown, M. Kato, D. Kuck, D. Slotnick, and R. Stokes. The ILLIAC IV computer. IEEE Transactions on Computers, C-17:746--757, 1968.
• 10: Kenneth E. Batcher. The Multi-Dimensional Access Memory in STARAN. IEEE Transactions on Computers, C-26(2):174--177, February 1977.
• 11: Siegfried Benkner , Guy Lonsdale , Hans P. Zima, The HPF+ Project: Supporting HPF for Advanced Industrial Applications, Proceedings of the 5th International Euro-Par Conference on Parallel Processing, p.1155-1165, August 31-September 03, 1999
• 12: Siegfried Benkner , Hans Zima, Compiling high performance Fortran for distributed-memory architectures, Parallel Computing, v.25 n.13-14, p.1785-1825, Dec. 1999 [doi>10.1016/S0167-8191(99)00074-5]
• 13: John Bircsak , Peter Craig , RaeLyn Crowell , Zarka Cvetanovic , Jonathan Harris , C. Alexander Nelson , Carl D. Offner, Extending OpenMP for NUMA machines, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.48-es, November 04-10, 2000, Dallas, Texas, United States
• 14: A. Malony , B. Mohr , P. Beckman , D. Gannon , S. Yang , F. Bodin , S. Kesavan, Implementing a parallel C++ runtime system for scalable parallel systems, Proceedings of the 1993 ACM/IEEE conference on Supercomputing, p.588-597, December 1993, Portland, Oregon, United States [doi>10.1145/169627.169802]
• 15: Zeki Bozkus , Larry Meadows , Steven Nakamoto , Vincent Schuster , Mark Young, PGHPF—an optimizing High Performance Fortran compiler for distributed memory machines, Scientific Programming, v.6 n.1, p.29-40, Spring 1997
• 16: Michael Burke , Ron Cytron, Interprocedural dependence analysis and parallelization, Proceedings of the 1986 SIGPLAN symposium on Compiler construction, p.162-175, June 25-27, 1986, Palo Alto, California, United States
• 17: D. Callahan, Bradford L. Chamberlain, and Hans P. Zima. The Cascade High Productivity Language. In Ninth International Workshop on High-Level Parallel Programming Models and Supportive Environments (HIPS'04), pages 52--60, April 2004.
• 18: D. Callahan and K. Kennedy. Compiling Programs for Distributed-Memory Multiprocessors. Journal of Supercomputing, 2:151--169, October 1988.
• 19: Bradford Lee Chamberlain , Lawrence Snyder, The design and implementation of a region-based parallel programming language, 2001
• 20: Bradford L. Chamberlain, D. Callahan, and H. P. Zima. Parallel Programmability and the Chapel Language. The International Journal of High Performance Computing Applications, 2007. Special Issue on High Productivity Programming Languages and Models (in print).
• 21: B. Chapman, P. Mehrotra, and H. Zima. Programming in Vienna Fortran. Scientific Programming, 1(1):31--50, Fall 1992.
• 22: Philippe Charles , Christian Grothoff , Vijay Saraswat , Christopher Donawa , Allan Kielstra , Kemal Ebcioglu , Christoph von Praun , Vivek Sarkar, X10: an object-oriented approach to non-uniform cluster computing, Proceedings of the 20th annual ACM SIGPLAN conference on Object oriented programming, systems, languages, and applications, October 16-20, 2005, San Diego, CA, USA
• 23: Siddhartha Chatterjee , John R. Gilbert , Robert Schreiber , Shang-Hua Teng, Automatic array alignment in data-parallel programs, Proceedings of the 20th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.16-28, March 1993, Charleston, South Carolina, United States [doi>10.1145/158511.158517]
• 24: Daniel Chavarría-Miranda, Guohua Jin, and John Mellor-Crummey. Assessing the US Supercomputing Strategy: An Application Study Using IMPACT-3D. In Proceedings of the 19th International Parallel and Distributed Processing Symposium (IPDPS), Denver, CO, April 2005.
• 25: Daniel G. Chavarría-Miranda , John M. Mellor-Crummey, An Evaluation of Data-Parallel Compiler Support for Line-Sweep Applications, Proceedings of the 2002 International Conference on Parallel Architectures and Compilation Techniques, p.7-17, September 22-25, 2002
• 26: A. Choudhary, G. Fox, S. Ranka, S. Hiranandani, K. Kennedy, C. Koelbel, and J. Saltz. Software Support for Irregular and Loosely Synchronous Problems. International Journal of Computing Systems in Engineering, 3(2):43--52, 1993. (also available as CRPC-TR92258).
• 27: Peter Christy. Software to Support Massively Parallel Computing on the MasPar MP-1. In Proceedings of the IEEE Compcon, 1990.
• 28: Stephanie Coleman , Kathryn S. McKinley, Tile size selection using cache organization and data layout, Proceedings of the ACM SIGPLAN 1995 conference on Programming language design and implementation, p.279-290, June 18-21, 1995, La Jolla, California, United States
• 29: K. D. Cooper, M. W. Hall, K. Kennedy, and L. Torczon. Interprocedural Analysis and Optimization. Communications in Pure and Applied Mathematics, 48:947--1003, 1995.
• 30: Cray history. http://www.cray.com/about-cray/history.html.
• 31: Cray-1 Computer System. Hardware Reference Manual 224004. Technical report, Cray Research, Inc., November 1977. Revision C.
• 32: David E. Culler , Anoop Gupta , Jaswinder Pal Singh, Parallel Computer Architecture: A Hardware/Software Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 33: Leonardo Dagum , Ramesh Menon, OpenMP: An Industry-Standard API for Shared-Memory Programming, IEEE Computational Science & Engineering, v.5 n.1, p.46-55, January 1998 [doi>10.1109/99.660313]
• 34: F. Darema-Rogers, D. A. George, V. A. Norton, and G. F. Pfister. VM/EPEX---A VM Environment for Parallel Execution. Technical Report RC 11225(#49161), IBM T. J. & Watson Research Center, Yorktown Heights, New York, January 1985.
• 35: Luiz De Rose , David Padua, A MATLAB to Fortran 90 translator and its effectiveness, Proceedings of the 10th international conference on Supercomputing, p.309-316, May 25-28, 1996, Philadelphia, Pennsylvania, United States [doi>10.1145/237578.237627]
• 36: R. L. Diaconescu and H. P. Zima. An Approach to Data Distributions in Chapel. The International Journal of High Performance Computing Applications, 2006. Special Issue on High Productivity Programming Languages and Models (in print).
• 37: M. Flynn. Some Computer Organisations and their Effectiveness. Trans. Computers, 21:948--960, 1972.
• 38: G. Fox, S. Hiranandani, K. Kennedy, C. Koelbel, U. Kremer, C.-W. Tseng, and M. Wu. The Fortran D Language Specification. Technical Report TR90--141, Department of Computer Science, December 1990.
• 39: Dennis Gannon , William Jalby , Kyle Gallivan, Strategies for cache and local memory management by global program transformation, Journal of Parallel and Distributed Computing, v.5 n.5, p.587-616, October 1988 [doi>10.1016/0743-7315(88)90014-7]
• 40: Michael Gerndt. Automatic Parallelization for Distributed-Memory Multiprocessing Systems. PhD thesis, University of Bonn, Germany, 1989.
• 41: Wolfgang K. Giloi. SUPRENUM: A Trendsetter in Modern Supercomputer Development. Parallel Computing, pages 283--296, 1988.
• 42: W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitsberg, W. Saphir, and M. Snir. MPI---The Complete Reference: Volume 2, The MPI Extensions. Scientific and Engineering Computation Series. MIT Press, Cambridge, Massachusetts, September 1998.
• 43: Manish Gupta , Sam Midkiff , Edith Schonberg , Ven Seshadri , David Shields , Ko-Yang Wang , Wai-Mee Ching , Ton Ngo, An HPF compiler for the IBM SP2, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.71-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224422]
• 44: Manish Gupta, Automatic data partitioning on distributed memory multicomputers, University of Illinois at Urbana-Champaign, Champaign, IL, 1992
• 45: Lutz H. Hamel , Philip J. Hatcher , Michael J. Quinn, An optimizing C* compiler for a hypercube multicomputer, Languages, compilers and run-time environments for distributed memory machines, Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 1992
• 46: Jonathan Harris , John A. Bircsak , M. Regina Bolduc , Jill Ann Diewald , Israel Gale , Neil W. Johnson , Shin Lee , C. Alexander Nelson , Carl D. Offner, Compiling high performance Fortran for distributed-memory systems, Digital Technical Journal, v.7 n.3, p.5-23, 1995
• 47: Philip J. Hatcher , Anthony J. Lapadula , Robert R. Jones , Michael J. Quinn , Ray J. Anderson, A production-quality C* compiler for Hypercube multicomputers, Proceedings of the third ACM SIGPLAN symposium on Principles and practice of parallel programming, p.73-82, April 21-24, 1991, Williamsburg, Virginia, United States
• 48: John L. Hennessy , David A. Patterson, Computer architecture (2nd ed.): a quantitative approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1996
• 49: High Performance Fortran Forum. High Performance Fortran Language Specification. Scientific Programming, 2(1--2): 1--170, 1993. (also available as CRPC-TR92225).
• 50: High Performance Fortran Forum. High Performance Fortran Language Specification, version 1.0. Technical Report CRPC-TR92225, Rice University, Center for Research on Parallel Computation, Houston, Tex., 1993.
• 51: R. Hill. MIMDizer: A New Tool for Parallelization. Supercomputing Review, 3(4), April 1990.
• 52: W. Daniel Hillis, The connection machine, MIT Press, Cambridge, MA, 1989
• 53: Seema Hiranandani , Ken Kennedy , Chau-Wen Tseng, Compiling Fortran D for MIMD distributed-memory machines, Communications of the ACM, v.35 n.8, p.66-80, Aug. 1992 [doi>10.1145/135226.135230]
• 54: Seema Hiranandani , Ken Kennedy , Chau-Wen Tseng, Evaluation of compiler optimizations for Fortran D on MIMD distributed memory machines, Proceedings of the 6th international conference on Supercomputing, p.1-14, July 19-24, 1992, Washington, D. C., United States [doi>10.1145/143369.143372]
• 55: P. Husbands, C. Isbell, and A. Edelman. MATLAB*P: A Tool for Interactive Supercomputing. In Proceedings of the 9th SIAM Conference on Parallel Processing. Scientific Computing, 1999.
• 56: ESA/390 Principles of Operation. Technical report, IBM Corporation. IBM Publication No. SA22--7201.
• 57: K. Ikudome, G. Fox, A. Kolawa, and J. Flower. An Automatic and Symbolic Parallelization System for Distributed Memory Parallel Computers. In Fifth Distributed-Memory Computing Conference, pages 1105--1114, Charleston, South Carolina, April 1990.
• 58: H. F. Jordan. The Force. In L. H. Jamieson, D. B. Gannon, and R. J. Douglass, editors, The Characteristics of Parallel Algorithms. MIT Press, 1987.
• 59: Kendall Square Systems. http://en.wikipedia.org/wiki/Kendall-Square-Research.
• 60: Ken Kennedy , Ulrich Kremer, Automatic data layout for distributed-memory machines, ACM Transactions on Programming Languages and Systems (TOPLAS), v.20 n.4, p.869-916, July 1998 [doi>10.1145/291891.291901]
• 61: Jeremy Kepner and Nadya Travinin. Parallel Matlab: The next generation. In Proceedings of the 2003 Workshop on High Performance Embedded Computing (HPEC03), 2003.
• 62: KSR1 Principles of Operation, 1991. Waltham, MA.
• 63: D. Kuck, Y. Muraoka, and S. Chen. On the Number of Operations Simultaneously Executable in Fortran-like Programs and Their Resulting Speedup. IEEE Transactions on Computers, C-21(12): 1293--1310, December 1972.
• 64: Monica D. Lam , Edward E. Rothberg , Michael E. Wolf, The cache performance and optimizations of blocked algorithms, Proceedings of the fourth international conference on Architectural support for programming languages and operating systems, p.63-74, April 08-11, 1991, Santa Clara, California, United States
• 65: Leslie Lamport, The parallel execution of DO loops, Communications of the ACM, v.17 n.2, p.83-93, Feb. 1974 [doi>10.1145/360827.360844]
• 66: B. Leasure. Parallel Processing Model for High-Level Programming Languages. Technical report, American National Standard for Information Processing, April 1994.
• 67: J. Li , M. Chen, Compiling Communication-Efficient Programs for Massively Parallel Machines, IEEE Transactions on Parallel and Distributed Systems, v.2 n.3, p.361-376, July 1991 [doi>10.1109/71.86111]
• 68: Jingke Li , Marina Chen, Generating explicit communication from shared-memory program references, Proceedings of the 1990 conference on Supercomputing, p.865-876, October 1990, New York, New York, United States
• 69: John H. Merlin, ADAPTing Fortran 90 Array Programs for Distributed Memory Architectures, Proceedings of the First International ACPC Conference on Parallel Computation, p.184-200, September 30-October 02, 1991
• 70: Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. International Journal of Super-computer Applications and High Performance Computing, 8(3/4):165--414, 1994. (special issue on MPI, also available electronically via ftp://www.netlib.org/mpi/mpi-report.ps).
• 71: Myrias Systems. http://en.allexperts.com/e/m/my/myrias-research-corporation.htm.
• 72: Dimitrios S. Nikolopoulos , Theodore S. Papatheodorou , Constantine D. Polychronopoulos , Jesus Labarta , Eduard Ayguade;eacute;, Is data distribution necessary in OpenMP?, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.47-es, November 04-10, 2000, Dallas, Texas, United States
• 73: OpenMP Application Program Interface. Version 2.5. Technical report, OpenMP Architecture Review Board, May 2005. http://www.openmp.org.
• 74: Anita Osterhaug, Guide to parallel programming on Sequent computer systems: 2nd edition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1989
• 75: D. Pase. MPP Fortran Programming Model. High Performance Fortran Forum, January 1991.
• 76: Results from a Parallel MATLAB Compiler, Proceedings of the 12th. International Parallel Processing Symposium on International Parallel Processing Symposium, p.81, March 30-April 03, 1998
• 77: A. P. Reeves and C. M. Chase. The Paragon Programming Paradigm and Distributed-Memory Multicomputers. In J. Saltz and P. Mehrotra, editors, Compilers and Runtime Software for Scalable Multiprocessors. Elsevier, Amsterdam, The Netherlands, 1991.
• 78: A. Rogers , K. Pingali, Process decomposition through locality of reference, Proceedings of the ACM SIGPLAN 1989 Conference on Programming language design and implementation, p.69-80, June 19-23, 1989, Portland, Oregon, United States
• 79: John R. Rose and Guy L. Steele. C*: An Extended C Language for Data Parallel Programming. In Proceedings Second International Conference on Supercomputing, volume Vol. II, 2--16, 1987. International Supercomputing Institute.
• 80: M. Rosing, R. W. Schnabel, and R. P. Weaver. Expressing Complex Parallel Algorithms in DINO. In Fourth Conference on Hypercubes, Concurrent Computers, and Applications, pages 553--560, 1989.
• 81: Roland Rühl , Marco Annaratone, Parallelization of FORTRAN code on distributed-memory parallel processors, Proceedings of the 4th international conference on Supercomputing, p.342-353, June 11-15, 1990, Amsterdam, The Netherlands
• 82: Hitoshi Sakagami , Hitoshi Murai , Yoshiki Seo , Mitsuo Yokokawa, 14.9 TFLOPS three-dimensional fluid simulation for fusion science with HPF on the Earth Simulator, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-14, November 16, 2002, Baltimore, Maryland
• 83: Joel Saltz , Kathleen Crowley , Ravi Mirchandaney , Harry Berryman, Run-time scheduling and execution of loops on message passing machines, Journal of Parallel and Distributed Computing, v.8 n.4, p.303-312, Apr. 1990 [doi>10.1016/0743-7315(90)90129-D]
• 84: V. Sarkar, Automatic selection of high-order transformations in the IBM XL FORTRAN compilers, IBM Journal of Research and Development, v.41 n.3, p.233-264, May 1997
• 85: Charles L. Seitz, The cosmic cube, Communications of the ACM, v.28 n.1, p.22-33, Jan. 1985 [doi>10.1145/2465.2467]
• 86: M. Shimasaki and Hans P. Zima. Special Issue on the Earth Simulator, November 2004.
• 87: D. Slotnick, W. Brock, and R. MacReynolds. The SOLOMON computer. In Proceedings of the AFIPS Fall Joint Computer Conference, pages 87--107, 1962.
• 88: Marc Snir , Steve Otto, MPI-The Complete Reference: The MPI Core, MIT Press, Cambridge, MA, 1998
• 89: Lawrence Snyder. A Programming Guide to ZPL. MIT Press, Scientific and Engineering Computation Series, March 1999.
• 90: Sun Microsystems, Inc., Burlington, Massachusetts. The Fortress Language Specification, Version 0.707, July 2005.
• 91: Thinking Machines Corporation, Cambridge, Massachusetts. CM Fortran Reference Manual, Version 1.0, February 1991.
• 92: Transputer: A Programmable Component that Gives Micros a New Name. Computer Design, 23:243--244, February 1984.
• 93: Rémi Triolet , Francois Irigoin , Paul Feautrier, Direct parallelization of call statements, Proceedings of the 1986 SIGPLAN symposium on Compiler construction, p.176-185, June 25-27, 1986, Palo Alto, California, United States
• 94: Chau-Wen Tseng, An optimizing Fortran D compiler for MIMD distributed-memory machines, Rice University, Houston, TX, 1993
• 95: P. S. Tseng. A Systolic Array Programming Language. In Fifth Distributed-Memory Computing Conference, pages 1125--1130, Charleston, South Carolina, April 1990.
• 96: Manuel Ujaldon , Emilio L. Zapata , Barbara M. Chapman , Hans P. Zima, Vienna-Fortran/HPF Extensions for Sparse and Irregular Problems and Their Compilation, IEEE Transactions on Parallel and Distributed Systems, v.8 n.10, p.1068-1083, October 1997 [doi>10.1109/71.629489]
• 97: Vax computers. http://h18000.www1.hp.com/alphaserver/vax/timeline/1986.html.
• 98: Michael E. Wolf , Monica S. Lam, A data locality optimizing algorithm, Proceedings of the ACM SIGPLAN 1991 conference on Programming language design and implementation, p.30-44, June 24-28, 1991, Toronto, Ontario, Canada
• 99: Michael Joseph Wolfe, Optimizing Supercompilers for Supercomputers, MIT Press, Cambridge, MA, 1990
• 100: H. Zima, H.-J. Bast, and M. Gerndt. SUPERB: A Tool For Semi-Automatic MIMD/SIMD Parallelization. Parallel Computing, 6:1--18, 1988.
• 101: H. Zima and B. Chapman. Compiling for Distributed-Memory Systems. Proceedings of the IEEE, 81(2):264--287, February 1993.
• 102: Hans P. Zima, P. Brezany, B. Chapman, P. Mehrotra, and A. Schwald. Vienna Fortran - A Language Specification. Internal Report 21, ICASE, NASA Langley Research Center, March 1992.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.2 HISTORY OF COMPUTING

General Terms:
Languages

Keywords:
High Performance Fortran (HPF), compilers, parallel computing