A case study in application I/O on Linux clusters

ABSTRACT

A critical but often ignored component of system performance is the I/O system. Today's applications demand a great deal from underlying storage systems and software, and both high-performance distributed storage and high level interfaces have been developed to fill these needs.In this paper we discuss the I/O performance of a parallel scientific application on a Linux cluster, the FLASH astrophysics code. This application relies on three I/O software components to provide high-performance parallel I/O on Linux clusters: the Parallel Virtual File System, the ROMIO MPI-IO implementation, and the Hierarchical Data Format library. Through instrumentation of both the application and underlying system software code we discover the location of major software bottlenecks. We work around the most inhibiting of these bottlenecks, showing substantial performance improvement. We point out similarities between the inefficiencies found here and those found in message passing systems, indicating that research in the message passing field could be leveraged to solve similar problems in high-level I/O interfaces.

REFERENCES

• 1: Tim Bray. Bonnie file system benchmark. http://www.textuality.com/bonnie/.
• 2: A. C. Calder , B. C. Curtis , L. J. Dursi , B. Fryxell , P. MacNeice , K. Olson , P. Ricker , R. Rosner , F. X. Timmes , H. M. Tufo , J. W. Turan , M. Zingale , G. Henry, High performance reactive fluid flow simulations using adaptive mesh refinement on thousands of processors, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.56-es, November 04-10, 2000, Dallas, Texas, United States
• 3: Philip H. Carns, Walter B. Ligon III, Robert B. Ross, and Rajeev Thakur. PVFS: A parallel file system for Linux clusters. In Proceedings of the 4th Annual Linux Showcase and Conference, pages 317-327, Atlanta, GA, October 2000. USENIX Association.
• 4: Matthew Cettei, Walter Ligon, and Robert Ross. Support for parallel out of core applications on beowulf workstations. In Proceedings of the 1998 IEEE Aerospace Conference, March 1998.
• 5: Chiba City, the Argonne scalable cluster. http://www.mcs.anl.gov/chiba/.
• 6: FLASH I/O benchmark. http://flash.uchicago.edu/~zingale/flash benchmark io/.
• 7: B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, and H. Tufo. FLASH: An adaptive mesh hydrodynamics code for modelling astrophysical thermonuclear flashes. Astrophysical Journal Suppliment, 131:273, 2000.
• 8: William Gropp , Ewing Lusk , Nathan Doss , Anthony Skjellum, A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, v.22 n.6, p.789-828, Sept. 1996 [doi>10.1016/0167-8191(96)00024-5]
• 9: William Gropp, Ewing Lusk, and Deborah Swider. Improving the performance of MPI derived datatypes. In MPIDC, 1999.
• 10: William Gropp , Rajeev Thakur , Ewing Lusk, Using MPI-2: Advanced Features of the Message Passing Interface, MIT Press, Cambridge, MA, 1999
• 11: IEEE/ANSI Std. 1003.1. Portable operating system interface (POSIX)-part 1: System application program interface (API) {C language}, 1996 edition.
• 12: P. MacNeice, K. M. Olson, C. Mobarry, R. de Fainchtein, and C. Packer. PARAMESH: A parallel adaptive mesh refinement community toolkit. Computer Physics Communications, 126, 2000.
• 13: Message Passing Interface Forum. MPI-2: Extensions to the message-passing interface, July 1997. http://www.mpi-forum.org/docs/docs.html.
• 14: ReiserFS. http://www.reiserfs.org.
• 15: Hakan Taki , Gil Utard, MPI-IO on a Parallel File System for Cluster of Workstations, Proceedings of the 1st IEEE Computer Society International Workshop on Cluster Computing, p.150, December 02-03, 1999
• 16: Test TCP. ftp://ftp.arl.mil/pub/ttcp/.
• 17: Rajeev Thakur , William Gropp , Ewing Lusk, On implementing MPI-IO portably and with high performance, Proceedings of the sixth workshop on I/O in parallel and distributed systems, p.23-32, May 05-05, 1999, Atlanta, Georgia, United States [doi>10.1145/301816.301826]
• 18: Jesper Larsson Träff , Rolf Hempel , Hubert Ritzdorf , Falk Zimmermann, Flattening on the Fly: Efficient Handling of MPI Derived Datatypes, Proceedings of the 6th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, p.109-116, September 26-29, 1999
• 19: Omer Zaki , Ewing Lusk , William Gropp , Deborah Swider, Toward Scalable Performance Visualization with Jumpshot, International Journal of High Performance Computing Applications, v.13 n.3, p.277-288, August 1999 [doi>10.1177/109434209901300310]

INDEX TERMS

Primary Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices
          Subjects: Channels and controllers

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.3 Interconnections (subsystems)
          Subjects: Parallel I/O

  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.4 Parallel Architectures

General Terms:
Design, Performance, Theory