Using MPI file caching to improve parallel write performance for large-scale scientific applications

ABSTRACT

Typical large-scale scientific applications periodically write checkpoint files to save the computational state throughout execution. Existing parallel file systems improve such write-only I/O patterns through the use of client-side file caching and write-behind strategies. In distributed environments where files are rarely accessed by more than one client concurrently, file caching has achieved significant success; however, in parallel applications where multiple clients manipulate a shared file, cache coherence control can serialize I/O. We have designed a thread based caching layer for the MPI I/O library, which adds a portable caching system closer to user applications so more information about the application's I/O patterns is available for better coherence control. We demonstrate the impact of our caching solution on parallel write performance with a comprehensive evaluation that includes a set of widely used I/O benchmarks and production application I/O kernels.

REFERENCES

• 1: Philip A. Bernstein , Vassco Hadzilacos , Nathan Goodman, Concurrency control and recovery in database systems, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1987
• 2: Toni Cortes , Sergi Girona , Jesús Labarta, PACA: A Cooperative File System Cache for Parallel Machines, Proceedings of the Second International Euro-Par Conference on Parallel Processing, p.477-486, August 26-29, 1996
• 3: Michael D. Dahlin , Randolph Y. Wang , Thomas E. Anderson , David A. Patterson, Cooperative caching: using remote client memory to improve file system performance, Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation, p.19-es, November 14-17, 1994, Monterey, California
• 4: J. del Rosario, R. Brodawekar, and A. Choudhary. Improved Parallel I/O via a Two-Phase Run-time Access Strategy. In the Workshop on I/O in Parallel Computer Systems at IPPS '93, pages 56--70, April 1993.
• 5: S. A. Fineberg , P. Wong , B. Nitzberg , C. Kuszmaul, PMPIO - A Portable Implementation of MPI-IO, Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation, p.188, March 27-31, 1996
• 6: B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, and H. Tufo. FLASH: An Adaptive Mesh Hydrodynamics Code for Modelling Astrophysical Thermonuclear Flashes. Astrophysical Journal Suppliment, pages 131--273, 2000.
• 7: HDF Group. Hierarchical Data Format, Version 5. The National Center for Supercomputing Applications, http://hdf.ncsa.uiuc.edu/HDF5.
• 8: James V. Huber, Jr. , Andrew A. Chien , Christopher L. Elford , David S. Blumenthal , Daniel A. Reed, PPFS: a high performance portable parallel file system, Proceedings of the 9th international conference on Supercomputing, p.385-394, July 03-07, 1995, Barcelona, Spain [doi>10.1145/224538.224638]
• 9: IEEE/ANSI Std. 1003.1. Portable Operating System Interface (POSIX)-Part 1: System Application Program Interface (API) {C Language}, 1996.
• 10: Florin Isaila , Guido Malpohl , Vlad Olaru , Gabor Szeder , Walter Tichy, Integrating collective I/O and cooperative caching into the "clusterfile" parallel file system, Proceedings of the 18th annual international conference on Supercomputing, June 26-July 01, 2004, Malo, France [doi>10.1145/1006209.1006219]
• 11: W. Liao, A. Ching, K. Coloma, A. Choudhary, and L. Ward. An Implementation and Evaluation of Client-Side File Caching for MPI-IO. In the International Parallel and Distributed Processing Symposium (IPDPS), March 2007.
• 12: Wei-keng Liao , K. Coloma , A. Choudhary , L. Ward , E. Russell , S. Tideman, Collective caching: application-aware client-side file caching, Proceedings of the High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium, p.81-90, July 24-27, 2005 [doi>10.1109/HPDC.2005.1520940]
• 13: Lustre: A Scalable, High-Performance File System. Whitepaper. Cluster File Systems, Inc., 2003.
• 14: Xiaosong Ma , Marianne Winslett , Jonghyun Lee , Shengke Yu, Improving MPI-IO Output Performance with Active Buffering Plus Threads, Proceedings of the 17th International Symposium on Parallel and Distributed Processing, p.68.2, April 22-26, 2003
• 15: Message Passing Interface Forum. MPI: A Message Passing Interface Standard, Version 1.1, June 1995. http://www.nipi-forum.org/docs/docs.html.
• 16: Message Passing Interface Forum. MPI-2: Extensions to the Message Passing Interface, July 1997. http://www.mpi-forum.org/docs/docs.html.
• 17: Jean-Pierre Prost , Richard Treumann , Richard Hedges , Bin Jia , Alice Koniges, MPI-IO/GPFS, an optimized implementation of MPI-IO on top of GPFS, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.17-17, November 10-16, 2001, Denver, Colorado [doi>10.1145/582034.582051]
• 18: Jean-Pierre Prost , Richard Treumann , Richard Hedges , Alice E. Koniges , Alison White, Towards a High-Performance Implementation of MPI-IO on Top of GPFS, Proceedings from the 6th International Euro-Par Conference on Parallel Processing, p.1253-1262, August 29-September 01, 2000
• 19: R. Ross , R. Latham , W. Gropp , R. Thakur , B. Toonen, Implementing MPI-IO atomic mode without file system support, Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05) - Volume 2, p.1135-1142, May 09-12, 2005
• 20: R. Sankaran, E. Hawkes, J. Chen, T. Lu, and C. Law. Direct Numerical Simulations of Turbulent Lean Premixed Combustion. Journal of Physics: conference series, 46:38--42, 2006.
• 21: Frank B. Schmuck , Roger L. Haskin, GPFS: A Shared-Disk File System for Large Computing Clusters, Proceedings of the Conference on File and Storage Technologies, p.231-244, January 28-30, 2002
• 22: Andrew S. Tanenbaum , Maarten van Steen, Distributed Systems: Principles and Paradigms (2nd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2006
• 23: Rajeev Thakur , Alok Choudhary, An extended two-phase method for accessing sections of out-of-core arrays, Scientific Programming, v.5 n.4, p.301-317, Winter 1996
• 24: R. Thakur , W. Gropp , E. Lusk, An Abstract-Device Interface for Implementing Portable Parallel-I/O Interfaces, Proceedings of the 6th Symposium on the Frontiers of Massively Parallel Computation, p.180, March 27-31, 1996
• 25: R. Thakur, W. Gropp, and E. Lusk. Users Guide for ROMIO: A High-Performance, Portable MPI-IO Implementation. Technical Report ANL/MCS-TM-234, Mathematics and Computer Science Division, Argonne National Laboratory, October 1997.
• 26: Rajeev Thakur , William Gropp , Ewing Lusk, On implementing MPI-IO portably and with high performance, Proceedings of the sixth workshop on I/O in parallel and distributed systems, p.23-32, May 05-05, 1999, Atlanta, Georgia, United States [doi>10.1145/301816.301826]
• 27: Geoffrey M. Voelker , Eric J. Anderson , Tracy Kimbrel , Michael J. Feeley , Jeffrey S. Chase , Anna R. Karlin , Henry M. Levy, Implementing cooperative prefetching and caching in a globally-managed memory system, Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, p.33-43, June 22-26, 1998, Madison, Wisconsin, United States
• 28: P. Wong and R. der Wijngaart. NAS Parallel Benchmarks I/O Version 2.4. Technical Report NAS-03-002, NASA Ames Research Center, Moffet Field, CA, January 2003.
• 29: M. Zingale. FLASH I/O Benchmark Routine -Parallel HDF 5, March 2001. http://flash.uchicago.edu/~zingale/flash_b enchmark_io.