Direct-pNFS: scalable, transparent, and versatile access to parallel file systems

ABSTRACT

Grid computations require global access to massive data stores. To meet this need, the GridNFS project aims to provide scalable, high-performance, transparent, and secure wide-area data management as well as a scalable and agile name space.

While parallel file systems give high I/O throughput, they are highly specialized, have limited operating system and hardware platform support, and often lack strong security mechanisms. Remote data access tools such as NFS and GridFTP overcome some of these limitations, but fail to provide universal, transparent, and scalable remote data access.

As part of GridNFS, this paper introduces Direct-pNFS, which builds on the NFSv4.1 protocol to meet a key challenge in accessing remote parallel file systems: high-performance and scalable data access without sacrificing transparency, security, orportability. Experiments with Direct-pNFS demonstrate I/O throughput that equals or out performs the exported parallel file system across a range of workloads.

REFERENCES

• 1: Nils Nieuwejaar , David Kotz , Apratim Purakayastha , Carla Schlatter Ellis , Michael L. Best, File-Access Characteristics of Parallel Scientific Workloads, IEEE Transactions on Parallel and Distributed Systems, v.7 n.10, p.1075-1089, October 1996 [doi>10.1109/71.539739]
• 2: Phyllis E. Crandall , Ruth A. Aydt , Andrew A. Chien , Daniel A. Reed, Input/output characteristics of scalable parallel applications, Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM), p.59-es, December 04-08, 1995, San Diego, California, United States [doi>10.1145/224170.224396]
• 3: F. Wang, Q. Xin, B. Hong, S. A. Brandt, E. L. Miller, D. D. E Long, and T. T. McLarty, "File System Workload Analysis For Large Scale Scientific Computing Applications," in Proc. of the 21st IEEE/12th NASA Goddard Conf. on Mass Storage Systems and Technologies, College Park, MD, 2004.
• 4: Darryl Strauss , Wook, Linux Helps Bring Titanic to Life, Linux Journal, v.1998 n.46es, Feb. 1998
• 5: B. Callaghan, B. Pawlowski, and P. Staubach, NFS Version 3 Protocol Specification. RFC 1813, 1995.
• 6: Common Internet File System File Access Protocol (CIFS), msdn.microsoft.com/library/en-us/cifs/ protocol/cifs.asp.
• 7: S. Shepler, M. Eisler, and D. Noveck, NFSv4 Minor Version 1. Internet Draft, 2006.
• 8: Dean Hildebrand , Peter Honeyman, Exporting Storage Systems in a Scalable Manner with pNFS, Proceedings of the 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST'05), p.18-27, April 11-14, 2005 [doi>10.1109/MSST.2005.14]
• 9: B. Halevy, B. Welch, and J. Zelenka, Object-based pNFS Operations. Internet Draft, 2007.
• 10: Mary G. Baker , John H. Hartman , Michael D. Kupfer , Ken W. Shirriff , John K. Ousterhout, Measurements of a distributed file system, Proceedings of the thirteenth ACM symposium on Operating systems principles, p.198-212, October 13-16, 1991, Pacific Grove, California, United States
• 11: R. O. Weber, SCSI Object-Based Storage Device Commands (OSD). Storage Networking Industry Association. ANSI/INCITS 400--2004, www.t10.org, 2004.
• 12: J. Satran, K. Meth, C. Sapuntzakis, M. Chadalapaka, and E. Zeidner, Internet Small Computer Systems Interface (iSCSI). RFC 3720, 2001.
• 13: Panasas Inc., "Panasas ActiveScale File System," www.panasas.com.
• 14: Cluster File Systems Inc., Lustre: A Scalable, High-Performance File System. www.lustre.org, 2002.
• 15: D. L. Black, S. Fridella, and J. Glasgow, pNFS Block/Volume Layout. Internet Draft, 2007.
• 16: D. Hildebrand , P. Honeyman, Scaling NFSv4 with parallel file systems, Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05) - Volume 2, p.1039-1046, May 09-12, 2005
• 17: Dean Hildebrand , Lee Ward , Peter Honeyman, Large files, small writes, and pNFS, Proceedings of the 20th annual international conference on Supercomputing, June 28-July 01, 2006, Cairns, Queensland, Australia [doi>10.1145/1183401.1183419]
• 18: Frank B. Schmuck , Roger L. Haskin, GPFS: A Shared-Disk File System for Large Computing Clusters, Proceedings of the Conference on File and Storage Technologies, p.231-244, January 28-30, 2002
• 19: Red Hat Software Inc., "Red Hat Global File System," www.redhat.com/software/rha/gfs.
• 20: S. R. Soltis, T. M. Ruwart, and M. T. O'Keefe, "The Global File System," in Proc. of the 5th NASA Goddard Conf. on Mass Storage Systems, College Park, MD, 1996.
• 21: Polyserve Inc., "Matrix Server Architecture," www.polyserve.com.
• 22: Parallel Virtual File System - Version 2, www.pvfs.org.
• 23: IBRIX Fusion, www.ibrix.com.
• 24: S. V. Anastasiadis, K. C. Sevcik, and M. Stumm, "Disk Striping Scalability in the Exedra Media Server," in Proc. of the ACM/SPIE Multimedia Computing and Networking, San Jose, CA, 2001.
• 25: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 26: Florin Isaila , Walter F. Tichy, Clusterfile: A Flexible Physical Layout Parallel File System, Proceedings of the 3rd IEEE International Conference on Cluster Computing, p.37, October 08-11, 2001
• 27: IOR Benchmark, www.llnl.gov/asci/purple/benchmarks/limited/ior.
• 28: ATLAS, atlasinfo.cern.ch.
• 29: J. Katcher, "PostMark: A New File System Benchmark," Network Appliance, Technical Report TR3022, 1997.
• 30: Margo I. Seltzer , Gregory R. Ganger , M. Kirk McKusick , Keith A. Smith , Craig A. N. Soules , Christopher A. Stein, Journaling versus soft updates: asynchronous meta-data protection in file systems, Proceedings of the Annual Technical Conference on 2000 USENIX Annual Technical Conference, p.6-6, June 18-23, 2000, San Diego, California
• 31: OpenSSH, www.openssh.org.
• 32: P. Andrews, C. Jordan, and W. Pfeiffer, "Marching Towards Nirvana: Configurations for Very High Performance Parallel File Systems," in Proc. of the HiperIO Workshop, Barcelona, Spain, 2006.
• 33: P. Andrews, C. Jordan, and H. Lederer, "Design, Implementation, and Production Experiences of a Global Storage Grid," in Proc. of the 23rd IEEE/14th NASA Goddard Conf. on Mass Storage Systems and Technologies, College Park, MD, 2006.
• 34: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 35: Osamu Tatebe , Youhei Morita , Satoshi Matsuoka , Noriyuki Soda , Satoshi Sekiguchi, Grid Datafarm Architecture for Petascale Data Intensive Computing, Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, p.102, May 21-24, 2002
• 36: Atul Adya , William J. Bolosky , Miguel Castro , Gerald Cermak , Ronnie Chaiken , John R. Douceur , Jon Howell , Jacob R. Lorch , Marvin Theimer , Roger P. Wattenhofer, Farsite: federated, available, and reliable storage for an incompletely trusted environment, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060291]
• 37: G. H. Kim, R. G. Minnich, and L. McVoy, "Bigfoot-NFS: A Parallel File-Striping NFS Server (Extended Abstract)," 1994, www.bitmover.com/lm.
• 38: F. Garcia-Carballeira, A. Calderon, J. Carretero, J. Fernandez, and J. M. Perez, "The Design of the Expand File System," Intl. J. of High Performance Computing Applications, 17(1):21--37, 2003.
• 39: Pierre Lombard , Yves Denneulin, nfsp: A Distributed NFS Server for Clusters of Workstations, Proceedings of the 16th International Parallel and Distributed Processing Symposium, p.352, April 15-19, 2002
• 40: A. Batsakis and R. Burns, "Cluster Delegation: High-Performance Fault-Tolerant Data Sharing in NFS," in Proc. of the 14th IEEE Intl. Symp. on High Performance Distributed Computing, 2005.
• 41: Bill Allcock , Joe Bester , John Bresnahan , Ann L. Chervenak , Ian Foster , Carl Kesselman , Sam Meder , Veronika Nefedova , Darcy Quesnel , Steven Tuecke, Data management and transfer in high-performance computational grid environments, Parallel Computing, v.28 n.5, p.749-771, May 2002 [doi>10.1016/S0167-8191(02)00094-7]
• 42: Chaitanya Baru , Reagan Moore , Arcot Rajasekar , Michael Wan, The SDSC storage resource broker, Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, p.5, November 30-December 03, 1998, Toronto, Ontario, Canada

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Measurements

General Terms:
Design, Experimentation, Performance

Keywords:
distributed file system, nFSv4, pNFS, parallel I/O