Gathering at the well: creating communities for grid I/O

ABSTRACT

Grid applications have demanding I/O needs. Schedulers must bring jobs and data in close proximity in order to satisfy throughput, scalability, and policy requirements. Most systems accomplish this by making either jobs or data mobile. We propose a system that allows jobs and data to meet by binding execution and storage sites together into I/O communities which then participate in the wide-area system. The relationships between participants in a community may be expressed by the ClassAd framework. Extensions to the framework allow community members to express indirect relations. We demonstrate our implementation of I/O communities by improving the performance of a key high-energy physics simulation on an international distributed system.

REFERENCES

• 1: Albert D. Alexandrov , Maximilian Ibel , Klaus E. Schauser , Chris J. Scheiman, UFO: a personal global file system based on user-level extensions to the operating system, ACM Transactions on Computer Systems (TOCS), v.16 n.3, p.207-233, Aug. 1998 [doi>10.1145/290409.290410]
• 2: Bill Allcock, Joe Bester, John Bresnahan, Ann Chervenak, Ian Foster, Carl Kesselman, Sam Meder, Veronika Nefedova, Darcy Quesnel, and Steven Tuecke. Secure, efficient data transport and replica management for high-performance data-intensive computing. Submitted for publication.
• 3: William Allcock, Ann Chervenak, Ian Foster, Carl Kesselman, Charles Salisbury, and Steve Tuecke. The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets. To appear in the Journal of Network and Computer Applications, 2001.
• 4: Darrell Anderson , Ken Yocum , Jeff Chase, A Case for Buffer Servers, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, p.82, March 28-30, 1999
• 5: Chaitanya Baru , Reagan Moore , Arcot Rajasekar , Michael Wan, The SDSC storage resource broker, Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, p.5, November 30-December 03, 1998, Toronto, Ontario, Canada
• 6: Jim Basney, Miron Livny, and Paolo Mazzanti. Utilizing widely distributed computational resources efficiently with execution domains. Submitted to Computer Physics Communications, 2001.
• 7: John Bent, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, and Miron Livny. NeST project. http://www.nestproject.com, 2001.
• 8: Joseph Bester , Ian Foster , Carl Kesselman , Jean Tedesco , Steven Tuecke, GASS: a data movement and access service for wide area computing systems, Proceedings of the sixth workshop on I/O in parallel and distributed systems, p.78-88, May 05-05, 1999, Atlanta, Georgia, United States [doi>10.1145/301816.301839]
• 9: Andrew D. Birrell , Bruce Jay Nelson, Implementing remote procedure calls, ACM Transactions on Computer Systems (TOCS), v.2 n.1, p.39-59, February 1984 [doi>10.1145/2080.357392]
• 10: J. Case, M. Fedor, M. Schoffstall, and J. Davin. A simple network management protocol. Request For Comments 1157, Internet Engineering Task Force, 1990.
• 11: EMC Corporation. http://www.emc.com.
• 12: Garth A. Gibson, David P. Nagle, Khalil Amiri, Fay W. Chang, Eugene Feinberg, Howard Gobioff Chen Lee, Berend Ozceri, Erik Riedel, and David Rochberg. A case for network-attached secure disks. Technical Report CMU-CS-96-142, Carnegie-Mellon University, 1996.
• 13: Gnutella. http://gnutella.wego.com.
• 14: D. Hitz. A storage networking appliance. Technical Report TR3001, Network Appliance, Inc., 2000.
• 15: John H. Howard , Michael L. Kazar , Sherri G. Menees , David A. Nichols , M. Satyanarayanan , Robert N. Sidebotham , Michael J. West, Scale and performance in a distributed file system, ACM Transactions on Computer Systems (TOCS), v.6 n.1, p.51-81, Feb. 1988 [doi>10.1145/35037.35059]
• 16: Galen Hunt and Doug Brubacher. Detours: Binary interception of Win32 functions. Technical Report MSR-TR-98-33, Microsoft Research, February 1999.
• 17: Network Appliance Inc. http://www.netapp.com.
• 18: Michael B. Jones, Interposition agents: transparently interposing user code at the system interface, Proceedings of the fourteenth ACM symposium on Operating systems principles, p.80-93, December 05-08, 1993, Asheville, North Carolina, United States
• 19: M. J. Litzkow. Remote UNIX: Turning Idle Workstations into Cycle Servers. In Proceedings of the 1987 Usenix Summer Conf., pages 381-384, 1987.
• 20: Michael Litzkow, Miron Livny, and Matthew Mutka. Condor --- A hunter of idle workstations. In Proceedings of the 8th International Conference on Distributed Computing Systems, pages 104-111, 1988.
• 21: Napster. http://www.napster.com.
• 22: V. Pai, P. Druschel, and W. Zwaenepoel. Flash: An efficient and portable web server. In Proceedings of the Usenix Technical Conference, 1999.
• 23: J. Plank, M. Beck, W. Elwasif, T. Moore, M. Swany, and R. Wolski. The Internet Backplane Protocol: Storage in the network. In Proceedings of the Network Storage Symposium, 1999.
• 24: Rajesh Raman , Miron Livny, Matchmaking frameworks for distributed resource management, 2000
• 25: R. Raman , M. Livny , M. Solomon, Matchmaking: Distributed Resource Management for High Throughput Computing, Proceedings of the The Seventh IEEE International Symposium on High Performance Distributed Computing, p.140, July 28-31, 1998
• 26: Rajesh Raman , Miron Livny , Marvin Solomon, Resource Management through Multilateral Matchmaking, Proceedings of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC'00), p.290, August 01-04, 2000
• 27: E. Riedel and G. Gibson. Active disks --- remote execution for network-attached storage. Technical Report CMU-CS-97-198, Carnegie-Mellon University, 1997.
• 28: Asad Samar and Heinz Stockinger. Grid Data Management Pilot. To appear in IASTED International Conference on Appliaed Informatics (AI2001), February 2001.
• 29: Russel Sandberg, David Goldberg, Steve Kleiman, Dan Walsh, and Bob Lyon. Design and implementation of the Sun network filesystem. In Proceedings of the Summer 1985 USENIX Conferece, pages 119-130, 1985.
• 30: Se-Chang Son, The Kangaroo Approach to Data Movement on the Grid, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing (HPDC-10'01), p.325, August 07-09, 2001
• 31: Douglas Thain and Miron Livny. The Pluggable File System. http://www.cs.wisc.edu/condor/pfs.
• 32: Douglas Thain , Miron Livny, Bypass: A Tool for Building Split Execution Systems, Proceedings of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC'00), p.79, August 01-04, 2000
• 33: Douglas Thain , Miron Livny, Multiple Bypass: Interposition Agents for Distributed Computing, Cluster Computing, v.4 n.1, p.39-47, March 2001 [doi>10.1023/A:1011412209850]
• 34: Sudharshan Vazhkudai , Steven Tuecke , Ian Foster, Replica Selection in the Globus Data Grid, Proceedings of the 1st International Symposium on Cluster Computing and the Grid, p.106, May 15-18, 2001
• 35: Sudharshan Vazhkudai, Steven Tuecke, and Ian Foster. Replica selection in the globus data grid, May 2001.
• 36: Jim Waldo, The Jini architecture for network-centric computing, Communications of the ACM, v.42 n.7, p.76-82, July 1999 [doi>10.1145/306549.306582]
• 37: Jia Wang, A survey of web caching schemes for the Internet, ACM SIGCOMM Computer Communication Review, v.29 n.5, October 1999 [doi>10.1145/505696.505701]
• 38: Brian S. White , Andrew S. Grimshaw , Anh Nguyen-Tuong, Grid-Based File Access: The Legion I/O Model, Proceedings of the Ninth IEEE International Symposium on High Performance Distributed Computing (HPDC'00), p.165, August 01-04, 2000
• 39: W. Yeong, T. Howes, and S. Kille. Lightweight directory access protocol. Request For Comments 1777, Internet Engineering Task Force, March 1995.
• 40: Victor C. Zandy , Barton P. Miller , Miron Livny, Process Hijacking, Proceedings of the The Eighth IEEE International Symposium on High Performance Distributed Computing, p.32, August 03-06, 1999

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Distributed networks

Additional Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.4 Parallel Architectures
          Subjects: Distributed architectures

  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features
          Subjects: Frameworks

General Terms:
Algorithms, Design, Performance