Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Data-intensive applications frequently transfer large amounts of data over wide-area networks. The performance achieved in such settings can often be improved by routing data via intermediate nodes chosen to increase aggregate bandwidth. We explore the benefits of overlay network approaches by designing and implementing a service-oriented architecture that incorporates two key optimizations -- multi-hop path splitting and multi-pathing - within the GridFTP file transfer protocol. We develop a file transfer scheduling algorithm that incorporates the two optimizations in conjunction with the use of available file replicas. The algorithm makes use of information from past GridFTP transfers to estimate network bandwidths and resource availability. The effectiveness of these optimizations is evaluated using several application file transfer patterns: one-to-all broadcast, all-to-one gather, and data redistribution, on a wide-area testbed. The experimental results show that our architecture and algorithm achieve significant performance improvement.

REFERENCES

• 1: GT C WS core. http://www.globus.org/toolkit/docs/4.0/common/cwscore/.
• 2: The Large Haldron Collider (LHC) http://Ihc.web.cern.ch/Ihc/.
• 3: Globus FTP Client API, 2002. http://www.globus.org/api/c/globus_ftp_client/html/index.html.
• 4: Predicting the Performance of Wide Area Data Transfers, Proceedings of the 16th International Symposium on Parallel and Distributed Processing, p.34, April 15-19, 2002
• 5: W. Allcock. Gridftp: Protocol extensions to ftp for the grid. In Global Grid ForumGFD-R-P.020, 2003.
• 6: William Allcock , John Bresnahan , Rajkumar Kettimuthu , Michael Link, The Globus Striped GridFTP Framework and Server, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.54, November 12-18, 2005 [doi>10.1109/SC.2005.72]
• 7: A. Bakre , B. R. Badrinath, I-TCP: indirect TCP for mobile hosts, Proceedings of the 15th International Conference on Distributed Computing Systems, p.136, May 30-June 02, 1995
• 8: M. Beck, T. Moore, J. S. Plank, and M. Swany. Logistical networking: Sharing more than the wires. In S. Hariri, C. A. Lee, and C. S. Raghavendra, editors, Active Middleware Services, Norwell, MA, 2000. Kluwer Academic.
• 9: Kevin Brown , Suresh Singh, M-TCP: TCP for mobile cellular networks, ACM SIGCOMM Computer Communication Review, v.27 n.5, p.19-43, Oct. 1997 [doi>10.1145/269790.269794]
• 10: Z. Cai, V. Kumar, and K. Schwan. Iq-paths: Self-regulating data streams across network overlays. In Proceedings of the The 15th IEEE International Symposium on High Performance Distributed Computing (HPDC '06), 2006.
• 11: A. Giersch, Y. Robert, and F. Vivien. Scheduling tasks sharing files from distributed repositories. In Euro-Par 2004: Parallel Processing: 10th International Euro-Par Conference, volume 3149 of LNCS, pages 246--253, Sept. 2004.
• 12: K. Holtman. Cms data grid system overview and requirements. In Computing in High Energy and Nuclear Physics (CHEP), 2001.
• 13: Oscar H. Ibarra , Chul E. Kim, Heuristic Algorithms for Scheduling Independent Tasks on Nonidentical Processors, Journal of the ACM (JACM), v.24 n.2, p.280-289, April 1977 [doi>10.1145/322003.322011]
• 14: G. Khanna, U. Catalyurek, T. Kurc, P. Sadayappan, and J. Saltz. Scheduling file transfers for data-intensive jobs on heterogeneous clusters. In A.-M. Kermarrec, L. Bougé, and T. Priol, editors, Euro-Par, volume 4641 of Lecture Notes in Computer Science, pages 214--223. Springer, 2007.
• 15: G. Khanna, T. Kurc, U. Catalyurek, R. Kettimuthu, P. Sadayappan, and J. Saltz. A dynamic scheduling approach for coordinated wide-area data transfers using gridftp. In Proc. of 22th International Parallel and Distributed Processing Symposium (IPDPS), Miami, Florida, 2008.
• 16: Gaurav Khanna , Nagavijayalakshmi Vydyanathan , T. Kurc , U. Catalyurek , P. Wyckoff , J. Saltz , P. Sadayappan, A hypergraph partitioning based approach for scheduling of tasks with batch-shared I/O, Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05) - Volume 2, p.792-799, May 09-12, 2005
• 17: Min Sik Kim , Taekhyun Kim , YongJune Shin , Simon S. Lam , Edward J. Powers, A wavelet-based approach to detect shared congestion, ACM SIGCOMM Computer Communication Review, v.34 n.4, October 2004
• 18: Tevfik Kosar , Miron Livny, Stork: Making Data Placement a First Class Citizen in the Grid, Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04), p.342-349, March 24-26, 2004
• 19: H. Pucha and Y. C. Hu. Overlay tcp: ending end-to-end transport for higher throughput. In Poster in ACM SIGCOMM, Philadelphia, PA, 2005.
• 20: P. Rizk, C. Kiddle, and R. Simmonds. A gridftp overlay network service. In Proceedings of the 7th IEEE/ACM International Conference on Grid Computing, Baercelona, Spain, 2007.
• 21: Dan Rubenstein , Jim Kurose , Don Towsley, Detecting shared congestion of flows via end-to-end measurement, Proceedings of the 2000 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.145-155, June 18-21, 2000, Santa Clara, California, United States
• 22: Martin Swany, Improving Throughput for Grid Applications with Network Logistics, Proceedings of the 2004 ACM/IEEE conference on Supercomputing, p.23, November 06-12, 2004 [doi>10.1109/SC.2004.30]
• 23: TeraGrid. http://www.teragrid.org.
• 24: Brian Tierney , Jason Lee , Ling Tony Chen , Hanan Herzog , Gary Hoo , Guojun Jin , William E. Johnston, Distributed parallel data storage systems: a scalable approach to high speed image servers, Proceedings of the second ACM international conference on Multimedia, p.399-405, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192709]
• 25: Brian L. Tierney , Jason Lee , Brian Crowley , Mason Holding , Jeremy Hylton , Fred L. Drake Jr, A Network-Aware Distributed Storage Cache for Data Intensive Environments, Proceedings of the 8th IEEE International Symposium on High Performance Distributed Computing, p.33, August 03-06, 1999
• 26: Rich Wolski, Dynamically forecasting network performance using the Network Weather Service, Cluster Computing, v.1 n.1, p.119-132, 1998 [doi>10.1023/A:1019025230054]
• 27: Lingyun Yang , J. M. Schopf , I. Foster, Improving parallel data transfer times using predicted variances in shared networks, Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid (CCGrid'05) - Volume 2, p.734-742, May 09-12, 2005

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General
          Subjects: Data communications

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Experimentation, Measurement, Performance