HTTP redirection for replica catalogue lookups in data grids

ABSTRACT

Data distribution and replication in distributed systems require special purpose middleware tools for accessing replicated data. Data Grids, special forms of systems distributed over wide-area networks, need to handle data management issues like distribution and replication of large amounts of data in the Tera- and Petabyte scale. Replica catalogues are used for cataloguing and locating replicated files in distributed sites all around the globe. We present a novel and administratively scalable approach for distributing a replica catalogue and resolving file location information by using HTTP redirection. HTTP redirection servers managing local file catalogues allow for greater flexibility and local file management autonomy whereas a global replica catalogue provides the necessary mapping of logical files to individual sites. By distributing the catalogues a site can autonomously move files for load balancing within a site without notifying a global replica catalogue. Our approach scales well in terms of catalogue administration to a large number of sites and file entries and thus establishes a powerful middleware service. We present the design and implementation of our catalogue redirection servers and report on promising experimental results.

REFERENCES

• 1: Bill Allcock , Joe Bester , John Bresnahan , Ann L. Chervenak , Carl Kesselman , Sam Meder , Veronika Nefedova , Darcy Quesnel , Steven Tuecke , Ian Foster, Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing, Proceedings of the Eighteenth IEEE Symposium on Mass Storage Systems and Technologies, p.13, April 17-20, 2001
• 2: Jim Basney, Miron Livny, and Todd Tannenbaum, "High Throughput Computing with Condor", HPCU news, Volume 1(2), June 1997.
• 3: Chaitanya Baru , Reagan Moore , Arcot Rajasekar , Michael Wan, The SDSC storage resource broker, Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, p.5, November 30-December 03, 1998, Toronto, Ontario, Canada
• 4: Jacek Becla, Andrew Hanushevsky. Creating Highly Performant Servers For Massively Large Databases. 9th IEEE International Symposium on High Performance Distributed Computing. Pittsburgh, Pennsylvania, August 1-4, 2000.
• 5: Ann Chervenak, Ian Foster, Carl Kesselman, Chuck Salisbury, Steve Tuecke. The Data Grid: Towards an Architecture for the Distributed Management and Analysis is of Large Scientific DataSets. Network Storage Symposium, Seattle 1999.
• 6: David M. Choy , Patricia G. Selinger, A distributed catalog for heterogeneous distributed database resources, Proceedings of the first international conference on Parallel and distributed information systems, p.236-244, December 1991, Miami, Florida, United States
• 7: Distributed Computing Environment (DCE), http://www.opengroup.org/dce/
• 8: European DataGrid Project, http://www.eu-datagrid.org
• 9: Roy Fielding, James Gettys, Jeffrey Mogul, Henrik Nielson, Larry Masinter, Paul Leach, Tim Berners-Lee. Hypertext Transfer Protocol --- HTTP/1.1, RFC 2616, June 1999.
• 10: Ian Foster , Carl Kesselman, The grid: blueprint for a new computing infrastructure, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 11: Garth A. Gibson , David F. Nagle , Khalil Amiri , Fay W. Chang , Eugene M. Feinberg , Howard Gobioff , Chen Lee , Berend Ozceri , Erik Riedel , David Rochberg , Jim Zelenka, File server scaling with network-attached secure disks, Proceedings of the 1997 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.272-284, June 15-18, 1997, Seattle, Washington, United States
• 12: Andrew S. Grimshaw , Wm. A. Wulf , CORPORATE The Legion Team, The Legion vision of a worldwide virtual computer, Communications of the ACM, v.40 n.1, p.39-45, Jan. 1997 [doi>10.1145/242857.242867]
• 13: Andrew Hanushevsky, Marcin Nowak. Pursuit of a Scalable High Performance Multi-Petabyte Database. IEEE Symposium on Mass Storage Systems, March 15-18, 1999.
• 14: Wolfgang Hoschek , Francisco Javier Jaén-Martínez , Asad Samar , Heinz Stockinger , Kurt Stockinger, Data Management in an International Data Grid Project, Proceedings of the First IEEE/ACM International Workshop on Grid Computing, p.77-90, December 17, 2000
• 15: High Performance Storage System (HPSS): http://www.sdsc.edu/hpss/
• 16: Hyeran Lim , Vikram Kapoor , Chirag Wighe , David H.-C. Du, Active Disk File System: A Distributed, Scalable File System, Proceedings of the Eighteenth IEEE Symposium on Mass Storage Systems and Technologies, p.101, April 17-20, 2001
• 17: Newsletter at National University of Singapore, Novel Framework Enhances Web Performance and Scalability, Volume 5 Number 2, Apr./Jun. 2000.
• 18: Heinz Stockinger, Distributed Database Management Systems and the Data Grid, Proceedings of the Eighteenth IEEE Symposium on Mass Storage Systems and Technologies, p.1, April 17-20, 2001
• 19: Heinz Stockinger , Asad Samar , Koen Holtman , Bill Allcock , Ian Foster , Brian Tierney, File and Object Replication in Data Grids, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing (HPDC-10'01), p.76, August 07-09, 2001
• 20: Wengyik Yeong, Tim Howes, Steve Kille. Lightweight Directory Access Protocol, RFC 1777, March 1995.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Distributed databases

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.7 Database Administration
          Subjects: Security, integrity, and protection
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Distributed systems

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.1 Document and Text Editing
          Subjects: Document management

General Terms:
Algorithms, Design, Performance, Theory

Keywords:
HTTP, distributed computing, file replication, grid, replica catalogue