A Metadata Catalog Service for Data Intensive Applications

ABSTRACT

Advances in computational, storage and network technologies as well as middle ware such as the Globus Toolkit allow scientists to expand the sophistication and scope of data-intensive applications. These applications produce and analyze terabytes and petabytes of data that are distributed in millions of files or objects. To manage these large data sets efficiently, metadata or descriptive information about the data needs to be managed. There are various types of metadata, and it is likely that a range of metadata services will exist in Grid environments that are specialized for particular types of metadata cataloguing and discovery. In this paper, we present the design of a Metadata Catalog Service (MCS) that provides a mechanism for storing and accessing descriptive metadata and allows users to query for data items based on desired attributes. We describe our experience in using the MCS with several applications and present a scalability study of the service.

REFERENCES

• 1: Ian Foster , Carl Kesselman, The grid: blueprint for a new computing infrastructure, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 2: [2] I. Foster, "Grid Computing," presented at Advanced Computing and Analysis Techniques in Physics Research (ACAT), 2000.
• 3: Ian Foster , Carl Kesselman , Steven Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, International Journal of High Performance Computing Applications, v.15 n.3, p.200-222, August 2001 [doi>10.1177/109434200101500302]
• 4: Ann Chervenak , Ewa Deelman , Ian Foster , Leanne Guy , Wolfgang Hoschek , Adriana Iamnitchi , Carl Kesselman , Peter Kunszt , Matei Ripeanu , Bob Schwartzkopf , Heinz Stockinger , Kurt Stockinger , Brian Tierney, Giggle: a framework for constructing scalable replica location services, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-17, November 16, 2002, Baltimore, Maryland
• 5: [5] ESG, "The Earth Systems Grid." http://www.earthsystemgrid.org
• 6: [6] E. Deelman, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, A. Arbree, R. Cavanaugh, K. Blackburn, A. Lazzarini, and S. Koranda, "Mapping Abstract Complex Workflows onto Grid Environments," Journal of Grid Computing, vol. 1, pp. 25-39, 2003.
• 7: Bill Allcock , Ian Foster , Veronika Nefedova , Ann Chervenak , Ewa Deelman , Carl Kesselman , Jason Lee , Alex Sim , Arie Shoshani , Bob Drach , Dean Williams, High-performance remote access to climate simulation data: a challenge problem for data grid technologies, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.46-46, November 10-16, 2001, Denver, Colorado [doi>10.1145/582034.582080]
• 8: L. Pearlman , V. Welch , I. Foster , C. Kesselman , S. Tuecke, A Community Authorization Service for Group Collaboration, Proceedings of the 3rd International Workshop on Policies for Distributed Systems and Networks (POLICY'02), p.50, June 05-07, 2002
• 9: [9] A. Chervenak, E. Deelman, C. Kesselman, L. Pearlman, and G. Singh, "A Metadata Catalog Service for Data Intensive Applications," GriPhyN technical report, 2002-11 2002.
• 10: [10] E. Deelman, J. Blythe, Y. Gil, and C. Kesselman, "Pegasus: Planning for Execution in Grids," GriPhyN 2002-20, 2002.
• 11: [11] A. Abramovici, W. E. Althouse, and e. al., "LIGO: The Laser Interferometer Gravitational-Wave Observatory (in Large Scale Measurements)," Science, vol. 256, pp. 325-333, 1992.
• 12: Ewa Deelman , Carl Kesselman , Gaurang Mehta , Leila Meshkat , Laura Pearlman , Kent Blackburn , Phil Ehrens , Albert Lazzarini , Roy Williams , Scott Koranda, GriPhyN and LIGO, Building a Virtual Data Grid for Gravitational Wave Scientists, Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing, p.225, July 24-26, 2002
• 13: [13] MCAT, "MCAT - A Meta Information Catalog (Version 1.1)."
• 14: Chaitanya Baru , Reagan Moore , Arcot Rajasekar , Michael Wan, The SDSC storage resource broker, Proceedings of the 1998 conference of the Centre for Advanced Studies on Collaborative research, p.5, November 30-December 03, 1998, Toronto, Ontario, Canada
• 15: [15] Guy, L., P. Kunszt, E. Laure, H. Stockinger, K. Stockinger (2002). Replica Management in Data Grids. Global Grid Forum 5.
• 16: Steven Fitzgerald, Grid Information Services for Distributed Resource Sharing, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing, p.181, August 07-09, 2001