A software architecture-based framework for highly distributed and data intensive scientific applications

ABSTRACT

Modern scientific research is increasingly conducted by virtual communities of scientists distributed around the world. The data volumes created by these communities are extremely large, and growing rapidly. The management of the resulting highly distributed, virtual data systems is a complex task, characterized by a number of formidable technical challenges, many of which are of a software engineering nature. In this paper we describe our experience over the past seven years in constructing and deploying OODT, a software framework that supports large, distributed, virtual scientific communities. We outline the key software engineering challenges that we faced, and addressed, along the way. We argue that a major contributor to the success of OODT was its explicit focus on software architecture. We describe several large-scale, real-world deployments of OODT, and the manner in which OODT helped us to address the domain-specific challenges induced by each deployment.

REFERENCES

• 1: A. Chervenak, I. Foster, et al., "The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Data Sets," J. of Network and Computer Applications, vol. 23, pp. 187--200, 2000.
• 2: Nenad Medvidovic , Richard N. Taylor, A Classification and Comparison Framework for Software Architecture Description Languages, IEEE Transactions on Software Engineering, v.26 n.1, p.70-93, January 2000 [doi>10.1109/32.825767]
• 3: Dewayne E. Perry , Alexander L. Wolf, Foundations for the study of software architecture, ACM SIGSOFT Software Engineering Notes, v.17 n.4, p.40-52, Oct. 1992 [doi>10.1145/141874.141884]
• 4: "The Globus Alliance (http://www.globus.org)," 2005.
• 5: "Webservices.org (http://www.webservices.org)," 2005.
• 6: A. Luther, R. Buyya, et al., "Alchemi: A.NET-based Enterprise Grid Computing System," in Proc. of Intl' Conference on Internet Computing, Las Vegas, NV, USA, pp. 2005.
• 7: "JCGrid Web Site (http://jcgrid.sourceforge.net)," 2005.
• 8: "LHC Computing Grid (http://lcg.web.cern.ch/LCG/)," 2005.
• 9: D. Bernholdt, S. Bharathi, et al., "The Earth System Grid: Supporting the Next Generation of Climate Modeling Research," Proceedings of the IEEE, vol. 93, pp. 485--495, 2005.
• 10: A. Finkelstein, C. Gryce, et al., "Relating Requirements and Architectures: A Study of Data Grids," J. of Grid Computing, vol. 2, pp. 207--222, 2004.
• 11: C. A. Mattmann, N. Medvidovic, et al., "Unlocking the Grid," in Proc. of CBSE, St. Louis, MO, pp. 322--336, 2005.
• 12: Joachim Hammer , Héctor García-Molina , Kelly Ireland , Yannis Papakonstantinou , Jeffrey Ullman , Jennifer Widom, Information translation, mediation, and mosaic-based browsing in the TSIMMIS system, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.483, May 22-25, 1995, San Jose, California, United States
• 13: T. Kirk, A. Y. Levy, et al., "The information manifold," Working Notes of the AAAI Spring Symposium on Information Gathering in Heterogeneous, Distributed Environments, Menlo Park, CA, Technical Report SS-95-08, 1995.
• 14: Oren Etzioni , Daniel Weld, A softbot-based interface to the Internet, Communications of the ACM, v.37 n.7, p.72-76, July 1994 [doi>10.1145/176789.176797]
• 15: Alfredo Goñi , Arantza Illarramendi , Eduardo Mena , José Miguel Blanco, An Optimal Cache for a Federated Database System, Journal of Intelligent Information Systems, v.9 n.2, p.125-155, Sept./Oct. 1997 [doi>10.1023/A:1008699315030]
• 16: Michael R. Genesereth , Arthur M. Keller , Oliver M. Duschka, Infomaster: an information integration system, Proceedings of the 1997 ACM SIGMOD international conference on Management of data, p.539-542, May 11-15, 1997, Tucson, Arizona, United States
• 17: A. Tomasic, L. Raschid, et al., "A data model and query processing techniques for scaling access to distributed heterogeneous databases in disco," IEEE Transactions on Computers, pp. 1997.
• 18: Yigal Arens , Craig A. Knoblock , Wei-Min Shen, Query reformulation for dynamic information integration, Journal of Intelligent Information Systems, v.6 n.2-3, p.99-130, June 1996 [doi>10.1007/BF00122124]
• 19: José Luis Ambite , Naveen Ashish , Greg Barish , Craig A. Knoblock , Steven Minton , Pragnesh J. Modi , Ion Muslea , Andrew Philpot , Sheila Tejada, Ariadne: a system for constructing mediators for Internet sources, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.561-563, June 01-04, 1998, Seattle, Washington, United States
• 20: G. Barish and C. A. Knoblock, "An Expressive and Efficient Language for Information Gathering on the Web," in Proc. of AIPS Workshop, Toulouse, France, pp. 2002.
• 21: Alon Y. Halevy, Answering queries using views: A survey, The VLDB Journal — The International Journal on Very Large Data Bases, v.10 n.4, p.270-294, December 2001 [doi>10.1007/s007780100054]
• 22: José Luis Ambite , Craig A. Knoblock , Ion Muslea , Andrew G. Philpot, Compiling Source Descriptions for Efficient and Flexible Information Integration, Journal of Intelligent Information Systems, v.16 n.2, p.149-187, March/April 2001 [doi>10.1023/A:1011289701371]
• 23: E. Lambrecht and S. Kambhampati, "Planning for Information Gathering: A Tutorial Survey," ASU CSE Technical Report 96-017, May 1997.
• 24: "Enterprise Java Beans (http://java.sun.com/ejb)," pp. 2005.
• 25: "Java RMI (http://java.sun.com/rmi/)," 2005.
• 26: C. A. Mattmann, S. Malek, et al., "GLIDE: A Grid-based Lightweight Infrastructure for Data-intensive Environments," in Proc. of European Grid Conference, Amsterdam, the Netherlands, pp. 68--77, 2005.
• 27: DCMI, "Dublin Core Metadata Element Set," 1999.
• 28: T. Berners-Lee, R. Fielding, et al., "Uniform Resource Identifiers (URI): Generic Syntax," 1998.
• 29: "Open Channel Foundation: Request Object Oriented Data Technology (OODT) - (http://openchannelsoftware.com/orders/index.php?group_id=332)," 2005.
• 30: J. Steven Hughes , Susan K. McMahon, The Planetary Data System. A Case Study in the Development and Management of Meta-Data for a Scientific Digital Library, Proceedings of the Second European Conference on Research and Advanced Technology for Digital Libraries, p.335-350, September 21-23, 1998
• 31: S. Srivastava, Informatics in proteomics. Boca Raton, FL: Taylor & Francis/CRC Press, 2005.
• 32: "UGS Products: TeamCenter (http://www.ugs.com/products/teamcenter/)," 2005.
• 33: "Document Management | Xerox Docushre (http://docushare.xerox.com/ds/)," 2005.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

General Terms:
Design, Standardization

Keywords:
OODT, data management, software architecture