BitDew: a programmable environment for large-scale data management and distribution

ABSTRACT

Desktop Grids use the computing, network and storage resources from idle desktop PC's distributed over multiple-LAN's or the Internet to compute a large variety of resource-demanding distributed applications. While these applications need to access, compute, store and circulate large volumes of data, little attention has been paid to data management in such large-scale, dynamic, heterogeneous, volatile and highly distributed Grids. In most cases, data management relies on ad-hoc solutions, and providing a general approach is still a challenging issue.

To address this problem, we propose the BitDew framework, a programmable environment for automatic and transparent data management on computational Desktop Grids. This paper describes the BitDew programming interface, its architecture, and the performance evaluation of its runtime components. BitDew relies on a specific set of meta-data to drive key data management operations, namely life cycle, distribution, placement, replication and fault-tolerance with a high level of abstraction. The Bitdew runtime environment is a flexible distributed service architecture that integrates modular P2P components such as DHT's for a distributed data catalog and collaborative transport protocols for data distribution. Through several examples, we describe how application programmers and Bitdew users can exploit Bitdew's features. The performance evaluation demonstrates that the high level of abstraction and transparency is obtained with a reasonable overhead, while offering the benefit of scalability, performance and fault tolerance with little programming cost.

REFERENCES

• 1: Atul Adya , William J. Bolosky , Miguel Castro , Gerald Cermak , Ronnie Chaiken , John R. Douceur , Jon Howell , Jacob R. Lorch , Marvin Theimer , Roger P. Wattenhofer, Farsite: federated, available, and reliable storage for an incompletely trusted environment, ACM SIGOPS Operating Systems Review, v.36 n.SI, Winter 2002 [doi>10.1145/844128.844130]
• 2: Luc Onana Alima , Sameh El-Ansary , Per Brand , Seif Haridi, DKS (N, k, f): A Family of Low Communication, Scalable and Fault-Tolerant Infrastructures for P2P Applications, Proceedings of the 3st International Symposium on Cluster Computing and the Grid, p.344, May 12-15, 2003
• 3: William Allcock , John Bresnahan , Rajkumar Kettimuthu , Michael Link, The Globus Striped GridFTP Framework and Server, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.54, November 12-18, 2005 [doi>10.1109/SC.2005.72]
• 4: David P. Anderson, BOINC: A System for Public-Resource Computing and Storage, Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, p.4-10, November 08-08, 2004 [doi>10.1109/GRID.2004.14]
• 5: R. Anderson. The Eternity Service. In Proceedings of Pragocrypt '96, 1996.
• 6: N. Andrade, W. Cirne, F. Brasileiro, and P. Roisenberg. OurGrid: An Approach to Easily Assemble Grids with Equitable Resource Sharing. In Proceedings of the 9th Workshop on Job Scheduling Strategies for Parallel Processing, June 2003.
• 7: Mario Antonioletti , Malcolm Atkinson , Rob Baxter , Andrew Borley , Neil P. Chue Hong , Brian Collins , Neil Hardman , Alastair C. Hume , Alan Knox , Mike Jackson , Amy Krause , Simon Laws , James Magowan , Norman W. Paton , Dave Pearson , Tom Sugden , Paul Watson , Martin Westhead, The design and implementation of Grid database services in OGSA-DAI: Research Articles, Concurrency and Computation: Practice & Experience, v.17 n.2-4, p.357-376, February 2005 [doi>10.1002/cpe.v17:2/4]
• 8: G. Antoniu. L. Bougé, and M. Jan. JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid. Scalable Computing: Practice and Experience, 6(3):45--55, September 2005.
• 9: Alessandro Bassi , Micah Beck , Graham Fagg , Terry Moore , James S. Plank , Martin Swany , Rich Wolski, The Internet Backplane Protocol: A Study in Resource Sharing, Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, p.194, May 21-24, 2002
• 10: Raphaël Bolze , Franck Cappello , Eddy Caron , Michel Daydé , Frédéric Desprez , Emmanuel Jeannot , Yvon Jégou , Stephane Lanteri , Julien Leduc , Noredine Melab , Guillaume Mornet , Raymond Namyst , Pascale Primet , Benjamin Quetier , Olivier Richard , El-Ghazali Talbi , Iréa Touche, Grid'5000: A Large Scale And Highly Reconfigurable Experimental Grid Testbed, International Journal of High Performance Computing Applications, v.20 n.4, p.481-494, November 2006 [doi>10.1177/1094342006070078]
• 11: Ali Raza Butt , Troy A. Johnson , Yili Zheng , Y. Charlie Hu, Kosha: A Peer-to-Peer Enhancement for the Network File System, Proceedings of the 2004 ACM/IEEE conference on Supercomputing, p.51, November 06-12, 2004 [doi>10.1109/SC.2004.32]
• 12: B. Cohen. Incentives Build Robustness in BitTorrent. In Workshop on Economics of Peer-to-Peer Systems, Berkeley, 2003.
• 13: Grid-Based Metadata Services, Proceedings of the 16th International Conference on Scientific and Statistical Database Management, p.393, June 21-23, 2004 [doi>10.1109/SSDBM.2004.39]
• 14: Enabling Grids for E-Science in Europe.
• 15: Gilles Fedak , Cecile Germain , Vincent Neri , Franck Cappello, XtremWeb: A Generic Global Computing System, Proceedings of the 1st International Symposium on Cluster Computing and the Grid, p.582, May 15-18, 2001
• 16: Y. Fernandess and D. Malkhi. On Collaborative Content Distribution using Multi-Message Gossip. In Proceeding of IEEE IPDPS, Rhodes Island, 2006.
• 17: David Gelernter, Generative communication in Linda, ACM Transactions on Programming Languages and Systems (TOPLAS), v.7 n.1, p.80-112, Jan. 1985 [doi>10.1145/2363.2433]
• 18: C. Gkantsidis, J. Miller, and P. Rodriguez. Anatomy of a P2P Content Distribution System with Network Coding. In IPTPS '06, California, U.S.A., 2006.
• 19: Christos Gkantsidis , John Miller , Pablo Rodriguez, Comprehensive view of a live network coding P2P system, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177104]
• 20: C. Gkantsidis and P. Rodriguez. Network Coding for Large Scale Content Distribution. In Proceedings of IEEE/INFOCOM 2005, Miami, USA, March 2005.
• 21: A. Iamnitchi, S. Doraimani, and G. Garzoglio. Filecules in High-Energy Physics: Characteristics and Impact on Resource Management. In proceeding of 15th IEEE International Symposium on High Performance Distributed Computing HPDC 15, Paris, 2006.
• 22: A. lamnitchi, S. Doraimani, and G. Garzoglio. Filecules in High-Energy Physics: Characteristics and Impact on Resource Management. In HPDC 2006, Paris, 2006.
• 23: H. Jin, M. Xiong, S. Wu, and D. Zou. Replica Based Distributed Metadata Management in Grid Environment. Computational Science - Lecture Notes in Computer Science, Springer-Verlag, 3994:1055--1062, 2006.
• 24: Katarzyna Keahey , Karl Doering , Ian Foster, From Sandbox to Playground: Dynamic Virtual Environments in the Grid, Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, p.34-42, November 08-08, 2004 [doi>10.1109/GRID.2004.32]
• 25: D. Kondo, F. Araujo, P. Malecot, P. Domingues, L. M. Silva, G. Fedak, and F. Cappello. Characterizing Result Errors in Internet Desktop Grids. In European Conference on Parallel and Distributed Computing (Euro-Par), 2007.
• 26: Derrick Kondo , Andrew A. Chien , Henri Casanova, Resource Management for Rapid Application Turnaround on Enterprise Desktop Grids, Proceedings of the 2004 ACM/IEEE conference on Supercomputing, p.17, November 06-12, 2004 [doi>10.1109/SC.2004.50]
• 27: Tevfik Kosar , Miron Livny, Stork: Making Data Placement a First Class Citizen in the Grid, Proceedings of the 24th International Conference on Distributed Computing Systems (ICDCS'04), p.342-349, March 24-26, 2004
• 28: John Kubiatowicz , David Bindel , Yan Chen , Steven Czerwinski , Patrick Eaton , Dennis Geels , Ramakrishna Gummadi , Sean Rhea , Hakim Weatherspoon , Chris Wells , Ben Zhao, OceanStore: an architecture for global-scale persistent storage, Proceedings of the ninth international conference on Architectural support for programming languages and operating systems, p.190-201, November 2000, Cambridge, Massachusetts, United States
• 29: M. J. Litzkow, M. Livny, and M. W. Mutka. Condor - A Hunter of Idle Workstations. In Proceedings of the 8th International Conference on Distributed Computing Systems (ICDCS), pages 104--111, Washington, DC, 1988. IEEE Computer Society.
• 30: J. Luna, M. Flouris. M. Marazakis, and A. Bilas. Providing security to the Desktop Data Grid. In 2nd Workshop on Desktop Grids and Volunteer Computing Systems (PCGrid '08), 2008.
• 31: Petar Maymounkov , David Mazières, Kademlia: A Peer-to-Peer Information System Based on the XOR Metric, Revised Papers from the First International Workshop on Peer-to-Peer Systems, p.53-65, March 07-08, 2002
• 32: Ekow Otoo , Doron Rotem , Alexandru Romosan, Optimal File-Bundle Caching Algorithms for Data-Grids, Proceedings of the 2004 ACM/IEEE conference on Supercomputing, p.6, November 06-12, 2004 [doi>10.1109/SC.2004.36]
• 33: PPDG. From Fabric to Physics. Technical report, The Particle Physics Data Grid, 2006.
• 34: A. Reinefeld, F. Schintke, and T. Schatt. Scalable and Self-Optimizing Data Grids. Annual Review of Scalable Computing, Singapore University Press, 6:30--60, 2004.
• 35: Antony I. T. Rowstron , Peter Druschel, Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems, Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms Heidelberg, p.329-350, November 12-16, 2001
• 36: E. Santos-Neto, W. Cirne, F. Brasileiro, and A. Lima. Exploiting Replication and Data Reuse to Efficiently Schedule Data-intensive Applications on Grids. In Proceedings of the 10th Workshop on Job Scheduling Strategies for Parallel Processing, 2004.
• 37: Luis F. G. Sarmenta, Sabotage-tolerance mechanisms for volunteer computing systems, Future Generation Computer Systems, v.18 n.4, p.561-572, March 2002 [doi>10.1016/S0167-739X(01)00077-2]
• 38: Gurmeet Singh , Shishir Bharathi , Ann Chervenak , Ewa Deelman , Carl Kesselman , Mary Manohar , Sonal Patil , Laura Pearlman, A Metadata Catalog Service for Data Intensive Applications, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.33, November 15-21, 2003
• 39: Ion Stoica , Robert Morris , David Karger , M. Frans Kaashoek , Hari Balakrishnan, Chord: A scalable peer-to-peer lookup service for internet applications, Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, p.149-160, August 2001, San Diego, California, United States
• 40: Osamu Tatebe , Youhei Morita , Satoshi Matsuoka , Noriyuki Soda , Satoshi Sekiguchi, Grid Datafarm Architecture for Petascale Data Intensive Computing, Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, p.102, May 21-24, 2002
• 41: Sudharshan S. Vazhkudai , Xiaosong Ma , Vincent W. Freeh , Jonathan W. Strickland , Nandan Tammineedi , Stephen L. Scott, FreeLoader: Scavenging Desktop Storage Resources for Scientific Data, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.56, November 12-18, 2005 [doi>10.1109/SC.2005.27]
• 42: Baohua Wei , G. Fedak , F. Cappello, Scheduling Independent Tasks Sharing Large Data Distributed with BitTorrent, Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing, p.219-226, November 13-14, 2005 [doi>10.1109/GRID.2005.1542745]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features
          Subjects: Frameworks

General Terms:
Design, Experimentation, Measurement, Performance