Exploring data reliability tradeoffs in replicated storage systems

ABSTRACT

This paper explores the feasibility of a cost-efficient storage architecture that offers the reliability and access performance characteristics of a high-end system. This architecture exploits two opportunities: First, scavenging idle storage from LAN-connected desktops not only offers a low-cost storage space, but also high I/O throughput by aggregating the I/O channels of the participating nodes. Second, the two components of data reliability - durability and availability - can be decoupled to control overall system cost. To capitalize on these opportunities, we integrate two types of components: volatile, scavenged storage and dedicated, yet low-bandwidth durable storage. On the one hand, the durable storage forms a low-cost back-end that enables the system to restore the data the volatile nodes may lose. On the other hand, the volatile nodes provide a high-throughput front-end.

While integrating these components has the potential to offer a unique combination of high throughput, low cost, and durability, a number of concerns need to be addressed to architect and correctly provision the system. To this end, we develop analytical- and simulation based tools to evaluate the impact of system characteristics (e.g., bandwidth limitations on the durable and the volatile nodes) and design choices (e.g., replica placement scheme) on data availability and the associated system costs (e.g., maintenance traffic). Further, we implement and evaluate a prototype of the proposed architecture: namely a GridFTP server that aggregates volatile resources. Our evaluation demonstrates an impressive, up to 800MBps transfer throughput for the new GridFTP service.

REFERENCES

• 1: Atul Adya , William J. Bolosky , Miguel Castro , Gerald Cermak , Ronnie Chaiken , John R. Douceur , Jon Howell , Jacob R. Lorch , Marvin Theimer , Roger P. Wattenhofer, Farsite: federated, available, and reliable storage for an incompletely trusted environment, ACM SIGOPS Operating Systems Review, v.36 n.SI, Winter 2002 [doi>10.1145/844128.844130]
• 2: Samer Al-Kiswany , Matei Ripeanu , Sudharshan S. Vazhkudai , Abdullah Gharaibeh, stdchk: A Checkpoint Storage System for Desktop Grid Computing, Proceedings of the 2008 The 28th International Conference on Distributed Computing Systems, p.613-624, June 17-20, 2008 [doi>10.1109/ICDCS.2008.19]
• 3: Allcock, W., et al. GridFTP: Protocol Extensions to FTP for the Grid. Global Grid ForumGFD-RP, 20(2003).
• 4: William Allcock , John Bresnahan , Rajkumar Kettimuthu , Michael Link , Catalin Dumitrescu , Ioan Raicu , Ian Foster, The Globus Striped GridFTP Framework and Server, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.54, November 12-18, 2005 [doi>10.1109/SC.2005.72]
• 5: Amazon Web Services. http:/s3.amazonaws.com.
• 6: Ranjita Bhagwan , Kiran Tati , Yu-Chung Cheng , Stefan Savage , Geoffrey M. Voelker, Total recall: system support for automated availability management, Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation, p.25-25, March 29-31, 2004, San Francisco, California
• 7: Charles Blake , Rodrigo Rodrigues, High availability, scalable storage, dynamic peer networks: pick two, Proceedings of the 9th conference on Hot Topics in Operating Systems, p.1-1, May 18-21, 2003, Lihue, Hawaii
• 8: William J. Bolosky , John R. Douceur , David Ely , Marvin Theimer, Feasibility of a serverless distributed file system deployed on an existing set of desktop PCs, Proceedings of the 2000 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.34-43, June 18-21, 2000, Santa Clara, California, United States
• 9: David W. Brubeck , Lawrence A. Rowe, Hierarchical Storage Management in a Distributed VOD System, IEEE MultiMedia, v.3 n.3, p.37-47, September 1996 [doi>10.1109/93.556538]
• 10: Philip H. Carns , Walter B. Ligon, III , Robert B. Ross , Rajeev Thakur, PVFS: a parallel file system for linux clusters, Proceedings of the 4th annual Linux Showcase & Conference, p.28-28, October 10-14, 2000, Atlanta, Georgia
• 11: Byung-Gon Chun , Frank Dabek , Andreas Haeberlen , Emil Sit , Hakim Weatherspoon , M. Frans Kaashoek , John Kubiatowicz , Robert Morris, Efficient replica maintenance for distributed storage systems, Proceedings of the 3rd conference on Networked Systems Design & Implementation, p.4-4, May 08-10, 2006, San Jose, CA
• 12: Frank Dabek , M. Frans Kaashoek , Robert T. Morris, A distributed hash table, Massachusetts Institute of Technology, Cambridge, MA, 2006
• 13: Frank Dabek , M. Frans Kaashoek , David Karger , Robert Morris , Ion Stoica, Wide-area cooperative storage with CFS, ACM SIGOPS Operating Systems Review, v.35 n.5, Dec. 2001
• 14: Peter Druschel , Antony Rowstron, PAST: A Large-Scale, Persistent Peer-to-Peer Storage Utility, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.75, May 20-22, 2001
• 15: Fuhrmann, P. dCache, the commodity cache. In proceedings of the Twelfth NASA Goddard and Twenty First IEEE Conference on Mass Storage Systems and Technologies, Washington DC., 2004.
• 16: Abdullah Gharaibeh , Samer Al-Kiswany , Matei Ripeanu, Configurable security for scavenged storage systems, Proceedings of the 4th ACM international workshop on Storage security and survivability, October 31-31, 2008, Alexandria, Virginia, USA [doi>10.1145/1456469.1456480]
• 17: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 18: Heath, T., et al. The Shape of Failure. In Proceedings of the First Workshop on Evaluating and Architecting System dependabilitY (EASY). 2001.
• 19: Leonard Kleinrock, Theory, Volume 1, Queueing Systems, Wiley-Interscience, 1975
• 20: John Kubiatowicz , David Bindel , Yan Chen , Steven Czerwinski , Patrick Eaton , Dennis Geels , Ramakrishna Gummadi , Sean Rhea , Hakim Weatherspoon , Chris Wells , Ben Zhao, OceanStore: an architecture for global-scale persistent storage, ACM SIGARCH Computer Architecture News, v.28 n.5, p.190-201, Dec. 2000
• 21: Geoffrey Lefebvre , Michael J. Feeley, Separating durability and availability in self-managed storage, Proceedings of the 11th workshop on ACM SIGOPS European workshop, p.28-es, September 19-22, 2004, Leuven, Belgium [doi>10.1145/1133572.1133576]
• 22: Andrew W. Leung , Ethan L. Miller , Stephanie Jones, Scalable security for petascale parallel file systems, Proceedings of the 2007 ACM/IEEE conference on Supercomputing, November 10-16, 2007, Reno, Nevada [doi>10.1145/1362622.1362644]
• 23: Microsystems, S. and others. NFS: Network file system protocol specification. Request for Comments, 1094(1988).
• 24: Athicha Muthitacharoen , Robert Morris , Thomer M. Gil , Benjie Chen, Ivy: a read/write peer-to-peer file system, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060293]
• 25: Nurmi, D., et al. Modeling Machine Availability in Enterprise and Wide-Area Distributed Computing Environments. LECTURE NOTES IN COMPUTER SCIENCE, 3648(2005), 432.
• 26: Mayur R. Palankar , Adriana Iamnitchi , Matei Ripeanu , Simson Garfinkel, Amazon S3 for science grids: a viable solution?, Proceedings of the 2008 international workshop on Data-aware distributed computing, p.55-64, June 24-24, 2008, Boston, MA, USA [doi>10.1145/1383519.1383526]
• 27: Qiao Lian , Wei Chen , Zheng Zhang, On the Impact of Replica Placement to the Reliability of Distributed Brick Storage Systems, Proceedings of the 25th IEEE International Conference on Distributed Computing Systems, p.187-196, June 06-10, 2005 [doi>10.1109/ICDCS.2005.56]
• 28: Ramabhadran, S. and Pasquale, J. Analysis of Long-Running Replicated Systems. In INFOCOM, 2006.
• 29: Rausand, M. and Hoyland, A. System Reliability Theory: Models, Statistical Methods, and Applications. Wiley-Interscience, 2004.
• 30: Simpy homepage. http://simpy.sourceforge.net/ (2009).
• 31: Sudharshan S. Vazhkudai , Xiaosong Ma , Vincent W. Freeh , Jonathan W. Strickland , Nandan Tammineedi , Stephen L. Scott, FreeLoader: Scavenging Desktop Storage Resources for Scientific Data, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.56, November 12-18, 2005 [doi>10.1109/SC.2005.27]
• 32: Hakim Weatherspoon , John Kubiatowicz, Design and evaluation of distributed wide-area on-line archival storage systems, University of California at Berkeley, Berkeley, CA, 2006
• 33: Sage A. Weil , Scott A. Brandt , Ethan L. Miller , Darrell D. E. Long , Carlos Maltzahn, Ceph: a scalable, high-performance distributed file system, Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, p.22-22, November 06-08, 2006, Seattle, WA

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.3 File Systems Management
          Subjects: Distributed file systems

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Modeling and prediction

General Terms:
Measurement, Performance, Reliability

Keywords:
gridFTP, mosastore, reliability, storage system