Building a database on S3

ABSTRACT

There has been a great deal of hype about Amazon's simple storage service (S3). S3 provides infinite scalability and high availability at low cost. Currently, S3 is used mostly to store multi-media documents (videos, photos, audio) which are shared by a community of people and rarely updated. The purpose of this paper is to demonstrate the opportunities and limitations of using S3 as a storage system for general-purpose database applications which involve small objects and frequent updates. Read, write, and commit protocols are presented. Furthermore, the cost ($), performance, and consistency properties of such a storage system are studied.

REFERENCES

• 1: M. Atwood. A storage engine for Amazon S3. MySQL Conference and Expo, 2007. http://fallenpegasus.com/code/mysql-awss3.
• 2: R. Bayer and M. Schkolnick. Concurrency of operations on B-trees. Acta Inf., 9(1):1--21, 1977.
• 3: Hal Berenson , Phil Bernstein , Jim Gray , Jim Melton , Elizabeth O'Neil , Patrick O'Neil, A critique of ANSI SQL isolation levels, Proceedings of the 1995 ACM SIGMOD international conference on Management of data, p.1-10, May 22-25, 1995, San Jose, California, United States
• 4: Philip A. Bernstein , Vassco Hadzilacos , Nathan Goodman, Concurrency control and recovery in database systems, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1987
• 5: Alexandros Biliris, The performance of three database storage structures for managing large objects, Proceedings of the 1992 ACM SIGMOD international conference on Management of data, p.276-285, June 02-05, 1992, San Diego, California, United States
• 6: D. Brunner. Scalability: Set Amazon?s servers on fire, not yours. Talk at ETech Conf., 2007. http://blogs.smugmug.com/don/files/ETech-SmugMug-Amazon-2007.pdf.
• 7: Giuseppe DeCandia , Deniz Hastorun , Madan Jampani , Gunavardhan Kakulapati , Avinash Lakshman , Alex Pilchin , Swaminathan Sivasubramanian , Peter Vosshall , Werner Vogels, Dynamo: amazon's highly available key-value store, Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, October 14-17, 2007, Stevenson, Washington, USA
• 8: Ian Foster , Carl Kesselman, The Grid 2: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2003
• 9: S. Garfinkel. An evaluation of Amazon?s grid computing services: EC2, S3, and SQS. Technical Report TR-08-07, Harvard University, 2007.
• 10: Seth Gilbert , Nancy Lynch, Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services, ACM SIGACT News, v.33 n.2, June 2002 [doi>10.1145/564585.564601]
• 11: Jim Gray , Andreas Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1992
• 12: Joseph M. Hellerstein, Architectures and algorithms for internet-scale (p2p) data management, Proceedings of the Thirtieth international conference on Very large data bases, p.1244-1244, August 31-September 03, 2004, Toronto, Canada
• 13: Philip L. Lehman , s. Bing Yao, Efficient locking for concurrent operations on B-trees, ACM Transactions on Database Systems (TODS), v.6 n.4, p.650-670, Dec. 1981 [doi>10.1145/319628.319663]
• 14: David Lomet, Replicated indexes for distributed data, Proceedings of the fourth international conference on on Parallel and distributed information systems, p.108-119, December 18-20, 1996, Miami Beach, Florida, United States
• 15: Mark L. McAuliffe , Michael J. Carey , Marvin H. Solomon, Towards effective and efficient free space management, Proceedings of the 1996 ACM SIGMOD international conference on Management of data, p.389-400, June 04-06, 1996, Montreal, Quebec, Canada
• 16: RightScale LLC. Redundant MySQL set-up for Amazon EC2, November 2007. http://info.rightscale.com/2007/8/20/redundant-mysql.
• 17: J. W. Ross , G. Westerman, Preparing for utility computing: The role of IT architecture and relationship management, IBM Systems Journal, v.43 n.1, p.5-19, January 2004
• 18: M. Stonebraker. The case for shared nothing. IEEE Data Eng. Bulletin, 9(1):4--9, 1986.
• 19: Andrew S. Tanenbaum , Maarten van Steen, Distributed Systems: Principles and Paradigms (2nd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2006
• 20: TPC. TPC Benchmark W. Specification Version 1.8 of TPC Council, 2002.
• 21: Werner Vogels, Data access patterns in the Amazon.com technology platform, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 22: Gerhard Weikum , Gottfried Vossen, Transactional information systems: theory, algorithms, and the practice of concurrency control and recovery, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2001

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.2 Physical Design

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Distributed databases; Concurrency

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
aws, cloud computing, concurrency, cost trade-off, database, ec2, eventual consistency, performance, s3, simpledb, sqs, storage system