How is the weather tomorrow?: towards a benchmark for the cloud

ABSTRACT

Traditionally, the goal of benchmarking a software system is to evaluate its performance under a particular workload for a fixed configuration. The most prominent examples for evaluating transactional database systems as well as other components on top (such as a application-servers or web-servers) are the various TPC benchmarks.

In this paper we argue that traditional benchmarks (like the TPC benchmarks) are not sufficient for analyzing the novel cloud services. Moreover, we present some initial ideas how such a new benchmark should look like that fits better to the characteristics of cloud computing (e.g., scalability, pay-per-use and fault-tolerance). The main challenge of such a new benchmark is to make the reported results comparable because different providers offer different services with different capabilities and guarantees.

REFERENCES

• 1: 28msec, Inc. Sausalito, Apr. 2009. http://www.28msec.com/.
• 2: Apache. HBase, Apr. 2009. http://hadoop.apache.org/hbase/.
• 3: M. Armbrust et al. Above the clouds: A berkeley view of cloud computing. Technical Report UCB/EECS-2009-28, 2009.
• 4: Matthias Brantner , Daniela Florescu , David Graf , Donald Kossmann , Tim Kraska, Building a database on S3, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada [doi>10.1145/1376616.1376645]
• 5: Eric A. Brewer, Towards robust distributed systems (abstract), Proceedings of the nineteenth annual ACM symposium on Principles of distributed computing, p.7, July 16-19, 2000, Portland, Oregon, United States [doi>10.1145/343477.343502]
• 6: Michael Cafarella , Edward Chang , Andrew Fikes , Alon Halevy , Wilson Hsieh , Alberto Lerner , Jayant Madhavan , S. Muthukrishnan, Data management projects at Google, ACM SIGMOD Record, v.37 n.1, March 2008 [doi>10.1145/1374780.1374789]
• 7: Jean-Daniel Cryans , Alain April , Alain Abran, Criteria to Compare Cloud Computing with Current Database Technology, Proceedings of the International Conferences on Software Process and Product Measurement, November 18-19, 2008, Munich, Germany [doi>10.1007/978-3-540-89403-2_11]
• 8: Facebook. Cassandra - A distributed structured storage system, Apr. 2009. http://cwiki.apache.org/confluence/display/CSDR/Index.
• 9: S. Garfinkel. An evaluation of Amazon's grid computing services: EC2, S3 and SQS, 2007.
• 10: Seth Gilbert , Nancy Lynch, Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services, ACM SIGACT News, v.33 n.2, June 2002 [doi>10.1145/564585.564601]
• 11: Brian Hayes, Cloud computing, Communications of the ACM, v.51 n.7, July 2008 [doi>10.1145/1364782.1364786]
• 12: Heroku. Heroku - Instant Ruby Platform, Apr. 2009. http://heroku.com/.
• 13: TPC. TPC-W Benchmark 1.8. http://www.tpc.org/tpcw/, 2002.
• 14: TPC. TPC-H Benchmark 2.8. http://www.tpc.org/tpch/, 2008.
• 15: TPC. TPC-C Benchmark 5.10.1. http://www.tpc.org/tpch/, 2009.
• 16: Werner Vogels, Data access patterns in the Amazon.com technology platform, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems
          Subjects: Distributed databases

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Performance attributes; Measurement techniques

General Terms:
Design, Performance