RUBiS revisited: why J2EE benchmarking is hard

ABSTRACT

We have replicated the experiments of Cecchet et al detailed in "Performance and Scalability of EJB Applications" at OOPSLA '02. We report on our experiences configuring, deploying and tuning Enterprise software, and provide evidence that many of the conclusions of the original work are misleading or cannot be generalized.

REFERENCES

• 1: Emmanuel Cecchet , Julie Marguerite , Willy Zwaenepoel, Performance and scalability of EJB applications, Proceedings of the 17th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, November 04-08, 2002, Seattle, Washington, USA
• 2: eBay. http://www.ebay.com.
• 3: JBoss Application Server. http://www.jboss.org.
• 4: Java Open Application Server. http://jonas.objectweb.org.
• 5: Jakarta Tomcat. http://jakarta.apache.org.
• 6: The Apache HTTPD Server Project. http://httpd.apache.org.
• 7: EJB 2.1 specification. http://java.sun.com/products/ejb/index.jsp.
• 8: One Billion Transactions a Day: A Java{tm} 2 Platform, Enterprise Edition (J2EE{tm}) System Architected with the Core J2EE Patterns. http://javaoneonline.mentorware.net/.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques; Performance attributes

  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Complexity measures; Performance measures

General Terms:
Design, Experimentation, Measurement, Performance

Keywords:
EJB container design, performance, profiling, scalability