A framework for measurement based performance modeling

ABSTRACT

Techniques for performance modeling are broadly classified into measurement, analytical and simulation based techniques. Measurement based performance modeling is commonly adopted in practice. Measurement based modeling requires the execution of a large number of performance tests to build accurate performance models. These performance tests must be repeated for every release or build of an application. This is a time consuming and error-prone manual process.

In this paper, we present a framework for the systematic and automated building of measurement based performance models. The framework is based on our experience in performance modeling of two large applications: the DVD Store application by Dell and another larger enterprise application. We use the Dell DVD Store application as a running example to demonstrate the various steps in our framework. We present the benefits and shortcomings of our framework. We discuss the expected reduction in effort due to adopting our framework.

REFERENCES

• 1: Jayshankar Sankarasetty , Kevin Mobley , Libby Foster , Tad Hammer , Terri Calderone, Software performance in the real world: personal lessons from the performance trauma team, Proceedings of the 6th international workshop on Software and performance, February 05-08, 2007, Buenes Aires, Argentina [doi>10.1145/1216993.1217027]
• 2: Jaffe, D., Muirhead T. 2005. The Open Source DVD Store Application. http://www.dell.com/downloads/global/power/ps3q05-20050217-Jaffe-OE.pdf
• 3: Simon F. Goldsmith , Alex S. Aiken , Daniel S. Wilkerson, Measuring empirical computational complexity, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287681]
• 4: Jain R. 1992. The art of computer systems performance analysis. John Wiley.
• 5: Jinlin Yang , David Evans , Deepali Bhardwaj , Thirumalesh Bhat , Manuvir Das, Perracotta: mining temporal API rules from imperfect traces, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134325]
• 6: Muirhead T., Jaffe, D. 2005. Migrating enterprise databases from Sun servers to the Dell PowerEdge 2850 running Microsoft Windows Server 2003. http://www.dell.com/downloads/global/power/ps1q05-20040270-Jaffe.pdf
• 7: Murray Woodside , Greg Franks , Dorina C. Petriu, The Future of Software Performance Engineering, 2007 Future of Software Engineering, p.171-187, May 23-25, 2007 [doi>10.1109/FOSE.2007.32]
• 8: Neil J. Gunther, Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services, Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 9: Monchai Sopitkamol , Daniel A. Menascé, A method for evaluating the impact of software configuration parameters on e-commerce sites, Proceedings of the 5th international workshop on Software and performance, p.53-64, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071027]
• 10: Cemal Yilmaz , Arvind S. Krishna , Atif Memon , Adam Porter , Douglas C. Schmidt , Aniruddha Gokhale , Balachandran Natarajan, Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062515]
• 11: Research In Motion. Capacity calculator for BlackBerry Enterprise Server 4.1 for Microsoft Exchange. http://www.blackberry.com/select/toolkit/dls/BlackBerry_Enterprise_Server_Version_4.1.0_for_Microsoft_Exchange_Capacity_Calculator.xls
• 12: Mania D. and Murphy J. 2002. Framework for predicting the performance of component-based systems. In Proceedings of IEEE 10th International Conference on Software, Telecommunications and Computer Networks (Italy, October 2002). SoftCOM 2002. pp. 46--50, ISBN 953 6114 52 6.
• 13: WebLOAD load testing stress testing tool. http://www.webload.org/
• 14: HP LoadRunner Software. https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-126-17%5E8_4000_100__
• 15: http://www.clarkware.com/software/JUnitPerf.html
• 16: http://www.junit.org
• 17: Tauseef A. Israr , Danny H. Lau , Greg Franks , Murray Woodside, Automatic generation of layered queuing software performance models from commonly available traces, Proceedings of the 5th international workshop on Software and performance, p.147-158, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071037]
• 18: S. E. Sim. 1998. Supporting multiple program comprehension strategies during software maintenance. Masters thesis, University of Toronto, 1998.
• 19: Research In Motion. BlackBerry Enterprise Server for Microsoft Exchange version 4.1 performance benchmarking. http://www.blackberry.com/knowledgecenterpublic/livelink.exe/fetch/2000/8067/645045/7963/7965/1180408/Performance_Benchmarking_Guide.pdf?nodeid=1367404&vernum=0
• 20: Odysseas Pentakalos , Mark Friedman, Windows 2000 performance guide: help for Windows 2000 administrators, O'Reilly & Associates, Inc., Sebastopol, CA, 2002
• 21: Connie U. Smith , Catalina M. Lladó , Vittorio Cortellessa , Antinisca Di Marco , Lloyd G. Williams, From UML models to software performance results: an SPE process based on XML interchange formats, Proceedings of the 5th international workshop on Software and performance, p.87-98, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071030]
• 22: Tao Xie , Darko Marinov , David Notkin, Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests, Proceedings of the 19th IEEE international conference on Automated software engineering, p.196-205, September 20-24, 2004 [doi>10.1109/ASE.2004.61]
• 23: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 24: The R project for statistical computing. http://www.r-project.org/
• 25: Juran, J. M., Godfrey A. B. 1988. Juran's Quality Handbook. McGraw-Hill Professional.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Modeling techniques

General Terms:
Measurement, Performance

Keywords:
framework, measurement, modeling, performance