The Future of Software Performance Engineering

ABSTRACT

Performance is a pervasive quality of software systems; everything affects it, from the software itself to all underlying layers, such as operating system, middleware, hardware, communication networks, etc. Software Performance Engineering encompasses efforts to describe and improve performance, with two distinct approaches: an early-cycle predictive modelbased approach, and a late-cycle measurement-based approach. Current progress and future trends within these two approaches are described, with a tendency (and a need) for them to converge, in order to cover the entire development cycle.

REFERENCES

• 1: [1] B. Abrahao, V. Almeida, J. Almeida, A. Zhang, D. Beyer, F. Safai, "Self-Adaptive SLA-Driven Capacity Management for Internet Services", Proc. 10th IEEE/IFIP Net Operations and Management Symp., 2006, pp. 557-568.
• 2: Characterizing the scalability of a large web-based shopping system, ACM Transactions on Internet Technology (TOIT), v.1 n.1, p.44-69, Aug. 2001 [doi>10.1145/383034.383036]
• 3: Alberto Avritzer , Elaine J. Weyuker, Deriving workloads for performance testing, Software—Practice & Experience, v.26 n.6, p.613-633, June 1996 [doi>10.1002/(SICI)1097-024X(199606)26:6<613::AID-SPE23>3.3.CO;2-X]
• 4: Alberto Avritzer , Joe Kondek , Danielle Liu , Elaine J. Weyuker, Software performance testing based on workload characterization, Proceedings of the 3rd international workshop on Software and performance, July 24-26, 2002, Rome, Italy [doi>10.1145/584369.584373]
• 5: Rajive L. Bagrodia , Chien-Chung Shen, MIDAS: Integrated Design and Simulation of Distributed Systems, IEEE Transactions on Software Engineering, v.17 n.10, p.1042-1058, October 1991 [doi>10.1109/32.99192]
• 6: [6] S. Balsamo and M. Marzolla. "Simulation Modeling of UML Software Architectures", Proc. ESM'03, Nottingham (UK), June 2003.
• 7: Simonetta Balsamo , Antinisca Di Marco , Paola Inverardi , Marta Simeoni, Model-Based Performance Prediction in Software Development: A Survey, IEEE Transactions on Software Engineering, v.30 n.5, p.295-310, May 2004 [doi>10.1109/TSE.2004.9]
• 8: Scott Barber, Creating Effective Load Models for Performance Testing with Incomplete Empirical Data, Proceedings of the Web Site Evolution, Sixth IEEE International Workshop on (WSE'04), p.51-59, September 11-11, 2004
• 9: [9] S. Barber, "Beyond performance testing", parts 1-14, IBM DeveloperWorks, Rational Technical Library, 2004, www-128.ibm.com/developerworks/rational/library/4169.html
• 10: [10] S. Barber, "User Community Modeling Language for performance test workloads", http://www-128.ibm.com/ developerworks/rational/library/5219.html, July 2004.
• 11: [11] S. Barber, "Tester PI: Performance Investigator", Better Software, March 2006, pp 20-25.
• 12: Antonia Bertolino , Raffaela Mirandola, Software performance engineering of component-based systems, Proceedings of the 4th international workshop on Software and performance, January 14-16, 2004, Redwood Shores, California
• 13: [13] T. L. Booth, C. A. Wiecek, "Performance Abstract Data Types as a Tool in Software Performance Analysis and Design", IEEE Trans. Software Engineering, v 6, 1980, pp. 138-151.
• 14: [14] L. Chung, B. A. Nixon, E. Yu, J. Mylopoulos, Non-Functional Requirements in Software Engineering, Kluwer, 2000.
• 15: [15] Compuware, Applied Performance Management Survey, Oct. 2006.
• 16: Vittorio Cortellessa , Raffaela Mirandola, Deriving a queueing network based performance model from UML diagrams, Proceedings of the 2nd international workshop on Software and performance, p.58-70, September 2000, Ottawa, Ontario, Canada [doi>10.1145/350391.350406]
• 17: Vittorio Cortellessa , Antinisca Di Marco , Paola Inverardi, Non-Functional Modeling and Validation in Model-Driven Architecture, Proceedings of the Sixth Working IEEE/IFIP Conference on Software Architecture, p.25, January 06-09, 2007 [doi>10.1109/WICSA.2007.30]
• 18: [18] Czarnecki, K. and U. Eisenecker, Generative Programming, Addison Wesley, 2000.
• 19: [19] M. Debusmann and K. Geihs, "Efficient and Transparent Instrumentation of Application Components Using an Aspect-Oriented Approach", in Self-Managing Distributed Systems, vol. LNCS 2867 Springer, 2003, pp. 209-220.
• 20: Robert France , Bernhard Rumpe, Model-driven Development of Complex Software: A Research Roadmap, 2007 Future of Software Engineering, p.37-54, May 23-25, 2007 [doi>10.1109/FOSE.2007.14]
• 21: Greg Franks , Dorina Petriu , Murray Woodside , Jing Xu , Peter Tregunno, Layered Bottlenecks and Their Mitigation, Proceedings of the 3rd international conference on the Quantitative Evaluation of Systems, p.103-114, September 11-14, 2006 [doi>10.1109/QEST.2006.23]
• 22: Vahid Garousi , Lionel C. Briand , Yvan Labiche, Traffic-aware stress testing of distributed systems based on UML models, Proceeding of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134340]
• 23: Christopher Gill , Ron Cytron , Douglas Schmidt, Middleware Scheduling Optimization Techniques for Distributed Real-Time and Embedded Systems, Proceedings of the The Seventh IEEE International Workshop on Object-Oriented Real-Time Dependable Systems (WORDS 2002), p.311, January 07-09, 2002
• 24: Curtis E. Hrischuk , C. Murray Woodside , Jerome A. Rolia , Rod Iversen, Trace-Based Load Characterization for Generating Performance Software Models, IEEE Transactions on Software Engineering, v.25 n.1, p.122-135, January 1999 [doi>10.1109/32.748921]
• 25: C. E. Hrischuk , C. M. Woodside, Logical Clock Requirements for Reverse Engineering Scenarios from a Distributed System, IEEE Transactions on Software Engineering, v.28 n.4, p.321-339, April 2002 [doi>10.1109/TSE.2002.995416]
• 26: [26] IBM, IBM Rational PurifyPlus, Purify, PureCoverage, and Quantify: Getting Started, May 2002. G126-5339-00.
• 27: Tauseef Israr , Murray Woodside , Greg Franks, Interaction tree algorithms to extract effective architecture and layered performance models from traces, Journal of Systems and Software, v.80 n.4, p.474-492, April, 2007 [doi>10.1016/j.jss.2006.07.019]
• 28: Xiaohua Jia , Deying Li , Hongwei Du , Jinli Cao, On Optimal Replication of Data Object at Hierarchical and Transparent Web Proxies, IEEE Transactions on Parallel and Distributed Systems, v.16 n.8, p.673-685, August 2005 [doi>10.1109/TPDS.2005.94]
• 29: Mark W. Johnson, Monitoring and Diagnosing Application Response Time with ARM, Proceedings of the IEEE Third International Workshop on Systems Management, p.4, April 22-24, 1998
• 30: Lennard Kerber, Scenario-Based Performance Evaluation of SDL/MSC-Specified Systems, Performance Engineering, State of the Art and Current Trends, p.185-201, January 2001
• 31: [31] D. E. Knuth, "An empirical study of FORTRAN programs", Software Practice and Experience, vol 1 no 2, Apr. 1971.
• 32: Jeff Kramer , Jeff Magee, Self-Managed Systems: an Architectural Challenge, 2007 Future of Software Engineering, p.259-268, May 23-25, 2007 [doi>10.1109/FOSE.2007.19]
• 33: Diwakar Krishnamurthy , Jerome A. Rolia , Shikharesh Majumdar, A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems, IEEE Transactions on Software Engineering, v.32 n.11, p.868-882, November 2006 [doi>10.1109/TSE.2006.106]
• 34: Marin Litoiu , Jerome A. Rolia, Object Allocation for Distributed Applications with Complex Workloads, Proceedings of the 11th International Conference on Computer Performance Evaluation: Modelling Techniques and Tools, p.25-39, March 27-31, 2000
• 35: [35] J. W. S. Liu, Real-Time Systems, Prentice-Hall, 2000.
• 36: Yan Liu , Alan Fekete , Ian Gorton, Design-Level Performance Prediction of Component-Based Applications, IEEE Transactions on Software Engineering, v.31 n.11, p.928-941, November 2005 [doi>10.1109/TSE.2005.127]
• 37: Juan Pablo López-Grao , José Merseguer , Javier Campos, From UML activity diagrams to Stochastic Petri nets: application to software performance engineering, Proceedings of the 4th international workshop on Software and performance, January 14-16, 2004, Redwood Shores, California
• 38: Ying Lu , Tarek Abdelzaher , Chenyang Lu , Lui Sha , Xue Liu, Feedback Control with Queueing-Theoretic Prediction for Relative Delay Guarantees in Web Servers, Proceedings of the The 9th IEEE Real-Time and Embedded Technology and Applications Symposium, p.208, May 27-30, 2003
• 39: Henry Lucas, Jr., Performance Evaluation and Monitoring, ACM Computing Surveys (CSUR), v.3 n.3, p.79-91, Sept. 1971 [doi>10.1145/356589.356590]
• 40: Michael R. Lyu, Software Reliability Engineering: A Roadmap, 2007 Future of Software Engineering, p.153-170, May 23-25, 2007 [doi>10.1109/FOSE.2007.24]
• 41: Peter Maly , C. Murray Woodside, Layered Modeling of Hardware and Software, with Application to a LAN Extension Router, Proceedings of the 11th International Conference on Computer Performance Evaluation: Modelling Techniques and Tools, p.10-24, March 27-31, 2000
• 42: Allen D. Malony , B. Robert Helm, A theory and architecture for automating performance diagnosis, Future Generation Computer Systems, v.18 n.1, p.189-200, September 2001 [doi>10.1016/S0167-739X(01)00052-8]
• 43: Patrick Martin , Wendy Powley , Xiaoyi Xu , Wenhu Tian, Automated Configuration of Multiple Buffer Pools, The Computer Journal, v.49 n.4, p.487-499, July 2006 [doi>10.1093/comjnl/bxl028]
• 44: [44] Stephen J. Mellor, "Agile MDA", Addison Wesley online article, July 23, 2004.
• 45: Daniel A. Menascé , Hassan Gomaa, A Method for Design and Performance Modeling of Client/Server Systems, IEEE Transactions on Software Engineering, v.26 n.11, p.1066-1085, November 2000 [doi>10.1109/32.881718]
• 46: Daniel A. Menascé , Honglei Ruan , Hassan Gomaa, A framework for QoS-aware software components, Proceedings of the 4th international workshop on Software and performance, January 14-16, 2004, Redwood Shores, California
• 47: Paulo Merson , Scott Hissam, Predictability by construction, Companion to the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 16-20, 2005, San Diego, CA, USA [doi>10.1145/1094855.1094900]
• 48: Barton P. Miller , Mark D. Callaghan , Jonathan M. Cargille , Jeffrey K. Hollingsworth , R. Bruce Irvin , Karen L. Karavanic , Krishna Kunchithapadam , Tia Newhall, The Paradyn Parallel Performance Measurement Tool, Computer, v.28 n.11, p.37-46, November 1995 [doi>10.1109/2.471178]
• 49: Andreas Mitschele-Thiel , Bruno Müller-Clostermann, Performance engineering of SDL/MSC systems, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.17, p.1801-1815, June 7, 1999 [doi>10.1016/S1389-1286(99)00061-4]
• 50: [50] N. Nethercote and J. Seward, "Valgrind: a program supervision framework" Electronic Notes in Theoretical Computer Science, 89(2), Oct. 2003, pp. 1-23.
• 51: [51] Object Management Group, MDA Guide, Version 1.0.1, OMG document omg/2003-06-01, 2003
• 52: [52] Object Management Group, UML Profile for Schedulability, Performance, and Time Specification, Version 1.1, OMG document formal/05-01-02, Jan 2005.
• 53: [53] Object Management Group, UML Profile for Modeling and Analysis of Real-Time and Embedded systems (MARTE) RFP, OMG doc. realtime/05-02-06, 2005.
• 54: [54] Object Management Group, Software Process Engineering Metamodel Specification, formal/05-01-06, 2006.
• 55: Andreas L. Opdahl, Sensitivity analysis of combined software and hardware performance models: open queueing networks, Performance Evaluation, v.22 n.1, p.75-92, Feb. 1995 [doi>10.1016/0166-5316(93)E0039-8]
• 56: Dorin Petriu , Murray Woodside, Analysing software requirements specifications for performance, Proceedings of the 3rd international workshop on Software and performance, July 24-26, 2002, Rome, Italy [doi>10.1145/584369.584371]
• 57: [57] Dorin B. Petriu, C.M. Woodside, "An intermediate metamodel with scenarios and resources for generating performance models from UML designs", Software and Systems Modeling, vol. 5, no. 4, 2006.
• 58: Dorina C. Petriu , Hui Shen, Applying the UML Performance Profile: Graph Grammar-Based Derivation of LQN Models from UML Specifications, Proceedings of the 12th International Conference on Computer Performance Evaluation, Modelling Techniques and Tools, p.159-177, April 14-17, 2002
• 59: [59] R. H. Reussner, V. Firus, S. Becker, "Parametric Performance Contracts for Software Components and their Compositionality", in 9th Int. Workshop on Component-Oriented Programming, Oslo, June 2004.
• 60: Jerome Rolia , Ludmila Cherkasova , Richard Friedrich, Performance engineering for EA systems in next generation data centresPerformance engineering for EA systems in next generation data centres, Proceedings of the 6th international workshop on Software and performance, February 05-08, 2007, Buenes Aires, Argentina [doi>10.1145/1216993.1217004]
• 61: Philip C. Roth , Barton P. Miller, On-line automated performance diagnosis on thousands of processes, Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programming, March 29-31, 2006, New York, New York, USA [doi>10.1145/1122971.1122984]
• 62: Praveen Kaushik Sharma , Joseph Loyall , Richard E. Schantz , Jianming Ye , Prakash Manghwani , Matthew Gillen , George T. Heineman, Managing End-to-End QoS in Distributed Embedded Applications, IEEE Internet Computing, v.10 n.3, p.16-23, May 2006 [doi>10.1109/MIC.2006.58]
• 63: [63] H. A. Sholl and T. L. Booth, "Software Performance Modeling Using Computation Structures", IEEE Trans on Software Engineering, v. 1, no. 4 Dec. 1975.
• 64: [64] H. Shen, D. C. Petriu, "Performance Analysis of UML Models using Aspect Oriented Modeling Techniques", In MoDELS 2005 (L. Briand and C. Williams, Eds.), LNCS Vol. 3713, Springer, 2005, pp. 156-170.
• 65: Connie U. Smith, Performance Engineering of Software Systems, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1990
• 66: [66] C. U. Smith, L. G. Williams, Performance Solutions, Addison-Wesley, 2002.
• 67: [67] C. U. Smith, "Software Performance Engineering", Encyclopedia of Software Engineering, Wiley, 2002.
• 68: Connie U. Smith , Catalina M. Llado, Performance Model Interchange Format (PMIF 2.0): XML Definition and Implementation, Proceedings of the The Quantitative Evaluation of Systems, First International Conference on (QEST'04), p.38-47, September 27-30, 2004 [doi>10.1109/QEST.2004.34]
• 69: Connie U. Smith , Catalina M. Lladó , Vittorio Cortellessa , Antinisca Di Marco , Lloyd G. Williams, From UML models to software performance results: an SPE process based on XML interchange formats, Proceedings of the 5th international workshop on Software and performance, p.87-98, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071030]
• 70: Monchai Sopitkamol , Daniel A. Menascé, A method for evaluating the impact of software configuration parameters on e-commerce sites, Proceedings of the 5th international workshop on Software and performance, p.53-64, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071027]
• 71: Clemens Szyperski, Component Software: Beyond Object-Oriented Programming, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 72: Ann T. Tai , John F. Meyer , Algirdas Avizienia , Algirdas Avizienis, Software Performability: From Concepts to Applications, Kluwer Academic Publishers, Norwell, MA, 1996
• 73: Tom Verdickt , Bart Dhoedt , Frank Gielen , Piet Demeester, Automatic Inclusion of Middleware Performance Attributes into Architectural UML Software Models, IEEE Transactions on Software Engineering, v.31 n.8, p.695-711, August 2005 [doi>10.1109/TSE.2005.88]
• 74: Kimberly M. Wasserman , George Michailidis , Nicholas Bambos, Optimal processor allocation to differentiated job flows, Performance Evaluation, v.63 n.1, p.1-14, January 2006 [doi>10.1016/j.peva.2004.11.001]
• 75: Elaine J. Weyuker , Filippos I. Vokolos, Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study, IEEE Transactions on Software Engineering, v.26 n.12, p.1147-1156, December 2000 [doi>10.1109/32.888628]
• 76: [76] G. Weikum, A. Moenkeberg, C. Hasse, P. Zabback, "Self-tuning database technology and information services: from wishful thinking to viable engineering", Proc. 28th International Conference on Very Large Data Bases (VLDB 2002), pp. 20-31, Hong Kong, CN, 2002.
• 77: C. M. Woodside, A Three-View Model for Performance Engineering of Concurrent Software, IEEE Transactions on Software Engineering, v.21 n.9, p.754-767, September 1995 [doi>10.1109/32.464545]
• 78: Murray Woodside , Curtis Hrishchuk , Bran Selic , Stefan Bayarov, Automated performance modeling of softwaree genrated by a design environment, Performance Evaluation, v.45 n.2-3, p.107-123, July 1 2001 [doi>10.1016/S0166-5316(01)00033-5]
• 79: Murray Woodside , Dorin Petriu , Khalid Siddiqui, Performance-related completions for software specifications, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581346]
• 80: Murray Woodside , Dorina C. Petriu , Dorin B. Petriu , Hui Shen , Toqeer Israr , Jose Merseguer, Performance by unified model analysis (PUMA), Proceedings of the 5th international workshop on Software and performance, p.1-12, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071022]
• 81: Murray Woodside , Tao Zheng , Marin Litoiu, The Use of Optimal Filters to Track Parameters of Performance Models, Proceedings of the Second International Conference on the Quantitative Evaluation of Systems (QEST'05) on The Quantitative Evaluation of Systems, p.74, September 19-22, 2005 [doi>10.1109/QEST.2005.40]
• 82: Xiuping Wu , Murray Woodside, Performance modeling from software components, Proceedings of the 4th international workshop on Software and performance, January 14-16, 2004, Redwood Shores, California
• 83: [83] J. Xu, M. Woodside, and D.C. Petriu, "Performance Analysis of a Software Design using the UML Profile for Schedulability, Performance and Time", in Proc. 13th Int. Conf. Modeling Techniques and Tools for Computer Performance Evaluation, Urbana, USA, Sept. 2003.
• 84: Cemal Yilmaz , Arvind S. Krishna , Atif Memon , Adam Porter , Douglas C. Schmidt , Aniruddha Gokhale , Balachandran Natarajan, Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062515]
• 85: [85] T. Zheng, M. Woodside, "Heuristic Optimization of Scheduling and Allocation for Distributed Systems with Soft Deadlines", in Proc. 13th Int. Conf. Modelling Techniques and Tools for Computer Performance Evaluation, Urbana, USA, 2003.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance

General Terms:
Performance, Theory