Profiling Deployed Software: Assessing Strategies and Testing Opportunities

ABSTRACT

An understanding of how software is employed in the field can yield many opportunities for quality improvements. Profiling released software can provide such an understanding. However, profiling released software is difficult due to the potentially large number of deployed sites that must be profiled, the transparency requirements at a user's site, and the remote data collection and deployment management process. Researchers have recently proposed various approaches to tap into the opportunities offered by profiling deployed systems and overcome those challenges. Initial studies have illustrated the application of these approaches and have shown their feasibility. Still, the proposed approaches, and the tradeoffs between overhead, accuracy, and potential benefits for the testing activity have been barely quantified. This paper aims to overcome those limitations. Our analysis of 1,200 user sessions on a 155 KLOC deployed system substantiates the ability of field data to support test suite improvements, assesses the efficiency of profiling techniques for released software, and the effectiveness of testing efforts that leverage profiled field data.

REFERENCES

• 1: Matthew Arnold , Barbara G. Ryder, A framework for reducing the cost of instrumented code, Proceedings of the ACM SIGPLAN 2001 conference on Programming language design and implementation, p.168-179, June 2001, Snowbird, Utah, United States
• 2: Thomas Ball , James R. Larus, Optimally profiling and tracing programs, ACM Transactions on Programming Languages and Systems (TOPLAS), v.16 n.4, p.1319-1360, July 1994 [doi>10.1145/183432.183527]
• 3: Thomas Ball , James R. Larus, Optimally profiling and tracing programs, Proceedings of the 19th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, p.59-70, January 19-22, 1992, Albuquerque, New Mexico, United States [doi>10.1145/143165.143180]
• 4: Brad Calder , Peter Feller , Alan Eustace, Value profiling, Proceedings of the 30th annual ACM/IEEE international symposium on Microarchitecture, p.259-269, December 01-03, 1997, Research Triangle Park, North Carolina, United States
• 5: William Dickinson , David Leon , Andy Podgurski, Finding failures by cluster analysis of execution profiles, Proceedings of the 23rd International Conference on Software Engineering, p.339-348, May 12-19, 2001, Toronto, Ontario, Canada
• 6: Sebastian Elbaum , Madeline Hardojo, An empirical study of profiling strategies for released software and their impact on testing activities, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 7: Sebastian Elbaum , Satya Kanduri , Anneliese Amschler Andrews, Anomalies as Precursors of Field Failures, Proceedings of the 14th International Symposium on Software Reliability Engineering, p.108, November 17-21, 2003
• 8: Sebastian Elbaum , Alexey G. Malishevsky , Gregg Rothermel, Test Case Prioritization: A Family of Empirical Studies, IEEE Transactions on Software Engineering, v.28 n.2, p.159-182, February 2002 [doi>10.1109/32.988497]
• 9: Michael D. Ernst , Jake Cockrell , William G. Griswold , David Notkin, Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Transactions on Software Engineering, v.27 n.2, p.99-123, February 2001 [doi>10.1109/32.908957]
• 10: Glenn Ammons , Thomas Ball , James R. Larus, Exploiting hardware performance counters with flow and context sensitive profiling, ACM SIGPLAN Notices, v.32 n.5, p.85-96, May 1997
• 11: Susan L. Graham , Peter B. Kessler , Marshall K. Mckusick, Gprof: A call graph execution profiler, ACM SIGPLAN Notices, v.17 n.6, p.120-126, June 1982
• 12: K. Gross S. McMaster A. Porter A. Urmanov and L. Votta, “Proactive System Maintenance Using Software Telemetry,” <i>Proc. Workshop Remote Analysis and Monitoring Software Systems,</i> pp. 24-26, 2003.
• 13: MIT Program Analysis Group, <i>The Daikon Invariant Detector User Manual.</i> http://pag.csail.mit.edu/daikon/download/doc/ daikon.html, 2005.
• 14: M. Harrold R. Lipton and A. Orso, “Gamma: Continuous Evolution of Software After Deployment,” cc.gatech.edu/aristotle /Research/Projects/gamma.html, 2003.
• 15: David M. Hilbert , David F. Redmiles, An approach to large-scale collection of application usage data over the Internet, Proceedings of the 20th international conference on Software engineering, p.136-145, April 19-25, 1998, Kyoto, Japan
• 16: D. Hilbert and D. Redmiles, “Separating the Wheat from the Chaff in Internet-Mediated User Feedback,” 1998.
• 17: InCert, “Rapid Failure Recovery to Eliminate Application Downtime,” www.incert.com, June 2001.
• 18: Jim Bowring , Alessandro Orso , Mary Jean Harrold, Monitoring deployed software using software tomography, Proceedings of the 2002 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, November 18-19, 2002, Charleston, South Carolina, USA
• 19: James F. Bowring , James M. Rehg , Mary Jean Harrold, Active learning for automatic classification of software behavior, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 20: David Leon , Andy Podgurski , Lee J. White, Multivariate visualization in observation-based testing, Proceedings of the 22nd international conference on Software engineering, p.116-125, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337195]
• 21: Don Libes , Tim O'Reilly , Tim C'Reilly, Exploring Expect, O'Reilly & Associates, Inc., Sebastopol, CA, 1996
• 22: Ben Liblit , Alex Aiken , Alice X. Zheng , Michael I. Jordan, Bug isolation via remote program sampling, Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation, June 09-11, 2003, San Diego, California, USA
• 23: A. Memon , A. Porter , C. Yilmaz , A. Nagarajan , D. Schmidt , B. Natarajan, Skoll: Distributed Continuous Quality Assurance, Proceedings of the 26th International Conference on Software Engineering, p.459-468, May 23-28, 2004
• 24: J. Musa, <i>Software Reliability Engineering.</i> McGraw-Hill, 1999.
• 25: Netscape, “Netscape Quality Feedback System,” home.netscape. com/communicator/navigator/v4.5/qfs1.html, 2000.
• 26: Nielsen, “Nielsen Net Ratings: Nearly 40 Million Internet Users Connect Via Broadband,” www.nielsen-netratings.com, 2003.
• 27: University of Washington, Pine Information Center, http://www.washington.edu/pine/, 2005.
• 28: Alessandro Orso , Taweesup Apiwattanapong , Mary Jean Harrold, Leveraging field data for impact analysis and regression testing, Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering, September 01-05, 2003, Helsinki, Finland
• 29: Alessandro Orso , Donglin Liang , Mary Jean Harrold , Richard Lipton, Gamma system: continuous evolution of software after deployment, Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis, July 22-24, 2002, Roma, Italy
• 30: Christina Pavlopoulou , Michal Young, Residual test coverage monitoring, Proceedings of the 21st international conference on Software engineering, p.277-284, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302637]
• 31: Steven P. Reiss , Manos Renieris, Encoding program executions, Proceedings of the 23rd International Conference on Software Engineering, p.221-230, May 12-19, 2001, Toronto, Ontario, Canada
• 32: D. Richardson L. Clarke L. Osterweil and M. Young, “Perpetual Testing Project,” http://www.ics.uci.edu/~djr/edcs/PerpTest. html, 1997.
• 33: André van der Hoek , Richard S. Hall , Dennis Heimbigner , Alexander L. Wolf, Software release management, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.159-175, September 22-25, 1997, Zurich, Switzerland
• 34: Cemal Yilmaz , Myra B. Cohen , Adam Porter, Covering arrays for efficient fault characterization in complex configuration spaces, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, July 11-14, 2004, Boston, Massachusetts, USA
• 35: C. Yilmaz A. Porter and A. Schmidt, “Distributed Continuous Quality Assurance: The Skoll Project,” <i>Proc. Workshop Remote Analysis and Monitoring Software Systems,</i> pp. 16-19, 2003.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

General Terms:
Experimentation

Keywords:
Index Terms- Profiling, instrumentation, software deployment, testing, empirical studies.