Representing unit test data for large scale software development

ABSTRACT

Large scale software projects rely on routine, automated testing to gauge progress towards its goals. The diversity and quantity of these tests grow as time and project scope increase. This is as a consequence of both experience and expanding audience. It becomes increasingly difficult to interpret testing results as the testing suites multiply and diversify. If interpretation becomes too difficult, testings results could become ignored all together. Visualization has proven to be an effective tool to aid the interpretation of large amounts of data. We have adapted visualization techniques based on small multiples to communicate the health of the software project across several levels of abstraction. The collective set of techniques we refer to as the SeeTest visualization schema. We applied this visualization technique to the Open MPI test results in order to assist developers in the software release cycle. Through the visualizations, developers found a variety of surprising mismatches between their data and their intuitions. This exploration did not involve collecting any data not already being collected, merely presenting it in manner that better supported their needs. In this paper, we detail the development of the representation we used and give more particular analysis of the insights gained by the Open MPI community. The techniques presented in this paper can be applied to other software projects.

REFERENCES

• 1: Benjamin B. Bederson , Jesse Grosjean , Jon Meyer, Toolkit Design for Interactive Structured Graphics, IEEE Transactions on Software Engineering, v.30 n.8, p.535-546, August 2004 [doi>10.1109/TSE.2004.44]
• 2: Cottam, J. A., and Lumsdaine, A. 2007. Tuple Space Mapper: Design, challenges and goals. Tech. Rep. TR648, Indiana University, Bloomington, IN, June.
• 3: Stephen G. Eick , Joseph L. Steffen , Eric E. Sumner, Jr., Seesoft-A Tool for Visualizing Line Oriented Software Statistics, IEEE Transactions on Software Engineering, v.18 n.11, p.957-968, November 1992 [doi>10.1109/32.177365]
• 4: Gabriel, E., and et al. 2004. Open MPI: Goals, concept, and design of a next generation MPI implementation. In Proceedings, 11th European PVM/MPI Users' Group Meeting, 97--104.
• 5: Mary Jean Harrold, Testing: a roadmap, Proceedings of the Conference on The Future of Software Engineering, p.61-72, June 04-11, 2000, Limerick, Ireland [doi>10.1145/336512.336532]
• 6: Hursey, J., Mallove, E., Squyres, J. M., and Lumsdaine, A. 2007. An extensible framework for distributed testing of MPI implementations. In Proceedings, Euro PVM/MPI.
• 7: James A. Jones , Mary Jean Harrold , John Stasko, Visualization of test information to assist fault localization, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581397]
• 8: JUnit, 2008. JUnit: Resources for Test Driven Development. http://www.junit.org/.
• 9: Kitware Inc., 2008. CDash: Open Source, Distributed, Software Quality System. http://www.cdash.org/.
• 10: Kitware Inc., 2008. Dart: Tests, Reports and Dashboards. http://public.kitware.com/Dart/.
• 11: CORPORATE The MPI Forum, MPI: a message passing interface, Proceedings of the 1993 ACM/IEEE conference on Supercomputing, p.878-883, December 1993, Portland, Oregon, United States [doi>10.1145/169627.169855]
• 12: Michael Meyer , Tudor Gîrba , Mircea Lungu, Mondrian: an agile information visualization framework, Proceedings of the 2006 ACM symposium on Software visualization, September 04-05, 2006, Brighton, United Kingdom [doi>10.1145/1148493.1148513]
• 13: North, S. 1998. Visualizing graph models of software. In Software Visualization: Programming as a Multimedia Experience, J. Stasko, J. Domingue, M. H. Brown, and B. A. Price, Eds. The MIT Press.
• 14: nUnit, 2008. nUnit. http://www.nunit.org/.
• 15: Open MPI, 2008. MTT: MPI Testing Tool. http://www.open-mpi.org/projects/mtt/.
• 16: Leon Osterweil, Strategic directions in software quality, ACM Computing Surveys (CSUR), v.28 n.4, p.738-750, Dec. 1996 [doi>10.1145/242223.242288]
• 17: Thomas Panas , Rüdiger Lincke , Welf Löwe, Online-configuration of software visualizations with Vizz3D, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056043]
• 18: Martin Pinzger , Harald Gall , Michael Fischer , Michele Lanza, Visualizing multiple evolution metrics, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056027]
• 19: Reiss, S., and Renieris, M. 2003. The Bloom software visualization system. In Software Visualization: From Theory to Practice, J. Cao, Z. Ren, A. Chan, L. Fang, L. Xie, and D. Chen, Eds. Kluwer Academic Publishers, Norwell, Massachusetts, 243--284.
• 20: Steven P. Reiss , Manos Renieris, Jove: java as it happens, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056034]
• 21: Lucian Voinea , Alex Telea , Jarke J. van Wijk, CVSscan: visualization of code evolution, Proceedings of the 2005 ACM symposium on Software visualization, May 14-15, 2005, St. Louis, Missouri [doi>10.1145/1056018.1056025]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Life cycle

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Screen design (e.g., text, graphics, color)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.8 Applications

Keywords:
MPI, project management, testing, visualization