Toward automatic artifact matching for tool evaluation

ABSTRACT

Before performing software maintenance and evolution tasks, developers must expend significant effort to understand the design of the subject software system. Reverse engineering and reengineering tools can extract artifacts, such as class diagrams, from source code so that the effort expended on program comprehension can be greatly reduced. Choosing suitable and appropriate tools is itself a difficult process. Many metrics and benchmarks have been proposed to evaluate the quality of reengineering tools, but the process is still complicated and requires much human effort. Furthermore, ignored by most evaluation experiments and approaches is the understanding of differences and similarities of the output artifacts produced by different tools. This paper proposes a novel approach for tool evaluation making a direct comparison of the output artifacts of the chosen tools. We apply our methodology on class diagrams extracted from source code. We utilize a class matching model to automate the process. In this study, we use ten open-source programs as test cases and two open source reverse engineering tools Doxygen and StarUML. We evaluate the output of these two tools based on class matching given the same input. This approach, we believe, provides developers with good guidance in tool selection.

REFERENCES

• 1: Ahmed K. Elmagarmid , Panagiotis G. Ipeirotis , Vassilios S. Verykios, Duplicate Record Detection: A Survey, IEEE Transactions on Knowledge and Data Engineering, v.19 n.1, p.1-16, January 2007 [doi>10.1109/TKDE.2007.9]
• 2: Lionel C. Briand, The Experimental Paradigm in Reverse Engineering: Role, Challenges, and Limitations, Proceedings of the 13th Working Conference on Reverse Engineering, p.3-8, October 23-27, 2006 [doi>10.1109/WCRE.2006.53]
• 3: Sabrina Fortsch and Bernhard Westfechtel, DIFFERENCING AND MERGING OF SOFTWARE DIAGRAMS; State of the Art and Challenges. International Conference on software and data technologies ICSOFT 2007.
• 4: Susan Elliott Sim , Richard C. Holt , Steve Easterbrook, On Using a Benchmark to Evaluate C++ Extractors, Proceedings of the 10th International Workshop on Program Comprehension, p.114, June 27-29, 2002
• 5: Steven Kearney, James F. Power. Benchmarking the accuracy of reverse engineering tools for Java programs: a study of eleven UML tools. Technical Report: NUIM-CS-TR-2007-1. June 6, 2007.
• 6: Elliot J. Chikofsky , James H. Cross II, Reverse Engineering and Design Recovery: A Taxonomy, IEEE Software, v.7 n.1, p.13-17, January 1990 [doi>10.1109/52.43044]
• 7: Yann-Gael Gueheneuc, A Systematic Study of UML Class Diagram Constituents for their Abstract and Precise Recovery, Proceedings of the 11th Asia-Pacific Software Engineering Conference, p.265-274, November 30-December 03, 2004 [doi>10.1109/APSEC.2004.13]
• 8: Richard C. Holt , Andy Schürr , Susan Elliott Sim , Andreas Winter, GXL: a graph-based standard exchange format for reengineering, Science of Computer Programming, v.60 n.2, p.149-170, April 2006 [doi>10.1016/j.scico.2005.10.003]
• 9: Nicholas A. Kraft , Brian A. Malloy , James F. Power, An infrastructure to support interoperability in reverse engineering, Information and Software Technology, v.49 n.3, p.292-307, March, 2007 [doi>10.1016/j.infsof.2006.10.014]
• 10: Daniel L. Moise, Kenny Wong. Issues in Integrating Schemas for Reverse Engineering. Proceedings of the International Workshop on Meta-Models and Schemas for Reverse Engineering (ateM 2003).
• 11: Nicholas A. Kraft , Errol L. Lloyd , Brian A. Malloy , Peter J. Clarke, The implementation of an extensible system for comparison and visualization of class ordering methodologies, Journal of Systems and Software, v.79 n.8, p.1092-1109, August 2006 [doi>10.1016/j.jss.2005.10.019]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Keywords:
class matching model, reverse engineering, tool evaluation