Experiments on quality evaluation of embedded software in Japan robot software design contest

ABSTRACT

As a practical opportunity for educating Japanese young developers in the field of embedded software development, a software design contest involving the design of software to automatically control a line-trace robot, and conduct running performance tests was held. In this paper,we give the results of the contest from the viewpoint of software quality evaluation. We create a framework for evaluating the software quality which integrated design model quality and the final system performance, and conduct analysis using the framework. As a result of analysis,it is found that the quantitative measurement of the structural complexity of the design models bears a strong relationship to qualitative evaluation of the design conducted by judges. It is also found that there is no strong correlation between design model quality evaluated by the judges and the final system performance. For embedded software development, it is particularly important to estimate and verify reliability and performance in the early stages,using the model. Based on the analysis result,we consider possible remedies with respect to the models submitted,the evaluation methods used and the contest specifications. In order to adequately measure several non-functional quality characteristics including performance on the model,it is necessary to improve the way of developing robot software (such as applying model driven development)and reexamine the evaluation methods.

REFERENCES

• 1: Japan System House Association: Embedded Technology (ET) Software Design Robot Contest 2005, http: //www.etrobo. jp/
• 2: The LEGO Group, LEGO Mindstorms, http: //mindstorms. lego. com/
• 3: Barry W. Boehm, A Spiral Model of Software Development and Enhancement, Computer, v.21 n.5, p.61-72, May 1988 [doi>10.1109/2.59]
• 4: OMG: UML 2. 0 Specification, 2004, http: //www.uml. org/
• 5: Y. Kobayashi, T. Yamazaki, T. Futagami and H. Watanabe: Robot Contest as Software Engineering Education, Proc. IPSJ/SIGSE Embedded Software Symposium (ESS2004), 2004. (in Japanese)
• 6: D. West: Metaphor, Architecture, and XP, Proc. 3rd International Conference on Extreme Programming and Agile Processes in Software Engineering, 2002.
• 7: J. Herbsleb, D. Root and J. Tomayko: The eXtreme Programming (XP) Metaphor and Software Architecture, CMU-CS-03-167, Carnegie Mellon University, 2003.
• 8: OMG: Software Process Engineering Metamodel Specification, Version 1. 1, 2005.
• 9: ISO/IEC TR 9126-1: Software engineering -Product Quality -Part 1: Quality model, 2001.
• 10: ISO/IEC TR 9126-2: Software engineering -Product Quality -Part 2: External metrics, 2003.
• 11: ISO/IEC TR 9126-3: Software engineering -Product Quality -Part 3: Internal metrics, 2003.
• 12: Kilsup Lee , Sung Jong Lee, A Quantitative Software Quality Evaluation Model for the Artifacts of Component Based Development, Proceedings of the Sixth International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing and First ACIS International Workshop on Self-Assembling Wireless Networks (SNPD/SAWN'05), p.20-25, May 23-25, 2005 [doi>10.1109/SNPD-SAWN.2005.7]
• 13: Mark Lorenz , Jeff Kidd, Object-oriented software metrics: a practical guide, Prentice-Hall, Inc., Upper Saddle River, NJ, 1994
• 14: F. B. Abreu, M. Gonlao and R. Esteves: Toward the Design Quality Evaluation of Object-Oriented Software Systems, Proc. 5th International Conference on Software Quality, 1995.
• 15: Rajiv D. Banker , Srikant M. Datar , Chris F. Kemerer , Dani Zweig, Software complexity and maintenance costs, Communications of the ACM, v.36 n.11, p.81-94, Nov. 1993 [doi>10.1145/163359.163375]
• 16: W. P. Stevens, G. J. Myers and L. L. Constantine: Structured Design, IBM Systems Journal, Vol. 13, No. 2, 1974.
• 17: A. Brown: An introduction to Model Driven Architecture, IBM developerWorks, May, 2004, http: //www-128. ibm. com/developerworks/rational/library/3100. html
• 18: Bran Selic, The Pragmatics of Model-Driven Development, IEEE Software, v.20 n.5, p.19-25, September 2003 [doi>10.1109/MS.2003.1231146]
• 19: K. Kang, S. Cohen, J. Hess, W. Novak and A. Peterson: Feature-Oriented Domain Analysis (FODA) Feasibility Study, CMU/SEI-90-TR-21, Carnegie Mellon University, 1990.
• 20: OMG: UML Pro. le for Schedulability, Performance, and Time (version 1. 1), 2005.
• 21: Moreno Marzolla , Simonetta Balsamo, UML-PSI: The UML Performance Simulator, Proceedings of the The Quantitative Evaluation of Systems, First International Conference on (QEST'04), p.340-341, September 27-30, 2004 [doi>10.1109/QEST.2004.50]
• 22: Simonetta Balsamo , Antinisca Di Marco , Paola Inverardi , Marta Simeoni, Model-Based Performance Prediction in Software Development: A Survey, IEEE Transactions on Software Engineering, v.30 n.5, p.295-310, May 2004 [doi>10.1109/TSE.2004.9]
• 23: Murray Woodside , Dorina C. Petriu , Dorin B. Petriu , Hui Shen , Toqeer Israr , Jose Merseguer, Performance by unified model analysis (PUMA), Proceedings of the 5th international workshop on Software and performance, p.1-12, July 12-14, 2005, Palma, Illes Balears, Spain [doi>10.1145/1071021.1071022]
• 24: Tiffany Winn , Paul Calder, Is This a Pattern?, IEEE Software, v.19 n.1, p.59-66, January 2002 [doi>10.1109/52.976942]
• 25: H. Washizaki, A. Kubo, A. Takasu and Y. Fukazawa: Relation Analysis among Patterns on Software Development Process, Proc. 6th International Conference on Product Focused Software Process Improvement, LNCS Vol. 3547, 2005.
• 26: William J. Brown , Raphael C. Malveau , Hays W. McCormick, III , Thomas J. Mowbray, AntiPatterns: refactoring software, architectures, and projects in crisis, John Wiley & Sons, Inc., New York, NY, 1998
• 27: Connie U. Smith , Lloyd G. Williams, Software performance antipatterns, Proceedings of the 2nd international workshop on Software and performance, p.127-136, September 2000, Ottawa, Ontario, Canada [doi>10.1145/350391.350420]
• 28: Bran Selic , Garth Gullekson , Paul T. Ward, Real-time object-oriented modeling, John Wiley & Sons, Inc., New York, NY, 1994
• 29: Sherif M. Yacoub , Hany H. Ammar , Tom Robinson, Dynamic Metrics for Object Oriented Designs, Proceedings of the 6th International Symposium on Software Metrics, p.50, November 04-06, 1999
• 30: M. Genero, D. Miranda and M. Piattini: Defining Metrics for UML Statechart Diagrams in a Methodological Way, International Workshop on Conceptual Modeling Quality, 2003.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Product metrics

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures; Complexity measures

General Terms:
Measurement

Keywords:
embedded software development, robot contest, software design, software model, software quality