Evaluating the ICRA 2008 HRI challenge

ABSTRACT

This paper reports on the evaluation of the ICRA 2008 Human-Robot Interaction (HRI) Challenge. Five research groups demonstrated state-of-the-art work on HRI with a special focus on social and learning abilities. The demonstrations were rated by expert evaluators, in charge of awarding the prize, and 269 participants, i.e. 20 percent of the conference attendees through a standardized questionnaire (semantic differential). The data was analyzed with respect to six independent variables: expert evaluators vs. attendees, nationality of participants, origin region of the demo, age, gender and knowledge level of the attendees. Conference attendees tended to give higher scores for Social Skills, General Impression, and Overall Score than the expert evaluators. Irrespectively of the level of knowledge, age, and gender, conference attendees rated all demos relatively homogeneously. However, a comparative analysis of the conference attendees's ratings nationality-wise showed that demonstrations were rated differently depending on the region of origin. Conference attendees for the USA and Asian countries tended to rate demos from the same country of origin more frequently and more positively.

REFERENCES

• 1: IEEE robotics & automation magazine - finding the right path. Robotics & Automation Magazine, IEEE, 15(3):10--15, September 2008.
• 2: S. Calinon and A. Billard. Active teaching in robot programming by demonstration. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 702--707, August 2007.
• 3: C. Kidd and C. Breazeal. Designing a sociable robot system for weight maintenance. Consumer Communications and Networking Conference, 2006.
• 4: CCNC 2006. 3rd IEEE, 1:253--257, Jan. 2006.
• 5: H. Kozima, Y. Yasuda, and C. Nakagawa. Social interaction facilitated by a minimally-designed robot: Findings from longitudinal therapeutic practices for autistic children. In Robot and Human interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on, pages 599--604, Aug. 2007.
• 6: N. Mitsunaga, T. Miyashita, H. Ishiguro, K. Kogure, and N. Hagita. Robovie-iv: A communication robot interacting with people daily in an office. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pages 5066--5072, Oct. 2006.
• 7: S. Woods, M. Walters, K. Koay, and K. Dautenhahn. Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In Advanced Motion Control, 2006. 9th IEEE International Workshop on, pages 750--755, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.9 Robotics

General Terms:
Design, Experimentation, Human Factors

Keywords:
evaluation, expert evaluation, human-robot interaction challenge, user-based evaluation