The oz of wizard: simulating the human for interaction research

ABSTRACT

The Wizard of Oz experiment method has a long tradition of acceptance and use within the field of human-robot interaction. The community has traditionally downplayed the importance of interaction evaluations run with the inverse model: the human simulated to evaluate robot behavior, or Oz of Wizard. We argue that such studies play an important role in the field of human-robot interaction. We differentiate between methodologically rigorous human modeling and placeholder simulations using simplified human models. Guidelines are proposed for when Oz of Wizard results should be considered acceptable. This paper also describes a framework for describing the various permutations of Wizard and Oz states.

REFERENCES

• 1: J. F. Kelley, An iterative design methodology for user-friendly natural language office information applications, ACM Transactions on Information Systems (TOIS), v.2 n.1, p.26-41, Jan. 1984 [doi>10.1145/357417.357420]
• 2: Masahiro Shiomi , Takayuki Kanda , Satoshi Koizumi , Hiroshi Ishiguro , Norihiro Hagita, Group attention control for communication robots with wizard of OZ approach, Proceedings of the ACM/IEEE international conference on Human-robot interaction, March 10-12, 2007, Arlington, Virginia, USA [doi>10.1145/1228716.1228733]
• 3: Peter H. Kahn , Nathan G. Freier , Takayuki Kanda , Hiroshi Ishiguro , Jolina H. Ruckert , Rachel L. Severson , Shaun K. Kane, Design patterns for sociality in human-robot interaction, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349836]
• 4: Masahiro Shiomi , Daisuke Sakamoto , Kanda Takayuki , Carlos Toshinori Ishi , Hiroshi Ishiguro , Norihiro Hagita, A semi-autonomous communication robot: a field trial at a train station, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349862]
• 5: J. Gregory Trafton , Alan C. Schultz , Dennis Perznowski , Magdalena D. Bugajska , William Adams , Nicholas L. Cassimatis , Derek P. Brock, Children and robots learning to play hide and seek, Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, March 02-03, 2006, Salt Lake City, Utah, USA [doi>10.1145/1121241.1121283]
• 6: Granda, T., Kirkpatrick, M., Julien, T., and Peterson, L. 1990. The evolutionary role of humans in the human-robot system. In Proc. Human Factors Society 34th Annual Meeting. 664--668.
• 7: Aaron Steinfeld , Terrence Fong , David Kaber , Michael Lewis , Jean Scholtz , Alan Schultz , Michael Goodrich, Common metrics for human-robot interaction, Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, March 02-03, 2006, Salt Lake City, Utah, USA [doi>10.1145/1121241.1121249]
• 8: David A. Forsyth , Jean Ponce, Computer Vision: A Modern Approach, Prentice Hall Professional Technical Reference, 2002
• 9: Jenkins, O., González, G., and Loper, M. 2007. Interactive human pose and action recognition using dynamical motion primitives. International Journal of Humanoid Robotics, 4(2):365--385.
• 10: Grollman, D. H., and Jenkins, O. C. 2008. Sparse incremental learning for interactive robot control policy estimation. In International Conference on Robotics and Automation (ICRA, Pasadena, CA, USA, May 2008).
• 11: Sonia Chernova , Manuela Veloso, Multi-thresholded approach to demonstration selection for interactive robot learning, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349852]
• 12: Finale Doshi , Nicholas Roy, Efficient model learning for dialog management, Proceedings of the ACM/IEEE international conference on Human-robot interaction, March 10-12, 2007, Arlington, Virginia, USA [doi>10.1145/1228716.1228726]
• 13: Sven R. Schmidt-Rohr , Steffen Knoop , Martin Lösch , Rüdiger Dillmann, Reasoning for a multi-modal service robot considering uncertainty in human-robot interaction, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349855]
• 14: Vondrak, M., Sigal, L., and Jenkins, O. 2008. Physical simulation for probabilistic motion tracking. In Computer Vision and Pattern Recognition (CVPR, Anchorage, AK, USA, Jun 2008).
• 15: Srinivasa, S., Ferguson, D., Vande Weghe, M., Diankov, R., Berenson, D., Helfrich, C. and Strasdat, H. 2008. The Robotic Busboy: steps towards developing a mobile robotic home assistant. In International Conference on Intelligent Autonomous Systems.
• 16: Curtis M. Humphrey , Julie A. Adams, Compass visualizations for human-robotic interaction, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349830]
• 17: Fumitaka Yamaoka , Takayuki Kanda , Hiroshi Ishiguro , Norihiro Hagita, How close?: model of proximity control for information-presenting robots, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349841]
• 18: Guy Hoffman , Cynthia Breazeal, Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team, Proceedings of the ACM/IEEE international conference on Human-robot interaction, March 10-12, 2007, Arlington, Virginia, USA [doi>10.1145/1228716.1228718]
• 19: Francine Gemperle , Carl DiSalvo , Jodi Forlizzi , Willy Yonkers, The Hug: a new form for communication, Proceedings of the 2003 conference on Designing for user experiences, June 06-07, 2003, San Francisco, California [doi>10.1145/997078.997103]
• 20: Leila Takayama , Wendy Ju , Clifford Nass, Beyond dirty, dangerous and dull: what everyday people think robots should do, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349827]
• 21: Scholtz, J., Young, J., Drury, J., and Yanco, H. 2004. Evaluation of human-robot interaction awareness in search and rescue. In Proc. IEEE International Conference on Robotics and Automation.
• 22: Casper, J., and Murphy, R. 2003. Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics B, 33.
• 23: Bilge Mutlu , Jodi Forlizzi, Robots in organizations: the role of workflow, social, and environmental factors in human-robot interaction, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349860]
• 24: Krishnan, H., Gibb, S., Steinfeld, A., and Shladover, S. E. 2001. Rear-end collision-warning system: Design and evaluation via simulation, Transportation Research Record - Journal of the Transportation Research Board 1759, 52--60.
• 25: Bainbridge, W., Hart, J., Kim, E. and Scassellati, B. 2008. The effect of presence on human-robot interaction. In IEEE International Symposium on Robot and Human Interactive Communication (Munich, Germany).
• 26: Tapus, A., Mataric, M. and Scassellati, B. 2007. The grand challenges in socially assistive robotics. IEEE Robotics and Automation Magazine. Vol. 4, No. 1. p. 35--427.
• 27: Odest Chadwicke Jenkins , German González , Matthew Maverick Loper, Tracking human motion and actions for interactive robots, Proceedings of the ACM/IEEE international conference on Human-robot interaction, March 10-12, 2007, Arlington, Virginia, USA [doi>10.1145/1228716.1228765]
• 28: Koenig, N., Chernova, S., Jones, C., Loper, M., and Jenkins, O. 2008. Hands-free interaction for human-robot teams. In ICRA 2008 Workshop on Social Interaction with Intelligent Indoor Robots (Pasadena, CA, USA, May 2008).
• 29: Christoph Bartneck, HRI caught on film 2, Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, March 12-15, 2008, Amsterdam, The Netherlands [doi>10.1145/1349822.1349873]
• 30: Kevin Gold , Brian Scassellati, Using context and sensory data to learn first and second person pronouns, Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, March 02-03, 2006, Salt Lake City, Utah, USA [doi>10.1145/1121241.1121262]
• 31: Gold K. and Scassellati, B. 2006. Grounded pronoun learning and pronoun reversal. In 5th International Conference on Development and Learning (ICDL-06, Bloomington, IN).
• 32: Gold, K., Doniec, M., and Scassellati, B. 2007. Learning grounded semantics with word trees: prepositions and pronouns. In Proceedings of the 6th IEEE International Conference on Development and Learning (ICDL, London, England, July 2007).
• 33: Gold, K. and Scassellati, B. 2007. A robot that uses existing vocabulary to infer non-visual word meanings from observation. In Proceedings of the Twenty-Second Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI. Vancouver, BC, Canada, August, 2007).
• 34: MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk, 3rd ed. Mahwah, NJ: Lawrence Erlbaum Associates.
• 35: Kevin Gold , Brian Scassellati, Using sentence context and implicit contrast to learn sensor-grounded meanings for relational and deictic words: the twig system, Yale University, New Haven, CT, 2008

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

General Terms:
Design, Experimentation, Human Factors, Measurement, Performance, Theory

Keywords:
evaluation, human-robot interaction, interaction, wizard of oz