A Wizard of Oz study for an AR multimodal interface

ABSTRACT

In this paper we describe a Wizard of Oz (WOz) user study of an Augmented Reality (AR) interface that uses multimodal input (MMI) with natural hand interaction and speech commands. Our goal is to use a WOz study to help guide the creation of a multimodal AR interface which is most natural to the user. In this study we used three virtual object arranging tasks with two different display types (a head mounted display, and a desktop monitor) to see how users used multimodal commands, and how different AR display conditions affect those commands. The results provided valuable insights into how people naturally interact in a multimodal AR scene assembly task. For example, we discovered the optimal time frame for fusing speech and gesture commands into a single command. We also found that display type did not produce a significant difference in the type of commands used. Using these results, we present design recommendations for multimodal interaction in AR environments.

REFERENCES

• 1: Swan II, J.E. and Gabbard, J. L. 2005. Survey of User-Based Experimentation in Augmented Reality. In Proc. of 1st Int'l Conf. on Virtual Reality. HCI International 2005, 2005.
• 2: Broll, W., Stoerring, M., and Mottram, C. 2003. The Augmented Round Table -- a New Interface to Urban Planning and Architectural Design, In Proc. INTERACT'03, pp. 1103--1104.
• 3: Kousuke Nakashima , Takashi Machida , Kiyoshi Kiyokawa , Haruo Takemura, A 2D-3D integrated environment for cooperative work, Proceedings of the ACM symposium on Virtual reality software and technology, November 07-09, 2005, Monterey, CA, USA [doi>10.1145/1101616.1101621]
• 4: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, Proceedings of the 7th annual conference on Computer graphics and interactive techniques, p.262-270, July 14-18, 1980, Seattle, Washington, United States
• 5: P. R. Cohen , M. Dalrymple , D. B. Moran , F. C. Pereira , J. W. Sullivan, Synergistic use of direct manipulation and natural language, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.227-233, March 1989
• 6: Sharon Oviatt , Rachel Coulston , Rebecca Lunsford, When do we interact multimodally?: cognitive load and multimodal communication patterns, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027957]
• 7: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 8: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 9: Ingmar Rauschert , Pyush Agrawal , Rajeev Sharma , Sven Fuhrmann , Isaac Brewer , Alan MacEachren, Designing a human-centered, multimodal GIS interface to support emergency management, Proceedings of the 10th ACM international symposium on Advances in geographic information systems, November 08-09, 2002, McLean, Virginia, USA [doi>10.1145/585147.585172]
• 10: Edward Tse , Saul Greenberg , Chia Shen, GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181012]
• 11: Alex Olwal , Hrvoje Benko , Steven Feiner, SenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System, Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.300, October 07-10, 2003
• 12: D. Weimer , S. K. Ganapathy, A synthetic visual environment with hand gesturing and voice input, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.235-240, March 1989
• 13: David B. Koons , Carlton J. Sparrell, Iconic: speech and depictive gestures at the human-machine interface, Conference companion on Human factors in computing systems, p.453-454, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260487]
• 14: Lucente, M., Zwart, G. J., and George, A. D. 1998. Visualization Space: A Testbed for Deviceless Multimodal User Interface. In AAAI Spring Symposium on Intelligent Environments. AAAI TR SS-98-02.
• 15: Irawati, S., Green, S., Billinghurst, M., Duenser, A., and Ko, H. 2006. "Move the Couch Where?": Developing an Augmented Reality Multimodal Interface". In Proc. ISMAR'06, 183 -- 186.
• 16: Billinghurst, M., Kato, H., Poupyrev, I., Imamoto, K., and Tachibana, K. 2000. Virtual Object Manipulation on a Table-Top AR Environment. In Proc. ISAR'00, 111--119.
• 17: Irawati, S., Green, S., Billinghurst, M., Duenser, A., and Ko, H. 2006. An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures. In Proc. ICAT'06, 272--283.
• 18: Mathias Koelsch , Ryan Bane , Tobias Hoellerer , Matthew Turk, Multimodal Interaction with a Wearable Augmented Reality System, IEEE Computer Graphics and Applications, v.26 n.3, p.62-71, May 2006 [doi>10.1109/MCG.2006.66]
• 19: Kölsch, M., Turk, M., and Tobias, H. 2004. Vision-Based Interfaces for Mobility. In Proc. MobiQuitous'04, 86-- 94.
• 20: Daniel Salber , Joëlle Coutaz, Applying the Wizard of Oz Technique to the Study of Multimodal Systems, Selected papers from the Third International Conference on Human-Computer Interaction, p.219-230, August 03-07, 1993
• 21: Joëlle Coutaz , Daniel Salber , Eric Carraux , Nathalie Portolan, NEIMO, a multiworkstation usability lab for observing and analyzing multimodal interaction, Conference companion on Human factors in computing systems: common ground, p.402-403, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/257089.257904]
• 22: Oviatt, S. L., Cohen, P. R., Fong, M. W., and Frank, M. P. 1992. A rapid semi-automatic simulation technique for interactive speech and handwriting. In Proc. Int'l Conf. on Spoken Language Processing, 2, 1351--1354.
• 23: Sharon L. Oviatt , Philip R. Cohen , Michelle Wang, Toward interface design for human language technology: modality and structure as determinants of linguistic complexity, Speech Communication, v.15 n.3-4, p.283-300, Dec. 1994
• 24: Corradini, A. and Cohen, P. R. 2002. On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence. In Proc. Int'l CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, 52--61.
• 25: Lennart Molin, Wizard-of-Oz prototyping for co-operative interaction design of graphical user interfaces, Proceedings of the third Nordic conference on Human-computer interaction, p.425-428, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028086]
• 26: OpenCV Library. 2008. http://sourceforge.net/projects/opencvlibrary/
• 27: Herbert Freeman, Computer Processing of Line-Drawing Images, ACM Computing Surveys (CSUR), v.6 n.1, p.57-97, March 1974 [doi>10.1145/356625.356627]
• 28: Gunilla Borgefors, Distance transformations in digital images, Computer Vision, Graphics, and Image Processing, v.34 n.3, p.344-371, June 1986 [doi>10.1016/S0734-189X(86)80047-0]
• 29: Looser, J., Grasset, R., Seichter, H., and Billinghurst, M. 2006. OSGART -- A Pragmatic Approach to MR. In Proc. Industrial AR Workshop, ISMAR'06.
• 30: Billinghurst, M., Campbell, S., Chinthammit, W., Hendrickson, D., Poupyrev, I., Takahashi, K., and Kato, H. 2000. Magic book: Exploring transitions in collaborative AR interfaces. Emerging Technologies Proposal, SIGGRAPH'00.
• 31: McNeil, D. 1992. Hand and Mind: What gestures reveal about thought. University of Chicago Press, Chicago, IL, USA.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

General Terms:
Experimentation, Human Factors

Keywords:
AR, augmented reality, multimodal interaction, multimodal interface, user study, wizard of oz