Multimodal interaction in an augmented reality scenario

ABSTRACT

We describe an augmented reality system designed for online acquisition of visual knowledge and retrieval of memorized objects. The system relies on a head mounted camera and display, which allow the user to view the environment together with overlaid augmentations by the system. In this setup, communication by hand gestures and speech is mandatory as common input devices like mouse and keyboard are not available. Using gesture and speech, basically three types of tasks must be handled: (i) Communication with the system about the environment, in particular, directing attention towards objects and commanding the memorization of sample views; (ii) control of system operation, e.g. switching between display modes; and (iii) re-adaptation of the interface itself in case communication becomes unreliable due to changes in external factors, such as illumination conditions. We present an architecture to manage these tasks and describe and evaluate several of its key elements, including modules for pointing gesture recognition, menu control based on gesture and speech, and control strategies to cope with situations when vision becomes unreliable and has to be re-adapted by speech.

REFERENCES

• 1: Y. Aloimonos. Active vision revisited. In Y. Aloimonos, editor, Active Perception, pages 1--18. Lawrence Efibaum, Hillsdale, 1993.
• 2: H. J. Andersen, M. Stoerring, and E. Granum. Physics-based modelling of human skin colour under mixed illuminants. Robotics and Autonomous Systems, 35(3-4):131--142, 2001.
• 3: C. Bauckhage, G. A. Fink, J. Fritsch, F. Kummert, F. Lomker, G. Sagerer, and S. Wachsmuth. An Integrated System for Cooperative Man-Machine Interaction. In IEEE Int'l Symp. Comput. Intelligence in Robotics and Automation, pages 328--333, Banff, Canada, 2001.
• 4: R. Bergevin , M. D. Levine, Generic Object Recognition: Building and Matching Coarse Descriptions from Line Drawings, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.15 n.1, p.19-36, January 1993 [doi>10.1109/34.184772]
• 5: H. I. Christensen. Cognitive (vision) systems. ERCIM News, pages 17--18, April April, 2003.
• 6: J. L. Crowley and H. I. Christensen, editors. Vision as Process. Springer, 1995.
• 7: James L. Crowley , Joëlle Coutaz , François Bérard, Perceptual user interfaces: things that see, Communications of the ACM, v.43 n.3, p.54-ff., March 2000 [doi>10.1145/330534.330540]
• 8: B. A. Draper, G. Kutlu, E. M. Riseman, and A. R. Hanson. ISR3: Communication and Data Storage for an Unmanned Ground Vehicle. In Proc. ICPR, volume~I, pages 833--836, 1994.
• 9: Gernot A. Fink, Developing HMM-Based Recognizers with ESMERALDA, Proceedings of the Second International Workshop on Text, Speech and Dialogue, p.229-234, September 01, 1999
• 10: A. R. Hanson and E. M. Riseman. VISIONS: A Computer System for Interpreting Scenes. In A.R. Hanson and E.M. Riseman, editors, Computer Vision Systems. Academic Press, 1978.
• 11: C. Harris and M. Stephens. A Combined Corner and Edge Detector. In Proc. 4th Alvey Vision Conf., pages 147--151, 1988.
• 12: G. Heidemann, R. Rae, H. Bekel, I. Bax, and H. Ritter. Integrating context-free and context-dependent attentional mechanisms for gestural object reference. In Proc. Int'l Conf. Cognitive Vision Systems, pages 22--33, Graz, Austria, 2003.
• 13: A. Hoogs, J. Rittscher, G. Stein, and J. Schmiederer. Video Content Annotation Using Visual analysis and a Large Semantic Knowledgebase. In Proc. CVPR 2003, volume 2, pages 327--334, 2003.
• 14: Thomas Kalinke , Werner von Seelen, Entropie als Maß des lokalen Informationsgehalts in Bildern zur Realisierung einer Aufmerksamkeitssteuerung, Mustererkennung 1996, 18. DAGM-Symposium, p.627-634, September 11-13, 1996
• 15: G. Lindegaard. Usability Testing and System Evaluation: A Guide for Designing Useful Computer Systems. Chapman & Hall, 1994.
• 16: P. J. Locher and C. F. Nodine. Symmetry Catches the Eye. In A. Levy-Schoen and J. K. O'Reagan, editors, Eye Movements: From Physiology to Cognition, pages 353--361. Elsevier Science Publishers B. V. (North Holland), 1987.
• 17: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 18: Sharon Oviatt , Philip Cohen, Perceptual user interfaces: multimodal interfaces that process what comes naturally, Communications of the ACM, v.43 n.3, p.45-53, March 2000 [doi>10.1145/330534.330538]
• 19: Alex Pentland, Perceptual user interfaces: perceptual intelligence, Communications of the ACM, v.43 n.3, p.35-44, March 2000 [doi>10.1145/330534.330536]
• 20: Daniel Reisfeld , Haim Wolfson , Yehezkel Yeshurun, Context-free attentional operators: the generalized symmetry transform, International Journal of Computer Vision, v.14 n.2, p.119-130, March 1995 [doi>10.1007/BF01418978]
• 21: D. Roy. Learning visually grounded words and syntax of natural spoken language. Evolution of Communication, 4(1):33--56, 2000.
• 22: H. Siegl, and A. Pinz. A Mobile AR Kit as a Human Computer Interface for Cognitive Vision. In Proc. WIAMIS'04, Lisbon, 2004.
• 23: Birgitta Martinkauppi, Skin Detection in Video under Changing Illumination Conditions, Proceedings of the International Conference on Pattern Recognition, p.1839, September 03-08, 2000
• 24: Thad E. Starner, Wearable Computers: No Longer Science Fiction, IEEE Pervasive Computing, v.1 n.1, p.86-88, January 2002 [doi>10.1109/MPRV.2002.993148]
• 25: Sven Wachsmuth , Gernot A. Fink , Franz Kummert , Gerhard Sagerer, Using Speech in Visual Object Recognition, Mustererkennung 2000, 22. DAGM-Symposium, p.428-435, September 13-15, 2000
• 26: Craig Wisneski , Hiroshi Ishii , Andrew Dahley , Matthew G. Gorbet , Scott Brave , Brygg Ullmer , Paul Yarin, Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information, Proceedings of the First International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture, p.22-32, February 01, 1998

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.m Miscellaneous

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Algorithms, Human Factors, Reliability, Verification

Keywords:
augmented reality, human-machine-interaction, image retrieval, interfaces, memory, mobile systems, object recognition