Expression constraints in multimodal human-computer interaction

ABSTRACT

Thanks to recent scientific advances, it is now possible to design multimodal interfaces allowing the use of speech and gestures on a touchscreen. However, present speech recognizers and natural language interpreters cannot yet process spontaneous speech accurately. These limitations make it necessary to impose constraints on users' speech inputs. Thus, ergonomic studies are needed to provide user interface designers with efficient guidelines for the definition of usable speech constraints. We evolved a method for designing oral and multimodal (speech + 2D gestures) command languages, which could be interpreted reliably by present systems, and easy to learn through human-computer interaction (HCI). The empirical study presented here contributes to assessing the usability of such artificial languages in a realistic software environment. Analyses of the multimodal protocols collected indicate that all subjects were able to assimilate rapidly the given expression constraints, mainly while executing simple interactive tasks; in addition, these constraints, which had no noticeable effect on the subjects' activities, had a limited influence on their use of modalities. These results contribute to the validation of the method we propose for the design of tractable and usable multimodal command languages.

REFERENCES

• 1: René Amalberti , Noëlle Carbonell , Pierre Falzon, User representations of computer systems in human-computer speech interaction, International Journal of Man-Machine Studies, v.38 n.4, p.547-566, April, 1993 [doi>10.1006/imms.1993.1026]
• 2: Bekker, M.M., Van Nes, F.L., and Juola, J.F. A comparison of mouse and speech input control of a textannotation system Behaviour & information Technology. 14, 1 (1995), 14-22.
• 3: Coutaz, J., and Caelen, J. A taxonomy for multimedia and multimodal user interface. Proceedings 1" ERCIM Workshop on Multimodal Human-Computer Interaction (Lisbon, November 1991), INESC.
• 4: R. I. Damper , S. D. Wood, Speech versus keying in command and control applications, International Journal of Human-Computer Studies, v.42 n.3, p.289-305, March 1995 [doi>10.1006/ijhc.1995.1014]
• 5: Dillon, T.W., and Norcio, A.F. User performance and acceptance of a speech input interface in a health assessment task. International Journal of Man-Machine Studies, 38 (January 1993), 547-566.
• 6: Thomas W. Dillon , A. F. Norcio, User performance and acceptance of a speech interface in a health assessment task, International Journal of Human-Computer Studies, v.47 n.4, p.591-602, Oct. 1997 [doi>10.1006/ijhc.1997.0153]
• 7: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 8: Mignot, C, and Carbonell, N. "Natural" multimodal HCI: Experimental results on the use of spontaneous speech and hand gestures. Proceedings 2"d ERCIM Workshop on Multimodal Human-Computer Interaction (Nancy, November 1994), INRIA, 97-l 12.
• 9: A. C. Murray , D. M. Jones , C. R. Frankish, Dialogue design in speech-mediated data-entry: the role of syntactic constraints and feedback, International Journal of Human-Computer Studies, v.45 n.3, p.263-286, Sept. 1996 [doi>10.1006/ijhc.1996.0052]
• 10: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 11: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 12: Sandrine Robbe , Noelle Carbonell , Pierre Dauchy, Constrained vs spontaneous speech and gestures for interacting with computers: A comparative empirical study, Proceedings of the IFIP TC13 Interantional Conference on Human-Computer Interaction, p.445-452, July 14-18, 1997

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Experimentation, Human Factors

Keywords:
multimodal user interfaces, speech constraints, usability