Explorative studies on multimodal interaction in a PDA- and desktop-based scenario

ABSTRACT

This paper presents two explorative case studies on multimodal interaction. Goal of this work is to find and underpin design recommendations to provide well proven decision support across all phases of the usability engineering lifecycle [1]. During this work, user interface patterns for multimodal interaction were identified [2, 3]. These patterns are closely related to other user interface patterns [4, 5, 6]. Two empirical case studies, one using a Wizard of Oz setting and another one using a stand-alone prototype linked to a speech recognition engine [7] were conducted to assess the acceptance of resulting interaction styles. Although the prototypes applied as well those interface patterns that increase usability by means of traditional interaction techniques and thus compete with multimodal interaction styles, multimodal interaction was preferred by most of the users.

REFERENCES

• 1: Deborah J. Mayhew, The usability engineering lifecycle: a practitioner's handbook for user interface design, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 2: Ratzka, A., Wolff, C.: A pattern-based methodology for multimiodal interaction design. In Sojka, P., KopeÇcek, I., Pala, K., eds.: Proc. of Text, Speech, and Dialogue, TSD'06. LNAI 4188, Berlin, Heidelberg, Springer (2006) 677--686
• 3: Ratzka, A.: Design patterns in the context of multi-modal interaction. In: To appear in: Proceedings of the 6th Nordic Conference on Pattern Languages of Programs 2007 VikingPLoP 2007. (2008)
• 4: Tidwell, J.: Common ground: A pattern language for human-computer interface design. (1999)
• 5: Jenifer Tidwell, Designing Interfaces, O'Reilly Media, Inc., 2005
• 6: van Welie, M., Trætteberg, H.: Interaction patterns in user interfaces. In: Proceedings of the Seventh Pattern Languages of Programs Conference. (2000)
• 7: Ratzka, A.: A wizard-of-oz setting for multimodal interaction. an approach to user-based elicitation of design patterns. In Osswald, A., Stempfhuber, M., Wolff, C., eds.: Open Innovation. Proc. 10th International Symposium for Information Science, Konstanz, Universit¨atsverlag Konstanz (2007) 159--170
• 8: Niedermaier, F.B.: Entwicklung und Bewertung eines Rapid-Prototyping Ansatzes zur multimodalen Mensch-Maschine-Interaktion im Kraftfahrzeug. PhD thesis, Fakultat fur Elektrotechnik und Informationstechnik der Technischen Universitat Munchen (2003)
• 9: DragiÇcevic, P.: Un modele d'interaction en entree pour des systemes interactifs multi-dispositifs hautement configurables. PhD thesis, Universite de Nantes, ecole doctorale sciences et technologies de l'information et des materieaux (Mars 2004)
• 10: Bernsen, N.O.: A toolbox of output modalities: Representing output information in multimodal interfaces. (1997)
• 11: Bernsen, N.O.: Multimodality in language and speech systems -- from theory to design support tool. In Granstrom, B., ed.: Multimodality in Language and Speech Systems, Dordrecht, Kluwer (2001)
• 12: Seifert, K.: Evaluation multimodaler Computer-Systeme in fruhen Entwicklungsphasen. PhD thesis, Fakultat V -- Verkehrs- und Maschinensysteme, Technische Universitat Berlin (2002)
• 13: Alexander, C.: The Timeless Way of Building. Oxford University Press (1979)
• 14: Gabriel, D.: A pattern definition. http://hillside.net/patterns/definition.html (checked on 2007-05-09)
• 15: Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., Angel, S.: A Pattern Language. Oxford University Press (1977)
• 16: Erich Gamma , Richard Helm , Ralph Johnson , John Vlissides, Design patterns: elements of reusable object-oriented software, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 17: Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M.: Pattern-orientierte Softwarearchitektur. Addison--Wesley, Bonn (1998)
• 18: Borchers, J.O.: A pattern approach to interaction design. AI & Society Journal of Human-Centered Systems and Machine Intelligence 15(4) (2001) 359--376
• 19: Douglas K. Van Duyne , James Landay , Jason I. Hong, The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2002
• 20: van Welie, M.: Gui design patterns. In: www.welie.com -- ...patterns in Interaction Design. (2003)
• 21: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 22: Oviatt, S., Olsen, E.: Integration themes in multimodal human-computer interaction. In Shirai, K., Furui, S., Kakehi, K., eds.: Proceedings of the International Conference on Spoken Language Processing. Volume 2., Yokohama, Japan, Acoustical Society of Japan (1994)
• 23: Sharon L. Oviatt , Philip R. Cohen , Michelle Wang, Toward interface design for human language technology: modality and structure as determinants of linguistic complexity, Speech Communication, v.15 n.3-4, p.283-300, Dec. 1994
• 24: Antonella De Angeli , Walter Gerbino , Giulia Cassano , Daniela Petrelli, Visual display, pointing, and natural language: the power of multimodal interaction, Proceedings of the working conference on Advanced visual interfaces, May 24-27, 1998, L'Aquila, Italy [doi>10.1145/948496.948519]
• 25: Sharon Oviatt, Multimodal interfaces for dynamic interactive maps, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.95-102, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238438]
• 26: Cohen, P., Johnston, M., McGee, D., Oviatt, S.L., Clow, J., Smith, I.: The efficiency of multimodal interaction: A case study. In: ICSLP'98. (1998)
• 27: Philip Cohen , David McGee , Josh Clow, The efficiency of multimodal interaction for a map-based task, Proceedings of the sixth conference on Applied natural language processing, p.331-338, April 29-May 04, 2000, Seattle, Washington [doi>10.3115/974147.974193]
• 28: Oviatt, S., Kuhn, K.: Referential features and linguistic indirection in multimodal language. In: Proceedings of the International Conference on Spoken Language Processing. Volume 6., ASSTA (1998) 2339--2342
• 29: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 30: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 31: Sharon Oviatt, Multimodal system processing in mobile environments, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.21-30, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354408]
• 32: Oviatt, S.L.: Multimodal signal processing in naturalistic noisy environments. In Yuan, B., Huang, T., Tang, X., eds.: Proceedings of the 6th International Conference on SPoken Language Processing (ICSLP). Volume 2., Peking, Chinese Friendship Publishers (2000) 696--699
• 33: Sharon Oviatt, Taming recognition errors with a multimodal interface, Communications of the ACM, v.43 n.9, p.45-51, Sept. 2000 [doi>10.1145/348941.348979]
• 34: Sanjeev Kumar , Philip R. Cohen , Rachel Coulston, Multimodal interaction under exerted conditions in a natural field setting, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027971]
• 35: Nishimoto, T., Shida, N., Kobayashi, T., Shirai, K.: Improving human interface in drawing tool using speech. In: Proceedings of 4th IEEE International Workshop on Robot and Human Communication, ROMAN'95. (1995) 107--112
• 36: Huang, X., Acero, A., Chelba, C., Deng, L., Duchene, D., Goodman, J., Hon, H., Jacoby, D., Jiang, L., Loynd, R., Mahajan, M., Mau, P., Meredith, S., Mughal, S., Neto, S., Plumpe, M., Wang, K., Wang, Y.: Mipad: A next generation pda prototype. In: ICSLP, Peking (2000)
• 37: Jennifer Lai, Facilitating mobile communication with multimodal access to email messages on a cell phone, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986038]
• 38: Matthias Jöst , Jochen Häußler , Matthias Merdes , Rainer Malaka, Multimodal interaction for pedestrians: an evaluation study, Proceedings of the 10th international conference on Intelligent user interfaces, January 10-13, 2005, San Diego, California, USA [doi>10.1145/1040830.1040852]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Experimentation, Human Factors, Verification

Keywords:
mobile computing, multimodality, user interface patterns