When do we interact multimodally?: cognitive load and multimodal communication patterns

ABSTRACT

Mobile usage patterns often entail high and fluctuating levels of difficulty as well as dual tasking. One major theme explored in this research is whether a flexible multimodal interface supports users in managing cognitive load. Findings from this study reveal that multimodal interface users spontaneously respond to dynamic changes in their own cognitive load by shifting to multimodal communication as load increases with task difficulty and communicative complexity. Given a flexible multimodal interface, users' ratio of multimodal (versus unimodal) interaction increased substantially from 18.6% when referring to established dialogue context to 77.1% when required to establish a new context, a +315% relative increase. Likewise, the ratio of users' multimodal interaction increased significantly as the tasks became more difficult, from 59.2% during low difficulty tasks, to 65.5% at moderate difficulty, 68.2% at high and 75.0% at very high difficulty, an overall relative increase of +27%. Analysis of users' task-critical errors and response latencies across task difficulty levels increased systematically and significantly as well, corroborating the manipulation of cognitive processing load. The adaptations seen in this study reflect users' efforts to self-manage limitations on working memory when task complexity increases. This is accomplished by distributing communicative information across multiple modalities, which is compatible with a cognitive load theory of multimodal interaction. The long-term goal of this research is the development of an empirical foundation for proactively guiding flexible and adaptive multimodal system design.

REFERENCES

• 1: Almor, A., Noun-phrase anaphora and focus: The informational load hypothesis. Psychological Review, 1999. 106: 748--765.
• 2: Baddeley, A., Working Memory. Science, 1992. 255: 556--559.
• 3: Benoit, C., J.-C. Martin, C. Pelachaud, L. Schomaker, & B. Suhm, Audio-visual and multimodal speech-based systems, Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation, R. Moore, ed. 2000, Kluwer Academic Publishers: Boston, MA. 102--203.
• 4: Calvert, G., C. Spence, & B.E. Stein, eds. The handbook of multisensory processing. 2004, MIT Press: Cambridge, MA.
• 5: Chandler, P. & J. Sweller, Cognitive load theory and the format of instruction. Cognition and Instruction, 1991. 8: 293--332.
• 6: Grant, K.W. & S. Greenberg. Speech intelligibility derived from asynchronous processing of auditory-visual information. Workshop on Audio-Visual Speech Processing (AVSP-2001). 2001. Scheelsminde, Denmark
• 7: Grice, H.P., Logic and conversation, Syntax and Semantics: Speech Acts, J.L. Morgan, ed. 1975, Acad Press: NY. 41--58.
• 8: Ken Hinckley , Jeff Pierce , Eric Horvitz , Mike Sinclair, Foreground and background interaction with sensor-enhanced mobile devices, ACM Transactions on Computer-Human Interaction (TOCHI), v.12 n.1, p.31-52, March 2005 [doi>10.1145/1057237.1057240]
• 9: Julie A. Jacko , Leon Barnard , Thitima Kongnakorn , Kevin P. Moloney , Paula J. Edwards , V. Kathlene Emery , Francois Sainfort, Isolating the effects of visual impairment: exploring the effect of AMD on the utility of multimodal feedback, Proceedings of the SIGCHI conference on Human factors in computing systems, p.311-318, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985732]
• 10: Anthony Jameson, Adaptive interfaces and agents, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 11: Mayer, R.E. & R. Moreno, A split-attention effect in multimedia learning: evidence for dual processing systems in working memory. Journal of Educational Psychology, 1998. 90(2): 312--320.
• 12: Mousavi, S.Y., R. Low, & J. Sweller, Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 1995. 87(2): 319--334.
• 13: Christian Müller , Barbara Großmann-Hutter , Anthony Jameson , Ralf Rummer , Frank Wittig, Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study, Proceedings of the 8th International Conference on User Modeling 2001, p.24-33, July 13-17, 2001
• 14: Oviatt, S.L., Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 1995. 9: 19--35.
• 15: Oviatt, S.L., Multimodal interactive maps: Designing for human performance. Human Computer Interaction, 1997. 12(1-2): 93--129.
• 16: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 17: Sharon Oviatt, Mutual disambiguation of recognition errors in a multimodel architecture, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.576-583, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303163]
• 18: Sharon Oviatt, Multimodal system processing in mobile environments, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.21-30, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354408]
• 19: Oviatt, S.L., P.R. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T.G. Holzman, T. Winograd, J. Landay, J. Larson, & D. Ferro, Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. Human Computer Interaction, 2000. 15(4): 263--322.
• 20: Sharon Oviatt , Rachel Coulston , Stefanie Tomko , Benfang Xiao , Rebecca Lunsford , Matt Wesson , Lesley Carmichael, Toward a theory of organized multimodal integration patterns during human-computer interaction, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958443]
• 21: Oviatt, S.L., T. Darrell, & M. Flickner, Multimodal Interfaces that flex, adapt, and persist, Comm. of the ACM. 2004. 30--33
• 22: Penney, C.G., Modality effects and the structure of short-term verbal memory. Memory and Cognition, 1989. 17: 398--422.
• 23: Prince, E., Toward a taxonomy of given-new information, Radical Pragmatics, P. Cole, ed. 1986, Academic: NY. 223--255.
• 24: Sweller, J., Cognitive load during problem solving: Effects on learning. Cognitive Science, 1988. 12: 257--285.
• 25: Technology for adaptive aging: Reports and papers. 2003, Nat. Acad. of Sci. Workshop: Nat. Acad. Press. http://www.nap.edu/books/0309091160/html/
• 26: Teder-Sälejärvi, W.A., J.J. McDonald, F. Di Russo, & S.A. Hillyard, An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 2002. 14: 106--114.
• 27: Tindall-Ford, S., P. Chandler, & J. Sweller, When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 1997. 3(3): 257--287.
• 28: Wickens, C., Sandry, D., and Vidulich, M., Compatibility and resource competition between modalities of input, central processing, and output. Human Factors, 1983. 25(2): 227--248.
• 29: Benfang Xiao , Rebecca Lunsford , Rachel Coulston , Matt Wesson , Sharon Oviatt, Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958480]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Theory and methods; Input devices and strategies (e.g., mouse, touchscreen); Prototyping; Voice I/O; Natural language; Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Human Factors, Performance, Reliability

Keywords:
cognitive load, dialogue context, human performance, individual differences, multimodal integration, multimodal interaction, speech and pen input, system adaptation, task difficulty, unimodal interaction