How pairs interact over a multimodal digital table

ABSTRACT

Co-located collaborators often work over physical tabletops using combinations of expressive hand gestures and verbal utterances. This paper provides the first observations of how pairs of people communicated and interacted in a multimodal digital table environment built atop existing single user applications. We contribute to the understanding of these environments in two ways. First, we saw that speech and gesture commands served double duty as both commands to the computer, and as implicit communication to others. Second, in spite of limitations imposed by the underlying single-user application, people were able to work together simultaneously, and they performed interleaving acts: the graceful mixing of inter-person speech and gesture actions as commands to the system. This work contributes to the intricate understanding of multi-user multimodal digital table interaction.

REFERENCES

• 1: Multimodal Interaction During Multiparty Dialogues: Initial Results, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.448, October 14-16, 2002 [doi>10.1109/ICMI.2002.1167037]
• 2: Clark, H. Using language. Cambridge Univ. Press, 1996.
• 3: Clow, J. and Oviatt, S. (1998) STAMP: A suite of tools for analyzing multimodal system processing, Proc. Int. Conf. Spoken Language Processing, 1998.
• 4: Sharon Oviatt, Ten myths of multimodal interaction, Communications of the ACM, v.42 n.11, p.74-81, Nov. 1999 [doi>10.1145/319382.319398]
• 5: David Pinelle , Carl Gutwin , Saul Greenberg, Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.4, p.281-311, December 2003 [doi>10.1145/966930.966932]
• 6: Segal, L. Effects of checklist interface on non-verbal crew communications, NASA Ames Research Center, Contractor Report 177639. 1994.
• 7: Stacey D. Scott , M. Sheelagh , T. Carpendale , Kori M. Inkpen, Territoriality in collaborative tabletop workspaces, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031655]
• 8: Anthony Tang , Melanie Tory , Barry Po , Petra Neumann , Sheelagh Carpendale, Collaborative coupling over tabletop displays, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124950]
• 9: John C. Tang, Findings from observational studies of collaborative work, International Journal of Man-Machine Studies, v.34 n.2, p.143-160, Feb. 1991 [doi>10.1016/0020-7373(91)90039-A]
• 10: Edward Tse , Chia Shen , Saul Greenberg , Clifton Forlines, Enabling interaction with single user applications through speech and gestures on a multi-user tabletop, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133336]
• 11: Edward Tse , Saul Greenberg , Chia Shen, GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181012]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

General Terms:
Design, Human Factors

Keywords:
digital tables, gestures, multimodal interaction, speech