Exploring true multi-user multimodal interaction over a digital table

ABSTRACT

True multi-user, multimodal interaction over a digital table lets co-located people simultaneously gesture and speak commands to control an application. We explore this design space through a case study, where we implemented an application that supports the KJ creativity method as used by industrial designers. Four key design issues emerged that have a significant impact on how people would use such a multi-user multimodal system. First, parallel work is affected by the design of multimodal commands. Second, individual mode switches can be confusing to collaborators, especially if speech commands are used. Third, establishing personal and group territories can hinder particular tasks that require artefact neutrality. Finally, timing needs to be considered when designing joint multimodal commands. We also describe our model view controller architecture for true multi-user multimodal interaction.

REFERENCES

• 1: Dzmitry Aliakseyeu , Sriram Subramanian , Andrés Lucero , Carl Gutwin, Interacting with piles of artifacts on digital tables, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133298]
• 2: Benjamin B. Bederson , Jesse Grosjean , Jon Meyer, Toolkit Design for Interactive Structured Graphics, IEEE Transactions on Software Engineering, v.30 n.8, p.535-546, August 2004 [doi>10.1109/TSE.2004.44]
• 3: Boyle, M. and Greenberg, S. (2005) Rapidly Prototyping Multimedia Groupware. Proc. Distributed Multimedia Systems '05, Knowledge Systems Institute.
• 4: Buxton, W. (2007) Sketching User Experiences: Getting the Design Right and the Right Design, Morgan Kaufmann, ISBN-13: 978-0123740373.
• 5: Donald Cox , Saul Greenberg, Supporting collaborative interpretation in distributed Groupware, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.289-298, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.359000]
• 6: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 7: Gregg Foster , Mark Stefik, Cognoter: theory and practice of a colab-orative tool, Proceedings of the 1986 ACM conference on Computer-supported cooperative work, December 03-05, 1986, Austin, Texas [doi>10.1145/637069.637072]
• 8: Saul Greenberg , Carl Gutwin , Mark Roseman, Semantic Telepointers for Groupware, Proceedings of the 6th Australian Conference on Computer-Human Interaction (OZCHI '96), p.54, November 24-27, 1996
• 9: Gutwin, C., and Greenberg, S. (2004) The importance of awareness for team cognition in distributed collaboration. In E. Salas, S. Fiore (Eds) Team Cognition: Understanding the Factors that Drive Process and Performance, APA Press, 177--201.
• 10: Hall, E. (1966) The Hidden Dimension. Anchor Books
• 11: Scott R. Klemmer , Mark W. Newman , Ryan Farrell , Mark Bilezikjian , James A. Landay, The designers' outpost: a tangible interface for collaborative web site, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502350]
• 12: Russell Kruger , Sheelagh Carpendale , Stacey D. Scott , Saul Greenberg, Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication, Computer Supported Cooperative Work, v.13 n.5-6, p.501-537, December 2004 [doi>10.1007/s10606-004-5062-8]
• 13: Lucero, A., Martens, J. (2005) Mood Boards: Industrial Designers' Perception of Using Mixed Reality. Proc. SIGCHI. NL Conference 2005, 13--16.
• 14: Rogers, Y. and Lindley, S. (2004) Collaborating around vertical and horizontal large interactive displays: which way is best? Interacting with Computers (16), 1133--1152. Elsevier.
• 15: Stacey D. Scott , M. Sheelagh , T. Carpendale , Kori M. Inkpen, Territoriality in collaborative tabletop workspaces, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031655]
• 16: Somer, R., (1969) Personal Space: The Behavioural Basis of Design, Spectrum, ISBN 0-13-657577-3.
• 17: Anthony Tang , Melanie Tory , Barry Po , Petra Neumann , Sheelagh Carpendale, Collaborative coupling over tabletop displays, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124950]
• 18: Tang, A., Neustaedter, C. and Greenberg, S. (2006) VideoArms: Embodiments for Mixed Presence Groupware. Proc. the 20th BCS-HCI British HCI 2006 Group Conference (Sept 11--15, Queen Mary, University of London, UK).
• 19: Deborah G. Tatar , Gregg Foster , Daniel G. Bobrow, Design for conversation: lessons from Cognoter, International Journal of Man-Machine Studies, v.34 n.2, p.185-209, Feb. 1991 [doi>10.1016/0020-7373(91)90041-5]
• 20: Edward Tse , Chia Shen , Saul Greenberg , Clifton Forlines, Enabling interaction with single user applications through speech and gestures on a multi-user tabletop, Proceedings of the working conference on Advanced visual interfaces, May 23-26, 2006, Venezia, Italy [doi>10.1145/1133265.1133336]
• 21: Edward Tse , Chia Shen , Saul Greenberg , Clifton Forlines, How pairs interact over a multimodal digital table, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240659]
• 22: Edward Tse , Saul Greenberg , Chia Shen, GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications, Proceedings of the 8th international conference on Multimodal interfaces, November 02-04, 2006, Banff, Alberta, Canada [doi>10.1145/1180995.1181012]
• 23: Tse, E., Greenberg, S., Shen, C., Barnwell, J., Shipman, S. and Leigh, D. (2007) Multimodal Split View Tabletop Interaction Over Existing Applications. Report 2007-869-21, Dept. of Computer Science, University of Calgary, Canada.
• 24: Daniel Vogel , Patrick Baudisch, Shift: a technique for operating pen-based interfaces using touch, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240727]
• 25: Daniel Wigdor , Ravin Balakrishnan, Empirical investigation into the effect of orientation on text readability in tabletop displays, Proceedings of the ninth conference on European Conference on Computer Supported Cooperative Work, p.205-224, September 18-22, 2005, Paris, France
• 26: T. Yuizono , J. Munemori , Y. Nagasawa, GUNGEN: Groupware for a New Idea Generation Consistent Support System, Proceedings of the Third Asian Pacific Computer and Human Interaction, p.357, July 15-17, 1998

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

General Terms:
Design, Human Factors

Keywords:
computer supported cooperative work, multimodal speech and gesture interfaces, tabletop interaction