Turn it <u>this</u> way: grounding collaborative action with remote gestures

ABSTRACT

Remote gesture systems have been shown to provide a significant enhancement to performance in collaborative physical tasks, an effect ascribed to the ability of remote gestures to help ground deictic references. The argument that this effect works by replacing complex referential descriptions with simple pointing behaviours has been drawn into question by recent research. In this paper we significantly unpack the effects of remote gesturing on collaborative language, arguing for a more complex role for remote gestures in interaction. We demonstrate how remote gestures influence the structure of collaborative discourse, and how their use can also influence the temporal nature of the grounding process. Through generating a deeper understanding of these effects of remote gesturing on collaborative language we derive implications for the development and deployment of these technologies.

REFERENCES

• 1: Mathilde M. Bekker , Judith S. Olson , Gary M. Olson, Analysis of gestures in face-to-face design teams provides guidance for how to use groupware in design, Proceedings of the 1st conference on Designing interactive systems: processes, practices, methods, & techniques, p.157-166, August 23-25, 1995, Ann Arbor, Michigan, United States [doi>10.1145/225434.225452]
• 2: S. A. Bly , S. L. Minneman, Commune: a shared drawing surface, Proceedings of the ACM SIGOIS and IEEE CS TC-OA conference on Office information systems, p.184-192, April 25-27, 1990, Cambridge, Massachusetts, United States
• 3: Clark, H. H. Using language. (1996). Cambridge: Cambridge University Press.
• 4: Clark, H. H., & Brennan, S.E. Grounding in Communication. In L.B. Resnick, R.M. Levine, & S.D. Teasley (Eds.). Perspectives on socially shared cognition, (1991). 127--149. Washington, DC: APA.
• 5: Clark, H. H., & Krych, M. A. Speaking while monitoring addressees for understanding. Journal of Memory & Language, 50 (1), (2004). 62--81.
• 6: Clark, H. H., & Wilkes-Gibbs, D. Referring as a collaborative process. Cognition, 22 (1), (1986). 1--39.
• 7: Duncan, S. and & Fiske, D.W. Interaction Structure and Strategy. (1985) NewYork: Cambridge University Press.
• 8: Fussell, S. R., Setlock, L. D., Yang, J., Ou, J., Mauer, E. M., & Kramer, A. Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction, 19, (2004). 273--309.
• 9: Susan R. Fussell , Robert E. Kraut , Jane Siegel, Coordination of communication: effects of shared visual context on collaborative work, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.21-30, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358947]
• 10: Susan R. Fussell , Leslie D. Setlock , Elizabeth M. Parker, Where do helpers look?: gaze targets during collaborative physical tasks, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765980]
• 11: Darren Gergle , Robert E. Kraut , Susan R. Fussell, Action as language in a shared visual space, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031687]
• 12: Hiroshi Ishii , Minoru Kobayashi, ClearBoard: a seamless medium for shared drawing and conversation with eye contact, Proceedings of the SIGCHI conference on Human factors in computing systems, p.525-532, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142977]
• 13: Karsenty, L. Cooperative work and shared visual context: An empirical study of comprehension problems and in side-by-side and remote help dialogues. Human-Computer Interaction, 14(3), (1999). 283--315.
• 14: David Kirk , Danae Stanton Fraser, Comparing remote gesture technologies for supporting collaborative physical tasks, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124951]
• 15: David Kirk , Andy Crabtree , Tom Rodden, Ways of the hands, Proceedings of the ninth conference on European Conference on Computer Supported Cooperative Work, p.1-21, September 18-22, 2005, Paris, France
• 16: Adam D. I. Kramer , Lui Min Oh , Susan R. Fussell, Using linguistic features to measure presence in computer-mediated communication, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124907]
• 17: Krauss, R. & Fussell, S. Constructing Shared Communicative Environments. In L.B. Resnick, R.M. Levine, & S.D. Teasley (Eds.). Perspectives on socially shared cognition, (1991). 127--149. Washington, DC: APA.
• 18: Kraut, R. E., Fussell, S. R., & Siegel, J. Visual information as a conversational resource in collaborative physical tasks. Human Computer Interaction, 18, 1 (2003). 13--49.
• 19: Robert E. Kraut , Darren Gergle , Susan R. Fussell, The use of visual information in shared visual spaces: informing the development of virtual co-presence, Proceedings of the 2002 ACM conference on Computer supported cooperative work, November 16-20, 2002, New Orleans, Louisiana, USA [doi>10.1145/587078.587084]
• 20: Robert E. Kraut , Mark D. Miller , Jane Siegel, Collaboration in performance of physical tasks: effects on outcomes and communication, Proceedings of the 1996 ACM conference on Computer supported cooperative work, p.57-66, November 16-20, 1996, Boston, Massachusetts, United States [doi>10.1145/240080.240190]
• 21: Hideaki Kuzuoka , Jun'ichi Kosaka , Keiichi Yamazaki , Yasuko Suga , Akiko Yamazaki , Paul Luff , Christian Heath, Mediating dual ecologies, Proceedings of the 2004 ACM conference on Computer supported cooperative work, November 06-10, 2004, Chicago, Illinois, USA [doi>10.1145/1031607.1031686]
• 22: Paul Luff , Christian Heath , Hideaki Kuzuoka , Keiichi Yamazaki , Jun Yamashita, Handling documents and discriminating objects in hybrid spaces, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124858]
• 23: McNeill, D. Hand and Mind. What gestures reveal about thought. (1996) Chicago: University of Chicago Press.
• 24: Yukiko I. Nakano , Gabe Reinstein , Tom Stocky , Justine Cassell, Towards a model of face-to-face grounding, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.553-561, July 07-12, 2003, Sapporo, Japan [doi>10.3115/1075096.1075166]
• 25: O'Conaill, B. and & Whittaker, S. Characterizing, Predicting and Measuring Video-Mediated Communication: A Conversational Approach. In K. Finn, A . Sellen and S. Wilbur (Eds.) Video Mediated Communication. (1997), LEA.
• 26: Ochsman, R. B., & Chapanis, A. The Effects of 10 Communication Modes on the Behavior of Teams During Cooperative Problem-solving. International Journal of Man-Machine Studies, 6, (1974) 579--619.
• 27: Jiazhi Ou , Susan R. Fussell , Xilin Chen , Leslie D. Setlock , Jie Yang, Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958477]
• 28: Rauscher, F. H., Krauss, R. M. & Chen, Y. (1996) Gesture, Speech and Lexical Access: The Role of Lexical Movements in Speech Production. Psychological Science. 7, 4, 226--231.
• 29: Nobuchika Sakata , Takeshi Kurata , Takekazu Kato , Masakatsu Kourogi , Hideaki Kuzuoka, WACL: Supporting Telecommunications Using Wearable Active Camera with Laser Pointer, Proceedings of the 7th IEEE International Symposium on Wearable Computers, p.53, October 21-23, 2003
• 30: Abigail J. Sellen, Speech patterns in video-mediated conversations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.49-59, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142756]
• 31: Anthony Tang , Michael Boyle , Saul Greenberg, Display and presence disparity in Mixed Presence Groupware, Proceedings of the fifth conference on Australasian user interface, p.73-82, January 01, 2004, Dunedin, New Zealand
• 32: John C. Tang , Scott L. Minneman, VideoDraw: a video interface for collaborative drawing, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.313-320, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97302]
• 33: John C. Tang , Scott Minneman, VideoWhiteboard: video shadows to support remote collaboration, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.315-322, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108932]
• 34: Elizabeth S. Veinott , Judith Olson , Gary M. Olson , Xiaolan Fu, Video helps remote work: speakers who need to negotiate common ground benefit from seeing each other, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.302-309, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303067]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Computer-supported cooperative work

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Computer conferencing, teleconferencing, and videoconferencing

General Terms:
Human Factors

Keywords:
CSCW, conversational grounding, language use, overlaps, remote gesturing