Presiding over accidents: system direction of human action

ABSTRACT

As human-computer interaction becomes more closely modeled on human-human interaction, new techniques and strategies for human-computer interaction are required. In response to the inevitable shortcomings of recognition technologies, researchers have studied mediation: interaction techniques by which users can resolve system ambiguity and error. In this paper we approach the human-computer dialogue from the other side, examining system-initiated direction and mediation of human action. We conducted contextual interviews with a variety of experts in fields involving human-human direction, including a film director, photographer, golf instructor, and 911 operator. Informed by these interviews and a review of prior work, we present strategies for directing physical human action and an associated design space for systems that perform such direction. We illustrate these concepts with excerpts from our interviews and with our implemented system for automated media capture or "Active Capture," in which an unaided computer system uses techniques identified in our design space to act as a photographer, film director, and cinematographer.

REFERENCES

• 1: W. A. Ainsworth , S. R. Pratt, Feedback strategies for error correction in speech recognition systems, International Journal of Man-Machine Studies, v.36 n.6, p.833-842, June 1992 [doi>10.1016/0020-7373(92)90075-V]
• 2: Ellen M. Ayoob , Richard Grace , Aaron Steinfeld, A user-centered drowsy-driver detection and warning system, Proceedings of the 2003 conference on Designing for user experiences, June 06-07, 2003, San Francisco, California [doi>10.1145/997078.997081]
• 3: C. Baber , K. S. Hone, Modeling error recovery and repair in automatic speech recognition, International Journal of Man-Machine Studies, v.39 n.3, p.495-515, Sept. 1993 [doi>10.1006/imms.1993.1070]
• 4: Hugh Beyer, Contextual Design: Defining Customer-Centered Systems, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 5: M. Brand , N. Oliver , A. Pentland, Coupled hidden Markov models for complex action recognition, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.994, June 17-19, 1997
• 6: Brennan, S.E. & Hulteen, E.A. "Interaction and Feedback in a Spoken Language System: A Theoretical Framework", Knowledge-Based Systems 8(2-3), 143--151, 1995.
• 7: Clark, H.H. and Brennan, S.E. "Grounding in Communication". Perspectives on Socially Shared Cognition. APA Books, 1991.
• 8: Marc Davis, Editing out Video Editing, IEEE MultiMedia, v.10 n.2, p.54-64, March 2003 [doi>10.1109/MMUL.2003.1195161]
• 9: Davis, M. "Active Capture: Integrating Human-Computer Interaction and Computer Vision/Audition to Automate Media Capture", in ICME 2003, Baltimore, MD, Vol. II, 185--188, 2003.
• 10: Marc Davis , Jeffrey Heer , Ana Ramirez, Active capture: automatic direction for automatic movies, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957028]
• 11: Dunn, R. and Dunn, K. Teaching Students Through Their Individual Learning Styles. Reston Publishing. 1978.
• 12: Gottlieb, H. "The Jack Principles of the Interactive Conversation Interface", Jellyvision, Inc. 2002.
• 13: Kenneth R. Koedinger, Cognitive tutors as modeling tools and instructional models, Smart machines in education: the coming revolution in educational technology, MIT Press, Cambridge, MA, 2001
• 14: Lumet, S. Making Movies. Random House, Inc. 1995.
• 15: Mamet, D. On Directing Film. Penguin Books. 1991.
• 16: Mankoff, J. and Abowd, G.D. "Error Correction Techniques for Handwriting, Speech, and other ambiguous or error prone systems." GVU TechReport GIT-GVU-99-18. June 1999.
• 17: Jennifer Mankoff , Scott E. Hudson , Gregory D. Abowd, Providing integrated toolkit-level support for ambiguity in recognition-based interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.368-375, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332459]
• 18: Sharon Oviatt, Taming recognition errors with a multimodal interface, Communications of the ACM, v.43 n.9, p.45-51, Sept. 2000 [doi>10.1145/348941.348979]
• 19: Bernhard Suhm , Brad Myers , Alex Waibel, Multimodal error correction for speech user interfaces, ACM Transactions on Computer-Human Interaction (TOCHI), v.8 n.1, p.60-98, March 2001 [doi>10.1145/371127.371166]
• 20: Nicole Yankelovich , Gina-Anne Levow , Matt Marx, Designing SpeechActs: issues in speech user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems, p.369-376, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223952]
• 21: Elizabeth Zoltan-Ford, How to get people to say and type what computers can understand, International Journal of Man-Machine Studies, v.34 n.4, p.527-547, April 1991 [doi>10.1016/0020-7373(91)90034-5]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Keywords:
active capture, direction, error, error-prone systems, mediation, multimedia systems design, recognition