Two-handed gesture in multi-modal natural dialog

ABSTRACT

Tracking both hands in free-space with accompanying speech input can augment the user's ability to communicate with computers. This paper discusses the kinds of situations which call for two-handed input and not just the single hand, and reports a prototype in which two-handed gestures serve to input concepts, both static and dynamic, manipulate displayed items, and specify actions to be taken. Future directions include enlargement of the vocabulary of two-handed “coverbal” gestures and the modulation by gaze of gestural intent.

REFERENCES

• 1: Bolt, R. A. "The integrated multi-modal interface." The Transactions of the Institute of Electronics, Information and Communication Engineers (Japan), November 1987, Vol. J70-D, No. 11, pp. 2017-2025.
• 2: Bolt, Richard A. "Conversing with computers." Technology Review, Vol. 88, No. 2, Feb/Mar 1985.
• 3: Bolt, Richard. A. The human interface. Van Nostrand Reinhold, New York, 1984.
• 4: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, Proceedings of the 7th annual conference on Computer graphics and interactive techniques, p.262-270, July 14-18, 1980, Seattle, Washington, United States
• 5: Buxton, William. There's more to interaction than meets the eye: some issues in manual Input. In D.A. Norman and S. W. Draper (eds.), User centered system design. Hillsdale, NJ: Lawrence Erlbaum, 1986.
• 6: W. Buxton , B. Myers, A study in two-handed input, Proceedings of the SIGCHI conference on Human factors in computing systems, p.321-326, April 13-17, 1986, Boston, Massachusetts, United States
• 7: P. R. Cohen , M. Dalrymple , D. B. Moran , F. C. Pereira , J. W. Sullivan, Synergistic use of direct manipulation and natural language, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.227-233, March 1989
• 8: James D. Foley, Interfaces for advanced computing, Scientific American, v.257 n.4, p.126-135, Oct. 1987
• 9: Barbara J. Grosz , Karen Sparck-Jones , Bonnie Lynn Webber, Readings in natural language processing, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1986
• 10: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 11: Philip J. Hayes, The utility of natural language interfaces (panel session), Proceedings of the SIGCHI conference on Human factors in computing systems, p.19, April 1985, San Francisco, California, United States
• 12: Herranz, Edward. Giving directions to computers using two-handed gestures, speech, and gaze. Unplubished MIT Masters's thesis, May 1992.
• 13: Hill, I. D. Natural language versus computer language. In M. E. Sime and M. J. Coombs (eds.), Designing for human-computer communication. New York: Academic Press, 1983, 55-72.
• 14: Kendon, Adam. Current issues in the study of gesture. In: The Biological Foundations of Gestures: Motor and Semiotic Aspects. Jean-Luc Nespoulous, Paul Perron, and Andr6 Roch Lecours, eds. Hillside, NJ: Lawrence Erlbaum Associates, inc., 1986, 23-47.
• 15: Krueger, Myron W. Artificial reality H. Reading, MA: Addison-Wesley Publishing Company, 1991.
• 16: J. G. Neal , C. Y. Thielman , Z. Dobes , S. M. Haller , S. C. Shapiro, Natural language with integrated deictic and graphic gestures, Proceedings of the workshop on Speech and Natural Language, October 15-18, 1989, Cape Cod, Massachusetts [doi>10.3115/1075434.1075499]
• 17: Nespoulous, Jean-Luc and Andr6 Roch Lecours. Gestures: Nature and function, in: The Biological Foundations of Gestures: Motor and Semiotic Aspects. Jean- Luc Nespoulous, Paul Perron, and Andr6 Roch Lecours, eds. Hillside, NJ: Lawrence Erlbaum Associates, Inc., 1986, 49- 62.
• 18: Olson, David R. Language and thought: aspects of a cognitive of semantics. Psychogical Review, Vol. 77, No. 4, July 1970, 257-273.
• 19: Rich, Elaine. Natural-language interfaces. IEEE Computer, September 1984, Vol. 17, No. 9, 39-47.
• 20: India Starker , Richard A. Bolt, A gaze-responsive self-disclosing display, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.3-10, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97245]
• 21: D. J. Sturman , D. Zeltzer , S. Pieper, Hands-on interaction with virtual environments, Proceedings of the 2nd annual ACM SIGGRAPH symposium on User interface software and technology, p.19-24, November 13-15, 1989, Williamsburg, Virginia, United States [doi>10.1145/73660.73663]
• 22: Colin Ware , Harutune H. Mikaelian, An evaluation of an eye tracker as a device for computer input2, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.183-188, April 05-09, 1987, Toronto, Ontario, Canada
• 23: D. Weimer , S. K. Ganapathy, A synthetic visual environment with hand gesturing and voice input, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.235-240, March 1989
• 24: Yorick Wilks, Theoretical Issues in Natural Language Processing, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1989
• 25: Thomas G. Zimmerman , Jaron Lanier , Chuck Blanchard , Steve Bryson , Young Harvill, A hand gesture interface device, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.189-192, April 05-09, 1987, Toronto, Ontario, Canada

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis

General Terms:
Design, Human Factors, Languages

Keywords:
gestural input, multi-modal interaction, natural dialog