Something from nothing: augmenting a paper-based work practice via multimodal interaction

ABSTRACT

In this paper, we describe Rasa: an environment designed to augment, rather than replace, the work habits of its users. These work habits include drawing on Post-it™ notes using a symbolic language. Rasa observes and understands this language, assigning meaning simultaneously to objects in both the physical and virtual worlds. With Rasa, users rollout a paper map, register it, and move the augmented objects from one place to another on it. Once an object is augmented, users can modify the meaning represented by it, ask questions about that representation, view it in virtual reality, or give directions to it, all with speech and gestures. We examine the way Rasa uses language to augment objects, and compare it with prior methods, arguing that language is a more visible, flexible, and comprehensible method for creating augmentations than other approaches.

REFERENCES

• 1: Richard A. Bolt, “Put-that-there”: Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics, v.14 n.3, p.262-270, July 1980
• 2: Carpenter, R., Typed feature structures: Inheritance, (in)equality, and extensionality, in the Proceedings of the ITK Workshop: Inheritance in Natural Language Processing, Institute for Language Technology and Artificial Intelligence, Tilburg University, Tilburg, 9-18.
• 3: Bob Carpenter, The logic of typed feature structures, Cambridge University Press, New York, NY, 1992
• 4: Michael H. Coen, Design principles for intelligent environments, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.547-554, July 1998, Madison, Wisconsin, United States
• 5: Philip R. Cohen, Integrated interfaces for decision-support with simulation, Proceedings of the 23rd conference on Winter simulation, p.1066-1072, December 08-11, 1991, Phoenix, Arizona, United States
• 6: Philip R. Cohen , Adam Cheyer , Michelle Wang , Soon Cheol Baeg, An open agent architecture, Readings in agents, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1997
• 7: Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay Pittman , Ira Smith , Liang Chen , Josh Clow, QuickSet: multimodal interaction for distributed applications, Proceedings of the fifth ACM international conference on Multimedia, p.31-40, November 09-13, 1997, Seattle, Washington, United States [doi>10.1145/266180.266328]
• 8: Philip Cohen , David McGee , Josh Clow, The efficiency of multimodal interaction for a map-based task, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633311]
• 9: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 10: Johnston, M., Unification-based multimodal parsing, in the Proceedings of the 17th International Conference on Computational Linguistics and the 36th An-nual Meeting of the Association for Computational Linguistics (COLING-ACL 98), August 98, ACL Press, 624-630.
• 11: Michael Johnston , Philip R. Cohen , David McGee , Sharon L. Oviatt , James A. Pittman , Ira Smith, Unification-based multimodal integration, Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, p.281-288, July 07-12, 1997, Madrid, Spain
• 12: David B. Koons , Carlton J. Sparrell , Kristinn R. Thorisson, Integrating simultaneous input from speech, gaze, and hand gestures, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 13: Krueger, M.W., Artificial Reality II. 1991, Reading, MA: Addison-Wesley. p. 286.
• 14: David R. McGee , Philip R. Cohen , Sharon Oviatt, Confirmation in multimodal systems, Proceedings of the 36th annual meeting on Association for Computational Linguistics, p.823-829, August 10-14, 1998, Montreal, Quebec, Canada
• 15: Thomas P. Moran , Eric Saund , William van Melle , Robert Bryll , Anuj U. Gujar , Kenneth P. Fishkin , Beverly L. Harrison, The ins and outs of collaborative walls: demonstrating the collaborage concept, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632837]
• 16: Jeannette G. Neal , Stuart C. Shapiro, Intelligent multi-media interface technology, Intelligent user interfaces, ACM Press, New York, NY, 1991 [doi>10.1145/107215.128690]
• 17: William Newman , Pierre Wellner, A desk supporting computer-based interaction with paper documents, Proceedings of the SIGCHI conference on Human factors in computing systems, p.587-592, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.143007]
• 18: Laurence Nigay , Joëlle Coutaz, A generic platform for addressing the multimodal challenge, Proceedings of the SIGCHI conference on Human factors in computing systems, p.98-105, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223917]
• 19: Oviatt, S.L., Multimodal interactive maps: Designing for human performance. Human-Computer Interaction, 1997. 12 (special issue on "Multimodal interfaces"): 93-129.
• 20: Sharon Oviatt , Antonella DeAngeli , Karen Kuhn, Integration and synchronization of input modes during multimodal human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.415-422, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258821]
• 21: Norbert A. Streitz , Jörg Geißler , Torsten Holmer, Roomware for Cooperative Buildings: Integrated Design of Architectural Spaces and Information Spaces, Proceedings of the First International Workshop on Cooperative Buildings, Integrating Information, Organization, and Architecture, p.4-21, February 01, 1998
• 22: Brygg Ullmer , Hiroshi Ishii, The metaDESK: models and prototypes for tangible user interfaces, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.223-232, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263551]
• 23: Brygg Ullmer , Hiroshi Ishii , Dylan Glas, mediaBlocks: physical containers, transports, and controls for online media, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.379-386, July 1998 [doi>10.1145/280814.280940]
• 24: John Underkoffler , Hiroshi Ishii, Urp: a luminous-tangible workbench for urban planning and design, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.386-393, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303114]
• 25: Roy Want , Kenneth P. Fishkin , Anuj Gujar , Beverly L. Harrison, Bridging physical and virtual worlds with electronic tags, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.370-377, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303111]
• 26: Mark Weiser, Some computer science issues in ubiquitous computing, Communications of the ACM, v.36 n.7, p.75-84, July 1993 [doi>10.1145/159544.159617]
• 27: Pierre Wellner, Interacting with paper on the DigitalDesk, Communications of the ACM, v.36 n.7, p.87-96, July 1993 [doi>10.1145/159544.159630]
• 28: Wu, L., Oviatt, S., and Cohen, P., Multimodal integration - A statistical view. IEEE Transactions on Multimedia, 1999. 1(4): 334-341.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality

General Terms:
Design, Human Factors, Management, Performance, Theory

Keywords:
augmented reality, invisible interfaces, mixed reality, multimodal interfaces, phicons, tangible interfaces, ubiquitous computing