WordsEye: an automatic text-to-scene conversion system

ABSTRACT

Natural language is an easy and effective medium for describing visual ideas and mental images. Thus, we foresee the emergence of language-based 3D scene generation systems to let ordinary users quickly create 3D scenes without having to learn special software, acquire artistic skills, or even touch a desktop window-oriented interface. WordsEye is such a system for automatically converting text into representative 3D scenes. WordsEye relies on a large database of 3D models and poses to depict entities and actions. Every 3D model can have associated shape displacements, spatial tags, and functional properties to be used in the depiction process. We describe the linguistic analysis and depiction techniques used by WordsEye along with some general strategies by which more abstract concepts are made depictable.

REFERENCES

• 1: Proceedings of the Sixth Message Understanding Conference (MUC-6), San Mateo, CA, 1995. Morgan Kaufmann.
• 2: Giovanni Adorni , Mauro Di Manzo , Fausto Giunchiglia, Natural language driven image generation, Proceedings of the 22nd annual meeting on Association for Computational Linguistics, p.495-500, July 02-06, 1984, Stanford, California
• 3: Norman I. Badler , Rama Bindiganavale , Jan Allbeck , William Schuler , Liwei Zhao , Martha Palmer, Parameterized action representation for virtual human agents, Embodied conversational agents, MIT Press, Cambridge, MA, 2001
• 4: Rama Bindiganavale , William Schuler , Jan M. Allbeck , Norman I. Badler , Aravind K. Joshi , Martha Palmer, Dynamically altering agent behaviors using natural language instructions, Proceedings of the fourth international conference on Autonomous agents, p.293-300, June 03-07, 2000, Barcelona, Spain [doi>10.1145/336595.337503]
• 5: C. Brugman. The Story of Over. Master's thesis, University of California, Berkeley, Berkeley, CA, 1980.
• 6: Yu-Kuang Chang , Alyn P. Rockwood, A generalized de Casteljau approach to 3D free-form deformation, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.257-260, July 1994 [doi>10.1145/192161.192220]
• 7: Kenneth Ward Church, A stochastic parts program and noun phrase parser for unrestricted text, Proceedings of the second conference on Applied natural language processing, February 09-12, 1988, Austin, Texas [doi>10.3115/974235.974260]
• 8: S. R. Clay and J. Wilhelms. Put: Language-Based Interactive Manipulation of Objects. IEEE Computer Graphics and Applications, pages 31-39, March 1996.
• 9: Michael John Collins , Mitchell P. Marcus, Head-driven statistical models for natural language parsing, 1999
• 10: C. Fellbaum, editor. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, 1998.
• 11: C. Freksa, C. Habel, and K. F. Wender, editors. Spatial Cognition. Springer, Berlin, 1998.
• 12: John Funge , Xiaoyuan Tu , Demetri Terzopoulos, Cognitive modeling: knowledge, reasoning and planning for intelligent characters, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.29-38, July 1999 [doi>10.1145/311535.311538]
• 13: Sanda Harabagiu and Steven Maiorano. Knowledge-lean coreference resolution and its relation to textual cohesion and coherence. In Proceedings of the ACL-99 Workshop on the Relation of Discourse/Dialogue Structure and Reference, pages 29-38, College Park, MD, 1999. Association for Computational Linguistics.
• 14: B. Hawkins. The Semantics of English Spatial Prepositions. PhD thesis, University of California, San Diego, San Diego, CA, 1984.
• 15: Annette Herskovits, Language and Spatial Cognition, Cambridge University Press, New York, NY, 1987
• 16: R. Hudson. Word Grammar. Blackwell, Oxford, 1984.
• 17: R. Langacker. Foundations of Cognitive Grammar : Theoretical Prerequisites. Stanford University Press, Stanford, CA, 1987.
• 18: B. Levin. English Verb Classes And Alternations: A Preliminary Investigation. University of Chicago Press, Chicago, IL, 1993.
• 19: Mitchell P. Marcus , Mary Ann Marcinkiewicz , Beatrice Santorini, Building a large annotated corpus of English: the penn treebank, Computational Linguistics, v.19 n.2, June 1993
• 20: Patrick Oliver , Klaus-Peter Gapp, Representation and processing of spatial expressions, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1998
• 21: Craig W. Reynolds, Flocks, herds and schools: A distributed behavioral model, Proceedings of the 14th annual conference on Computer graphics and interactive techniques, p.25-34, August 1987
• 22: G. Senft, editor. Referring to Space: Studies in Austronesian and Papuan Languages. Clarendon Press, Oxford, 1997.
• 23: T. Winograd. Understanding Natural Language. PhD thesis, Massachusetts Institute of Technology, 1972.
• 24: A. Yamada. Studies on Spatial Description Understanding based on Geometric Constraints Satisfaction. PhD thesis, Kyoto University, Kyoto, 1993.
• 25: Jianmin Zhao , Norman I. Badler, Inverse kinematics positioning using nonlinear programming for highly articulated figures, ACM Transactions on Graphics (TOG), v.13 n.4, p.313-336, Oct. 1994 [doi>10.1145/195826.195827]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language parsing and understanding
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Design, Experimentation, Performance, Theory

Keywords:
HCI, applications, multimedia, scene generation, text-to-scene conversion