Representing coordination and non-coordination in an american sign language animation

ABSTRACT

While strings and syntax trees are used by the Natural Language Processing community to represent the structure of spoken languages, these encodings are difficult to adapt to a signed language like American Sign Language (ASL). In particular, the multichannel nature of an ASL performance makes it difficult to encode in a linear single-channel string. This paper will introduce the Partition/Constitute (P/C) Formalism, a new method of computationally representing a linguistic signal containing multiple channels. The formalism allows coordination and non-coordination relationships to be encoded between different portions of a signal. The P/C formalism will be compared to representations used in related research in gesture animation. The way in which P/C is used by this project to build an English-to-ASL machine translation system will also be discussed.

REFERENCES

• 1: William A. Baldwin , George O. Strawn, Multidimensional trees, Theoretical Computer Science, v.84 n.2, p.293-311, July 29, 1991 [doi>10.1016/0304-3975(91)90165-X]
• 2: Steven Bird , Mark Liberman, A formal framework for linguistic annotation, Speech Communication, v.33 n.1-2, p.23-60, Jan. 2001 [doi>10.1016/S0167-6393(00)00068-6]
• 3: Justine Cassell , Hannes Högni Vilhjálmsson , Timothy Bickmore, BEAT: the Behavior Expression Animation Toolkit, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.477-486, August 2001 [doi>10.1145/383259.383315]
• 4: Justine Cassell , Matthew Stone , Hao Yan, Coordination and context-dependence in the generation of embodied conversation, Proceedings of the first international conference on Natural language generation, June 12-16, 2000, Mitzpe Ramon, Israel [doi>10.3115/1118253.1118277]
• 5: Elliott, R., Glauert, J., Jennings, V., and Kennaway, J. 2004. An Overview of the SiGML Notation and SiGML Signing Software System. Workshop on the Representation and Processing of Signed Languages, 4th Int'l Conf. on Language Resources and Evaluation.
• 6: Holt, J. 1991. Demographic, Stanford Achievement Test - 8th Edition for Deaf and Hard of Hearing Students: Reading Comprehension Subgroup Results.
• 7: Huenerfauth, M. 2003. Survey and Critique of ASL Natural Language Generation and Machine Translation Systems. Technical Report MS-CIS-03-32, Computer and Information Science, University of Pennsylvania.
• 8: Huenerfauth, M. 2005. American Sign Language Generation: Multimodal NLG with Multiple Linguistic Channels. Association for Computational Linguistics, 43rd Annual Meeting, Student Research Workshop, Ann Arbor, MI, USA.
• 9: Huenerfauth, M. 2005. American Sign Language Spatial Representations for an Accessible User-Interface. 3rd International Conference on Universal Access in Human-Computer Interaction. Las Vegas, NV, USA.
• 10: Stefan Kopp , Paul Tepper , Justine Cassell, Towards integrated microplanning of language and iconic gesture for multimodal output, Proceedings of the 6th international conference on Multimodal interfaces, October 13-15, 2004, State College, PA, USA [doi>10.1145/1027933.1027952]
• 11: Liddell, S. 2003. Grammar, Gesture, and Meaning in American Sign Language. UK: Cambridge U. Press.
• 12: Marriot, K. & Meyer, B. 1996. Towards a Hierarchy of Visual Languages. AVI'96 Workshop on the Theory of Visual Languages and Computing, 2:311--331.
• 13: Martell, C. 2002. Form: An extensible, kinematically-based gesture annotation scheme. 3rd International Conference on Language Resources and Evaluation.
• 14: Mitchell, R. 2004. How many deaf people are there in the United States. Gallaudet Research Institute, Graduate School and Professional Programs, Gallaudet University. June 28, 2004. http://gri.gallaudet.edu/Demographics/deaf-US.php
• 15: Neidle, C., Kegl, J., MacLaughlin, D., Bahan, B., and Lee R.G. 2000. The Syntax of American Sign Language: Functional Categories and Hierarchical Structure. Cambridge, MA: The MIT Press.
• 16: Neidle, C., Sclaroff, S., and Athitsos, V. 2001. SignStream™: A Tool for Linguistic and Computer Vision Research on Visual-Gestural Language Data. In Behavior Research Methods, Instruments, and Computers 33:3, 311--320.
• 17: Maurizio Tucci , Giuliana Vitiello , Gennaro Costagliola, Parsing Nonlinear Languages, IEEE Transactions on Software Engineering, v.20 n.9, p.720-739, September 1994 [doi>10.1109/32.317427]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Language generation

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Machine translation

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.2 Social Issues
          Subjects: Assistive technologies for persons with disabilities

General Terms:
Design

Keywords:
accessibility technology for the deaf, american sign language, gesture generation, multimodal generation