An automatic lip-synchronization algorithm for synthetic faces

ABSTRACT

This paper addresses the problem of automatically synchronizing computer-generated faces with synthetic speech. The complete process provides a novel form of face-to-face communication and the ability to create a new range of talking personable synthetic characters. Based on plain ASCII text input, a synthetic speech segment is generated and synchronized in real-time to a graphical display of an articulating mouth and face. The key component of the algorithm is the run-time facility that adaptively synchronizes the graphical display of the face to the audio.

REFERENCES

• 1: P. Bergeron and P. Lachapelle. Techniques for animating characters. In Advanced Computer Animation, volume 2 of SIGGIiAPH '85 Tutorials, pages 61-79. ACM, 1985.
• 2: H. Choi, S.C. Harashima and T. Takebe. Analysis and synthesis of facial expressions in know}edge-based coding of facial image sequences. In International Conference on Acoustics Speech and Signal Processing, pages 2737-2740, 1991.
• 3: J. Destandes. Histoire Comparee du Cinema, volume 1. Castarman, Paris, 1966.
• 4: P. Ekman and W.V. Friesen. Manual for the Facial Action Coding System. Consulting Psychologists Press, Palo Alto CA, 1977.
• 5: Ingrid Carlbom , William M. Hsu , Gudrun Klinker , Richard Szeliski , Keith Waters , Michael Doyle , Jim Gettys , Kristen M. Harris , Thomas M. Levergood , Ricky Palmer , Larry Palmer , Marc Picart , Demetri Terzopoulos , David Tonnesen , Michael Vannier , Greg Wallace, Modeling and analysis of empirical data in collaborative environments, Communications of the ACM, v.35 n.6, p.74-84, June 1992 [doi>10.1145/129888.129893]
• 6: G. Farin, Curves and surfaces for computer aided geometric design, Academic Press Professional, Inc., San Diego, CA, 1990
• 7: V. Fromkin. Lip positions in American English vowels. Language and Speech, 7(3):215-225, 1964.
• 8: H. Gouraud. Continuous shading of curved surfaces. IEEE Trans on Computers, 20(6), 1971.
• 9: Paul S Heckbert, Survey of texture mapping, IEEE Computer Graphics and Applications, v.6 n.11, p.56-67, November 1986 [doi>10.1109/MCG.1986.276672]
• 10: D.R. Hill, A. Pearce, and B. WyviU. Animating speech: An automated aproach using speech synthesis by rules. The Visual Computer, 3:277-289, 1988.
• 11: Dennis H. Klatt. Software for a cascade/parallel formant synthesizer. J. Acoust. Soc. Am., 67(3):971-995, 1980.
• 12: Dennis H. Ktatt. Review of text-to-speech conversion for English. J. Acoust. Soc. Am., 82(3):737-793, 1987.
• 13: Doris H. U. Kochanek , Richard H. Bartels, Interpolating splines with local tension, continuity, and bias control, ACM SIGGRAPH Computer Graphics, v.18 n.3, p.33-41, July 1984
• 14: Thomas M. Levergood, Andrew C. Payne, James Gettys, G. Win_6eld Treese, and Lawrence C. Stewart. AudioFile' A network-transparent system for distributed audio applications. In Proceedings o/the USENIX Summet Conference, June 1993.
• 15: J. P. Lewis , F. I. Parke, Automated lip-synch and speech synthesis for character animation, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.143-147, April 05-09, 1987, Toronto, Ontario, Canada
• 16: N. Magnenat-Thalmann, N.E. Primeau, and D. Thaimann. Abstract muscle actions procedures for human face animation. Visual Computer, 3(5):290-297, 1988.
• 17: M. McGrath. An gzamination o! Cues /or Visual and Audio-Visual Speech Perception using Natural and Computer Generated Faces. PhD thesis, University of Nottingham, England, November 1985.
• 18: H. McGurk and J. MacDonald. Heating lips and seeing voices. Nature, 264:126-130, 1986.
• 19: Masaaki Oka , Kyoya Tsutsui , Akio Ohba , Yoshitaka Kurauchi , Takashi Tago, Real-time manipulation of texture-mapped surfaces, ACM SIGGRAPH Computer Graphics, v.21 n.4, p.181-188, July 1987
• 20: John K. Ousterhout. Tel: An embeddable command language. In Proceedings of the USENIX Winter Conference, January 1990.
• 21: John K. Ousterhout. An Xll toolkit based on the Tcl language. In Proceedings of the USENIX Winter Conference, January 1991.
• 22: F.I. Parke. Computer generated animation of faces. Master's thesis, University of Utah, Salt Lake City, June 1972. UTEC-CSc-72-120.
• 23: F.I. Parke. Parameterized models for facial animation. IEEE Computer Graphics and Applications, 2(9):61-68, 1982.
• 24: F.I. Parke. State of the art in facial animation. A CM SIGGRAPH Course Notes, 26, 1990.
• 25: Stephen M. Platt , Norman I. Badler, Animating facial expressions, ACM SIGGRAPH Computer Graphics, v.15 n.3, p.245-252, August 1981
• 26: William H. Press , Brian P. Flannery , Saul A. Teukolsky , William T. Vetterling, Numerical recipes: the art of scientific computing, Cambridge University Press, New York, NY, 1986
• 27: F. Thomas and O. Johnson. Disney Animation: the Illusion of Life. Abbeville Press, New York, 1981.
• 28: Kristinn R. Thorisson, Dialogue control in social interface agents, INTERACT '93 and CHI '93 conference companion on Human factors in computing systems, p.139-140, April 24-29, 1993, Amsterdam, The Netherlands [doi>10.1145/259964.260151]
• 29: C.T. Waite. The facial action control editor, face: A parametric facial expression editor for computer generated animation. Master's thesis, MIT, Feb 1989.
• 30: Janet H. Walker , Lee Sproull , R. Subramani, Using a human face in an interface, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.85-91, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191708]
• 31: E.F. Walther. Lipreading. Nelson-Hall Inc, Chicago, 1982.
• 32: Keith Waters, A muscle model for animation three-dimensional facial expression, ACM SIGGRAPH Computer Graphics, v.21 n.4, p.17-24, July 1987
• 33: K. Waters and T. Levergood. DECface: An automatic lip-synchronization algorithm for synthetic faces. Technical Report 93/4, Digital Equipment Corp, Cambridge Research Laboratory, Sept 1993.
• 34: P. Weil. About face. Master's thesis, MIT, Aug 1982.
• 35: Lance Williams, Performance-driven facial animation, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.235-242, Aug. 1990

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Animations

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing
          Subjects: Speech recognition and synthesis
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

General Terms:
Algorithms, Human Factors, Languages