FoleyAutomatic: physically-based sound effects for interactive simulation and animation

ABSTRACT

We describe algorithms for real-time synthesis of realistic sound effects for interactive simulations (e.g., games) and animation. These sound effects are produced automatically, from 3D models using dynamic simulation and user interaction. We develop algorithms that are efficient, physically-based, and can be controlled by users in natural ways. We develop effective techniques for producing high quality continuous contact sounds from dynamic simulations running at video rates which are slow relative to audio synthesis. We accomplish this using modal models driven by contact forces modeled at audio rates, which are much higher than the graphics frame rate. The contact forces can be computed from simulations or can be custom designed. We demonstrate the effectiveness with complex realistic simulations.

REFERENCES

• 1: T. Ananthapadmanaban and V. Radhakrishnan. An investigation on the role of surface irregularities in the noise spectrum of rolling and sliding contacts. Wear, 83:399-409, 1982.
• 2: Durand R. Begault, 3-D sound for virtual reality and multimedia, Academic Press Professional, Inc., San Diego, CA, 1994
• 3: W. Buxton. Introduction to this special issue on nonspeech audio. Human Computer Interaction, 4(1):1-9, 1989.
• 4: W. Buxton. Using our ears: an introduction to the use of nonspeech audio cues. In E. Farrell, editor, Extracting meaning from complex data: processing, display, interaction. Proceedings of the SPIE, volume Vol 1259, pages 124-127, 1990.
• 5: A. Chaigne and V. Doutaut. Numerical simulations of xylophones. I. Time domain modeling of the vibrating bars. J. Acoust. Soc. Am., 101(1):539-557, 1997.
• 6: A. Chaudhary, A. Freed, S. Khoury, and D. Wessel. A 3- D Graphical User Interface for Resonance Modeling. In Proceedings of the International Computer Music Conference, Ann Arbor, 1998.
• 7: P. R. Cook. Integration of physical modeling for synthesis and animation. In Proceedings of the International Computer Music Conference, pages 525-528, Banff, 1995.
• 8: P. R. Cook. Physically informed sonic modeling (PhISM): Percussive synthesis. In Proceedings of the International Computer Music Conference, pages 228- 231, Hong Kong, 1996.
• 9: P. R. Cook and G. Scavone. The synthesis toolkit (STK), version 2.1. In Proc. of the International Computer Music Conference, Beijing, 1999.
• 10: Derek DiFilippo , Dinesh K. Pai, The AHI: an audio and haptic interface for contact interactions, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.149-158, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354437]
• 11: A. Freed and X. Depalle. Synthesis of Hundreds of Sinusoidal Partials on a Desktop Computer without Custom Hardware. In Proceedings of The International Conference on Signal Processing Applications and Technology, Santa Clara, 1993.
• 12: D. J. Fried. Auditory correlates of perceived mallet hardness for a set of recorded percussive sound events. J. Acoust. Soc. Am., 87(1):311-321, 1990.
• 13: Thomas Funkhouser , Patrick Min , Ingrid Carlbom, Real-time acoustic modeling for distributed virtual environments, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.365-374, July 1999 [doi>10.1145/311535.311590]
• 14: William W. Gaver , Donald A. Norman, Everyday listening and auditory icons, 1988
• 15: William W. Gaver, Synthesizing auditory icons, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.228-235, May 1993, Amsterdam, The Netherlands
• 16: W. W. Gaver. What in the world do we hear?: An ecological approach to auditory event perception. Ecological Psychology, 5(1):1-29, 1993.
• 17: J. K. Hahn, H. Fouad, L. Gritz, and J. W. Lee. Integrating sounds and motions in virtual environments. In Sound for Animation and Virtual Reality, SIGGRAPH 95 Course 10 Notes, 1995.
• 18: D. J. Hermes. Synthesis of the sounds produced by rolling balls. In Internal IPO report no. 1226, IPO, Center for User-System Interaction, Eindhoven, The Netherlands, 2000.
• 19: D. P. Hess and A. Soom. Normal vibrations and friction under harmonic loads. II. Rough planar contacts. Transactions of the ASME. Journal of Tribology, 113:87-92, 1991.
• 20: M. M. J. Houben, D. J. Hermes, and A. Kohlrausch. Auditory perception of the size and velocity of rolling balls. In IPO Annual Progress Report, volume 34, 1999.
• 21: D. Jaffe. Ten criteria for evaluating synthesis and processing techniques. Computer Music Journal, 19(1):76- 87, 1995.
• 22: K. L. Johnson. Contact Mechanics. Cambridge University Press, Cambridge, 1985.
• 23: R. L. Klatzky, D. K. Pai, and E. P. Krotkov. Hearing material: Perception of material from contact sounds. PRESENCE: Teleoperators and Virtual Environments, 9(4):399-410, 2000.
• 24: M. V. Mathews. The Technology of Computer Music. MIT Press, Cambridge, 1969.
• 25: J. D. Morrison and J.-M. Adrien. Mosaic: A framework for modal synthesis. Computer Music Journal, 17(1), 1993.
• 26: P. R. Nayak. Contact vibrations. Journal of Sound and Vibration, 22:297-322, 1972.
• 27: D. D. Nelson, D. E. Johnson, and E. Cohen. Haptic rendering of surface-to-surface sculpted model interaction. In Proceedings of the ASME Dynamic Systems and Control Division, volume DSC-Vol. 67, pages 101-108, 1999.
• 28: James F. O'Brien , Perry R. Cook , Georg Essl, Synthesizing sounds from physically based motion, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.529-536, August 2001 [doi>10.1145/383259.383321]
• 29: D. K. Pai, U. M. Ascher, and P. G. Kry. Forward Dynamics Algorithms for Multibody Chains and Contact. In Proceedings of the 2000 IEEE International Conference on Robotics and Automation, pages 857-863, 2000.
• 30: Dinesh K. Pai , Kees van den Doel , Doug L. James , Jochen Lang , John E. Lloyd , Joshua L. Richmond , Som H. Yau, Scanning physical interaction behavior of 3D objects, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.87-96, August 2001 [doi>10.1145/383259.383268]
• 31: X. Rodet. Musical Sound Signal Analysis/Synthesis: Sinusoidal + Residual and Elementary Waveform Models. In IEEE Time-Frequency and Time-Scale Workshop 97, Coventry, Grande Bretagne, 1997.
• 32: L. Savioja, J. Huopaniemi, T. Lokki, and R. Vnnen. Virtual environment simulation - Advances in the DIVA project. In Proc. Int. Conf. Auditory Display, Palo Alto, USA, 1997.
• 33: R. S. Sayles and T. R. Thomas. Surface topography as a non-stationary random process. Nature, 271:431-434, 1978.
• 34: X. Serra. A System for Sound Analysis / Transformation / Synthesis Based on a Deterministic Plus Stochastic Decomposition. PhD thesis, Dept. of Music, Stanford University, 1989.
• 35: J. O. Smith. Physical modeling using digital waveguides. Computer Music Journal, 16(4):75-87, 1992.
• 36: J. Stam. Evaluation of loop subdivision surfaces. In SIGGRAPH 98, 1998. Included on course notes CD-ROM.
• 37: Ken Steiglitz, A digital signal processing primer, with applications to digital audio and computer music, Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, 1997
• 38: D. Stoianovici and Y. Hurmuzlu. A critical study of the applicability of rigid-body collision theory. ASME Journal of Applied Mechanics, 63:307-316, 1996.
• 39: Tapio Takala , James Hahn, Sound rendering, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.211-220, July 1992
• 40: T. R. Thomas. Rough Surfaces. Imperial College Press, London, second edition, 1999.
• 41: D. J. Thompson, D. F. Fodiman, and H. Mahe. Experimental validation of the twins prediction program for rolling noise, parts I and II. Journal of Sound and Vibration, 193:123-147, 1996.
• 42: Nicolas Tsingos , Thomas Funkhouser , Addy Ngan , Ingrid Carlbom, Modeling acoustics in virtual environments using the uniform theory of diffraction, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.545-552, August 2001 [doi>10.1145/383259.383323]
• 43: K. van den Doel. Sound Synthesis for Virtual Reality and Computer Games. PhD thesis, University of British Columbia, 1998.
• 44: K. van den Doel and D. K. Pai. Synthesis of shape dependent sounds with physical modeling. In Proceedings of the International Conference on Auditory Displays 1996, Palo Alto, 1996.
• 45: K. van den Doel and D. K. Pai. The sounds of physical shapes. Presence, 7(4):382-395, 1998.
• 46: John Wawrzynek, VLSI models for sound synthesis, Current directions in computer music research, MIT Press, Cambridge, MA, 1989

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Animation

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Auditory (non-speech) feedback
      H.5.5 Sound and Music Computing
          Subjects: Methodologies and techniques; Modeling

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.5 Computational Geometry and Object Modeling
          Subjects: Physically based modeling
  I.6 SIMULATION AND MODELING
      I.6.5 Model Development
          Subjects: Modeling methodologies
      I.6.8 Types of Simulation
          Subjects: Animation

General Terms:
Algorithms, Design, Experimentation

Keywords:
animation systems, computer games, head mounted displays, multimedia, physically based animation, physically based modeling, sound visualization, virtual reality