Audiovisual 3d rendering as a tool for multimodal interfaces

ABSTRACT

In this talk, we will start with a short overview of 3D audiovisual rendering and its applicability to multimodal interfaces. In recent years, we have seen the generalization of 3D applications, ranging from computer games, which involve a high level of realism, to applications such as SecondLife, in which the visual and auditory quality of the 3D environment leaves much to be desired. In our introduction will attempt to examine the relationship between the audiovisual rendering of the environment and the interface. We will then review some of the audio-visual rendering algorithms we have developed in the last few years. We will discuss four main challenges we have addressed. The first is the development of realistic illumination and shadow algorithms which contribute greatly to the realism of 3D scenes, but could also be important for interfaces. The second involves the application of these illumination algorithms to augmented reality settings. The third concerns the development of perceptually-based techniques, and in particular using audio-visual cross-modal perception. The fourth challenge has been the development of approximate but "plausible", interactive solutions to more advanced rendering effects, both for graphics and audio. On the audio side, our review will include the introduction of clustering, masking and perceptual rendering for 3D spatialized audio and our recently developed solution for the treatment of contact sounds. On the graphics side, our discussion will include a quick overview of our illumination and shadow work, its application to augmented reality, our work on interactive rendering approximations and perceptually driven algorithms. For all these techniques we will discuss their relevance to multimodal interfaces, including our experience in a urban design case-study and attempt to relate them to recent interface research. We will close with a broad reflection on the potential for closer collaboration between 3D audiovisual rendering and multimodal interfaces.

REFERENCES

• 1: Nicolas Bonneel , George Drettakis , Nicolas Tsingos , Isabelle Viaud-Delmon , Doug James, Fast modal sounds with scalable frequency-domain synthesis, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 2: Carsten Dachsbacher , Marc Stamminger , George Drettakis , Frédo Durand, Implicit visibility and antiradiance for interactive global illumination, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 3: Oliver Deussen , Carsten Colditz , Marc Stamminger , George Drettakis, Interactive visualization of complex plant ecosystems, Proceedings of the conference on Visualization '02, October 27-November 01, 2002, Boston, Massachusetts
• 4: George Drettakis, Nicolas Bonneel, Carsten Dachsbacher, Sylvain Lefebvre, Michael Schwarz, and Isabelle Viaud-Delmon. An interactive perceptual rendering pipeline using contrast and spatial masking. In Rendering Techniques (Proceedings of the Eurographics Symposium on Rendering). Eurographics, June 2007.
• 5: George Drettakis , Maria Roussou , Alex Reche , Nicolas Tsingos, Design and Evaluation of a Real-World Virtual Environment for Architecture and Urban Planning, Presence: Teleoperators and Virtual Environments, v.16 n.3, p.318-332, June 2007 [doi>10.1162/pres.16.3.318]
• 6: Céline Loscos , George Drettakis , Luc Robert, Interactive Virtual Relighting of Real Scenes, IEEE Transactions on Visualization and Computer Graphics, v.6 n.4, p.289-305, October 2000 [doi>10.1109/2945.895874]
• 7: Céline Loscos, Marie-Claude Frasson, George Drettakis, Bruce Walter, Xavier Granier, and Pierre Poulin. Interactive virtual relighting and remodeling of real scenes. In D. Lischinski and G. W. Larson, editors, Rendering Techniques (Proceedings of the Eurographics Workshop on Rendering), volume 10, pages 235--246, New York, NY, Jun 1999. Springer-Verlag/Wien.
• 8: Nicolas Tsingos , Emmanuel Gallo , George Drettakis, Perceptual audio rendering of complex virtual environments, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

General Terms:
Algorithms

Keywords:
3d audio, computer graphics