Real-time remote rendering of 3D video for mobile devices

ABSTRACT

At the convergence of computer vision, graphics, and multimedia, the emerging 3D video technology promises immersive experiences in a truly seamless environment. However, the requirements of huge network bandwidth and computing resources make it still a big challenge to render 3D video on mobile devices at real-time. In this paper, we present how remote rendering framework can be used to solve the problem. The differences between dynamic 3D video and static graphic models are analyzed. A general proxy-based framework is presented to render 3D video streams on the proxy and transmit the rendered scene to mobile devices over a wireless network. An image-based approach is proposed to enhance 3D interactivity and reduce the interaction delay. Experiments prove that the remote rendering framework can be effectively used for quality 3D video rendering on mobile devices in real time.

REFERENCES

• 1: Bumblebee 2 camera. http://www.ptgrey.com/products/bumblebee2/index.asp.
• 2: Chromium. http://chromium.sourceforge.net/.
• 3: Equalizer: Parallel rendering. http://www.equalizergraphics.com/.
• 4: Glintercept. http://glintercept.nutty.org/.
• 5: Gltrace. http://www.hawksoft.com/gltrace/.
• 6: P. Bao and D. Gourlay. Remote walkthrough over mobile networks using 3-D image warping and streaming. Vision, Image and Signal Processing, IEE Proceedings, 151(4):329--336, Aug. 2004.
• 7: A. Boukerche and R. Pazzi. Remote rendering and streaming of progressive panoramas for mobile devices. In Multimedia '06, pages 691--694. ACM, 2006.
• 8: C.-F. Chang and S.-H. Ger. Enhancing 3D graphics on mobile devices by image-based rendering. In PCM '02, pages 1105--1111, London, UK, 2002.
• 9: H. Kim and et al. An immersive free-viewpoint video system using multiple outer/inner cameras. In 3DPVT, pages 782--789, 2006.
• 10: S.-U. Kum and K. Mayer-Patel. Real-time multidepth stream compression. TOMCCAP, 1(2):128--150, 2005.
• 11: F. Lamberti and A. Sanna. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Trans. Vis. Comput. Graph., 13(2):247--260, 2007.
• 12: J.-M. Lien and et al. Skeleton-based data compression for multi-camera tele-immersion system. In ISVC'07, pages 714--723, November 2007.
• 13: W. R. Mark and et al. Post-rendering 3D warping. In SI3D '97: Proceedings of the 1997 symposium on Interactive 3D graphics, 1997.
• 14: W. Matusik and H. Pfister. 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. In SIGGRAPH '04, pages 814--824, 2004.
• 15: L. McMillan. An image-based approach to three-dimensional computer graphics. In Ph.D. Dissertation. University of North Carolina at Chapel Hill, Department of Computer Science, 1997.
• 16: L. McMillan and G. Bishop. Plenoptic modeling: an image-based rendering system. In SIGGRAPH '95, pages 39--46, 1995.
• 17: S. K. Penta and P. J. Narayanan. Compression of multiple depth maps for IBR. The Visual Computer, 21(8-10):611--618, 2005.
• 18: A. Redert and et al. ATTEST: Advanced three-dimensional television system technologies. In 3DPVT '02, pages 313--319, June 2002.
• 19: J. Shade and et al. Layered depth images. In SIGGRAPH '98, pages 231--242, 1998.
• 20: S. Shi and et al. View-dependent real-time 3D video compression for mobile devices. In Multimedia '08, pages 781--784. ACM, 2008.
• 21: H.-Y. Shum, S. B. Kang, and S.-C. Chan. Survey of image-based representations and compression techniques. IEEE Trans. Circuits Syst. Video Techn., 13(11):1020--1037, 2003.
• 22: A. Smolic and et al. 3D video and free viewpoint video - technologies, applications and mpeg standards. In ICME, pages 2161--2164. IEEE, 2006.
• 23: S. Wurmlin and et al. 3D video recorder: a system for recording and playing free-viewpoint video. Comput. Graph. Forum, 22(2):181--194, 2003.
• 24: Z. Yang and et al. TEEVE: The next generation architecture for tele-immersive environment. In ISM '05, pages 112--119, December 2005.
• 25: Z. Yang and et al. Real-time 3D video compression for tele-immersive environments. In MMCN '06, 2006.
• 26: Z. Yang and et al. Viewcast: view dissemination and management for multi-party 3D tele-immersive environments. In Multimedia '07. ACM, 2007.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Design, Measurement

Keywords:
3D video, mobile devices, remote rendering