VirtualStudio2Go: digital video composition for real environments

ABSTRACT

We synchronize film cameras and LED lighting with off-the-shelf video projectors. Radiometric compensation allows displaying keying patterns and other spatial codes on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables professional keying, environment matting, displaying moderator information, scene reconstruction, and camera tracking for non-studio film sets without being limited to the constraints of a virtual studio. This makes digital video composition more flexible, since static studio equipment, such as blue screens, teleprompters, or tracking devices, is not required. Authentic film locations can be supported with our portable system without causing a lot of installation effort.

REFERENCES

• 1: Oliver Bimber , Andreas Emmerling , Thomas Klemmer, Embedded Entertainment with Smart Projectors, Computer, v.38 n.1, p.48-55, January 2005 [doi>10.1109/MC.2005.17]
• 2: Bimber, O., Iwai, D., Wetzstein, G., and Grundhoefer, A. 2007. The Visual Computing of Projector-Camera Systems. In Proc. Eurographics (State-of-the-Art Report), 23--46.
• 3: Biswarup Choudhury , Deepali Singla , Sharat Chandran, Fast color-space decomposition based environment matting, Proceedings of the 2008 symposium on Interactive 3D graphics and games, February 15-17, 2008, Redwood City, California [doi>10.1145/1342250.1357013]
• 4: Yung-Yu Chuang , Douglas E. Zongker , Joel Hindorff , Brian Curless , David H. Salesin , Richard Szeliski, Environment matting extensions: towards higher accuracy and real-time capture, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.121-130, July 2000 [doi>10.1145/344779.344844]
• 5: Daniel Cotting , Martin Naef , Markus Gross , Henry Fuchs, Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display, Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.100-109, November 02-05, 2004 [doi>10.1109/ISMAR.2004.30]
• 6: Cotting, D., Ziegler, R., Gross, M. H., and Fuchs, H. 2005. Adaptive instant displays: Continuously calibrated projections using per-pixel light control. In Proceedings Eurographics 2005, 705--714. Eurographics 2005, Dublin, Ireland, August 29 -- September 2, 2005.
• 7: Fukaya, T., Fujikake, H., Yamanouchi, Y., Mitsumine, H., Yagi, N., Inoue, S., and Kikuchi, H. 2003. An effective interaction tool for performance in the virtual studio -- invisible light projection system. NHK Science and Technical Research Laboratories.
• 8: Graham, T., and Richard, R. 2005. Flash-based keying. European Patent Application EP1499117.
• 9: Grau, O., Pullen, T., and Thomas, G. A. 2004. A combined studio production system for 3-d capturing of live action and immersive actor feedback. IEEE Transactions on Circuits and Systems for Video Technology 14, 3, 370--380.
• 10: Grundhöfer, A., Seeger, M., Häntsch, F., and Bimber, O. 2007. Dynamic Adaptation of Projected Imperceptible Codes. Proc. of IEEE International Symposium on Mixed and Augmented Reality.
• 11: Nayar, S. K., Peri, H., Grossberg, M. D., and Belhumeur, P. N. 2003. A Projection System with Radiometric Compensation for Screen Imperfections. In Proc. of IEEE International Workshop on Projector-Camera Systems (ProCams).
• 12: Park, H., Lee, M.-H., Jin, B.-K. S. Y., and Park, J.-I. 2007. Content adaptive embedding of complementary patterns for non-intrusive direct-projected augmented reality. In HCI International 2007, vol. 14.
• 13: Pieter Peers , Philip Dutré, Wavelet environment matting, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 14: Ramesh Raskar , Greg Welch , Matt Cutts , Adam Lake , Lev Stesin , Henry Fuchs, The office of the future: a unified approach to image-based modeling and spatially immersive displays, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.179-188, July 1998 [doi>10.1145/280814.280861]
• 15: Pradeep Sen , Billy Chen , Gaurav Garg , Stephen R. Marschner , Mark Horowitz , Marc Levoy , Hendrik P. A. Lensch, Dual photography, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 16: Shirai, A., Takahashi, M., Kobayashi, K., Mitsumine, H., and Richir, S. 2005. Lumina studio: Supportive information display for virtual studio environments. In Proc. of IEEE VR Workshop on Emerging Display Technologies, 17--20.
• 17: C. Tomasi , R. Manduchi, Bilateral Filtering for Gray and Color Images, Proceedings of the Sixth International Conference on Computer Vision, p.839, January 04-07, 1998
• 18: Marcelo Bernardes Vieira , Luiz Velho , Asla Sa , Paulo Cezar Carvalho, A Camera-Projector System for Real-Time 3D Video, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, p.96, June 20-26, 2005 [doi>10.1109/CVPR.2005.385]
• 19: Waschbüsch, M., Würmlin, S., Cotting, D., Sadlo, F., and Gross, M. H. 2005. Scalable 3d video of dynamic scenes. The Visual Computer 21, 8-10, 629--638.
• 20: Gordon Wetzstein , Oliver Bimber, Radiometric Compensation through Inverse Light Transport, Proceedings of the 15th Pacific Conference on Computer Graphics and Applications, p.391-399, October 29-November 02, 2007 [doi>10.1109/PG.2007.52]
• 21: Wojdala, A., Gruszsewski, M., and Olech, R. 2002. Real-time shadow casting in virtual studio. Machine Graphics and Vision (MGV) 9, 1/2, 315--329.
• 22: Yoshida, T., Horii, C., and Sato, K. 2003. A Virtual Color Reconstruction System for Real Heritage with Light Projection. In Proc. of International Conference on Virtual Systems and Multimedia (VSMM), 161--168.
• 23: Jiayuan Zhu , Yee-Hong Yang, Frequency-Based Environment Matting, Proceedings of the Computer Graphics and Applications, 12th Pacific Conference, p.402-410, October 06-08, 2004
• 24: Zollmann, S., and Bimber, O. 2007. Imperceptible calibration for radiometric compensation. In Proceedings Eurographics 2007, Short Paper.
• 25: Douglas E. Zongker , Dawn M. Werner , Brian Curless , David H. Salesin, Environment matting and compositing, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.205-214, July 1999 [doi>10.1145/311535.311558]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
          Subjects: Display algorithms

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.0 General
          Subjects: Image displays
      I.4.1 Digitization and Image Capture

Keywords:
digital video composition, projector-camera systems, radiometric compensation