Free viewpoint video synthesis and presentation of sporting events for mixed reality entertainment

ABSTRACT

This paper presents a new framework for arbitrary view synthesis and presentation of sporting events for mixed reality entertainment. In accordance with the viewpoint position of an observer, virtual view image of sporting scene is generated by view interpolation among multiple videos captured at real stadium. Then the synthesized sporting scene is overlaid onto a desktop stadium model in the real world via HMD. This makes it possible to watch the event in front of the observer. Projective geometry between cameras is used for virtual view generation of the dynamic scene and geometric registration between the real world and the virtual view image of sporting scene. The proposed method does not need to calibrate multiple video cameras for capturing the event and the HMD camera. Therefore it can be applied even to dynamic events in a large space and enables observation with immersive impression. The proposed approach leads to make a new type of mixed reality entertainment for sporting events.

REFERENCES

• 1: Shai Avidan , Amnon Shashua, Novel View Synthesis by Cascading Trilinear Tensors, IEEE Transactions on Visualization and Computer Graphics, v.4 n.4, p.293-306, October 1998 [doi>10.1109/2945.765324]
• 2: R. T. Azuma. A survey of augmented reality. Presence, 6(4):355--385, 1997.
• 3: M. Bajura, H. Fuchs, and R. Ohbuchi. Merging virtual objects with the real world: Seeing utlrasound. Commun of the ACM, 36(7):52--62, 1993.
• 4: Michael Bajura , Ulrich Neumann, Dynamic Registration Correction in Video-Based Augmented Reality Systems, IEEE Computer Graphics and Applications, v.15 n.5, p.52-60, September 1995 [doi>10.1109/38.403828]
• 5: Thaddeus Beier , Shawn Neely, Feature-based image metamorphosis, Proceedings of the 19th annual conference on Computer graphics and interactive techniques, p.35-42, July 1992
• 6: J Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.8 n.6, p.679-698, Nov. 1986 [doi>10.1109/TPAMI.1986.4767851]
• 7: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• 8: CyberPlay. http://www.orad.co.il.
• 9: W. Du, H. Li, and A. Gagalowicz. Video based 3d soccer scene reconstruction. Proc. of Mirage 2003, pages 70--75, Mar 2003.
• 10: Markerless Augmented Reality with a Real-Time Affine Region Tracker, Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR'01), p.87, October 29-30, 2001
• 11: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 12: N. Inamoto and H. Saito. Fly through view video generation of soccer scene. International Workshop on Entertainment Computing (IWEC2002) Workshop Note, pages 94--101, May 2002.
• 13: N. Inamoto and H. Saito. Intermediate view generation of soccer scene from multiple videos. Proc. of International Conference on Pattern Recognition (ICPR2002), 2:713--716, August 2002.
• 14: N. Inamoto and H. Saito. Fly-through viewpoint video system for multi-view soccer movie using viewpoint interpolation. Proc. of Visual Communications and Image Processing (VCIP2003), SPIE, 5150(122), July 2003.
• 15: Naho Inamoto , Hideo Saito, Immersive Observation of Virtualized Soccer Match at Real Stadium Model, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.188, October 07-10, 2003
• 16: T. Kanade , P. J. Narayanan , P. W. Rander, Virtualized reality: concepts and early results, Proceedings of the IEEE Workshop on Representation of Visual Scenes, p.69, June 21-21, 1995
• 17: I. Kitahara, Y. Ohta, H. Saito, S. Akimichi, T. Ono, and T. Kanade. Recording multiple videos in a large-scale space for large-scale virtualized reality. Proc. of International Display Workshops (AD/IDW'01)s, pages 1377--1380, 2001.
• 18: Takayoshi Koyama , Itaru Kitahara , Yuichi Ohta, Live Mixed-Reality 3D Video in Soccer Stadium, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.178, October 07-10, 2003
• 19: Kiriakos N. Kutulakos , James Vallino, Affine Object Representations for Calibration-Free Augmented Reality, Proceedings of the 1996 Virtual Reality Annual International Symposium (VRAIS 96), p.25, March 30-April 03, 1996
• 20: Ulrich Neumann , Suya You , Jinhui Hu , Bolan Jiang , JongWeon Lee, Augmented Virtual Environments (AVE): Dynamic Fusion of Imagery and 3D Models, Proceedings of the IEEE Virtual Reality 2003, p.61, March 22-26, 2003
• 21: Simon Prince , Adrian David Cheok , Farzam Farbiz , Todd Williamson , Nik Johnson , Mark Billinghurst , Hirokazu Kato, 3D Live: Real Time Captured Content for Mixed Reality, Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR'02), p.7, September 30-October 01, 2002
• 22: H. Saito, S. Baba, and T. Kanade. Appearance-based virtual view generation from multicamera videos captured in the 3-d room. IEEE Trans. on Multimedia, 5(3):303--316, September 2003.
• 23: Steven M. Seitz , Charles R. Dyer, View morphing, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.21-30, August 1996 [doi>10.1145/237170.237196]
• 24: Steven M. Seitz , Charles R. Dyer, Photorealistic Scene Reconstruction by Voxel Coloring, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.1067, June 17-19, 1997
• 25: Yongduek Seo , Ki Sang Hong, Calibration-Free Augmented Reality in Perspective, IEEE Transactions on Visualization and Computer Graphics, v.6 n.4, p.346-359, October 2000 [doi>10.1109/2945.895879]
• 26: G. Simon, A. W. Fitzgibbob, and A. Zisserman. Markerless tracking using planar structures in the scene. Proc. of the International Symposium on Augmented Reality, pages 120--128, Oct 2000.
• 27: R. Y. Tsai. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, RA-3(4):323--344, August 1987.
• 28: M. D. Wheeler, Y. Sato, and K. Ikeuchi. Consensus surfaces for modeling 3d objects from multiple range images. DARPA Image Understanding Workshop, 1997.
• 29: S. Yaguchi and H. Saito. Arbitrary viewpoint video synthesis from multiple uncalibrated cameras. IEEE Trans. on Systems, Man and Cybernetics, PartB, 34(1):430--439, February 2004.