An image-warping VR-architecture: design, implementation and applications

ABSTRACT

We describe an architecture that provides a programmable display layer in order to allow the execution of custom programs on consecutive display frames. This replaces the default display behavior of repeating application frames until an update is available. The architecture is implemented using a multi-GPU system. We will show three applications of the architecture typical to VR. First, smooth motion is provided by generating intermediate display frames by per-pixel depth-image warping using 3D motion fields. Smooth motion can be beneficial for walk-throughs of large scenes. Second, we implement fine-grained latency reduction at the display frame level using a synchronized prediction of simulation objects and the viewpoint. This improves the average quality and consistency of latency reduction. Third, a crosstalk reduction algorithm for consecutive display frames is implemented, which improves the quality of stereoscopic images.

REFERENCES

• 1: Bex, P. J., Edgar, G. K., and Smith, A. T. 1995. Multiple images appear when motion energy detection fails. Journal of Experimental Psychology: Human Perception and Performance 21, 231--238.
• 2: Bles, W., and Wertheim, A. 2000. Appropriate use of virtual environments to minimise motion sickness. RTO MP58, 7.1--7.9.
• 3: Cheng, L., Bhushan, A., Pajarola, R., and Zarki, M. E. 2004. Real-time 3d graphics streaming using MPEG-4. In Proc. IEEE/ACM Workshop on Broadband Wireless Services and Applications.
• 4: Farrell, J. E., Pavel, M., and Sperling, G. 1990. The visible persistence of stimuli in stroboscopic motion. Vision Research 30, 6, 921--936.
• 5: Ryugo Kijima , Takeo Ojika, Reflex HMD to Compensate Lag and Correction of Derivative Deformation, Proceedings of the IEEE Virtual Reality Conference 2002, p.172, March 24-28, 2002
• 6: William R. Mark , Leonard McMillan , Gary Bishop, Post-rendering 3D warping, Proceedings of the 1997 symposium on Interactive 3D graphics, p.7-ff., April 27-30, 1997, Providence, Rhode Island, United States [doi>10.1145/253284.253292]
• 7: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 8: Mark R. Mine, Characterization of End-to-End Delays in Head-Mounted Display Systems, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1993
• 9: Marc Olano , Jon Cohen , Mark Mine , Gary Bishop, Combatting rendering latency, Proceedings of the 1995 symposium on Interactive 3D graphics, p.19-ff., April 09-12, 1995, Monterey, California, United States [doi>10.1145/199404.199407]
• 10: Voicu Popescu , Anselmo Lastra , Daniel Aliaga , Manuel de Oliveira Neto, Efficient warping for architectural walkthroughs using layered depth images, Proceedings of the conference on Visualization '98, p.211-215, October 18-23, 1998, Research Triangle Park, North Carolina, United States
• 11: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 12: Chris Shaw , Mark Green , Jiandong Liang , Yunqi Sun, Decoupled simulation in virtual reality with the MR toolkit, ACM Transactions on Information Systems (TOIS), v.11 n.3, p.287-317, July 1993 [doi>10.1145/159161.173948]
• 13: F. A. Smit , R. van Liere , B. Fröhlich, The design and implementation of a VR-architecture for smooth motion, Proceedings of the 2007 ACM symposium on Virtual reality software and technology, November 05-07, 2007, Newport Beach, California [doi>10.1145/1315184.1315212]
• 14: Smit, F., van Liere, R., and Froehlich, B. 2007. Non-uniform crosstalk reduction for dynamic scenes. In Proc. IEEE Virtual Reality, March 2007, 139--146.
• 15: Smit, F., van Liere, R., and Froehlich, B. 2007. Three extensions to subtractive crosstalk reduction. In Proc. Eurographics EGVE, 85--92.
• 16: Springer, J. P., Beck, S., Weiszig, F., Reiners, D., and Froehlich, B. 2007. Multi-frame rate rendering and display. In Proc. IEEE VR, 195--202.
• 17: J. Stewart , E. P. Bennett , L. McMillan, PixelView: a view-independent graphics rendering architecture, Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardware, August 29-30, 2004, Grenoble, France [doi>10.1145/1058129.1058141]
• 18: Dan S. Wallach , Sharma Kunapalli , Michael F. Cohen, Accelerated MPEG compression of dynamic polygonal scenes, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.193-196, July 1994 [doi>10.1145/192161.192198]
• 19: Woods, A. J., and Tan, S. S. 2002. Characteristic sources of ghosting in time-sequential stereoscopic video displays. In Proc. SPIE Vol. 4660, 66--77.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation
          Subjects: Display algorithms

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Virtual reality
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Motion

Keywords:
VR, crosstalk, image-warping, judder, latency, motion, stereoscopic display