TensorTextures: multilinear image-based rendering

ABSTRACT

This paper introduces a tensor framework for image-based rendering. In particular, we develop an algorithm called TensorTextures that learns a parsimonious model of the bidirectional texture function (BTF) from observational data. Given an ensemble of images of a textured surface, our nonlinear, generative model explicitly represents the multifactor interaction implicit in the detailed appearance of the surface under varying photometric angles, including local (per-texel) reflectance, complex mesostructural self-occlusion, interreflection and self-shadowing, and other BTF-relevant phenomena. Mathematically, TensorTextures is based on multilinear algebra, the algebra of higher-order tensors, hence its name. It is computed through a decomposition known as the N-mode SVD, an extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the application of TensorTextures to the image-based rendering of natural and synthetic textured surfaces under continuously varying viewpoint and illumination conditions.

REFERENCES

• 1: Kristin J. Dana , Bram van Ginneken , Shree K. Nayar , Jan J. Koenderink, Reflectance and texture of real-world surfaces, ACM Transactions on Graphics (TOG), v.18 n.1, p.1-34, Jan. 1999 [doi>10.1145/300776.300778]
• 2: R. Furukawa , H. Kawasaki , K. Ikeuchi , M. Sakauchi, Appearance based object modeling using texture database: acquisition, compression and rendering, Proceedings of the 13th Eurographics workshop on Rendering, June 26-28, 2002, Pisa, Italy
• 3: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 4: Jefferson Y. Han , Ken Perlin, Measuring bidirectional texture reflectance with a kaleidoscope, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 5: KOENDERINK, J. J., AND VAN DOORN, A. J. 1996. Illuminance texture due to surface mesostructure. Journal of the Optical Society of America 13, 3, 452--463.
• 6: Tamara G. Kolda, Orthogonal Tensor Decompositions, SIAM Journal on Matrix Analysis and Applications, v.23 n.1, p.243-255, 2001 [doi>10.1137/S0895479800368354]
• 7: KOUDELKA, M. L., MAGDA, S., BELHUMEUR, P. N., AND KRIEGMAN, D. J. 2003. Acquisition, compression, and synthesis of bidirectional texture functions. In Proc. 3rd Int. Workshop on Texture Analysis and Synthesis (Oct), Nice, France, 59--64.
• 8: KROONENBERG, P., ANDDE LEEUW, J. 1980. Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika 45, 69--97.
• 9: Lieven De Lathauwer , Bart De Moor , Joos Vandewalle, On the Best Rank-1 and Rank-(R₁,R₂,. . .,R_N) Approximation of Higher-Order Tensors, SIAM Journal on Matrix Analysis and Applications, v.21 n.4, p.1324-1342, March – May 2000 [doi>10.1137/S0895479898346995]
• 10: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 11: Xinguo Liu , Yizhou Yu , Heung-Yeung Shum, Synthesizing bidirectional texture functions for real-world surfaces, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.97-106, August 2001 [doi>10.1145/383259.383269]
• 12: Tom Malzbender , Dan Gelb , Hans Wolters, Polynomial texture maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.519-528, August 2001 [doi>10.1145/383259.383320]
• 13: Wojciech Matusik , Hanspeter Pfister , Matt Brand , Leonard McMillan, A data-driven reflectance model, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 14: MESETH, J., MULLER, G., SATTLER, M., AND KLEIN, R. 2003. BTF rendering for virtual environments. In Proc. Virtual Concepts 2003 (Nov), 356--363.
• 15: Mirko Sattler , Ralf Sarlette , Reinhard Klein, Efficient and realistic visualization of cloth, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 16: SUYKENS, F., BERGE, K. VON, LAGAE, A., AND KLEIN, R. 2003. Interactive rendering with bidirectional texture functions. Comp. Graphics Forum (Proc. Eurographics) 22, 3, 463--472.
• 17: TEO, P. AND HEEGER, P. 1994. Perceptual image distortion. In Proc. IEEE Conf. Image Processing (Nov), 982--986.
• 18: Xin Tong , Jingdan Zhang , Ligang Liu , Xi Wang , Baining Guo , Heung-Yeung Shum, Synthesis of bidirectional texture functions on arbitrary surfaces, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002
• 19: TUCKER, L. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 279--311.
• 20: M. A. O. Vasilescu , Demetri Terzopoulos, Multilinear Analysis of Image Ensembles: TensorFaces, Proceedings of the 7th European Conference on Computer Vision-Part I, p.447-460, May 28-31, 2002
• 21: Lifeng Wang , Xi Wang , Xin Tong , Stephen Lin , Shimin Hu , Baining Guo , Heung-Yeung Shum, View-dependent displacement mapping, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture

Keywords:
Bidirectional Texture Function, Image-Based Rendering, Multilinear Algebra, Statistical Learning, Tensor Decomposition, Tensors, Textured Surface Rendering