Non-photorealistic virtual environments

ABSTRACT

We describe a system for non-photorealistic rendering (NPR) of virtual environments. In real time, it synthesizes imagery of architectural interiors using stroke-based textures. We address the four main challenges of such a system — interactivity, visual detail, controlled stroke size, and frame-to-frame coherence — through image based rendering (IBR) methods. In a preprocessing stage, we capture photos of a real or synthetic environment, map the photos to a coarse model of the environment, and run a series of NPR filters to generate textures. At runtime, the system re-renders the NPR textures over the geometry of the coarse model, and it adds dark lines that emphasize creases and silhouettes. We provide a method for constructing non-photorealistic textures from photographs that largely avoids seams in the resulting imagery. We also offer a new construction, art-maps, to control stroke size across the images. Finally, we show a working system that provides an immersive experience rendered in a variety of NPR styles.

REFERENCES

• 1: Ian Buck , Adam Finkelstein , Charles Jacobs , Allison Klein , David H. Salesin , Joshua Seims , Richard Szeliski , Kentaro Toyama, Performance-driven hand-drawn animation, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.101-108, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340929]
• 2: Shenchang Eric Chen, QuickTime VR: an image-based approach to virtual environment navigation, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.29-38, September 1995 [doi>10.1145/218380.218395]
• 3: Shenchang Eric Chen , Lance Williams, View interpolation for image synthesis, Proceedings of the 20th annual conference on Computer graphics and interactive techniques, p.279-288, September 1993 [doi>10.1145/166117.166153]
• 4: Cassidy J. Curtis , Sean E. Anderson , Joshua E. Seims , Kurt W. Fleischer , David H. Salesin, Computer-generated watercolor, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.421-430, August 1997 [doi>10.1145/258734.258896]
• 5: Paul E. Debevec, Image-based modeling and lighting, ACM SIGGRAPH Computer Graphics, v.33 n.4, p.46-50, Nov.2000 [doi>10.1145/345370.345406]
• 6: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 7: Debevec, P. E., Yu, Y., and Borshukov, G. D. Efficient view-dependent image-based rendering with projective texture-mapping. Eurographics Rendering Workshop (June 1998), 105-116.
• 8: Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
• 9: Funkhouser, T. A. A visibility algorithm for hybrid geometry- and image-based modeling and rendering. Computers and Graphics, Special Issue on Visibility (1999).
• 10: Funkhouser, T. A., Teller, S. J., Sequin, C. H., and Khorramabadi, D. The UC Berkeley system for interactive visualization of large architectural models. Presence 5, 1 (January 1996).
• 11: Amy Gooch , Bruce Gooch , Peter Shirley , Elaine Cohen, A non-photorealistic lighting model for automatic technical illustration, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.447-452, July 1998 [doi>10.1145/280814.280950]
• 12: Steven J. Gortler , Radek Grzeszczuk , Richard Szeliski , Michael F. Cohen, The lumigraph, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.43-54, August 1996 [doi>10.1145/237170.237200]
• 13: Paul Haeberli, Paint by numbers: abstract image representations, ACM SIGGRAPH Computer Graphics, v.24 n.4, p.207-214, Aug. 1990
• 14: Paul S Heckbert, Survey of texture mapping, IEEE Computer Graphics and Applications, v.6 n.11, p.56-67, November 1986 [doi>10.1109/MCG.1986.276672]
• 15: Aaron Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.453-460, July 1998 [doi>10.1145/280814.280951]
• 16: Hewlett Packard. HP PEX Texture Mapping, www.hp.com/mhm/WhitePapers/PEXtureMapping/PEXtureMapping.html.
• 17: Youichi Horry , Ken-Ichi Anjyo , Kiyoshi Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.225-232, August 1997 [doi>10.1145/258734.258854]
• 18: Michael A. Kowalski , Lee Markosian , J. D. Northrup , Lubomir Bourdev , Ronen Barzel , Loring S. Holden , John F. Hughes, Art-based rendering of fur, grass, and trees, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.433-438, July 1999 [doi>10.1145/311535.311607]
• 19: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 20: Peter Litwinowicz, Processing images and video for an impressionist effect, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.407-414, August 1997 [doi>10.1145/258734.258893]
• 21: Manocha, D. Interactive walkthroughs of large geometric databases. Course #18, SIGGRAPH 2000 Course Notes (July 2000).
• 22: Lee Markosian , Michael A. Kowalski , Daniel Goldstein , Samuel J. Trychin , John F. Hughes , Lubomir D. Bourdev, Real-time nonphotorealistic rendering, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.415-420, August 1997 [doi>10.1145/258734.258894]
• 23: Leonard McMillan , Gary Bishop, Plenoptic modeling: an image-based rendering system, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.39-46, September 1995 [doi>10.1145/218380.218398]
• 24: Barbara J. Meier, Painterly rendering for animation, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.477-484, August 1996 [doi>10.1145/237170.237288]
• 25: Mizuno, S., Okada, M., and ichiro Toriwaki, J. Virtual sculpting and virtual woodcut printing. The Visual Computer 14, 2 (1998), 39-51.
• 26: Victor Ostromoukhov, Digital facial engraving, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.417-424, July 1999 [doi>10.1145/311535.311604]
• 27: Takafumi Saito , Tokiichiro Takahashi, NC machining with G-buffer method, ACM SIGGRAPH Computer Graphics, v.25 n.4, p.207-216, July 1991
• 28: Michael P. Salisbury , Michael T. Wong , John F. Hughes , David H. Salesin, Orientable textures for image-based pen-and-ink illustration, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.401-406, August 1997 [doi>10.1145/258734.258890]
• 29: Jonathan Shade , Steven Gortler , Li-wei He , Richard Szeliski, Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.231-242, July 1998 [doi>10.1145/280814.280882]
• 30: Tsai, R. Y. A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses. IEEE Journal of Robotics and Automation 3, 4 (Aug. 1987), 323-344.
• 31: Lance Williams, Pyramidal parametrics, Proceedings of the 10th annual conference on Computer graphics and interactive techniques, p.1-11, July 25-29, 1983, Detroit, Michigan, United States
• 32: Georges Winkenbach , David H. Salesin, Computer-generated pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.91-100, July 1994 [doi>10.1145/192161.192184]
• 33: Georges Winkenbach , David H. Salesin, Rendering parametric surfaces in pen and ink, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.469-476, August 1996 [doi>10.1145/237170.237287]
• 34: Michael T. Wong , Douglas E. Zongker , David H. Salesin, Computer-generated floral ornament, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.423-434, July 1998 [doi>10.1145/280814.280948]
• 35: Daniel N. Wood , Adam Finkelstein , John F. Hughes , Craig E. Thayer , David H. Salesin, Multiperspective panoramas for cel animation, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.243-250, August 1997 [doi>10.1145/258734.258859]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism
          Subjects: Color, shading, shadowing, and texture; Virtual reality
  I.4 IMAGE PROCESSING AND COMPUTER VISION

General Terms:
Design, Experimentation, Measurement, Performance, Theory

Keywords:
image-based rendering, interactive virtual environments, non-photorealistic rendering, texture mapping