Real-time painterly rendering for MR applications

ABSTRACT

In this paper we describe a real-time system for AR/MR rendering applications in a painterly style. Impressionistic images are created using a large number of brush strokes, which are organized as 3d particles to achieve frame-to-frame coherence. Reference pictures are used to compute the properties of each stroke.The presented technique is based on B. J. Meier's "Painterly Rendering for Animation". We modified the algorithm of Meier for real-time AR/MR environments by extensively using modern 3d hardware. Vertex and pixel shaders allow both the rendering of thousands of brush strokes per frame and the correct application of their properties. Direct rendering to textures allows rapid generation of reference pictures and assures great flexibility, since arbitrary rendering systems can be combined (e. g. painterly rendering of toon shaded objects, etc.).

REFERENCES

• 1: Cassidy J. Curtis , Sean E. Anderson , Joshua E. Seims , Kurt W. Fleischer , David H. Salesin, Computer-generated watercolor, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.421-430, August 1997 [doi>10.1145/258734.258896]
• 2: DIETRICH, S. 2000. Cartoon rendering and advanced texture features of the geforce 256 texture matrix, projective textures, cube maps, texture coordinate generation and dotproduct3 texture blending. Tech. rep., nVIDIA.
• 3: DRONE, S., KRISHNAMACHARI, P., LAPAN, D., MICHAELS, J., SCHMIDT, D., AND SMITH, P. 2000. Project aember: Real-time painterly rendering of 3d scenery. Tech. rep., UIUC ACM SIGGRAPH, August.
• 4: Frédo Durand, An invitation to discuss computer depiction, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508550]
• 5: Feng Dong , Gordon J. Clapworthy , Hai Lin , Meleagros A. Krokos, Nonphotorealistic Rendering of Medical Volume Data, IEEE Computer Graphics and Applications, v.23 n.4, p.44-52, July 2003 [doi>10.1109/MCG.2003.1210864]
• 6: Randima Fernando , Mark J. Kilgard, The Cg Tutorial: The Definitive Guide to Programmable Real-Time Graphics, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2003
• 7: Bert Freudenberg , Maic Masuch , Niklas Röber , Thomas Strothotte, The Computer-Visualistik-Raum: veritable and inexpensive presentation of a virtual reconstruction, Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, November 28-30, 2001, Glyfada, Greece [doi>10.1145/584993.585010]
• 8: Bruce Gooch , Greg Coombe , Peter Shirley, Artistic Vision: painterly rendering using computer vision techniques, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508545]
• 9: Paul Haeberli, Paint by numbers: abstract image representations, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.207-214, September 1990, Dallas, TX, USA
• 10: Aaron Hertzmann , Ken Perlin, Painterly rendering for video and interaction, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.7-12, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340917]
• 11: Matthew Kaplan , Bruce Gooch , Elaine Cohen, Interactive artistic rendering, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.67-74, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340925]
• 12: KATO, H., BILLINGHURST, M., BLANDING, B., AND MAY, R. 1999. ARToolKit.
• 13: Ken-ichi Anjyo , Katsuaki Hiramitsu, Stylized Highlights for Cartoon Rendering and Animation, IEEE Computer Graphics and Applications, v.23 n.4, p.54-61, July 2003 [doi>10.1109/MCG.2003.1210865]
• 14: KIRK, D. 2003. Cg Toolkit - User's Manual, Release 1.1, 1 ed. nVIDIA Corporation, March.
• 15: LANDER, J. 2000. Shades of disney: Opaquing a 3d world. Game Developers Magazine (March).
• 16: Aditi Majumder , M. Gopi, Hardware accelerated real time charcoal rendering, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508541]
• 17: MARK, W. R., GLANVILLE, R. S., AKELEY, K., AND KILGARD, M. J. 2003. Cg: A system for programming graphics hardware in a c-like language. In Proceedings of SIGGRAPH 2003, 896--907.
• 18: Lee Markosian , Michael A. Kowalski , Daniel Goldstein , Samuel J. Trychin , John F. Hughes , Lubomir D. Bourdev, Real-time nonphotorealistic rendering, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.415-420, August 1997 [doi>10.1145/258734.258894]
• 19: Barbara J. Meier, Painterly rendering for animation, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.477-484, August 1996 [doi>10.1145/237170.237288]
• 20: Emil Praun , Hugues Hoppe , Matthew Webb , Adam Finkelstein, Real-time hatching, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.581, August 2001 [doi>10.1145/383259.383328]
• 21: William T. Reeves , Ricki Blau, Approximate and probabilistic algorithms for shading and rendering structured particle systems, Proceedings of the 12th annual conference on Computer graphics and interactive techniques, p.313-322, July 1985
• 22: Michael P. Salisbury , Sean E. Anderson , Ronen Barzel , David H. Salesin, Interactive pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.101-108, July 1994 [doi>10.1145/192161.192185]
• 23: Georges Winkenbach , David H. Salesin, Computer-generated pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.91-100, July 1994 [doi>10.1145/192161.192184]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.5 Computational Geometry and Object Modeling

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Keywords:
augmented reality, mixed reality, non-photorealistic rendering, painterly rendering