A loose and sketchy approach in a mediated reality environment

ABSTRACT

In this paper, we present sketchy-ar-us, a modified, real-time version of the Loose and Sketchy algorithm used to render graphics in an AR environment. The primary challenge was to modify the original algorithm to produce a NPR effect at interactive frame rate. Our algorithm renders moderately complex scenes at multiple frames per second. Equipped with a handheld visor, visitors can see the real environment overlaid with virtual objects with both the real and virtual content rendered in a non-photorealistic style.

REFERENCES

• 1: Kusuma Agusanto , Li Li , Zhu Chuangui , Ng Wan Sing, Photorealistic rendering for augmented reality using environment illumination, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.208, October 07-10, 2003
• 2: Oliver Bimber , Anselm Grundhöfer , Gordon Wetzstein , Sebastian Knödel, Consistent Illumination within Optical See-Through Augmented Environments, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.198, October 07-10, 2003
• 3: Card, D., and Mitchell, J. 2002. Non-Photorealistic Rendering with Pixel and Vertex Shaders. In Direct3D ShaderX, Wordware, W. Esngel, Ed.
• 4: Collomosse, J. P., Rowntree, D., and Hall, P. M. 2003. Stroke surfaces: A spatio-temporal framework for temporally coherent non-photorealistic animations. Tech. Rep. 2003--01, University of Bath, U.K., Bath, June.
• 5: Collomosse, J. P., Rowntree, D., and Hall, P. M. 2003. Video analysis for cartoon-like special effects. In 14th British Machine Vision Conference, 749--758.
• 6: Cassidy J. Curtis , Sean E. Anderson , Joshua E. Seims , Kurt W. Fleischer , David H. Salesin, Computer-generated watercolor, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.421-430, August 1997 [doi>10.1145/258734.258896]
• 7: Curtis, C. 1998. Non-photorealistic animation. In Proc. ACM SIGGRAPH 1998, ACM Press.
• 8: Frédo Durand, An invitation to discuss computer depiction, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508550]
• 9: Fernando, R. 2004. GPU Gems. Addison-Wesley.
• 10: Ferwerda, J. 2003. Three varieties of realism in computer graphics. In Proceedings SPIE Human Vision and Electronic Imaging '03.
• 11: Jan Fischer , Dirk Bartz, Stylized Augmented Reality for Improved Immersion, Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality, p.195-202, 325, March 12-16, 2005 [doi>10.1109/VR.2005.71]
• 12: Simon Gibson , Jon Cook , Toby Howard , Roger Hubbold, Rapid shadow generation in real-world lighting environments, Proceedings of the 14th Eurographics workshop on Rendering, June 25-27, 2003, Leuven, Belgium
• 13: Gogolin, H. 2004. Wenn weniger mehr ist. GEE Magazin 6 (August), 64--68.
• 14: Bruce Gooch , Amy Gooch, Non-Photorealistic Rendering, A. K. Peters, Ltd., Natick, MA, 2001
• 15: Bruce Gooch , Greg Coombe , Peter Shirley, Artistic Vision: painterly rendering using computer vision techniques, Proceedings of the 2nd international symposium on Non-photorealistic animation and rendering, June 03-05, 2002, Annecy, France [doi>10.1145/508530.508545]
• 16: Gouraud, H. 1971. Continuous shading of curved surfaces. In IEEE Transactions on Computers, 623--629.
• 17: Raphael Grasset , Jean-Dominique Gascuel , Dieter Schmalstieg, Interactive Mediated Reality, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.302, October 07-10, 2003
• 18: Paul Haeberli, Paint by numbers: abstract image representations, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.207-214, September 1990, Dallas, TX, USA
• 19: Michael Haller , Daniel Sperl, Real-time painterly rendering for MR applications, Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, June 15-18, 2004, Singapore [doi>10.1145/988834.988839]
• 20: Michael Haller , Stephan Drab , Werner Hartmann, A real-time shadow approach for an augmented reality application using shadow volumes, Proceedings of the ACM symposium on Virtual reality software and technology, October 01-03, 2003, Osaka, Japan [doi>10.1145/1008653.1008665]
• 21: Michael Haller, Photorealism or/and non-photorealism in augmented reality, Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry, June 16-18, 2004, Singapore [doi>10.1145/1044588.1044627]
• 22: Aaron Hertzmann , Ken Perlin, Painterly rendering for video and interaction, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.7-12, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340917]
• 23: Robert D. Kalnins , Philip L. Davidson , Lee Markosian , Adam Finkelstein, Coherent stylized silhouettes, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 24: Hirokazu Kato , Mark Billinghurst, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, p.85, October 20-21, 1999
• 25: Michael A. Kowalski , Lee Markosian , J. D. Northrup , Lubomir Bourdev , Ronen Barzel , Loring S. Holden , John F. Hughes, Art-based rendering of fur, grass, and trees, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.433-438, July 1999 [doi>10.1145/311535.311607]
• 26: Landerl, F. 2005. Skizzenhaftes Rendering in Echtzeitanwendungen. Master's thesis, Fachhochschule Hagenberg, Medientechnik und -design, Hagenberg, Austria.
• 27: Mann, S., and Fung, J. 2001. VideoOrbits on Eye Tap devices for deliberately Diminished Reality or altering the visual perception of rigid planar patches of a real world scene. In International Symposium on Mixed Reality (ISMR).
• 28: Mann, S. 1994. Mediated reality. TR 260, M.I.T. Media Lab Perceptual Computing Section, Cambridge, Massachusetts, http://wearcam.org/mr.htm.
• 29: Masuch, M., Schlechtweg, S., and Schönwälder, B. 1997. dali! - drawing animated lines! 87--95. ISBN 1-56555-111-7.
• 30: McCloud, S. 1994. Understanding Comics. Perennial Currents.
• 31: Barbara J. Meier, Painterly rendering for animation, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.477-484, August 1996 [doi>10.1145/237170.237288]
• 32: T. Naemura , T. Nitta , A. Mimura , H. Harashima, Virtual Shadows - Enhanced Interaction in Mixed Reality Environment, Proceedings of the IEEE Virtual Reality Conference 2002, p.293, March 24-28, 2002
• 33: Marc Nienhaus , Jürgen Döllner, Blueprints: illustrating architecture and technical parts using hardware-accelerated non-photorealistic rendering, Proceedings of the 2004 conference on Graphics interface, p.49-56, May 17-19, 2004, London, Ontario, Canada
• 34: J. D. Northrup , Lee Markosian, Artistic silhouettes: a hybrid approach, Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, p.31-37, June 05-07, 2000, Annecy, France [doi>10.1145/340916.340920]
• 35: Emil Praun , Hugues Hoppe , Matthew Webb , Adam Finkelstein, Real-time hatching, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.581, August 2001 [doi>10.1145/383259.383328]
• 36: Ramesh Raskar , Michael Cohen, Image precision silhouette edges, Proceedings of the 1999 symposium on Interactive 3D graphics, p.135-140, April 26-29, 1999, Atlanta, Georgia, United States [doi>10.1145/300523.300539]
• 37: Ramesh Raskar , Kar-Han Tan , Rogerio Feris , Jingyi Yu , Matthew Turk, Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 38: Roussou, M., and Drettakis, G. 2003. Photorealism and non-photorealism in virtual heritage representation. In VAST 2003 and First Eurographics Workshop on Graphics and Cultural Heritage, A. Chalmers, D. Arnold, and F. Niccolucci, Eds., Eurographics.
• 39: Takafumi Saito , Tokiichiro Takahashi, Comprehensible rendering of 3-D shapes, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.197-206, September 1990, Dallas, TX, USA
• 40: Michael P. Salisbury , Sean E. Anderson , Ronen Barzel , David H. Salesin, Interactive pen-and-ink illustration, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.101-108, July 1994 [doi>10.1145/192161.192185]
• 41: Thomas Strothotte , Stefan Schlechtweg, Non-photorealistic computer graphics: modeling, rendering, and animation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2002
• 42: Natsuki Sugano , Hirokazu Kato , Keihachiro Tachibana, The Effects of Shadow Representation of Virtual Objects in Augmented Reality, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, p.76, October 07-10, 2003
• 43: Wynn, C. 2002. OpenGL Render-to-Texture. TR, nVIDIA Corporation, March.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.3 Picture/Image Generation

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.5 Computational Geometry and Object Modeling

Keywords:
loose and sketchy, mediated reality, non-photorealistic rendering