Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments

ABSTRACT

We present a method to accelerate global illumination computation in prerendered animations by taking advantage of limitations of the human visual system. A spatiotemporal error tolerance map, constructed from psychophysical data based on velocity dependent contrast sensitivity, is used to accelerate rendering. The error map is augmented by a model of visual attention in order to account for the tracking behavior of the eye. Perceptual acceleration combined with good sampling protocols provide a global illumination solution feasible for use in animation. Results indicate an order of magnitude improvement in computational speed.

REFERENCES

• 1: Maneesh Agrawala , Andrew C. Beers , Navin Chaddha, Model-based motion estimation for synthetic animations, Proceedings of the third ACM international conference on Multimedia, p.477-488, November 05-09, 1995, San Francisco, California, United States [doi>10.1145/217279.215312]
• 2: Mark R. Bolin , Gary W. Meyer, A frequency based ray tracer, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.409-418, September 1995 [doi>10.1145/218380.218497]
• 3: Mark R. Bolin , Gary W. Meyer, A perceptually based adaptive sampling algorithm, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.299-309, July 1998 [doi>10.1145/280814.280924]
• 4: BURT,P.J.AND ADELSON, E. H. 1983. The Laplacian pyramid as a compact image code. In IEEE Trans. Commun. Com-31, 4 (Apr.), pp. 532-540.
• 5: CAMPBELL,F.W.AND ROBSON, J. G. 1968. Application of Fourier analysis to the visibility of gratings. In J. Phys. (London) 197, pp. 551-566.
• 6: Scott Daly, The visible differences predictor: an algorithm for the assessment of image fidelity, Digital images and human vision, MIT Press, Cambridge, MA, 1993
• 7: DALY, S. 1998. Engineering observations from spatiovelocity and spatiotemporal visual Models. In IS&T/SPIE Conference on Human Vision and Electronic Imaging III. SPIE, Vol. 3299, (Jan.), pp. 180-191.
• 8: DALY, S., MATTHEWS, K., AND RIBAS-CORBERA, J. 1999. Visual eccentricity models in face-based video compression. In IS&T/SPIE Conference on Human Vision and Electronic Imaging IV. SPIE, Vol. 3644, (Jan.), pp. 152-166.
• 9: James A. Ferwerda , Peter Shirley , Sumanta N. Pattanaik , Donald P. Greenberg, A model of visual masking for computer graphics, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.143-152, August 1997 [doi>10.1145/258734.258818]
• 10: GIBSON,S.AND HUBBOLD, R. 1997. Perceptually-driven radiosity. Compute. Graph. Forum 16, 2 (June), pp. 192-140.
• 11: GREENSPAN, H., BELONGIE, S., GOODMAN, R., PERONA, P., RAKSHIT, S., ANDERSON, C. H. 1994. Overcomplete steerable pyramid filters and rotation invariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Seattle, Wash. June). IEEE Computer Society Press, Los Alamitos, Calif., pp. 222-228.
• 12: HORVITZ,E.AND LENGYEL, J. 1997. Perception, attention, and resources: A decision-theoretic approach to graphics rendering. In Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence (Providence, R.I., Aug.), Don Gelger and Prakash Stenoy, Eds, Morgan Kaufmann Publishers, Inc., San Francisco, Calif., pp. 238-249.
• 13: ITTI,L.AND KOCH, C. 1999a. A comparison of feature combination strategies for saliency-based visual attention systems. In IS&T/SPIE Conference on Human Vision and Electronic Imaging IV. SPIE, Vol. 3644, (Jan.), pp. 373-382.
• 14: ITTI,L.AND KOCH, C. 1999b. Learning to detect salient objects in natural scenes using visual attention. In Image Understanding Workshop. (In press. A preprint version of this article is available from http://www.klab.caltech.edul/citti/attention).
• 15: ITTI,L.AND KOCH, C. 2000. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Res. 40, 10-12, pp. 1489-1506.
• 16: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 17: KELLY, D. H. 1979. Motion and vision II. Stabilized Spatio-temporal Threshold Surface. Opti. Soci. Ameri. 69, 10 (Oct.), pp. 1340-1349.
• 18: KOCH,C.AND ULLMAN, S. 1985. Shifts in selective visual attention: Towards the underlying neural circuitry. Human Neurobiology 4, pp. 219-227.
• 19: LEVENTHAL, A. G. 1991. The neural basis of visual function. In Vision and Visual Dysfunction, vol. 4. CRC Press, Boca Raton, Fla.
• 20: LUBIN, J. 1995. A visual discrimination model for imaging system design and evaluation. In Vision Models for Target Detection and Recognition, E. Peli, Ed. World Scientific, New Jersey, pp. 245-283.
• 21: Leonard McMillan, Jr., An image-based approach to three-dimensional computer graphics, University of North Carolina at Chapel Hill, Chapel Hill, NC, 1997
• 22: MEYER,G.W.AND LIU, A. 1992 Color spatial acuity control of a screen subdivision image synthesis algorithm. In Human Vision, Visual Processing and Digital Display III, B. E. Rogowitz, Ed. SPIE, vol. 1666, pp. 387-399.
• 23: MYSZKOWSKI, K. 1998. The visible differences predictor: Applications to global illumination problems. In Proceedings of the 9th Eurographics Workshop on Rendering (Vienna, Austria, June). Springer-Verlag, New York, pp. 223-236.
• 24: MYSZKOWSKI, K., ROKITA,P.,AND TAWARA, T. 1999. Perceptually-informed accelerated rendering of high quality walkthrough sequences. In Proceedings of the 10th Eurographics Workshop on Rendering (Grenada, Spain. June). Springer-Verlag, New York, pp. 5-18.
• 25: NIEBUR,E.AND KOCH, C. 1998. Computational architectures for attention. In The Attentive Brain, R. Parasuraman, Ed. MIT Press, Cambridge, Mass., pp. 164-186.
• 26: Mahesh Ramasubramanian , Sumanta N. Pattanaik , Donald P. Greenberg, A perceptually based physical error metric for realistic image synthesis, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.73-82, July 1999 [doi>10.1145/311535.311543]
• 27: A. Murat Tekalp, Digital video processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1995
• 28: John K. Tsotsos , Sean M. Culhane , Winky Yan Kei Wai , Yuzhong Lai , Neal Davis , Fernando Nuflo, Modeling visual attention via selective tuning, Artificial Intelligence, v.78 n.1-2, p.507-545, Oct. 1995 [doi>10.1016/0004-3702(95)00025-9]
• 29: Dan S. Wallach , Sharma Kunapalli , Michael F. Cohen, Accelerated MPEG compression of dynamic polygonal scenes, Proceedings of the 21st annual conference on Computer graphics and interactive techniques, p.193-196, July 1994 [doi>10.1145/192161.192198]
• 30: WANDELL, B. 1995. Foundations of Vision. Sinauer Associates, Inc., Sunderland, MA.
• 31: Gregory J. Ward , Francis M. Rubinstein , Robert D. Clear, A ray tracing solution for diffuse interreflection, Proceedings of the 15th annual conference on Computer graphics and interactive techniques, p.85-92, June 1988
• 32: WARD,G.AND HECKBERT, P. S. 1992. Irradiance gradients. In Proceedings of the 3rd Annual Eurographics Workshop on Rendering. Springer-Verlag, New York, pp. 85-98.
• 33: WARD-LARSON,G.AND SHAKESPEARE, R. 1998. Rendering with Radiance. Morgan-Kaufmann. San Francisco, Calif.
• 34: Gregory Ward Larson , Holly Rushmeier , Christine Piatko, A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes, IEEE Transactions on Visualization and Computer Graphics, v.3 n.4, p.291-306, October 1997 [doi>10.1109/2945.646233]
• 35: YANTIS, S. 1996. Attentional capture in vision. In Converging Operations in the Study of Selective Visual Attention, A. Kramer, M. Coles, and G. Logan, Eds, American Psychological Association, Washington, D.C., pp. 45-76.
• 36: YARBUS, A. L. 1967. Eye Movements and Vision. Plenum Press, New York.
• 37: Ramin Zabih , John Woodfill, Non-parametric local transforms for computing visual correspondence, Proceedings of the third European conference on Computer Vision (Vol. II), p.151-158, May 1994, Stockholm, Sweden

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

General Terms:
Algorithms