Real-world vision: Selective perception and task

ABSTRACT

Visual perception is an inherently selective process. To understand when and why a particular region of a scene is selected, it is imperative to observe and describe the eye movements of individuals as they go about performing specific tasks. In this sense, vision is an active process that integrates scene properties with specific, goal-oriented oculomotor behavior. This study is an investigation of how task influences the visual selection of stimuli from a scene. Four eye tracking experiments were designed and conducted to determine how everyday tasks affect oculomotor behavior. A portable eyetracker was created for the specific purpose of bringing the experiments out of the laboratory and into the real world, where natural behavior is most likely to occur. The experiments provide evidence that the human visual system is not a passive collector of salient environemental stimuli, nor is vision general-purpose. Rather, vision is active and specific, tightly coupled to the requirements of a task and a plan of action. The experiments support the hypothesis that the purpose of selective attention is to maximize task efficiency by fixating relevant objects in the scene. A computational model of visual attention is presented that imposes a high-level constraint on the bottom-up salient properties of a scene for the purpose of locating regions that are likely to correspond to foreground objects rather than background or other salient nonobject stimuli. In addition to improving the correlation to human subject fixation densities over a strictly bottom-up model [Itti et al. 1998; Parkhurst et al. 2002], this model predicts a central fixation tendency when that tendency is warranted, and not as an artificially primed location bias.

REFERENCES

• 1: Andrews, T. J. and Coppola, D. M. 1999. Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments. Vis. Res. 39, 2947--2953.
• 2: Arrington, K. F. 2000. History of eye tracking methods. http://www.arringtonresearch.com.
• 3: Dana H. Ballard , Mary M. Hayhoe , Jeff B. Pelz, Memory representations in natural tasks, Journal of Cognitive Neuroscience, v.7 n.1, p.66-80, Winter 1995 [doi>10.1162/jocn.1995.7.1.66]
• 4: J. J. Bonaiuto , L. Itti, Combining attention and recognition for rapid scene analysis, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, p.90, June 20-26, 2005 [doi>10.1109/CVPR.2005.432]
• 5: Bruce, N. and Tsotsos, J. 2006. Saliency based on information maximization. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS '05). 155--162.
• 6: Burt, P. J. and Adelson, E. H. 1983. The laplacian pyramimd as a compact image code. IEEE Trans. Comm. 31, 4, 532--540.
• 7: Buswell, G. T. 1935. How People Look at Pictures: A Study of the Psychology of Perception in Art. University of Chicago Press, Chicago.
• 8: J Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.8 n.6, p.679-698, Nov. 1986 [doi>10.1109/TPAMI.1986.4767851]
• 9: Canosa, R. L. 2005. Modeling selective perception of complex natural scenes. Int. J. AI. 14, 1(2), 233--260.
• 10: Carmi, R. and Itti, L. 2006. Visual causes versus correlates of attentional selection in dynamic scenes. Vis. Res. 46, 26, 4333--4345.
• 11: Carpenter, D. H. S. 1988. Movements of the Eyes. Pion, London.
• 12: David Chapman, Vision, instruction, and action, MIT Press, Cambridge, MA, 1991
• 13: Desimone, R. and Duncan, J. 1995. Neural mechanisms of selective visual attention. Ann. Rev. Neurosci. 18, 193--222.
• 14: Egeth, H. E. and Yantis, S. 1997. Visual attention: Control, representation, and time course. Ann. Rev. Psych. 48, 269--297.
• 15: Epelboim, J. and Suppes, P. 2001. A model of eye movements and visual working memory during problem solving in geometry. Vis. Res. 41, 1561--1574.
• 16: Fairchild, M. D. 1998. Color Appearance Models. Addison-Wesley, Reading, MA.
• 17: Fecteau, J. H. Andmunoz, D. P. 2006. Salience, relevance, and firing: A priority map for target selection. TRENDS Cogn. Sci. 10, 8, 382--390.
• 18: Findlay, J. M. and Gilchrist, I. D. 2003. Active Vision: The Psychology of Looking and Seeing. Oxford University Press, Oxford, UK.
• 19: Gabor, D. 1946. Theory of communication. IEEE Proc. 93, 429--441.
• 20: Godijn, R. and Theeuwes, J. 2002. Programming of endogenous and exogenous saccades: Evidence for a competitive integration model. J. Exper. Psych. Hum. Percep. Perform. 28, 5, 1039--1054.
• 21: Goldberg, M. E., Eggers, H. M., and Gouras, P. 1991. The ocular motor system. In Principles of Neural Science 3rd Ed. Appleton & Lange, Norwalk, CT, 660--679.
• 22: Gottlieb, J. P., Kusunoki, M., and Goldberg, M. E. 1998. The representation of visual salience in monkey parietal cortex. Nature 391, 29, 481--484.
• 23: Stevan Harnad, The symbol grounding problem, Physica D, v.42 n.1-3, p.335-346, June 1990
• 24: Hayhoe, M. M. 2000. Vision using routines: A functional account of vision. Vis. Cogn. 7, 43--64.
• 25: Hayhoe, M. M., Ballard, D. H., and Pelz, J. B. 1994. Visual representations in natural tasks. In Proceedings of the Workshop on Visual Behaviors. IEEE, Los Alamitos, CA.
• 26: Hebb, D. O. 1949. The Organization of Behavior. John Wiley & Sons, New York.
• 27: Hubel, D. H. and Wiesel, T. N. 1968. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215--243.
• 28: Hunt, R. W. G. 1995. The Reproduction of Color 5th ed. Fountain Press, Kingston-Upon-Thames, England.
• 29: Laurent Itti , Christof Koch , Ernst Niebur, A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.11, p.1254-1259, November 1998 [doi>10.1109/34.730558]
• 30: Itti, L., Rees, G., and J. K. Tsotsos, E. 2005. Neurobiology of Attention. Elsevier, San Diego, CA.
• 31: Yuri A. Ivanov , Aaron F. Bobick, Recognition of Visual Activities and Interactions by Stochastic Parsing, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.852-872, August 2000 [doi>10.1109/34.868686]
• 32: Kaufman, L. and Richards, W. 1969. Spontaneous fixation tendencies for visual forms. Percept. Psychophys. 5, 2, 85--88.
• 33: Kowler, E. 1995. Eye movements. In Visual Cognition 39 2nd Ed. MIT Press, Cambridge, MA.
• 34: Land, M. L., Mennie, N., and Rusted, J. 1999. The roles of vision and eye movements in the control of activities of daily living. Perception 28, 1311--1328.
• 35: Olivier Le Meur , Patrick Le Callet , Dominique Barba , Dominique Thoreau, A Coherent Computational Approach to Model Bottom-Up Visual Attention, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.5, p.802-817, May 2006 [doi>10.1109/TPAMI.2006.86]
• 36: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 37: Magee, D., Needham, C. J., Santos, P., Cohn, A. G., and Hogg, D. C. 2004. Autonomous learning for a cognitive agent using continuous models and inductive logic programming from audio-visual input. In Proceedings of the Workshop on Anchoring Symbols to Sensor Data 4 (AAI-04). AAAI, Menlo Park, CA, 17--24.
• 38: Manno, J. L. and Sakrison, D. J. 1974. The effects of a visual fidelity criterion on the encoding of images. IEEE Trans. Inform. Theory 20, 4, 525--535.
• 39: Milanese, R., Wechsler, H., Gil, S., Bost, J., and Pun, T. 1994. Integration of bottom-up and top-down cues for visual attention using non-linear relaxation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 781--785.
• 40: Moran, J. and Desimone, R. 1985. Selective attention gates visual processing in the extrastriate cortex. Science 229, 782--784.
• 41: Morgan, M. J., Hole, G. J., and Glennerstner, A. 1990. Biases and sensitivities in geometrical illusions. Vis. Res. 30, 11, 1793--1810.
• 42: Vidhya Navalpakkam , Laurent Itti, A Goal Oriented Attention Guidance Model, Proceedings of the Second International Workshop on Biologically Motivated Computer Vision, p.453-461, November 22-24, 2002
• 43: Navalpakkam, V. and Itti, L. 2006. Top-down attention selection is fine grained. J. Vis. 6, 1180--1193.
• 44: Navalpakkam, V. and Itti, L. 2007. Search goal tunes visual features optimally. Neuron 53, 4, 605--617.
• 45: Northdurft, H. C. 2005. Salience of feature contrast. Neurobiol Attention, 233--239.
• 46: Noton, D. and Stark, L. 1971. Scan paths and saccadic eye movements while viewing and recognizing patterns. Vis. Res. 11, 929--942.
• 47: Palmer, S. E. 1999. Vision Science: Photons to Phenomenology. MIT Press, Cambridge, MA.
• 48: Parkhurst, D., Law, K., and Niebur, E. 2002. Modeling the role of salience in the allocation of overt visual attention. Vis. Res. 42, 1, 107--123.
• 49: Sumanta N. Pattanaik , James A. Ferwerda , Mark D. Fairchild , Donald P. Greenberg, A multiscale model of adaptation and spatial vision for realistic image display, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.287-298, July 1998 [doi>10.1145/280814.280922]
• 50: Pelz, J. B. and Canosa, R. L. 2001. Oculomotor behavior and perceptual strategies in complex tasks. Vis. Res. 41, 3587--3596.
• 51: Jeff B. Pelz , Roxanne Canosa , Jason Babcock, Extended tasks elicit complex eye movement patterns, Proceedings of the 2000 symposium on Eye tracking research & applications, p.37-43, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355023]
• 52: Peters, R. J. and Itti, L. 2007. Beyond bottom-up: Incorporating task-dependent influences into a computational model of spatial attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 1--8.
• 53: Robert J. Peters , Laurent Itti, Applying computational tools to predict gaze direction in interactive visual environments, ACM Transactions on Applied Perception (TAP), v.5 n.2, p.1-19, May 2008 [doi>10.1145/1279920.1279923]
• 54: Peters, R. J., Iyer, A., Itti, L., and Koch, C. 2005. Components of bottom-up gaze allocation in natural images. Vis. Res. 45, 18, 2397--2416.
• 55: Posner, M. I. 1980. Orienting of attention. Q. J. Exper. Psych 32, 3--25.
• 56: Claudio M. Privitera , Lawrence W. Stark, Algorithms for Defining Visual Regions-of-Interest: Comparison with Eye Fixations, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.9, p.970-982, September 2000 [doi>10.1109/34.877520]
• 57: Keith Rayner, Eye movements in reading: perceptual and language processes, Academic Press Professional, Inc., San Diego, CA, 1983
• 58: Rensink, R. A. 2000a. The dynamic representation of scenes. Vis. Cogn. 7, 17--42.
• 59: Rensink, R. A. 2000b. Seeing, sensing, and scrutinizing. Vis. Res. 40, 1469--1487.
• 60: Rensink, R. A., O'Regan, J. K., and Clark, J. J. 1995. Image flicker is as good as saccades in making large scene changes invisible. Perception 24, 26--27.
• 61: Rubin, E. 1921. Visuell Wahrgenommene Figuren. Glydenalske Boghandel, Copenhagen.
• 62: Santos, P., Magee, D., Cohn, A., and Hogg, D. 2004. Combining multiple answers for learning mathematical structures from visual observations. In Proceedings of the 16th European Conference on Artificial Intelligence 16. IOS Press, The Netherlands, 544--554.
• 63: Tatler, B. W., Baddeley, R. J., and Gilchrist, I. D. 2005. Visual correlates of fixation selection: Effects of scale and time. Vis. Res. 45, 643--659.
• 64: Theeuwes, J. 2004. Top-down search strategies cannot override attentional capture. Psychonomic Bull. Rev. 11, 65--70.
• 65: Torralba, A. 2003. Modeling global scene factors in attention. J. Opt. Soc. Am. 20, 7, 1407--1418.
• 66: Torralba, A., Oliva, A., Castelhano, M. S., and Henderson, J. M. 2006. Contextual guidance of eye movements and attention in real-world scenes: The role of global features on object search. Psych. Rev. 113, 4, 766--786.
• 67: Treisman, A. and Gelade, G. 1980. A feature integration theory of attention. Cogn. Psych. 12, 97--136.
• 68: Ullman, S. 1984. Visual routines. Cognition 18, 97--106.
• 69: van Zoest,W., Donk, M., and Theeuwes, J. 2004. The role of stimulus-driven and goal-driven control in saccadic visual selection. J. Exper. Psych. Hum. Percept. Perform. 30, 4, 746--759.
• 70: Dirk Walther , Christof Koch, 2006 Special Issue: Modeling attention to salient proto-objects, Neural Networks, v.19 n.9, p.1395-1407, November, 2006 [doi>10.1016/j.neunet.2006.10.001]
• 71: Dirk Walther , Ueli Rutishauser , Christof Koch , Pietro Perona, Selective visual attention enables learning and recognition of multiple objects in cluttered scenes, Computer Vision and Image Understanding, v.100 n.1-2, p.41-63, October 2005 [doi>10.1016/j.cviu.2004.09.004]
• 72: Yarbus, A. 1967. Eye Movements and Vision L. A. Riggs, translator. Plenum Press, New York.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Representations, data structures, and transforms
  I.5 PATTERN RECOGNITION
      I.5.4 Applications
          Subjects: Computer vision

  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Psychology

General Terms:
Algorithms, Experimentation

Keywords:
Active vision, eye-tracking, saliency modeling