Causal saliency effects during natural vision

ABSTRACT

Salient stimuli, such as color or motion contrasts, attract human attention, thus providing a fast heuristic for focusing limited neural resources on behaviorally relevant sensory inputs. Here we address the following questions: What types of saliency attract attention and how do they compare to each other during natural vision? We asked human participants to inspect scene-shuffled video clips, tracked their instantaneous eye-position, and quantified how well a battery of computational saliency models predicted overt attentional selections (saccades). Saliency effects were measured as a function of total viewing time, proximity to abrupt scene transitions (jump cuts), and inter-participant consistency. All saliency models predicted overall attentional selection well above chance, with dynamic models being equally predictive to each other, and up to 3.6 times more predictive than static models. The prediction accuracy of all dynamic models was twice higher than their average for saccades that were initiated immediately after jump cuts, and led to maximal inter-participant consistency. Static models showed mixed results in these circumstances, with some models having weaker prediction accuracy than their average. These results demonstrate that dynamic visual cues play a dominant causal role in attracting attention, while static visual cues correlate with attentional selection mostly due to top-down causes.

REFERENCES

• 1: Abrams, R. A., and Christ, S. E. (2005). The onset of receding motion captures attention: comment on Franconeri and Simons (2003). Perception & Psychophysics 67, 219--223.
• 2: Anderson, J. D. (1996). The reality of illusion: an ecological approach to cognitive film theory (Carbondale, Southern Illinois University Press).
• 3: Atkinson, J., and Braddick, O. (2003). Neurobiological Models of Normal and Abnormal Visual Development (Hove, East Sussex; New York, Psychology Press).
• 4: Bex, P. J., and Makous, W. (2002). Spatial frequency, phase, and the contrast of natural images. Journal of the Optical Society of America A. Optics, Image Science, and Vision 19, 1096--1106.
• 5: Bisley, J. W., and Goldberg, M. E. (2003). Neuronal activity in the lateral intraparietal area and spatial attention. Science 299, 81--86.
• 6: Caspi, A., Beutter, B. R., and Eckstein, M. P. (2004). The time course of visual information accrual guiding eye movement decisions. Proceedings of the National Academy of Sciences of the United States of America 101, 13086--13090.
• 7: Connolly, M., and Van Essen, D. (1984), The representation of the visual field in parvicellular and magnocellular layers of the lateral geniculate nucleus in the macaque monkey. Journal of Comparative Neurology 226, 544--564.
• 8: Curcio, C. A., Sloan, K. R., Jr., Packer, O., Hendrickson, A. E., and Kalina, R. E. (1987). Distribution of cones in human and monkey retina: individual variability and radial asymmetry. Science 236, 579--582.
• 9: Curio, E. (1976). The ethology of predation (Berlin; New York, Springer-Verlag).
• 10: Driver, J., Mcleod, P., and Dienes, Z. (1992). Motion coherence and conjunction search: implications for guided search theory. Perception & Psychophysics 51, 79--85.
• 11: Efron, B., and Tibshirani, R. (1993). An introduction to the bootstrap (New York, Chapman & Hall).
• 12: Fecteau, J. H., Bell, A. H., and Munoz, D. P. (2004). Neural correlates of the automatic and goal-driven biases in orienting spatial attention. Journal of Neurophysiology 92, 1728--1737.
• 13: Findlay, J. M. (2004). Eye scanning and visual search. In The Interface of Language, Vision, and Action: Eye Movements and the Visual World, J. M. Henderson, and F. Ferreira, eds. (UK, Psychology Press).
• 14: Findlay, J. M., and Walker, R. (1999). A model of saccade generation based on parallel processing and competitive inhibition. Behavioral and Brain Sciences 22, 661--674.
• 15: Finlay, D., and Ivinskis, A. (1984). Cardiac and Visual Responses to Moving Stimuli Presented Either Successively or Simultaneously to the Central and Peripheral Visual-Fields in 4-Month-Old Infants. Developmental Psychology 20, 29--36.
• 16: Folk, C. L., Remington, R. W., and Johnston, J. C. (1992). Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human Perception and Performance 18, 1030--1044.
• 17: Franconeri, S. L., Hollingworth, A., and Simons, D. J. (2005). Do new objects capture attention? Psychological Science 16, 275--281.
• 18: Frost, B. J., and Nakayama, K. (1983). Single Visual Neurons Code Opposing Motion Independent of Direction. Science 220, 744--745.
• 19: Gibson, J. J. (1979/1986). The ecological approach to visual perception (Hillsdale, N.J., Lawrence Erlbaum Associates).
• 20: Gottlieb, J. P., Kusunoki, M., and Goldberg, M. E. (1998). The representation of visual salience in monkey parietal cortex. Nature 391, 481--484.
• 21: Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences 7, 498--504.
• 22: Henderson, J. M., and Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology 50, 243--271.
• 23: Hernandez-Peon, R., Scherrer, H., and Jouvet, M. (1956). Modification of electric activity in cochlear nucleus during attention in unanesthetized cats. Science 123, 331--332.
• 24: Hillstrom, A. P., and Yantis, S. (1994). Visual motion and attentional capture. Perception and Psychophysics 55, 399--411.
• 25: Hochberg, J. E. (1986). Representation of motion and space in video and cinematic displays. In Handbook of perception and human performance: Vol. 1. Sensory processes and perception, K. R. Boff, R. Kaufman, and J. P. Thomas, eds. (New York, Wiley), pp. 22--21 to 22--64.
• 26: Itti, L. (2005). Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Visual Cognition 12, 1093--1123.
• 27: Itti, L., and Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research 40, 1489--1506.
• 28: Jacobs, R. A. (2002). What determines visual cue reliability? Trends in Cognitive Sciences 6, 345--350.
• 29: James, W. (1890). Principles of Psychology (Oxford, England, Henry Holt).
• 30: Jonides, J., and Yantis, S. (1988). Uniqueness of Abrupt Visual Onset in Capturing Attention. Perception and Psychophysics 43, 346--354.
• 31: Klin, A., Jones, W., Schultz, R., Volkmar, F., and Cohen, D. (2002). Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry 59, 809--816.
• 32: Koch, C., and Ullman, S. (1985). Shifts in Selective Visual-Attention-Towards the Underlying Neural Circuitry. Human Neurobiology 4, 219--227.
• 33: Krieger, G., Rentschler, I., Hauske, G., Schill, K., and Zetzsche, C. (2000). Object and scene analysis by saccadic eye-movements: an investigation with higher-order statistics. Spatial Vision 13, 201--214.
• 34: Kustov, A. A., and Robinson, D. L. (1996). Shared neural control of attentional shifts and eye movements. Nature 384, 74--77.
• 35: Land, M. F., and Mcleod, P. (2000). From eye movements to actions: how batsmen hit the ball. Nature Neuroscience 3, 1340--1345.
• 36: Mannan, S. K., Ruddock, K. H., and Wooding, D. S. (1997). Fixation patterns made during brief examination of two-dimensional images. Perception 26, 1059--1072.
• 37: Melcher, D., and Kowler, E. (2001). Visual scene memory and the guidance of saccadic eye movements. Vision Research 41, 3597--3611.
• 38: Mizutani, A., Chahl, J. S., and Srinivasan, M. V. (2003). Motion camouflage in dragonflies. Nature 423, 604--604.
• 39: Najemnik, J., and Geisler, W. S. (2005). Optimal eye movement strategies in visual search. Nature 434, 387--391.
• 40: Oliva, A., Torralba, A., Castelhano, M. S., and Henderson, J. M. (2003). Top-down control of visual attention in object detection. Paper presented at: International Conference on Image Processing.
• 41: Parkhurst, D., Law, K., and Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research 42, 107--123.
• 42: Parkhurst, D. J., and Niebur, E. (2003). Scene content selected by active vision. Spatial Vision 16, 125--154.
• 43: Peters, R. J., Iyer, A., Itti, L., and Koch, C. (2005). Components of bottom-up gaze allocation in natural images. Vision Research 45, 2397--2416.
• 44: Polat, U., and Sagi, D. (1994). Spatial interactions in human vision: from near to far via experience-dependent cascades of connections. Proceedings of the National Academy of Sciences of the United States of America 91, 1206--1209.
• 45: Regan, B. C., Julliot, C., Simmen, B., Vienot, F., Charles-Dominique, P., and Mollon, J. D. (2001). Fruits, foliage and the evolution of primate colour vision, Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 356, 229--283.
• 46: Reinagel, P., and Zador, A. M. (1999). Natural scene statistics at the centre of gaze. Network 10, 341--350.
• 47: Sheinberg, D. L., and Logothetis, N. K. (2001). Noticing familiar objects in real world scenes: The role of temporal cortical neurons in natural vision. Journal of Neuroscience 21, 1340--1350.
• 48: Sillito, A. M., Grieve, K. L., Jones, H. E., Cudeiro, J., and Davis, J. (1995). Visual Cortical Mechanisms Detecting Focal Orientation Discontinuities. Nature 378, 492--496.
• 49: Sperling, G., and Weichselgartner, E. (1995). Episodic Theory of the Dynamics of Spatial Attention. Psychological Review 102, 503--532.
• 50: Tatler, B. W., Baddeley, R. J., and Gilchrist, I. D. (2005). Visual correlates of fixation selection: effects of scale and time. Vision Research 45, 643--659.
• 51: Theeuwes, J. (1994). Stimulus-Driven Capture and Attentional Set - Selective Search for Color and Visual Abrupt Onsets. Journal of Experimental Psychology: Human Perception and Performance 20, 799--806.
• 52: Thompson, K., and Bordwell, D. (2003). Film history: an introduction, 2nd edn (Boston, McGraw-Hill).
• 53: Torralba, A. (2003). Modeling global scene factors in attention. Journal of the Optical Society of America A. Optics, Image Science, and Vision 20, 1407--1418.
• 54: Treisman, A. M., and Gelade, G. (1980). Feature-Integration Theory of Attention. Cognitive Psychology 12, 97--136.
• 55: Wolfe, J. M., Alvarez, G. A., and Horowitz, T. S. (2000). Attention is fast but volition is slow. Nature 406, 691.
• 56: Yantis, S., and Egeth, H. E. (1999). On the distinction between visual salience and stimulus-driven attentional capture. Journal of Experimental Psychology: Human Perception and Performance 25, 661--676.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis

General Terms:
Algorithms, Design, Theory

Keywords:
attention, computational modeling, eye-movements, natural vision, saliency