Eye-movement-contingent release of speech and “SEE”- multi-modalities eye tracking system (poster session)

ABSTRACT

A novel research method, the eye-movement-contingent release of speech method, was developed to examine the use and nature of the sound code in visual word recognition in reading. A new multi-modalities eye tracking system (“SEE”) was developed to implement the eye-movement-contingent release of speech method. The SEE system provides the experimenter synchronized orthogonal manipulations of visual and auditory signals, accurate measurement of oculomotor responses, post-hoc data replay facilities and statistical analysis tool.

A new method, eye-movement-contingent release of speech, was developed to study the use of sound codes during visual word recognition. This new method examines effects of spoken -rather then visual/orthographic- words on the concurrent recognition of visual words. Speech is to be presented while text are viewed and presentation of the speech signal is coordinated with the viewing of a preselected visual target word. During sentence reading, the reader's eye positions are continuously sampled, and a real time fixation detection algorithm is developed to present the auditory signal when a certain pre-specified eye position criterion (called boundary) is fulfilled. The auditory stimuli are presented from eye tracking data, that is, eye movement are contingent on the change of heard/seen stimuli, so that the location of the eye position during visual object perception is calculated relative to the auditory presentation of speech.

“SEE”-multi-modalities eye tracking system was developed to implement this new method. SEE system was mainly built on a fifth-generation Dual-Purkinje SRI eye tracker but also provides general API supports for use with other eye trackers (i.e. ICAN and SM systems). SEE software toolkit was installed on an IBM compatible personal computer, which was connected with the eye tracker through an Analogue-to-Digital converter.

SEE software toolkit is a 32-bit windows application running under Windows 95/98/NT/2000. It consists of two running phases: real-time data collection phase and post-hoc analysis phase. Data collection phase includes modules of calibration, real time eye data processing and raw data recording. Real time data processing module has sub-modules to process the raw data to filter noise, detect fixation/saccade, and manually/dynamically recalibrate to correct drift over time. Post-hoc analysis phase provides replay and statistic analysis. SEE software toolkit also provides visual and audio stimuli presentation.

Calibration. During the calibration, the reader is instructed to fixate a consequential displayed series of 5-9 target points on the screen whose coordinates are known; the raw data from the eye tracker are collected for each point over a period of time and are used to calculate parameters of a mapping transformation equation. After successful calibration, the raw digital stream are continuously mapped to X/Y coordinates on the display.

Eye-movement-contingent control. Movements of the eyes control the release of visual and acoustic information. Visual and/or sound boundaries are pre-defined in the sentence for contingent control. When a reader's eye moves across the release boundary, a visual and/or speech signal is presented. In the following illustration table, visual/sound boundary are defined at space before `yellow' and `stripe' respectively. * indicates the fixation location; ( ) indicates the spoken duration.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES
      Subjects: Health

General Terms:
Design, Human Factors, Languages, Management, Performance, Theory