Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder

ABSTRACT

Many first-hand accounts from individuals diagnosed with autism spectrum disorders (ASD) highlight the challenges inherent in processing high-speed, complex, and unpredictable social information such as facial expressions in real-time. In this paper, we describe a new technology aimed at helping people capture, analyze, and reflect on a set of social-emotional signals communicated by facial and head movements in live social interaction that occurs with their everyday social companions. We describe our development of a new combination of hardware using a miniature camera connected to an ultramobile PC together with custom software developed to track, capture, interpret, and intuitively present various interpretations of the facial-head movements (e.g., presenting that there is a high probability the person looks "confused"). This paper describes this new technology together with the results of a series of pilot studies conducted with adolescents diagnosed with ASD who used the technology in their peer-group setting and contributed to its development via their feedback.

REFERENCES

• 1: American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders DSM-IV-TR. 2000.
• 2: National Institute of Neurological Disorders and Stroke. Autistic Disorder Fact Sheet. December 12, 2007 {cited 2008 March 5}; Available from: http://www.ninds.nih.gov/disorders/autism/detail_autism.htm.
• 3: National Institute of Neurological Disorders and Stroke. Asperger's Syndrome Fact Sheet. December 12, 2007 {cited 2008 March 5}; Available from: http://www.ninds.nih.gov/disorders/asperger/detail_asperger.htm.
• 4: National Institute of Neurological Disorders and Stroke. Autism Spectrum Disorders (Pervasive Developmental Disorders.). {cited 2008 March 5}; Available from: http://www.nimh.nih.gov/health/publications/autism/complete-publication.shtml.
• 5: Bernard-Opitz, V., N. Sriram, and S. Nakhoda-Sapuan, Enhancing social problem solving in children with autism and normal children through computer-assisted instruction. Journal of Autism and Developmental Disorders, 2001. 31: p. 377--398.
• 6: Swettenham, J., Can children with autism be taught to understand false belief using computers? Journal of Child Psychology and Psychiatry, 1996. 37: p. 157--165.
• 7: Bolte, S., et al., The development and evaluation of a computer-based program to test and to teach the recognition of facial affect. International Journal of Circumpolar Health, 2002. 61(2): p. 61--68.
• 8: Howlin, P., S. Baron-Cohen, and J. Hadwin, Teaching Children with Autism to Mind-read: A Practical Guide for Teachers and Parents. 1999: John Wiley and Sons.
• 9: Rutherford, M.D., S. Baron-Cohen, and S. Wheelwright, Reading the mind in the voice: A study with normal adults and adults with Asperger syndrome and high functioning autism. Journal Autism and Developmental Disorders, 2002. 32: p. 189--194.
• 10: Baron-Cohen, S., et al., Mind Reading: The Interactive Guide to Emotions. 2004, Jessica Kingsley Publishers: London.
• 11: el Kaliouby, R. and P. Robinson, Real-time Inference of Complex Mental States from Facial Expressions and Head Gestures, in Real-Time Vision for Human-Computer Interaction. 2005, Springer-Verlag. p. 181--200.
• 12: Zacks, J.M., B. Tversky, and G. Iyer, Perceiving, Remembering, and Communicating Structure in Events. Journal of Experimental Psychology, 2001. 130(1): p. 29--58.
• 13: Ekman, P. and W.V. Friesen, Facial Action Coding System. 1977: Consulting Psychologist Press.
• 14: Baron-Cohen, S., The hyper-systemizing, assortative mating theory of autism. Progress in Neuro-Psychopharmacology & Biological Psychiatry, 2006. 30: p. 865--872.
• 15: Mottron, L., et.al., Enhanced Perceptual Functioning in Autism. Journal of Autism and Developmental Disorders, 2006. 30 (1): p. 27--43.

INDEX TERMS

Primary Classification:
  J. Computer Applications
  J.4 SOCIAL AND BEHAVIORAL SCIENCES
      Subjects: Psychology

General Terms:
Design, Experimentation, Human Factors

Keywords:
affective computing, asperger's syndrome, autism spectrum disorders, facial affect, facial expressions