Emotionally aware automated portrait painting

ABSTRACT

We combine a machine vision system that recognises emotions and a non-photorealistic rendering (NPR) system to automatically produce portraits which heighten the emotion of the sitter. To do this, the vision system analyses a short video clip of a person expressing an emotion, then tracks the movement of facial features and uses this tracking data to analyse which emotion was expressed and what the temporal dynamics of the expression were. The image where the emotion is expressed strongest, the location of the facial features in that image and a keyword describing the emotion detected are passed to the NPR software. This keyword is used to choose appropriate (simulated) art materials, colour palettes, abstraction methods and painting styles, so that the rendered image may heighten the emotion being expressed. We describe the vision and rendering systems and their combination, and provide examples of portraits produced in this emotionally aware fashion.

REFERENCES

• 1: K. Anderson and P. McOwan. A real-time automated system for recognition of human facial expressions. IEEE Trans. Systems, Man and Cybernetics, Part B, 36(1):96--105, 2006.
• 2: M. Bartlett, G. Littlewort, C. Lainscsek, I. Fasel, and J. Movellan. Machine learning methods for fully automatic recognition of facial expressions and actions. IEEE Int'l Conf. Systems, Man, and Cybernetics, 1:592--597, 2004.
• 3: S. Brooks. Mixed media painting and portraiture. IEEE Transactions on Visualization and Computer Graphics, 13(5):1041--1054, 2007.
• 4: J. Collomosse and P. Hall. Salience-adaptive painterly rendering using genetic search. Intl. Journal on Artificial Intelligence Tools (IJAIT), 15(4):551--576, 2006.
• 5: S. Colton. Creativity versus the perception of creativity in computational systems. In Proceedings of the AAAI Spring Symposium on Creative Systems, 2008.
• 6: P. Ekman, W. V. Friesen, and J. C. Hager. Facial Action Coding System. A Human Face, 2002. Salt Lake City.
• 7: Beat Fasel , Florent Monay , Daniel Gatica-Perez, Latent semantic analysis of facial action codes for automatic facial expression recognition, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026742]
• 8: Ian Fasel , Bret Fortenberry , Javier Movellan, A generative framework for real time object detection and classification, Computer Vision and Image Understanding, v.98 n.1, p.182-210, April 2005 [doi>10.1016/j.cviu.2004.07.014]
• 9: J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: A statistical view of boosting. The Annals of Statistics, 28(2):337--374, 2000.
• 10: Takeo Kanade , Yingli Tian , Jeffrey F. Cohn, Comprehensive Database for Facial Expression Analysis, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.46, March 26-30, 2000
• 11: D. Keltner and P. Ekman. Handbook of Emotions. Guilford Press, 2004. New York.
• 12: I. Kotsia, I.; Pitas. Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans. Image Processing, 16(1):172--187, Jan. 2007.
• 13: M. Pantic and L. J. M. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370--1390, 2003.
• 14: M. Pantic, M. F. Valstar, R. Rademaker, and L. Maat. Web-based database for facial expression analysis. In Int'l IEEE Conf. Multimedia and Expo, pages 317--321, 2005.
• 15: I. Patras and M. Pantic. Particle filtering with factorized likelihoods for tracking facial features. Proc. Int'l Conf. Automatic Face & Gesture Recognition, pages 97--102, 2004.
• 16: I. Patras and M. Pantic. Tracking deformable motion. Proc. Int'l Conf. Systems, Man and Cybernetics, 2005.
• 17: J. Platt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schoelkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 61--74. MIT press, 2000. Cambridge, MA.
• 18: Maria Shugrina , Margrit Betke , John Collomosse, Empathic painting: interactive stylization through observed emotional state, Proceedings of the 4th international symposium on Non-photorealistic animation and rendering, June 05-07, 2006, Annecy, France [doi>10.1145/1124728.1124744]
• 19: Thomas Strothotte , Stefan Schlechtweg, Non-photorealistic computer graphics: modeling, rendering, and animation, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2002
• 20: Ying-li Tian , Takeo Kanade , Jeffrey F. Cohn, Recognizing Action Units for Facial Expression Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.2, p.97-115, February 2001 [doi>10.1109/34.908962]
• 21: Anil K. Jain , Stan Z. Li, Handbook of Face Recognition, Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 22: M. F. Valstar and M. Pantic. Biologically vs. logic inspired encoding of facial actions and emotions in video. IEEE Int'l. Conf. on Multimedia and Expo, pages 325--328, 2006.
• 23: M. F. Valstar and M. Pantic. Combined support vector machines and hidden markov models for modeling facial action temporal dynamics. In IEEE W'shop Human Computer Interaction (HCI'07), IEEE ICCV'07, 2007.
• 24: D. Vukdadinovic and M. Pantic. Fully automatic facial feature point detection using gabor feature based boosted features. In IEEE Int'l Conf. Systems, Man, and Cybernetics, pages 1692--1698, 2005.
• 25: Yongmian Zhang , Qiang Ji, Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.5, p.699-714, May 2005 [doi>10.1109/TPAMI.2005.93]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Keywords:
affective computing, artificial creativity, automatic facial expression recognition, computational creativity, emotion detection, machine vision, non-photorealistic rendering