Visio-lization: generating novel facial images

ABSTRACT

Our goal is to generate novel realistic images of faces using a model trained from real examples. This model consists of two components: First we consider face images as samples from a texture with spatially varying statistics and describe this texture with a local non-parametric model. Second, we learn a parametric global model of all of the pixel values. To generate realistic faces, we combine the strengths of both approaches and condition the local non-parametric model on the global parametric model. We demonstrate that with appropriate choice of local and global models it is possible to reliably generate new realistic face images that do not correspond to any individual in the training data. We extend the model to cope with considerable intra-class variation (pose and illumination). Finally, we apply our model to editing real facial images: we demonstrate image in-painting, interactive techniques for improving synthesized images and modifying facial expressions.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Michael Ashikhmin, Synthesizing natural textures, Proceedings of the 2001 symposium on Interactive 3D graphics, p.217-226, March 2001 [doi>10.1145/364338.364405]
• 3: Christopher M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 4: Dmitri Bitouk , Neeraj Kumar , Samreen Dhillon , Peter Belhumeur , Shree K. Nayar, Face swapping: automatically replacing faces in photographs, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 5: Volker Blanz , Thomas Vetter, A morphable model for the synthesis of 3D faces, Proceedings of the 26th annual conference on Computer graphics and interactive techniques, p.187-194, July 1999 [doi>10.1145/311535.311556]
• 6: Blanz, V., Albrecht, I., Haber, J., and Seidel, H.-P. 2006. Creating face models from vague mental images. Computer Graphics Forum 25, 3, 645--654.
• 7: Brand, M., and Pletscher, P. 2008. A conditional random field for photo editing. In Proceedings of CVPR, 187--194.
• 8: Christoph Bregler , Michele Covell , Malcolm Slaney, Video Rewrite: driving visual speech with audio, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.353-360, August 1997 [doi>10.1145/258734.258880]
• 9: Dedeoglu, G., Kanade, T., and August, J. 2004. Highzoom video hallucination by exploiting spatio-temporal regularities. In Proceedings of CVPR, 151--158.
• 10: Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood for incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39 (B), 1, 1--38.
• 11: Diakopoulos, N., Essa, I., and Jain, R. 2004. Content based image synthesis. In CIVR 04, 299--307.
• 12: Alexei A. Efros , William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.341-346, August 2001 [doi>10.1145/383259.383296]
• 13: Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
• 14: Tony Ezzat , Gadi Geiger , Tomaso Poggio, Trainable videorealistic speech animation, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 15: Ghahramani, Z., and Hinton, G. E. 1997. The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, Dept. of Computer Science, University of Toronto, Canada.
• 16: James Hays , Alexei A. Efros, Scene completion using millions of photographs, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 17: Vivek Kwatra , Arno Schödl , Irfan Essa , Greg Turk , Aaron Bobick, Graphcut textures: image and video synthesis using graph cuts, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 18: Jean-François Lalonde , Derek Hoiem , Alexei A. Efros , Carsten Rother , John Winn , Antonio Criminisi, Photo clip art, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 19: Wei Liu , Dahua Lin , Xiaoou Tang, Hallucinating Faces: TensorPatch Super-Resolution and Coupled Residue Compensation, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.478-484, June 20-26, 2005 [doi>10.1109/CVPR.2005.172]
• 20: Ce Liu , Heung-Yeung Shum , William T. Freeman, Face Hallucination: Theory and Practice, International Journal of Computer Vision, v.75 n.1, p.115-134, October 2007 [doi>10.1007/s11263-006-0029-5]
• 21: Messer, K., Matas, J., Kittler, J., Luettin, J., and Maitre, G. 1999. XM2VTSbd: The extended MTVTS database. In Proceedings 2nd Conference on Audio and Videobase Biometric Personal Verification (AVBPA99), 72--77.
• 22: Nguyen, M., Lalonde, J., Efros, A., and La Torre, F. D. 2008. Image-based shaving. Computer Graphics Forum (Eurographics) 27, 2, 627--635.
• 23: Patrick Pérez , Michel Gangnet , Andrew Blake, Poisson image editing, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 24: Simon J. D. Prince , James H. Elder , Jonathan Warrell , Fatima M. Felisberti, Tied Factor Analysis for Face Recognition across Large Pose Differences, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.6, p.970-984, June 2008 [doi>10.1109/TPAMI.2008.48]
• 25: Joshua B. Tenenbaum , William T. Freeman, Separating Style and Content with Bilinear Models, Neural Computation, v.12 n.6, p.1247-1283, June 2000 [doi>10.1162/089976600300015349]
• 26: Turk, M. A., and Pentland, A. P. 1991. Face recognition using eigenfaces. In Proceedings of CVPR, 586--591.
• 27: Daniel Vlasic , Matthew Brand , Hanspeter Pfister , Jovan Popović, Face transfer with multilinear models, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 28: Li-Yi Wei , Marc Levoy, Fast texture synthesis using tree-structured vector quantization, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.479-488, July 2000 [doi>10.1145/344779.345009]
• 29: Tim Weyrich , Wojciech Matusik , Hanspeter Pfister , Bernd Bickel , Craig Donner , Chien Tu , Janet McAndless , Jinho Lee , Addy Ngan , Henrik Wann Jensen , Markus Gross, Analysis of human faces using a measurement-based skin reflectance model, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.8 Applications

Keywords:
face, non-parametric sampling, texture synthesis