Face swapping: automatically replacing faces in photographs

ABSTRACT

In this paper, we present a complete system for automatic face replacement in images. Our system uses a large library of face images created automatically by downloading images from the internet, extracting faces using face detection software, and aligning each extracted face to a common coordinate system. This library is constructed off-line, once, and can be efficiently accessed during face replacement. Our replacement algorithm has three main stages. First, given an input image, we detect all faces that are present, align them to the coordinate system used by our face library, and select candidate face images from our face library that are similar to the input face in appearance and pose. Second, we adjust the pose, lighting, and color of the candidate face images to match the appearance of those in the input image, and seamlessly blend in the results. Third, we rank the blended candidate replacements by computing a match distance over the overlap region. Our approach requires no 3D model, is fully automatic, and generates highly plausible results across a wide range of skin tones, lighting conditions, and viewpoints. We show how our approach can be used for a variety of applications including face de-identification and the creation of appealing group photographs from a set of images. We conclude with a user study that validates the high quality of our replacement results, and a discussion on the current limitations of our system.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Shai Avidan , Ariel Shamir, Seam carving for content-aware image resizing, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 3: Ronen Basri , David W. Jacobs, Lambertian Reflectance and Linear Subspaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.2, p.218-233, February 2003 [doi>10.1109/TPAMI.2003.1177153]
• 4: Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging Faces in Images. Computer Graphics Forum 23, 669--676.
• 5: Michael Boyle , Christopher Edwards , Saul Greenberg, The effects of filtered video on awareness and privacy, Proceedings of the 2000 ACM conference on Computer supported cooperative work, p.1-10, December 2000, Philadelphia, Pennsylvania, United States [doi>10.1145/358916.358935]
• 6: Timothy F. Cootes , Gareth J. Edwards , Christopher J. Taylor, Active Appearance Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.681-685, June 2001 [doi>10.1109/34.927467]
• 7: Paul Debevec , Tim Hawkins , Chris Tchou , Haarm-Pieter Duiker , Westley Sarokin , Mark Sagar, Acquiring the reflectance field of a human face, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.145-156, July 2000 [doi>10.1145/344779.344855]
• 8: Paul Debevec, Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, p.189-198, July 1998 [doi>10.1145/280814.280864]
• 9: Alexei A. Efros , William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.341-346, August 2001 [doi>10.1145/383259.383296]
• 10: Rob Fergus , Barun Singh , Aaron Hertzmann , Sam T. Roweis , William T. Freeman, Removing camera shake from a single photograph, ACM SIGGRAPH 2006 Papers, July 30-August 03, 2006, Boston, Massachusetts
• 11: Gross, R., Sweeney, L., de la Torre, F., and Baker, S. 2006. Model-Based Face De-Identification. 161--168.
• 12: James Hays , Alexei A. Efros, Scene completion using millions of photographs, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 13: Kundur, D., and Hatzinakos, D. 1996. Blind image deconvolution. IEEE Signal Processing Magazine, 3, 43--64.
• 14: Lanitis, I., Draganova, C., and Christodoulou, C. 2004. Comparing different classifiers for automatic age estimation. IEEE Trans. on Systems, Man, and Cybernetics, B 34, 621--628.
• 15: Zicheng Liu , Ying Shan , Zhengyou Zhang, Expressive expression mapping with ratio images, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.271-276, August 2001 [doi>10.1145/383259.383289]
• 16: Malik, S., 2003. Digital face replacement in photographs. http://www.cs.toronto.edu/~smalik/2530/project/results.html.
• 17: Baback Moghaddam , Ming-Hsuan Yang, Learning Gender with Support Faces, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.707-711, May 2002 [doi>10.1109/34.1000244]
• 18: Elaine M. Newton , Latanya Sweeney , Bradley Malin, Preserving Privacy by De-Identifying Face Images, IEEE Transactions on Knowledge and Data Engineering, v.17 n.2, p.232-243, February 2005 [doi>10.1109/TKDE.2005.32]
• 19: Omron, 2007. OKAO vision. http://omron.com/rd/vision/01.html.
• 20: Ravi Ramamoorthi , Pat Hanrahan, An efficient representation for irradiance environment maps, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.497-500, August 2001 [doi>10.1145/383259.383317]
• 21: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, The Earth Mover's Distance as a Metric for Image Retrieval, International Journal of Computer Vision, v.40 n.2, p.99-121, Nov. 2000 [doi>10.1023/A:1026543900054]
• 22: Hongcheng Wang , Ning Xu , Ramesh Raskar , Narendra Ahuja, Videoshop: A new framework for spatio-temporal video editing in gradient domain, Graphical Models, v.69 n.1, p.57-70, January, 2007 [doi>10.1016/j.gmod.2006.06.002]
• 23: Wang, Y., Liu, Z., Hua, G., Wen, Z., Zhang, Z., and Samaras, D. 2007. Face re-lighting from a single image under harsh lighting conditions. CVPR '07.
• 24: Wen, Z., Liu, Z., and Huang., T. S. 2003. Face Relighting with Radiance Environment Maps. In CVPR '03, 158--165.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.9 Applications

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process; Search process

Keywords:
computational photography, face replacement, image databases, image-based rendering