Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

What can you do with a million images? In this paper, we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless, but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks, we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data driven, requiring no annotations or labeling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of image completions and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Agrawal, A., Raskar, R., and Chellappa, R. What is the range of surface reconstructions from a gradient field? European Conference on Computer Vision (ECCV), May 2006.
• 3: Yuri Boykov , Olga Veksler , Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.11, p.1222-1239, November 2001 [doi>10.1109/34.969114]
• 4: Criminisi, A., Perez, P., and Toyama, K. Object removal by exemplar-based inpainting. CVPR, 02:721, 2003.
• 5: Diakopoulos, N., Essa, I., and Jain, R. Content based image synthesis. Conference on Image and Video Retrieval (CIVR), 2004.
• 6: Iddo Drori , Daniel Cohen-Or , Hezy Yeshurun, Fragment-based image completion, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 7: Alexei A. Efros , William T. Freeman, Image quilting for texture synthesis and transfer, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.341-346, August 2001 [doi>10.1145/383259.383296]
• 8: Alexei A. Efros , Thomas K. Leung, Texture Synthesis by Non-Parametric Sampling, Proceedings of the International Conference on Computer Vision-Volume 2, p.1033, September 20-25, 1999
• 9: James Hays , Alexei A. Efros, Scene completion using millions of photographs, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 10: Hays, J. and Efros, A. A. im2gps: estimating geographic information from a single image. CVPR, 2008.
• 11: Jiaya Jia , Jian Sun , Chi-Keung Tang , Heung-Yeung Shum, Drag-and-drop pasting, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 12: Johnson, M., Brostow, G. J., Shotton, J.: Arandjelovik, O., Kwatra, V., and Cipolla, R. Semantic photo synthesis. Comput. Graph. Forum (Proa Eurographics), 25(3):407--413, 2006.
• 13: King, D. The Commissar Vanishes. Henry Holt and Company, London, 1997.
• 14: Vivek Kwatra , Irfan Essa , Aaron Bobick , Nipun Kwatra, Texture optimization for example-based synthesis, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 15: Vivek Kwatra , Arno Schödl , Irfan Essa , Greg Turk , Aaron Bobick, Graphcut textures: image and video synthesis using graph cuts, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 16: Oliva, A. and Torralba, A. Building the gist of a scene: The role of global image features in recognition. Visual perception. Progr. Brain Res., 155:23--36, 2006.
• 17: Patrick Pérez , Michel Gangnet , Andrew Blake, Poisson image editing, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 18: Russell, B. C., Torralba, A., Murphy K. P., and Freeman, W. T. LabelMe: A database and web-based tool for image annotation. Technical Report, MIT, 2005, 2005.
• 19: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 20: Jian Sun , Lu Yuan , Jiaya Jia , Heung-Yeung Shum, Image completion with structure propagation, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 21: Torralba, A., Fergus, R., and Freeman, W. T. Tiny images. Technical Report MIT-CSAIL-TR-2007-024, 2007
• 22: Antonio Torralba , Kevin P. Murphy , William T. Freeman , Mark A. Rubin, Context-based vision system for place and object recognition, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.273, October 13-16, 2003
• 23: Wertheimer, M. Laws of organization in perceptual forms (partial translation). In: W. Ellis, editor, A Sourcebook of Gestatt Psychology, pages 71--88. Harcourt Brace and Company, London, 1938.
• 24: Wexler, Y., Shechtman, E., and Irani, M. Space-time video completion. CVPR. 01:120--127, 2004.
• 25: Wilczkowiak, M., Brostow, G. J., Tordoff, B., and Cipolla, R. Hole filling through photomontage. BMVC, 492--501, July 2005.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture

Additional Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.8 Applications
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.3 Enhancement
      I.4.7 Feature Measurement
      I.4.8 Scene Analysis
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology