SkyFinder: attribute-based sky image search

ABSTRACT

In this paper, we present SkyFinder, an interactive search system of over a half million sky images downloaded from the Internet. Using a set of automatically extracted, semantic sky attributes (category, layout, richness, horizon, etc.), the user can find a desired sky image, such as "a landscape with rich clouds at sunset" or "a whole blue sky with white clouds". The system is fully automatic and scalable. It computes all sky attributes offline, then provides an interactive online search engine. Moreover, we build a sky graph based on the sky attributes, so that the user can smoothly explore and find a path within the space of skies. We also show how our system can be used for controllable sky replacement.

REFERENCES

• 1: Dmitri Bitouk , Neeraj Kumar , Samreen Dhillon , Peter Belhumeur , Shree K. Nayar, Face swapping: automatically replacing faces in photographs, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 2: Boykov, Y., and Jolly, M. P. 2001. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. ICCVI, 105--112.
• 3: Dance, C., Willamowski, J., Fan, L., Bray, C., and Csurka, G. 2004. Visual categorization with bags of keypoint. In ECCV Workshop on Statistical Learning in Computer Vision.
• 4: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 5: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 6: James Hays , Alexei A. Efros, Scene completion using millions of photographs, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 7: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 8: Charles E. Jacobs , Adam Finkelstein , David H. Salesin, Fast multiresolution image querying, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.277-286, September 1995 [doi>10.1145/218380.218454]
• 9: Johnson, M., Brostow, G., Shotton, J., Arandjelovic, O., Kwatra, V., and Cipolla, R. 2006. Semantic photo synthesis. Computer Graphics Forum.
• 10: Neeraj Kumar , Peter Belhumeur , Shree Nayar, FaceTracer: A Search Engine for Large Collections of Images with Faces, Proceedings of the 10th European Conference on Computer Vision: Part IV, October 12-18, 2008, Marseille, France [doi>10.1007/978-3-540-88693-8_25]
• 11: Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In ICCV.
• 12: Jean-François Lalonde , Derek Hoiem , Alexei A. Efros , Carsten Rother , John Winn , Antonio Criminisi, Photo clip art, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 13: Jean-François Lalonde , Srinivasa G. Narasimhan , Alexei A. Efros, What Does the Sky Tell Us about the Camera?, Proceedings of the 10th European Conference on Computer Vision: Part IV, October 12-18, 2008, Marseille, France [doi>10.1007/978-3-540-88693-8_26]
• 14: Yin Li , Jian Sun , Chi-Keung Tang , Heung-Yeung Shum, Lazy snapping, ACM SIGGRAPH 2004 Papers, August 08-12, 2004, Los Angeles, California
• 15: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 16: Wojciech Matusik , Matthias Zwicker , Frédo Durand, Texture design using a simplicial complex of morphable textures, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 17: Moosmann, F., Triggs, B., and Jurie., F. 2006. Fast discriminative visual codebooks using randomized clustering forests. In NIPS.
• 18: Erik Reinhard , Michael Ashikhmin , Bruce Gooch , Peter Shirley, Color Transfer between Images, IEEE Computer Graphics and Applications, v.21 n.5, p.34-41, September 2001
• 19: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, A Metric for Distributions with Applications to Image Databases, Proceedings of the Sixth International Conference on Computer Vision, p.59, January 04-07, 1998
• 20: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 21: Noah Snavely , Rahul Garg , Steven M. Seitz , Richard Szeliski, Finding paths through the world's photos, ACM Transactions on Graphics (TOG), v.27 n.3, August 2008
• 22: Jessi Stumpfel , Chris Tchou , Andrew Jones , Tim Hawkins , Andreas Wenger , Paul Debevec, Direct HDR capture of the sun and sky, Proceedings of the 3rd international conference on Computer graphics, virtual reality, visualisation and interaction in Africa, November 03-05, 2004, Stellenbosch, South Africa [doi>10.1145/1029949.1029977]
• 23: Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, Inc., New York, NY, 1995

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  F. Theory of Computation
  F.2 ANALYSIS OF ALGORITHMS AND PROBLEM COMPLEXITY
      F.2.2 Nonnumerical Algorithms and Problems
          Subjects: Pattern matching

  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.7 Feature Measurement
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Algorithms, Design, Measurement