Interactive 3D architectural modeling from unordered photo collections

ABSTRACT

We present an interactive system for generating photorealistic, textured, piecewise-planar 3D models of architectural structures and urban scenes from unordered sets of photographs. To reconstruct 3D geometry in our system, the user draws outlines overlaid on 2D photographs. The 3D structure is then automatically computed by combining the 2D interaction with the multi-view geometric information recovered by performing structure from motion analysis on the input photographs. We utilize vanishing point constraints at multiple stages during the reconstruction, which is particularly useful for architectural scenes where parallel lines are abundant. Our approach enables us to accurately model polygonal faces from 2D interactions in a single image. Our system also supports useful operations such as edge snapping and extrusions.

Seamless texture maps are automatically generated by combining multiple input photographs using graph cut optimization and Poisson blending. The user can add brush strokes as hints during the texture generation stage to remove artifacts caused by unmodeled geometric structures. We build models for a variety of architectural scenes from collections of up to about a hundred photographs.

REFERENCES

• 1: Aseem Agarwala , Mira Dontcheva , Maneesh Agrawala , Steven Drucker , Alex Colburn , Brian Curless , David Salesin , Michael Cohen, Interactive digital photomontage, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 2: Daniel G. Aliaga , Paul A. Rosen , Daniel R. Bekins, Style Grammars for Interactive Visualization of Architecture, IEEE Transactions on Visualization and Computer Graphics, v.13 n.4, p.786-797, July 2007 [doi>10.1109/TVCG.2007.1024]
• 3: Baillard, C., and Zisserman, A. 2001. Automatic reconstruction of piecewise planar models from multiple views. In CVPR, 559--565.
• 4: M. Brown , D. G. Lowe, Unsupervised 3D Object Recognition and Reconstruction in Unordered Datasets, Proceedings of the Fifth International Conference on 3-D Digital Imaging and Modeling, p.56-63, June 13-16, 2005 [doi>10.1109/3DIM.2005.81]
• 5: Roberto Cipolla , Duncan Robertson, 3D Models of Architectural Scenes from Uncalibrated Images and Vanishing Points, Proceedings of the 10th International Conference on Image Analysis and Processing, p.824, September 27-29, 1999
• 6: A. Criminisi , I. Reid , A. Zisserman, Single View Metrology, International Journal of Computer Vision, v.40 n.2, p.123-148, Nov. 2000 [doi>10.1023/A:1026598000963]
• 7: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 8: A. R. Dick , P. H. S. Torr , R. Cipolla, Modelling and Interpretation of Architecture from Several Images, International Journal of Computer Vision, v.60 n.2, p.111-134, November 2004 [doi>10.1023/B:VISI.0000029665.07652.61]
• 9: El-Hakim, S., Whiting, E., and Gonzo, L. 2005. 3d modeling with reusable and integrated building blocks. The 7th Conference on Optical 3-D Measurement Techniques.
• 10: Martin A. Fischler , Robert C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, v.24 n.6, p.381-395, June 1981 [doi>10.1145/358669.358692]
• 11: Gibson, S., Hubbold, R., Cook, J., and Howard, T. 2003. Interactive reconstruction of virtual environments from video sequences. Computers and Graphics 27, 2.
• 12: Goesele, M., Snavely, N., Curless, B., Hoppe, H., and Seitz, S. M. 2007. Multi-view stereo for community photo collections. In ICCV.
• 13: Derek Hoiem , Alexei A. Efros , Martial Hebert, Automatic photo pop-up, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 14: Takeo Igarashi , John F. Hughes, A suggestive interface for 3D drawing, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502379]
• 15: Seon Joo Kim , Marc Pollefeys, Robust Radiometric Calibration and Vignetting Correction, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.4, p.562-576, April 2008 [doi>10.1109/TPAMI.2007.70732]
• 16: Lempitsky, V. S., and Ivanov, D. V. 2007. Seamless mosaicing of image-based texture maps. In CVPR.
• 17: Pascal Müller , Gang Zeng , Peter Wonka , Luc Van Gool, Image-based procedural modeling of facades, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 18: Byong Mok Oh , Max Chen , Julie Dorsey , Frédo Durand, Image-based modeling and photo editing, Proceedings of the 28th annual conference on Computer graphics and interactive techniques, p.433-442, August 2001 [doi>10.1145/383259.383310]
• 19: Patrick Pérez , Michel Gangnet , Andrew Blake, Poisson image editing, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003
• 20: Marc Pollefeys , Luc Van Gool , Maarten Vergauwen , Frank Verbiest , Kurt Cornelis , Jan Tops , Reinhard Koch, Visual Modeling with a Hand-Held Camera, International Journal of Computer Vision, v.59 n.3, p.207-232, September-October 2004 [doi>10.1023/B:VISI.0000025798.50602.3a]
• 21: M. Pollefeys , D. Nistér , J. -M. Frahm , A. Akbarzadeh , P. Mordohai , B. Clipp , C. Engels , D. Gallup , S. -J. Kim , P. Merrell , C. Salmi , S. Sinha , B. Talton , L. Wang , Q. Yang , H. Stewénius , R. Yang , G. Welch , H. Towles, Detailed Real-Time Urban 3D Reconstruction from Video, International Journal of Computer Vision, v.78 n.2-3, p.143-167, July 2008 [doi>10.1007/s11263-007-0086-4]
• 22: Rother, C. 2002. A new approach for vanishing point detection in architectural environments. In BMVC, vol. 20, 382--391.
• 23: Sketchup. http://www.sketchup.com.
• 24: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 25: Bill Triggs , Philip F. McLauchlan , Richard I. Hartley , Andrew W. Fitzgibbon, Bundle Adjustment - A Modern Synthesis, Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, p.298-372, September 21-22, 1999
• 26: van den Hengel, A., Dick, A., Thormahlen, T., Torr, P. H. S., and B. Ward. 2006. Fitting multiple models to multiple images with minimal user interaction. In the Intl. Workshop on the Representation and use of Prior Knowledge in Vision.
• 27: Anton van den Hengel , Anthony Dick , Thorsten Thormählen , Ben Ward , Philip H. S. Torr, VideoTrace: rapid interactive scene modelling from video, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 28: Tomás Werner , Andrew Zisserman, New Techniques for Automated Architectural Reconstruction from Photographs, Proceedings of the 7th European Conference on Computer Vision-Part II, p.541-555, May 28-31, 2002
• 29: Marta Wilczkowiak , Peter Sturm , Edmond Boyer, Using Geometric Constraints through Parallelepipeds for Calibration and 3D Modeling, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.2, p.194-207, February 2005 [doi>10.1109/TPAMI.2005.40]
• 30: Robert C. Zeleznik , Kenneth P. Herndon , John F. Hughes, SKETCH: an interface for sketching 3D scenes, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.163-170, August 1996 [doi>10.1145/237170.237238]
• 31: Remo Ziegler , Wojciech Matusik , Hanspeter Pfister , Leonard McMillan, 3D reconstruction using labeled image regions, Proceedings of the 2003 Eurographics/ACM SIGGRAPH symposium on Geometry processing, June 23-25, 2003, Aachen, Germany

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.7 Three-Dimensional Graphics and Realism

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding