Image-based façade modeling

ABSTRACT

We propose in this paper a semi-automatic image-based approach to façade modeling that uses images captured along streets and relies on structure from motion to recover camera positions and point clouds automatically as the initial stage for modeling. We start by considering a building façade as a flat rectangular plane or a developable surface with an associated texture image composited from the multiple visible images. A façade is then decomposed and structured into a Directed Acyclic Graph of rectilinear elementary patches. The decomposition is carried out top-down by a recursive subdivision, and followed by a bottom-up merging with the detection of the architectural bilateral symmetry and repetitive patterns. Each subdivided patch of the flat façade is augmented with a depth optimized using the 3D points cloud. Our system also allows for an easy user feedback in the 2D image space for the proposed decomposition and augmentation. Finally, our approach is demonstrated on a large number of façades from a variety of street-side images.

REFERENCES

• 1: Aseem Agarwala , Maneesh Agrawala , Michael Cohen , David Salesin , Richard Szeliski, Photographing long scenes with multi-viewpoint panoramas, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 2: Berg, A. C., Grabler, F., and Malik, J. 2007. Parsing images of architectural scenes. In Proceedings of IEEE International Conference on Computer Vision, 1--8.
• 3: J Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.8 n.6, p.679-698, Nov. 1986 [doi>10.1109/TPAMI.1986.4767851]
• 4: Nico Cornelis , Bastian Leibe , Kurt Cornelis , Luc Gool, 3D Urban Scene Modeling Integrating Recognition and Reconstruction, International Journal of Computer Vision, v.78 n.2-3, p.121-141, July 2008 [doi>10.1007/s11263-007-0081-9]
• 5: Criminisi, A., Perez, P., and Toyama, K. 2003. Object removal by exemplar-based inpainting. In Proceedings of IEEE Computer Vision and Pattern Recognition, vol. 2, 721--728.
• 6: Paul E. Debevec , Camillo J. Taylor , Jitendra Malik, Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.11-20, August 1996 [doi>10.1145/237170.237191]
• 7: A. R. Dick , P. H. S. Torr , R. Cipolla, Modelling and Interpretation of Architecture from Several Images, International Journal of Computer Vision, v.60 n.2, p.111-134, November 2004 [doi>10.1023/B:VISI.0000029665.07652.61]
• 8: Richard O. Duda , Peter E. Hart, Use of the Hough transformation to detect lines and curves in pictures, Communications of the ACM, v.15 n.1, p.11-15, Jan. 1972 [doi>10.1145/361237.361242]
• 9: Früh, C., and Zakhor, A. 2003. Constructing 3d city models by merging ground-based and airborne views. In Proceedings of IEEE Computer Vision and Pattern Recognition, vol. 2, 562--569.
• 10: Furukawa, Y., and Ponce, J. 2007. Accurate, dense, and robust multi-view stereopsis. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, 1--8.
• 11: Geman, S., and Geman, D. 1984. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6, 6, 721--741.
• 12: Goesele, M., Snavely, N., Curless, B., Seitz, S. M., and Hoppe, H. 2007. Multi-view stereo for community photo collections. In Proceeding of IEEE International Conference in Computer Vision, 1--8.
• 13: Richard Hartley , Andrew Zisserman, Multiple view geometry in computer visiond, Cambridge University Press, New York, NY, 2001
• 14: Kullback, S., and Leibler, R. A. 1951. On information and sufficiency. Annals of Mathematical Statistics 22, 79--86.
• 15: Maxime Lhuillier , Long Quan, A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.3, p.418-433, March 2005 [doi>10.1109/TPAMI.2005.44]
• 16: Yin Li , Jian Sun , Chi-Keung Tang , Heung-Yeung Shum, Lazy snapping, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 17: Lukas, Z., Andreas, K., Barbara, G.-G., and Konrad, K. 2006. Towards 3d map generation from digital aerial images. International Journal of Photogrammetry and Remote Sensing 60, 413--427.
• 18: Lukas, Z., Joachim, B., Konrad, K., and Horst, B. 2008. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In Proceedings of the European Conference on Computer Vision.
• 19: Pascal Müller , Peter Wonka , Simon Haegler , Andreas Ulmer , Luc Van Gool, Procedural modeling of buildings, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 20: Pascal Müller , Gang Zeng , Peter Wonka , Luc Van Gool, Image-based procedural modeling of facades, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 21: Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. ACM Transactions on Graphics 1, 433--442.
• 22: M. Pollefeys , D. Nistér , J. -M. Frahm , A. Akbarzadeh , P. Mordohai , B. Clipp , C. Engels , D. Gallup , S. -J. Kim , P. Merrell , C. Salmi , S. Sinha , B. Talton , L. Wang , Q. Yang , H. Stewénius , R. Yang , G. Welch , H. Towles, Detailed Real-Time Urban 3D Reconstruction from Video, International Journal of Computer Vision, v.78 n.2-3, p.143-167, July 2008 [doi>10.1007/s11263-007-0086-4]
• 23: Daniel Scharstein , Richard Szeliski, A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, International Journal of Computer Vision, v.47 n.1-3, p.7-42, April-June 2002 [doi>10.1023/A:1014573219977]
• 24: Noah Snavely , Steven M. Seitz , Richard Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
• 25: Jian Sun , Lu Yuan , Jiaya Jia , Heung-Yeung Shum, Image completion with structure propagation, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 26: Anton van den Hengel , Anthony Dick , Thorsten Thormählen , Ben Ward , Philip H. S. Torr, VideoTrace: rapid interactive scene modelling from video, ACM Transactions on Graphics (TOG), v.26 n.3, July 2007
• 27: George Vogiatzis , Philip H. S. Torr , Steven M. Seitz , Roberto Cipolla, Reconstructing relief surfaces, Image and Vision Computing, v.26 n.3, p.397-404, March, 2008 [doi>10.1016/j.imavis.2007.01.006]
• 28: Weiss, Y., and Freeman, W. T. 2001. On the optimality of solutions of the max-product belief propagation algorithm in arbitrary graphs. IEEE Transactions on Information Theory 47, 2, 723--735.
• 29: Tomás Werner , Andrew Zisserman, New Techniques for Automated Architectural Reconstruction from Photographs, Proceedings of the 7th European Conference on Computer Vision-Part II, p.541-555, May 28-31, 2002
• 30: Peter Wonka , Michael Wimmer , François Sillion , William Ribarsky, Instant architecture, ACM Transactions on Graphics (TOG), v.22 n.3, July 2003

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.5 Computational Geometry and Object Modeling
          Subjects: Modeling packages

Additional Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.5 Reconstruction

Keywords:
building modeling, city modeling, façade modeling, image-based modeling, photography