Outdoors augmented reality on mobile phone using loxel-based visual feature organization

ABSTRACT

We have built an outdoors augmented reality system for mobile phones that matches camera-phone images against a large database of location-tagged images using a robust image retrieval algorithm. We avoid network latency by implementing the algorithm on the phone and deliver excellent performance by adapting a state-of-the-art image retrieval algorithm based on robust local descriptors. Matching is performed against a database of highly relevant features, which is continuously updated to reflect changes in the environment. We achieve fast updates and scalability by pruning of irrelevant features based on proximity to the user. By compressing and incrementally updating the features stored on the phone we make the system amenable to low-bandwidth wireless connections. We demonstrate system robustness on a dataset of location-tagged images and show a smart-phone implementation that achieves a high image matching rate while operating in near real-time.

REFERENCES

• 1: Ritendra Datta , Jia Li , James Z. Wang, Content-based image retrieval: approaches and trends of the new age, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101866]
• 2: R. Fergus, P. Perona, and A. Zisserman, "Object class recognition by unsupervised scale-invariant learning," in Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), vol. 02. IEEE Computer Society, 2003, p. 264.
• 3: A. Ferencz, E. Learned-Miller, and J. Malik, "Learning Hyper-Features for Visual Identification," in Neural Information Processing Systems, vol. 18, 2004.
• 4: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 5: Y. Zhou, X. Fan, X. Xie, Y. Gong, and W.-Y. Ma, "Inquiring of the Sights from the Web via Camera Mobiles," ph2006 IEEE International Conference on Multimedia and Expo, pp. 661--664, July 2006.
• 6: Krystian Mikolajczyk , Cordelia Schmid, A Performance Evaluation of Local Descriptors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.10, p.1615-1630, October 2005 [doi>10.1109/TPAMI.2005.188]
• 7: H. Bay, T. Tuytelaars, and L. V. Gool, "SURF: Speeded Up Robust Features," in ECCV (1), 2006, pp. 404--417.
• 8: J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, "Object Retrieval with Large Vocabularies and Fast Spatial Matching," in Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
• 9: K. Greene, "Hyperlinking Reality via Phones," MIT Technology Review, Nov/Dec 2006.
• 10: E. Bruns and O. Bimber, "Adaptive Training of Video Sets for Image Recognition on Mobile Phones," Journal of Personal and Ubiquitous Computing, 2008, to appear.
• 11: C. Seifert, L. Paletta, A. Jeitler, E. Hödl, J.-P. Andreu, P. M. Luley, and A. Almer, "Visual Object Detection for Mobile Road Sign Inventory," in Proc. of Mobile Human-Computer Interaction - Mobile HCI 2004, 6th International Symposium, 2004, pp. 491--495.
• 12: Gerald Fritz , Christin Seifert , Lucas Paletta, A Mobile Vision System for Urban Detection with Informative Local Descriptors, Proceedings of the Fourth IEEE International Conference on Computer Vision Systems, p.30, January 04-07, 2006 [doi>10.1109/ICVS.2006.5]
• 13: H. Bay, B. Fasel, and L. V. Gool, "Interactive Museum Guide: Fast and Robust Recognition of Museum Objects," in Proceedings of the First International Workshop on Mobile Vision, May 2006.
• 14: Iryna Skrypnyk , David G. Lowe, Scene Modelling, Recognition and Tracking with Invariant Image Features, Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.110-119, November 02-05, 2004 [doi>10.1109/ISMAR.2004.53]
• 15: G. Takacs, V. Chandrasekhar, B. Girod, and R. Grzeszczuk, "Feature Tracking for Mobile Augmented Reality Using Video Coder Motion Vectors," in ISMAR '07: Proceedings of the Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'07), 2007.
• 16: T. Yeh, K. Tollmar, and T. Darrell, "Searching the Web with Mobile Images for Location Recognition," in Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2. IEEE Computer Society, 2004, pp. 76--81.
• 17: Sunil Arya , David M. Mount , Nathan S. Netanyahu , Ruth Silverman , Angela Y. Wu, An optimal algorithm for approximate nearest neighbor searching fixed dimensions, Journal of the ACM (JACM), v.45 n.6, p.891-923, Nov. 1998 [doi>10.1145/293347.293348]
• 18: L. V. G. H. Shao, T. Svoboda, "Zubud-Zürich buildings database for image based recognition." ETH Zürich, Tech. Rep. 260, 2003.
• 19: Y. Ke and R. Sukthankar, "PCA-SIFT: A More Distinctive Representation for Local Image Descriptors," in Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), vol. IEEE Computer Society, 2004, pp. 506--513.
• 20: W.-C. Chen, Y. Xiong, J. Gao, N. Gelfand, and R. Grzeszczuk, "Efficient Extraction of Robust Image Features on Mobile Devices," in ISMAR '07: Proceedings of the Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'07), 2007.
• 21: P. Viola and M. J. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. of Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2001, pp. I:511--518.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Design, Performance

Keywords:
augmented reality, geo-referenced photos, image search, local search, mobile computing, photo collections