PhoneGuide: museum guidance supported by on-device object recognition on mobile phones

ABSTRACT

We present PhoneGuide -- an enhanced museum guidance system that uses camera-equipped mobile phones and on-device object recognition.Our main technical achievement is a simple and light-weight object recognition approach that is realized with single-layer perceptron neuronal networks. In contrast to related systems which perform computationally intensive image processing tasks on remote servers, our intention is to carry out all computations directly on the phone. This ensures little or even no network traffic and consequently decreases cost for online times. Our laboratory experiments and field surveys have shown that photographed museum exhibits can be recognized with a probability of over 90%.We have evaluated different feature sets to optimize the recognition rate and performance. Our experiments revealed that normalized color features are most effective for our method. Choosing such a feature set allows recognizing an object below one second on up-to-date phones. The amount of data that is required for differentiating 50 objects from multiple perspectives is less than 6KBytes.

REFERENCES

• 1: {And03} Andreasson, H. and Duckett, T., "Object Recognition by a Mobile Robot using Omni-directional Vision", Proceedings of the Eighth Scandinavian Conference on Artificial Intelligence, Norway, 2003.
• 2: Paul M. Aoki , Allison Woodruff, Improving electronic guidebook interfaces using a task-oriented design approach, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.319-325, August 17-19, 2000, New York City, New York, United States [doi>10.1145/347642.347779]
• 3: {Ass03} Assad, M., Carmichael, D. J., Cutting, D., and Hudson, A., "AR phone: Accessible Augmented Reality in the Intelligent Environment", In OZCHI'03, pp. 232--235, 2003.
• 4: {Bom03} Bombara, M, Cali, D., and Santoro, C, "KORE: A Multi-Agent System to Assist Museum Visitors", Joint Workshop "From Objects to Agents": Intelligent Systems and Pervasive Computing, pp. 175--178, 2003.
• 5: {Bon04} University of Bonn, "Das Fotohandy als Fremdenführer", retrieved from WWW, http://www.ipb.uni-bonn.de/FotoNav/andhttp://www.geosciencenline.de/index.php?cmd=wissen_details&id=1713&datum=2004-10-12, 2004.
• 6: {Cla05} C-Lab, "Kickreal", retrieved from WWW, www.c-lab.de or www.kickreal.de, 2005.
• 7: {Cor00} Coors, V., Huch, T., and Kretschmar, U., "Matching Buildings: Pose Estimation in an Urban Environment", Proc. ISMAR'00, Munich, Germany, pp. 89--92, 2000.
• 8: Steven Feiner , Blair MacIntyre , Tobias Hollerer , Anthony Webster, A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment, Proceedings of the 1st IEEE International Symposium on Wearable Computers, p.74, October 13-14, 1997
• 9: {Fri04a} Fritz, G., Seifert, C., Paletta, L., and Bischof H., "Rapid Object Recognition from Discriminative Regions of Interest", Proc. National Conference on Artificial Intelligence, AAAI, San Rose, CA, 2004.
• 10: {Fri04b} Fritz, G., Seifert, C, Luley, P., Paletta, L., and Almer, A., "Mobile Vision for Ambient Learning in Urban Enviroments", In MLEARN 2004, Lake Bracciano, Rome, July 2004.
• 11: {Har88} Harris, C. and Stephens, M., "A combined corner and edge detector", In Alvey Vision Conference, pages 147--151, 1988.
• 12: {Hel04} Helmer, S. and Lowe, D. G., "Object Class Recognition with Many Local Features", In GMBV 2004, Washington, D.C., July 2004.
• 13: {Iqb02} Iqbal, Q. and Aggarwal, J. K., "CIRES: A System for content-based retrieval in digital image libraries", In ICARCV'02, 2002.
• 14: {Leh00} Lehmann, T. M., Wein, B. B., Dahmen, J., Bredno, J., Vogelsang, F., and Kohnen, M., "Content-Based Image Retrieval in Medical Applications": A Novel Multi-Step Approach, Proceedings SPIE'00, vol. 3972, pp. 312--320, 2002.
• 15: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 16: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 17: Michael Macedonia, Small is Beautiful, Computer, v.37 n.12, p.122-123, December 2004 [doi>10.1109/MC.2004.256]
• 18: K. Mikolajczyk , C. Schmid, An Affine Invariant Interest Point Detector, Proceedings of the 7th European Conference on Computer Vision-Part I, p.128-142, May 28-31, 2002
• 19: {MIT04} MIT's Technology Review, "Markets and Trends", p. 16, February 2004.
• 20: {Mnh04} Museum of Natural History, "Museum Puts Tags on Stuffed Birds", retrieved from WWW, http://rfidjournal.com/article/articleview/1110/1/1/, 2004.
• 21: {Moe04} Moehring, M., Lessig, C, and Bimber, O., "Optical Tracking and Video See-Through AR on Consumer Cell Phones", In proceedings of Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR, pp. 193--204, 2004.
• 22: {Pol03} Porikli, F. M., "Inter-Camera Color Calibration by Cross-Correlation Model Function", IEEE International Conference on Image Processing (ICIP), Vol. 2, pp. 133--136, September 2003.
• 23: Bernt Schiele , James L. Crowley, Object Recognition Using Multidimensional Receptive Field Histograms, Proceedings of the 4th European Conference on Computer Vision-Volume I, p.610-619, April 15-18, 1996
• 24: Cordelia Schmid , Roger Mohr, Local Grayvalue Invariants for Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.5, p.530-535, May 1997 [doi>10.1109/34.589215]
• 25: {Sei04} Seifert, C, Paletta, L., Jeitler, A., Hoedl, E., Andreu, J. P., Luley, P., and Almer A., "Visual Object Detection for Mobile Road Sign Inventory", In: Brewster S. and Dunlop M., (Eds.): Mobile HCI, LNCS 3160, pp. 491--495, Springer Verlag Berlin, 2004.
• 26: {Sem04} Semacode Cooperation, "Semacode", retrieved from WWW, http://www.semacode.org, 2004.
• 27: Michael J. Swain , Dana H. Ballard, Color indexing, International Journal of Computer Vision, v.7 n.1, p.11-32, Nov. 1991 [doi>10.1007/BF00130487]
• 28: Daniel Wagner , Dieter Schmalstieg, First Steps Towards Handheld Augmented Reality, Proceedings of the 7th IEEE International Symposium on Wearable Computers, p.127, October 21-23, 2003

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding

Keywords:
mobile phones, museum guidance, neural networks, object recognition