iScope: personalized multi-modality image search for mobile devices

ABSTRACT

Mobile devices are becoming a primary medium for personal information gathering, management, and sharing. Managing personal image data on mobile platforms is a difficult problem due to large data set size, content diversity, heterogeneous individual usage patterns, and resource constraints. This article presents a user-centric system, called iScope, for personal image management and sharing on mobile devices. iScope uses multi-modality clustering of both content and context information for efficient image management and search, and online learning techniques for predicting images of interest. It also supports distributed content-based search among networked devices while maintaining the same intuitive interface, enabling efficient information sharing among people. We have implemented iScope and conducted in-field experiments using networked Nokia N810 portable Internet tablets. Energy efficiency was a primary design focus during the design and implementation of the iScope search algorithms. Experimental results indicate that iScope improves search time and search energy by 4.1X and 3.8X on average, relative to browsing.

REFERENCES

• 1: "American museum," http://americanhistory.si.edu/.
• 2: "Wikipedia," http://en.wikipedia.org/wiki/Philippe Kahn/.
• 3: "Global mobile forecast to 2012," in Informa Telecomms&Media Report, November 2007.
• 4: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 5: Charles E. Jacobs , Adam Finkelstein , David H. Salesin, Fast multiresolution image querying, Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, p.277-286, September 1995 [doi>10.1145/218380.218454]
• 6: Qin Lv , Moses Charikar , Kai Li, Image similarity search with compact data structures, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA [doi>10.1145/1031171.1031213]
• 7: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 8: Paul Alexandru Chirita , Andrei Damian , Wolfgang Nejdl , Wolf Siberski, Search strategies for scientific collaboration networks, Proceedings of the 2005 ACM workshop on Information retrieval in peer-to-peer networks, November 04-04, 2005, Bremen, Germany [doi>10.1145/1096952.1096959]
• 9: Mukesh Dalal, Personalized social & real-time collaborative search, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242810]
• 10: Shenghua Bao , Guirong Xue , Xiaoyuan Wu , Yong Yu , Ben Fei , Zhong Su, Optimizing web search using social annotations, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242640]
• 11: Beth E. Kolko , Emma J. Rose , Erica J. Johnson, Communication as information-seeking: the case for mobile social software for developing regions, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242689]
• 12: Paul Heymann , Georgia Koutrika , Hector Garcia-Molina, Can social bookmarking improve web search?, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341558]
• 13: SA. Joels, "RFID security and privacy: A research survey," IEEE Journal on Selected Areas in Communications, vol. 24, no. 2, pp. 381--393, Feb. 2006.
• 14: M. J. Beller, L.-F. Chang, and Y. Yacobi, "Privacy and authentication on a portable communications system," IEEE Journal on Selected Areas in Communications, vol. 11, no. 6, pp. 821--829, Aug. 1993.
• 15: Faith M. Heikkila, Encryption: Security Considerations for Portable Media Devices, IEEE Security and Privacy, v.5 n.4, p.22-27, July 2007 [doi>10.1109/MSP.2007.80]
• 16: Adrian Perrig , John Stankovic , David Wagner, Security in wireless sensor networks, Communications of the ACM, v.47 n.6, June 2004 [doi>10.1145/990680.990707]
• 17: E. Berry, et al., "The use of a wearable camera, sensecam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report."
• 18: Chan Young Kim , Jae Kyu Lee , Yoon Ho Cho , Deok Hwan Kim, VISCORS: A Visual-Content Recommender for the Mobile Web, IEEE Intelligent Systems, v.19 n.6, p.32-39, November 2004 [doi>10.1109/MIS.2004.75]
• 19: Tom Yeh , Kristen Grauman , Konrad Tollmar , Trevor Darrell, A picture is worth a thousand keywords: image-based object search on a mobile platform, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057083]
• 20: I. Ahmad, et al., "Content-based image retrieval on mobile devices," in Proceedings of SPIE (Multimedia on Mobile Devices), vol. 5684, San Jose, CA, USA, January 2005.
• 21: Suckchul Kim , Yoonsik Tak , Yunyoung Nam , Eenjun Hwang, mCLOVER: mobile content-based leaf image retrieval system, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101183]
• 22: Menglei Jia , Xin Fan , Xing Xie , Mingjing Li , Wei-Ying Ma, Photo-to-Search: Using Camera Phones to Inquire of the Surrounding World, Proceedings of the 7th International Conference on Mobile Data Management, p.46, May 10-12, 2006 [doi>10.1109/MDM.2006.122]
• 23: James Fogarty , Desney Tan , Ashish Kapoor , Simon Winder, CueFlik: interactive concept learning in image search, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357061]
• 24: Giovanni Maria Sacco, Research Results in Dynamic Taxonomy and Faceted Search Systems, Proceedings of the 18th International Conference on Database and Expert Systems Applications, p.201-206, September 03-07, 2007 [doi>10.1109/DEXA.2007.122]
• 25: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 26: T. Yeh, K. Tollmar, and T. Darrell, "Searching the web with mobile images for location recognition," in Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, July 2004, pp. 76--81.
• 27: Deng Cai , Xiaofei He , Zhiwei Li , Wei-Ying Ma , Ji-Rong Wen, Hierarchical clustering of WWW image search results using visual, textual and link information, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027747]
• 28: Neil O'Hare , Cathal Gurrin , Gareth J. F. Jones , Hyowon Lee , Noel E. O'Connor , Alan F. Smeaton, Using text search for personal photo collections with the MediAssist system, Proceedings of the 2007 ACM symposium on Applied computing, March 11-15, 2007, Seoul, Korea [doi>10.1145/1244002.1244195]
• 29: Huan Wang , Liang-Tien Chia , Song Liu, Image retrieval ++--web image retrieval with an enhanced multi-modality ontology, Multimedia Tools and Applications, v.39 n.2, p.189-215, September 2008 [doi>10.1007/s11042-008-0202-7]
• 30: Kaushik Veeraraghavan , Edmund B. Nightingale , Jason Flinn , Brian Noble, quFiles: a unifying abstraction for mobile data management, Proceedings of the 9th workshop on Mobile computing systems and applications, February 25-26, 2008, Napa Valley, California [doi>10.1145/1411759.1411777]
• 31: Timothy Sohn , William G. Griswold , James Scott , Anthony LaMarca , Yatin Chawathe , Ian Smith , Mike Chen, Experiences with place lab: an open source toolkit for location-aware computing, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134351]
• 32: Tzvetan Horozov , Nitya Narasimhan , Venu Vasudevan, Using Location for Personalized POI Recommendations in Mobile Environments, Proceedings of the International Symposium on Applications on Internet, p.124-129, January 23-27, 2006 [doi>10.1109/SAINT.2006.55]
• 33: Cathal Gurrin , Gareth J. F. Jones , Hyowon Lee , Neil O'Hare , Alan F. Smeaton , Noel Murphy, Mobile access to personal digital photograph archives, Proceedings of the 7th international conference on Human computer interaction with mobile devices & services, September 19-22, 2005, Salzburg, Austria [doi>10.1145/1085777.1085842]
• 34: Rui Jesus , Ricardo Dias , Rute Frias , Nuno Correia, Geographic image retrieval in mobile guides, Proceedings of the 4th ACM workshop on Geographical information retrieval, November 09-09, 2007, Lisbon, Portugal [doi>10.1145/1316948.1316958]
• 35: M. Kahari and D. J. Murphy, "MARA, sensor based augmented reality system for mobile imaging device," in ISMAR, 2006.
• 36: D. Byrne, et al., "Using bluetooth&GPS metadata to measure event similarity in sensecam images," 2007.
• 37: Xavier Anguera , JieJun Xu , Nuria Oliver, Multimodal photo annotation and retrieval on a mobile phone, Proceeding of the 1st ACM international conference on Multimedia information retrieval, October 30-31, 2008, Vancouver, British Columbia, Canada [doi>10.1145/1460096.1460127]
• 38: Srijan Chakraborty , Yu Dong , David K. Y. Yau , John C. S. Lui, On the Effectiveness of Movement Prediction to Reduce Energy Consumption in Wireless Communication, IEEE Transactions on Mobile Computing, v.5 n.2, p.157-169, February 2006 [doi>10.1109/TMC.2006.24]
• 39: Jari Korhonen , Ye Wang, Power-efficient streaming for mobile terminals, Proceedings of the international workshop on Network and operating systems support for digital audio and video, June 13-14, 2005, Stevenson, Washington, USA [doi>10.1145/1065983.1065994]
• 40: Thomas Karagiannis , Jean-Yves Le Boudec , Milan Vojnović, Power law and exponential decay of inter contact times between mobile devices, Proceedings of the 13th annual ACM international conference on Mobile computing and networking, September 09-14, 2007, Montréal, Québec, Canada [doi>10.1145/1287853.1287875]
• 41: Karthik Kumar , Yamini Nimmagadda , Yu-Ju Hong , Yung-Hsiang Lu, Energy conservation by adaptive feature loading for mobile content-based image retrieval, Proceeding of the 13th international symposium on Low power electronics and design, August 11-13, 2008, Bangalore, India [doi>10.1145/1393921.1393963]
• 42: Changbo Yang , Ming Dong , Farshad Fotouhi, Semantic feedback for interactive image retrieval, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101240]
• 43: Steven C. H. Hoi , Michael R. Lyu , Rong Jin, A Unified Log-Based Relevance Feedback Scheme for Image Retrieval, IEEE Transactions on Knowledge and Data Engineering, v.18 n.4, p.509-524, April 2006 [doi>10.1109/TKDE.2006.53]
• 44: Wei Liu , Wei Jiang , Shih-Fu Chang, Relevance aggregation projections for image retrieval, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386372]
• 45: Sanjoy K. Saha , Amit K. Das , Bhabatosh Chanda, Image retrieval based on indexing and relevance feedback, Pattern Recognition Letters, v.28 n.3, p.357-366, February, 2007 [doi>10.1016/j.patrec.2006.04.005]
• 46: A. Demers , K. Petersen , M. Spreitzer , D. Terry , M. Theimer , B. Welch, The Bayou Architecture: Support for Data Sharing Among Mobile Users, Proceedings of the 1994 First Workshop on Mobile Computing Systems and Applications, p.2-7, December 08-09, 1994 [doi>10.1109/WMCSA.1994.37]
• 47: D. B. Terry , M. M. Theimer , Karin Petersen , A. J. Demers , M. J. Spreitzer , C. H. Hauser, Managing update conflicts in Bayou, a weakly connected replicated storage system, Proceedings of the fifteenth ACM symposium on Operating systems principles, p.172-182, December 03-06, 1995, Copper Mountain, Colorado, United States
• 48: Sumeet Sobti , Nitin Garg , Fengzhou Zheng , Junwen Lai , Yilei Shao , Chi Zhang , Elisha Ziskind , Arvind Krishnamurthy , Randolph Y. Wang, Segank: A Distributed Mobile Storage System, Proceedings of the 3rd USENIX Conference on File and Storage Technologies, March 31-31, 2004, San Francisco, CA
• 49: Daniel Peek , Jason Flinn, EnsemBlue: integrating distributed storage and consumer electronics, Proceedings of the 7th symposium on Operating systems design and implementation, November 06-08, 2006, Seattle, Washington
• 50: M. A. Smith, D. Davenport, and H. Hwa, "AURA: A mobile platform for object and location annotation," in UbiComp, 2003.
• 51: Shravan Gaonkar , Jack Li , Romit Roy Choudhury , Landon Cox , Al Schmidt, Micro-Blog: sharing and querying content through mobile phones and social participation, Proceeding of the 6th international conference on Mobile systems, applications, and services, June 17-20, 2008, Breckenridge, CO, USA [doi>10.1145/1378600.1378620]
• 52: Trevor Pering , David H. Nguyen , John Light , Roy Want, Face-to-Face Media Sharing Using Wireless Mobile Devices, Proceedings of the Seventh IEEE International Symposium on Multimedia, p.269-276, December 12-14, 2005 [doi>10.1109/ISM.2005.57]
• 53: Trevor Pering , Roy Want , Lamar Gardere , Kristin Vadas , Evan Welbourne, Musicology: Bringing Personal Music into Shared Spaces, Proceedings of the 2007 Fourth Annual International Conference on Mobile and Ubiquitous Systems: Networking&Services (MobiQuitous), p.1-8, August 06-10, 2007 [doi>10.1109/MOBIQ.2007.4450978]
• 54: Emiliano Miluzzo , Nicholas D. Lane , Kristóf Fodor , Ronald Peterson , Hong Lu , Mirco Musolesi , Shane B. Eisenman , Xiao Zheng , Andrew T. Campbell, Sensing meets mobile social networks: the design, implementation and evaluation of the CenceMe application, Proceedings of the 6th ACM conference on Embedded network sensor systems, November 05-07, 2008, Raleigh, NC, USA [doi>10.1145/1460412.1460445]
• 55: David Dearman , Melanie Kellar , Khai N. Truong, An examination of daily information needs and sharing opportunities, Proceedings of the ACM 2008 conference on Computer supported cooperative work, November 08-12, 2008, San Diego, CA, USA [doi>10.1145/1460563.1460668]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.5 COMPUTER SYSTEM IMPLEMENTATION
      C.5.0 General

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Design, Management

Keywords:
energy, management, performance, retrieval