Combining image descriptors to effectively retrieve events from visual lifelogs

ABSTRACT

The SenseCam is a wearable camera that passively captures approximately 3,000 images per day, which equates to almost one million images per year. It is used to create a personal visual recording of the wearer's life and generates information which can be helpful as a human memory aid. For such a large amount of visual information to be of any use, it is accepted that it should be structured into "events", of which there are about 8,000 in a wearer's average year. In automatically segmenting SenseCam images into events, it will then be useful for users to locate other events similar to a given event e.g. "what other times was I walking in the park?", "show me other events when I was in a restaurant". On two datasets of 240k and 1.8M images containing topics with a variety of information needs, we evaluate the fusion of MPEG-7, SIFT, and SURF content-based retrieval techniques to address the event search issue. We have found that our proposed fusion approach of MPEG-7 and SURF offers an improvement on using either of those sources or SIFT individually, and we have also shown how a lifelog event is modeled has a large effect on the retrieval performance.

REFERENCES

• 1: H. Bay, T. Tuytelaars, and L. Van Gool. SURF: speeded up robust features. In Proceedings of the 9th European Conference on Computer Vision (ECCV'06), May 2006.
• 2: G. Bell and J. Gemmell. A digital life. Scientific American, 2007.
• 3: Michael Blighe , Aiden Doherty , Alan F. Smeaton , Noel E. O'Connor, Keyframe detection in visual lifelogs, Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments, July 16-18, 2008, Athens, Greece [doi>10.1145/1389586.1389652]
• 4: M. Blighe, H. le Borgne, N. O'Connor, A. F. Smeaton, and G. Jones. Exploiting context information to aid landmark detection in SenseCam images. In ECHISE 2006 -- 2nd International Workshop on Exploiting Context Histories in Smart Environments (Ubicomp 2006), Orange County, CA, 2006.
• 5: Michael Blighe , Sorin Sav , Hyowon Lee , Noel E. O'Connor, Mo Músaem Fíorúil: A Web-Based Search and Information Service for Museum Visitors, Proceedings of the 5th international conference on Image Analysis and Recognition, June 25-27, 2008, P=voa de Varzim, Portugal [doi>10.1007/978-3-540-69812-8_48]
• 6: D. Byrne, A. R. Doherty, C. Snoek, G. J. Jones, and A. F. Smeaton. Validating the detection of everyday concepts in visual lifelogs. In submission, 2008.
• 7: D. Byrne, B. Lavelle, A. R. Doherty, G. Jones, and A. F. Smeaton. Using bluetooth and GPS metadata to measure event similarity in SenseCam images. In IMAI'07, pages 1454--1460, Salt Lake City, Utah, 2007.
• 8: Aiden R. Doherty , Daragh Byrne , Alan F. Smeaton , Gareth J.F. Jones , Mark Hughes, Investigating keyframe selection methods in the novel domain of passively captured visual lifelogs, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386389]
• 9: Aiden R. Doherty , Alan F. Smeaton, Automatically Segmenting LifeLog Data into Events, Proceedings of the 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services, p.20-23, May 07-09, 2008 [doi>10.1109/WIAMIS.2008.32]
• 10: E. Fox and J. Shaw. Combination of multiple searches. In TREC 2: Text REtrieval Conference, Gaithersberg, Maryland, USA, 1993.
• 11: S. Hodges, L. Williams, E. Berry, S. Izadi, J. Srinivasan, A. Butler, G. Smyth, N. Kapur, and K. Wood. SenseCam: a retrospective memory aid. In UbiComp: 8th International Conference on Ubiquitous Computing, volume 4602 of LNCS, pages 177--193, California, USA, 2006. Springer.
• 12: Tetsuro Hori , Kiyoharu Aizawa, Context-based video retrieval system for the life-log applications, Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrieval, November 07-07, 2003, Berkeley, California [doi>10.1145/973264.973270]
• 13: Alexandar Jaffe , Mor Naaman , Tamir Tassa , Marc Davis, Generating summaries and visualization for large collections of geo-referenced photographs, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178692]
• 14: H. Lee, A. F. Smeaton, N. O'Connor, G. Jones, M. Blighe, D. Byrne, A. Doherty, and C. Gurrin. Constructing a SenseCam visual diary as a media process multimedia systems. Multimedia Systems Journal, Special Issue on Canonical Processes of Media Production (in press), 2008.
• 15: W.-H. Lin and A. Hauptmann. Structuring continuous video recordings of everyday life using time-constrained clustering. In Multimedia Content Analysis, Management, and Retieval: SPIE IST Electronic Imaging, San Jose, California, USA, 2006.
• 16: David G. Lowe, Object Recognition from Local Scale-Invariant Features, Proceedings of the International Conference on Computer Vision-Volume 2, p.1150, September 20-25, 1999
• 17: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 18: Mark Montague , Javed A. Aslam, Relevance score normalization for metasearch, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502657]
• 19: David Nister , Henrik Stewenius, Scalable Recognition with a Vocabulary Tree, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2161-2168, June 17-22, 2006 [doi>10.1109/CVPR.2006.264]
• 20: C. Ó Conaire, N. E. O'Connor, A. Smeaton, and G. J. F. Jones. Organising a daily visual diary using multi-feature clustering. In Proc. of 19th annual Symposium on Electronic Imaging, 2007.
• 21: N. O'Hare, H. Lee, S. Cooray, C. Gurrin, G. Jones, J. Malobabic, N. O'Connor, A. F. Smeaton, and B. Uscilowski. Automatic text searching for personal photos. In SAMT 2006, Athens, Greece, 2006.
• 22: Datchakorn Tancharoen , Toshihiko Yamasaki , Kiyoharu Aizawa, Practical experience recording and indexing of Life Log video, Proceedings of the 2nd ACM workshop on Continuous archival and retrieval of personal experiences, November 11-11, 2005, Hilton, Singapore [doi>10.1145/1099083.1099092]
• 23: Ellen M. Voorhees, TREC: Continuing information retrieval's tradition of experimentation, Communications of the ACM, v.50 n.11, November 2007 [doi>10.1145/1297797.1297822]
• 24: Zhe Wang , Matthew D. Hoffman , Perry R. Cook , Kai Li, VFerret: content-based similarity search tool for continuous archived video, Proceedings of the 3rd ACM workshop on Continuous archival and retrival of personal experences, October 28-28, 2006, Santa Barbara, California, USA [doi>10.1145/1178657.1178663]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
fusion, image retrieval, lifelogging, sensecam