Efficient retrieval of life log based on context and content

ABSTRACT

In this paper, we present continuous capture of our life log with various sensors plus additional data and propose effective retrieval methods using this context and content. Our life log system contains video, audio, acceleration sensor, gyro, GPS, annotations, documents, web pages, and emails. In our previous studies, we showed our retrieval methodology [8], [9], which mainly depends on context information from sensor data. In this paper, we extend our methodology with additional functions. They are (1) spatio-temporal sampling for extraction of key frames for summarization; and (2) conversation scene detection. With the first of these, key frames for the summarization are extracted using time and location data (GPS). Because our life log captures dense location data, we can also make use of derivatives of location data, that is, speed and acceleration in the movement of the person. The summarizing key frames are made using them. We also introduce content analysis for conversation scene detection. In our previous work, we have investigated context-based retrieval, which differs from the majority of studies in image/video retrieval focusing on content-based retrieval. In this paper, we introduce visual and audio data content analysis for conversation scene detection. The detection of conversation scenes will be very important tags for our life log data retrieval. We describe our present system and additional functions, as well as preliminary results for the additional functions.

REFERENCES

• 1: Lamming, M., and Flynn, M. "Forget-me-not" Intimate Computing in Support of Human Memory. Proceedings of FRIEND21, 94 Int. Symp. Next Generation Human Interface, (Feb. 1994), 125--128.
• 2: Steve Mann, `WearCam' (The Wearable Camera): Personal Imaging Systems for long--term use in wearable tetherless computer--mediated reality and personal Photo/Videographic Memory Prosthesis, Proceedings of the 2nd IEEE International Symposium on Wearable Computers, p.124, October 19-20, 1998
• 3: Gemmell, J., Lueder, R., and Bell, G. Living with a Lifetime Store. ATR Workshop on Ubiquitous Experience Media, (Sep. 2003).
• 4: Jennifer Healey , Rosalind W. Picard, StartleCam: A Cybernetic Wearable Camera, Proceedings of the 2nd IEEE International Symposium on Wearable Computers, p.42, October 19-20, 1998
• 5: Brian Clarkson , Alex Pentland , Kenji Mase, Recognizing User Context via Wearable Sensors, Proceedings of the 4th IEEE International Symposium on Wearable Computers, p.69, October 18-21, 2000
• 6: Aizawa, K., Ishijima, K., and Shiina, M. Summarizing wearable video. Proceedings of ICIP 2001, (Oct. 2001), 398--401.
• 7: Sawahata, Y., and Aizawa, K. Wearable imaging system for summarizing personal experiences. Proceedings of ICME 2003, (Jul. 2003), I45--I48.
• 8: Tetsuro Hori , Kiyoharu Aizawa, Context-based video retrieval system for the life-log applications, Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrieval, November 07-07, 2003, Berkeley, California [doi>10.1145/973264.973270]
• 9: Hori, T. and Aizawa, K., Capturing Life Log and Retrieval based on Context, IEEE ICME2004, June, 2004.
• 10: Pan, H., Liang, Z. P., and Huang, T. S. Fusing Audio and Visual Features of Speech. Proceedings of ICIP 2000 (Sep. 2000), 214--217.
• 11: Cuetos, P., and Neti, C. Audio-Visual Intent-to-Speak Detection For Human-Computer Interaction. Proceedings of ICASSP'00(Jun. 2000), 2373--2376.
• 12: Tancharoen, D., and Jitapunkul, S. Automatic Face Color Segmentation Based Rate Control For Low Bit Rate Video Coding. Proceedings of ISCAS 2003 (May. 2003), 384--387.
• 13: Tsekeridou, S., and Pitas, I. Content-Based Video Parsing and Indexing Based on Audio-Visual Interaction. IEEE Trans. Circuits and System for Video Technology, 11, 4 (Apr. 2001), 522--535.
• 14: Kenichi Minami , Akihito Akutsu , Hiroshi Hamada , Yoshinobu Tonomura, Video Handling with Music and Speech Detection, IEEE MultiMedia, v.5 n.3, p.17-25, July 1998 [doi>10.1109/93.713301]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

Keywords:
content based retrieval, context based retrieval, life log, video retrieval, wearable computing