Less talk, more rock: automated organization of community-contributed collections of concert videos

ABSTRACT

We describe a system for synchronization and organization of user-contributed content from live music events. We start with a set of short video clips taken at a single event by multiple contributors, who were using a varied set of capture devices. Using audio fingerprints, we synchronize these clips such that overlapping clips can be displayed simultaneously. Furthermore, we use the timing and link structure generated by the synchronization algorithm to improve the findability and representation of the event content, including identifying key moments of interest and descriptive text for important captured segments of the show. We also identify the preferred audio track when multiple clips overlap. We thus create a much improved representation of the event that builds on the automatic content match. Our work demonstrates important principles in the use of content analysis techniques for social media content on the Web, and applies those principles in the domain of live music capture.

REFERENCES

• 1: Michael G. Christel , Alexander G. Hauptmann , Howard D. Wactlar , Tobun D. Ng, Collages as dynamic summaries for news video, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641120]
• 2: Sally Jo Cunningham , David M. Nichols, How people find videos, Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, June 16-20, 2008, Pittsburgh PA, PA, USA [doi>10.1145/1378889.1378924]
• 3: Adrian Graham , Hector Garcia-Molina , Andreas Paepcke , Terry Winograd, Time as essence for photo browsing through personal digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544301]
• 4: J. Haitsma and T. Kalker. A Highly Robust Audio Fingerprinting System With an Efficient Search Strategy. Journal of New Music Research, 32(2):211--221, 2003.
• 5: Cormac Herley, Accurate repeat finding and object skipping using fingerprints, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101295]
• 6: C. Herley. ARGOS: automatically extracting repeating objects from multimedia streams. IEEE Transactions on Multimedia, 8(1):115--129, 2006.
• 7: V. Kaplun, P. Vora, M. Naaman, P. Mead, and A. Moed. Understanding media capture and consumption for live music events. Technical report, Yahoo! Inc., 2008. In Submission.
• 8: Lyndon S. Kennedy , Mor Naaman, Generating diverse and representative image search results for landmarks, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367539]
• 9: Mor Naaman , Yee Jiun Song , Andreas Paepcke , Hector Garcia-Molina, Automatic organization for digital photographs with geographic coordinates, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA [doi>10.1145/996350.996366]
• 10: Suphi Umut Naci , Alan Hanjalic, Intelligent browsing of concert videos, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291264]
• 11: Rahul Nair , Nick Reid , Marc Davis, Photo LOI: browsing multi-user photo collections, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101187]
• 12: J. Ogle and D. Ellis. Fingerprinting to Identify Repeated Sound Events in Long-Duration Personal Audio Recordings. Acoustics, Speech and Signal Processing, 2007.
• 13: ICASSP 2007. IEEE International Conference on, 1, 2007.
• 14: Prarthana Shrstha , Mauro Barbieri , Hans Weda, Synchronization of multi-camera video recordings based on audio, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291367]
• 15: David A. Shamma , Ryan Shaw , Peter L. Shafton , Yiming Liu, Watch what I watch: using community activity to understand content, Proceedings of the international workshop on Workshop on multimedia information retrieval, September 24-29, 2007, Augsburg, Bavaria, Germany [doi>10.1145/1290082.1290120]
• 16: Ryan Shaw , Patrick Schmitz, Community annotation and remix: a research platform and pilot deployment, Proceedings of the 1st ACM international workshop on Human-centered multimedia, October 27-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178745.1178761]
• 17: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688]
• 18: Prarthana Shrestha , Hans Weda , Mauro Barbieri , Dragan Sekulovski, Synchronization of multiple video recordings based on still camera flashes, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180679]
• 19: C. Snoek, M. Worring, A. Smeulders, and B. Freiburg. The role of visual content and style for concert video indexing. Multimedia and Expo, 2007 IEEE International Conference on, pages 252--255, 2-5 July 2007.
• 20: Shingo Uchihashi , Jonathan Foote , Andreas Girgensohn , John Boreczky, Video Manga: generating semantically meaningful video summaries, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.383-392, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319654]
• 21: Y. van Houten, U. Naci, B. Freiburg, R. Eggermont, S. Schuurman, D. Hollander, J. Reitsma, M. Markslag, J. Kniest, M. Veenstra, and A. Hanjalic. The multimedian concert-video browser. Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on, pages 1561--1564, 6-6 July 2005.
• 22: A. Wang. An Industrial Strength Audio Search Algorithm. In Proceedings of the International Conference on Music Information Retrieval, 2003.
• 23: Utz Westermann , Ramesh Jain, Toward a Common Event Model for Multimedia Applications, IEEE MultiMedia, v.14 n.1, p.19-29, January 2007 [doi>10.1109/MMUL.2007.23]
• 24: Youtube.com, google inc. http://www.youtube.com.
• 25: Amit Zunjarwad , Hari Sundaram , Lexing Xie, Contextual wisdom: social relations and correlations for multimedia event annotation, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291382]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.m MISCELLANEOUS

General Terms:
Algorithms, Human Factors

Keywords:
audio fingerprinting, social media, synchronization, video