Cross-media correlation: a case study of navigated hypermedia documents

ABSTRACT

The research issues on multiple media correlation have arisen with more and more integrated multimedia applications. The multimedia correlation is used to coordinate different media and facilitate cross-media access. This paper presents our work on two types of multimedia correlation: explicit and implicit relations. We develop a system to carefully capture explicit relations and devise some computed synchronization processes to discover implicit relations between media objects. The proposed computed synchronization techniques, including speech-text alignment process in temporal domain, automatic scrolling process in spatial domain, and content dependency check process in content domain, will be addressed. Experimental results show that in the speech-text alignment process 80% of forced alignment are in-sync even the speech recognition accuracy is as low as 25%. The automatic scrolling process effectively maintains a resynchronization mechanism in different displaying environments.

REFERENCES

• 1: Herng-Yow Chen , Gin-Yi Chen , Jen-Shin Hong, Design of a Web-Based Synchronized Multimedia Lecture System for Distance Education, Proceedings of the IEEE International Conference on Multimedia Computing and Systems Volume II-Volume 2, p.887, June 07-11, 1999
• 2: Sugata Mukhopadhyay , Brian Smith, Passive capture and structuring of lectures, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.477-487, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319690]
• 3: Gregory D. Abowd , Christopher G. Atkeson , Jason Brotherton , Tommy Enqvist , Paul Gulley , Johan LeMon, Investigating the capture, integration and access problem of ubiquitous computing in an educational setting, Proceedings of the SIGCHI conference on Human factors in computing systems, p.440-447, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274704]
• 4: Charles B. Owen, Multiple Media Correlation: Theory and Applications, Dartmouth College, Hanover, NH, 1998
• 5: Synchronized Multimedia Integration Language (SMIL) Specification. http://www.w3.org/TR/REC-smil/
• 6: Chu, W.T., Hsu, K.T., and Chen, H.Y. Design of an Alignment System for Synchronized Speech-Text Presentation. Proceedings of Distributed Multimedia Systems, 2001, pp. 86--93.
• 7: Huang, X., Alleva, F., Hon, H. W, Hwang, M.Y., and Rosenfeld, R. The SPHINX II Speech Recognition System: An Overview. Computer Speech and Language, 2(7), 1993, pp. 137--148.
• 8: The CMU pronouncing dictionary http://www.speech.cs.cmu.edu/cgi-bin/cmudict
• 9: Ney, H. and Ortmanns S. Progress in Dynamic Programming Search for LVCSR. Proceedings of the IEEE, Vol. 88, No. 8, 2000, pp. 1224--1240.
• 10: John D. Kececioglu , Weiqing Zhang, Aligning Alignments, Proceedings of the 9th Annual Symposium on Combinatorial Pattern Matching, p.189-208, July 20-22, 1998
• 11: Eric L. Anson , Eugene W. Myers, ReAligner: a program for refining DNA sequence multi-alignments, Proceedings of the first annual international conference on Computational molecular biology, p.9-16, January 20-23, 1997, Santa Fe, New Mexico, United States [doi>10.1145/267521.267524]
• 12: Daniel Lopresti , Gordon Wilfong, Cross-Domain Approximate String Matching, Proceedings of the String Processing and Information Retrieval Symposium & International Workshop on Groupware, p.120, September 21-24, 1999
• 13: Alexander G. Hauptmann , Michael J. Witbrock, Informedia: news-on-demand multimedia information acquisition and retrieval, Intelligent multimedia information retrieval, MIT Press, Cambridge, MA, 1997
• 14: Owen, C.B. and Makedon, F. The Handbook of Multimedia Computing, chapter Cross-Modal Information Retrieval. CRC Press, Boca Raton, FL, 1998.
• 15: Justin Zobel , Philip Dart, Phonetic string matching: lessons from information retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.166-172, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243258]
• 16: Chen, T., Graf, H.P., and Wang, K. Lip synchronization using speech-assisted video processing. IEEE Signal Processing Letters, 2(4), 1995, pp. 57--59.
• 17: Bacher, C., Muller, R., Ottmann, T., and Will M. Authoring on the Fly: A new way of integrating Telepresentation and Courseware production. Proceedings of ICCCE'97, 1997, pp. 89--96.
• 18: Fred J. Damerau, A technique for computer detection and correction of spelling errors, Communications of the ACM, v.7 n.3, p.171-176, March 1964 [doi>10.1145/363958.363994]

INDEX TERMS

Keywords:
computed synchronization process, cross-media correlation, explicit relation, implicit relation