Exploring media correlation and synchronization for navigated hypermedia documents

ABSTRACT

This paper is devoted to explore media correlation and media synchronization in a composite multimedia document, the so-called navigated hypermedia document in our language learning system, to facilitate the multimedia authoring, presentation, and access. Two levels of media correlation in temporal, spatial, and content domains are investigated: syntactic level correlation and semantic level correlation. We devise a capturing mechanism to record all the media streams and relations between them, including voice and event streams, for replaying the lecturing in a form as close as possible to the original classroom experience. The syntactic level correlation is based on specific timestamps of the media stream and used to reconstruct the recorded lecture for synchronized presentation. Furthermore, to integrate media objects with specific segments within the media stream, some computed synchronization processes are required to discover semantic content of the media. The proposed computed synchronization techniques, including speech-event binding process for temporal domain, tele-pointer (i.e. cursor) movement interpolation and adaptable handwriting presentation for spatial domain, and erasing handling for content domain, will be addressed. Experimental results show that in the speech-event binding process 74% of speech access entries for accessible visualized events are found. The acceptable rate of human perception on tele-pointer movement is higher than 85% if time interval is selected carefully. Finally, the accuracy of erasing handling for content removing is about 71%.

REFERENCES

• 1: Harry W. Agius , Marios C. Angelides, Modeling Content for Semantic-Level Querying of Multimedia, Multimedia Tools and Applications, v.15 n.1, p.5-37, September 2001 [doi>10.1023/A:1011386102507]
• 2: Allan, J., Carbonell, J., Doddington, G., Yamron, J., and Yang, Y. Topic Detection and Tracking Pilot Study Final report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop (Lansdowne, Virginia, Feb. 8-11), 1998, 194--218.
• 3: Blakowski, G., and Steinmetz, R. A Media Synchronization Survey: Reference Model, Specification, and Case Studies. IEEE Journal on Selected Areas in Communications, 14, 1 (Jan. 1996), 5--35.
• 4: Jason A. Brotherton , Gregory D. Abowd, Lessons learned from eClass: Assessing automated capture and access in the classroom, ACM Transactions on Computer-Human Interaction (TOCHI), v.11 n.2, p.121-155, June 2004 [doi>10.1145/1005361.1005362]
• 5: Herng-Yow Chen , Gin-Yi Chen , Jen-Shin Hong, Design of a Web-Based Synchronized Multimedia Lecture System for Distance Education, Proceedings of the IEEE International Conference on Multimedia Computing and Systems Volume II-Volume 2, p.887, June 07-11, 1999
• 6: Chu, W.T., and Chen, H.Y. Toward better retrieval and presentation by exploring cross-media correlations. Multimedia Systems, 10, 3 (March 2005), 183--198.
• 7: Farkas, D.K., and Poltrock, S.E. Online editing, mark-up models and the workplace lives of editors and writers. IEEE Transactions on Professional Communication, 38, 2 (Jun. 1995), 110--117.
• 8: Athula Ginige , David B. Lowe , John Robertson, Hypermedia Authoring, IEEE MultiMedia, v.2 n.4, p.24-35, August 1995 [doi>10.1109/93.482293]
• 9: Hardman, L. Modeling and Authoring Hypermedia Documents. Ph.D. Thesis, University of Amsterdam, Amsterdam, Netherlands.
• 10: Gary Hardock , Gordon Kurtenbach , William Buxton, A marking based interface for collaborative writing, Proceedings of the 6th annual ACM symposium on User interface software and technology, p.259-266, December 1993, Atlanta, Georgia, United States [doi>10.1145/168642.168669]
• 11: Knoy, T. An Editing Workbook for Chinese Technical Writers, 2000.
• 12: Lauer, T., and Ottmann, T. Means and Methods in Automatic Courseware Production: Experience and Technical Challenges. In Proceedings of the E-Learn (Montreal, Canada, Oct. 15-19), 2002, 553--560.
• 13: Lauer, T., Muller, R., and Ottmann, T. Animations for Teaching Purposes: Now and Tomorrow. Journal of Universal Computer Science, 7, 5 (2001), 420--433.
• 14: Jin-Yi Wang , Herng-Yow Chen, A Marking-Based Synchronized Multimedia Tutoring System for Composition Studies, Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies, p.41-45, July 05-08, 2005 [doi>10.1109/ICALT.2005.14]
• 15: Dalila Mekhaldi , Denis Lalanne , Rolf Ingold, Thematic alignment of recorded speech with documents, Proceedings of the 2003 ACM symposium on Document engineering, November 20-22, 2003, Grenoble, France [doi>10.1145/958220.958230]
• 16: R. Müller , T. Ottmann, The "Authoring on the Fly" system for automated recording and replay of (tele)presentations, Multimedia Systems, v.8 n.3, p.158-176, October 2000 [doi>10.1007/s005300000042]
• 17: Muller, R., Ottmann, T., and Zhang, H. Presentation Recording as a Means to go Virtual for Campus-based Universities. In Proceedings of 13th International Conference on Information Resources Management Association (Seattle, WA, May 19-22), 2002, 567--570.
• 18: Ogata, H., Yano, Y., and Wakita, R. COCOA: A Communicative Correction Assisting System for Composition Studies, In Proceedings of International Conference on Computers in Education (Sarawak, Malaysia, Dec. 2-6), 1997, 461--468.
• 19: Charles B. Owen, Multiple Media Correlation: Theory and Applications, Dartmouth College, Hanover, NH, 1998
• 20: Sissel Guttormsen Schär , Helmut Krueger, Using New Learning Technologies with Multimedia, IEEE MultiMedia, v.7 n.3, p.40-51, July 2000 [doi>10.1109/93.879767]
• 21: Lisa Joy Stifelman , Christopher M. Schmandt, The audio notebook: paper and pen interaction with structured speech, 1997
• 22: Synchronized Multimedia Integration Language (SMIL) Specification. Available on http://www.w3.org/TR/REC-smil

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

Additional Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Distance learning

General Terms:
Design, Human Factors

Keywords:
computed synchronization process, media correlation, media synchronization, semantic level correlation, syntactic level correlation