Computational models for experiences in the arts, and multimedia

ABSTRACT

In this paper, we develop formal computational models for three aspects of experiential systems for browsing media -- (a) context (b) interactivity through hyper-mediation and (c) context evolution using a memory model. Experiential systems deal with the problem of developing context adaptive mechanisms for knowledge acquisition and insight. Context is modeled as a union of graphs whose nodes represent concepts and where the edges represent the semantic relationships. The system context is the union of the contexts of the user, the environment and the media being accessed. We also develop a novel concept dissimilarity. We then develop algorithms to determine the optimal hyperlink for each media element by determining the relationship between the user context and the media. As the user navigates through the hyper-linked sources, the memory model captures the interaction of the user with the hyper-linked sources and updates the user context. Finally, this results in new hyper-links for the media. Our pilot user studies show excellent results, validating our framework.

REFERENCES

• 1: Merriam Webster Dictionary <u>http://www.m-w.com</u>.
• 2: Google <u>http://www.google.com</u>.
• 3: house{raw} <u>http://isa.asu.edu/projects_house.html</u>.
• 4: OpenCyc <u>http://www.opencyc.org</u>.
• 5: L. AGNIHOTRI, K. DEVARA, T. MCGEE, et al. (2001). Summarization of Video Programs Based on Closed Captioning, SPIE Conf. on Storage and Retrieval in Media Databases, San Jose, CA, 599--607, January 2001.
• 6: B. ARONS (1994). Pitch-Based Emphasis Detection For Segmenting Speech Recordings, Proc. ICSLP, Yokohama Japan, 1931--1934, Sep. 1994.
• 7: R. ASCOT (1966). Behaviourist Art and Cybernetic Vision. <u>Cybernetica</u> 9: pp. 247--264.
• 8: R. BARTHES and S. HEATH (1988). <u>Image, music, text</u>. New York, Noonday Press.
• 9: A. B. BENITEZ, J. R. SMITH and S.-F. CHANG (2000). MediaNet: A Multimedia Information Network for Knowledge Representation, Proceedings of the 2000 SPIE Conference on Internet Multimedia Management Systems (IS&T/SPIE-2000), Boston MA, Nov 6--8, 2000.
• 10: A. B. BENITEZ and S.-F. CHANG (2002). Multimedia Knowledge Integration, Summarization and Evaluation, Proc. of the 2002 International Workshop On Multimedia Data Mining in conjuction with the International Conference on Knowledge Discovery and Data Mining (MDM/KDD-2002), Edmonton, Alberta, Canada, July 2002.
• 11: Jay David Bolter , Richard Grusin, Remediation: understanding new media, MIT Press, Cambridge, MA, 1998
• 12: F. BRAUDEL, R. D. AYALA, P. BRAUDEL, et al. (1998). <u>Les mémoires de la Méditerranée: préhistoire et antiquité</u>. Paris, Editions de Fallois.
• 13: V. BUSH (1945). As We May Think. <u>The Atlantic Monthly</u>. 176: 101--108, <u>http://www.theatlantic.com/unbound/flashbks/computer/bushf.htm</u>.
• 14: J. CAGE (1966). Diary: Audience 1966. A year from Monday, Middletown CT, Weslyan University Press, 1966.
• 15: E. CHANG, K. GOH, G. SYCHAY, et al. (2003). CBSA: content-based soft annotation for multimodal image retrieval using Bayes point machines. <u>IEEE Transactions on Circuits and Systems for Video Technology</u> 13(1): 26--38.
• 16: M. CHRISTEL, S. STEVENS, T. KANADE, et al. (1996). Techniques for the Creation and Exploration of Digital Video Libraries. Multimedia Tools and Applications. B. FURHT. Boston MA, Kluwer Academic Publishers.
• 17: Michael G. Christel , Michael A. Smith , C. Roy Taylor , David B. Winkler, Evolving video skims into useful multimedia abstractions, Proceedings of the SIGCHI conference on Human factors in computing systems, p.171-178, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274670]
• 18: Michael G. Christel, Visual digests for news video libraries, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.303-311, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319633]
• 19: Michael G. Christel , Alexander G. Hauptmann , Howard D. Wactlar , Tobun D. Ng, Collages as dynamic summaries for news video, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641120]
• 20: B. CLARKSON, N. SAWHNEY and A. PENTLAND (1998). Auditory Context Awareness in Wearable Computing, Workshop on Perceptual User Interfaces, San Francisco, CA,, November 5--6, 1998.
• 21: R. W. DEVAUL, S. J. SCHWARTZ and A. PENTLAND (2001) MIThril: context-aware computing for daily life <u>http://www.media.mit.edu/wearables/mithril/MIThril.pdf</u>.
• 22: A. K. DEY and G. D. ABOWD (1999). Towards a Better Understanding of Context and Context-Awareness, Proceedings of the 3rd International Symposium on Wearable Computers, San Francisco, CA, pp. 21--28, October 20--21, 1999.
• 23: Anind K. Dey , Daniel Salber , Gregory D. Abowd , Masayasu Futakawa, The Conference Assistant: Combining Context-Awareness with Wearable Computing, Proceedings of the 3rd IEEE International Symposium on Wearable Computers, p.21, October 18-19, 1999
• 24: Anind K. Dey, Understanding and Using Context, Personal and Ubiquitous Computing, v.5 n.1, p.4-7, February 2001 [doi>10.1007/s007790170019]
• 25: Ramesh Jain, Experiential computing, Communications of the ACM, v.46 n.7, July 2003 [doi>10.1145/792704.792729]
• 26: Ramesh Jain, Folk computing, Communications of the ACM, v.46 n.4, April 2003 [doi>10.1145/641205.641223]
• 27: Michael I. Jordan, Learning in graphical models, MIT Press, Cambridge, MA, 1999
• 28: I. KANT (1920). <u>Critique de la raison pure</u>. Paris, Flammarion.
• 29: I. KANT, P. GUYER and A. W. WOOD (1998). <u>Critique of pure reason</u>. Cambridge; New York, Cambridge University Press.
• 30: M. KRUGER (1996). Responsive Environments. Theories and Documents of Contemporary Arts: A sourcebook for Artists' Writings, Berkeley CA, University of California Press, 1996.
• 31: L. MANOVICH (2001). <u>The language of new media</u>. Cambridge, Mass., MIT Press.
• 32: M. MCLUHAN (1994). <u>Understanding media: the extensions of man</u>. Cambridge, Mass., MIT Press.
• 33: G. A. MILLER, R. BECKWITH, C. FELLBAUM, et al. (1993). Introduction to WordNet: An On-line Lexical Database. <u>International Journal of Lexicography</u> 3(4): 235--244.
• 34: M. R. NAPHADE and T. S. HUANG (2001). Recognizing high-level audio-visual concepts using context, Proc. IEEE Conference on Image Processing, 2001, Thessaloniki, Greece, pp. 46--49, Oct. 2001.
• 35: M. R. NAPHADE and T. S. HUANG (2002). Discovering recurrent events in video using unsupervised methods, Proceedings IEEE International Conference on Image Processing. 2002., pp. 13--16, 2002.
• 36: M. R. NAPHADE, C.-Y. LIN and J. R. SMITH (2002). Learning semantic multimedia representations from a small set of examples, Proc. IEEE International Conference on Multimedia and Expo, 2002, Lausanne, Switzerland, pp. 513--516, Aug. 2002.
• 37: Theodore H. Nelson, Computer Lib/Dream Machines, Microsoft Press, Redmond, WA, 1987
• 38: Randall Packer , Ken Jordan, Multimedia: From Wagner to Virtual Reality, W. W. Norton & Co., Inc., New York, NY, 2002
• 39: N. J. PAIK (1996). Cybernated Art. Theories and documents of Contemporary Art: A Sourcebook of Artists' Writings, Berkeley, CA, University of California Press, 1996.
• 40: S. PFEIFFER, R. LIENHART, S. FISCHER, et al. (1996). Abstracting Digital Movies Automatically. <u>Journal of Visual Communication and Image Representation</u> 7(4): 345--353.
• 41: John C. Platt, AutoAlbum: Clustering Digital Photographs using Probabilistic Model Merging, Proceedings of the IEEE Workshop on Content-based Access of Image and Video Libraries (CBAIVL'00), p.96, June 16-16, 2000
• 42: J. RAMAMOORTHY, S. WONG and H. SUNDARAM (2003). Dynamic Adaptive Visualizations. Arts Media and Engineering Center, ASU, AME-TR-2003-03, 2003.
• 43: Nitin Sawhney , Chris Schmandt, Nomadic radio: scaleable and contextual notification for wearable audio messaging, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.96-103, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303005]
• 44: B. N. T. SCHILIT, M. M. (1994). Disseminating active map information to mobile hosts. <u>IEEE Network</u> 8(5): 22--32.
• 45: B. SHEVADE and H. SUNDARAM (2003). Vidya: An Experiential Annotation System. Arts Media and Engineering Center, Arizona State University, AME-TR-2003-04, 2003.
• 46: H. SRIDHARAN, H. SUNDARAM and T. RIKAKIS (2003). Context, memory and Hyper-mediation in Experiential Systems. Arts Media and Engineering Center, ASU, AME-TR-2003-02, 2003.
• 47: Hari Sundaram , Shih-Fu Chang, Segmentation, structure detection and summarization of multimedia sequences, 2002
• 48: H. SUNDARAM and S.-F. CHANG (2002). Video skims: taxonomies and an optimal generation framework, Proc. of the International Conference on Image Processing 2002, Rochester, NY, 21--24, Sep. 2002.
• 49: Hari Sundaram , Lexing Xie , Shih-Fu Chang, A utility framework for the automatic generation of audio-visual skims, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641042]
• 50: H. SUNDARAM and T. RIKAKIS (2003). An Introduction to Experiential Systems. Arts Media and Engineering Center, ASU, AME-2003-01, 2003.
• 51: Shingo Uchihashi , Jonathan Foote , Andreas Girgensohn , John Boreczky, Video Manga: generating semantically meaningful video summaries, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.383-392, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319654]
• 52: B. VIOLA (1980). Will there be condominiums in data space? <u>Video</u> 80(5).
• 53: Howard D. Wactlar, The next generation electronic library—capturing the experience, ACM Computing Surveys (CSUR), v.28 n.4es, Dec. 1996 [doi>10.1145/242224.242372]
• 54: Howard D. Wactlar , Michael G. Christel , Alexander G. Hauptmann , Yihong Gong, Informedia Experience-on-Demand: capturing, integrating and communicating experiences across people, time and space, ACM Computing Surveys (CSUR), v.31 n.2es, June 1999 [doi>10.1145/323216.323356]
• 55: Ye Lu , Chunhui Hu , Xingquan Zhu , HongJiang Zhang , Qiang Yang, A unified framework for semantics and feature based relevance feedback in image retrieval systems, Proceedings of the eighth ACM international conference on Multimedia, p.31-37, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354403]
• 56: B. L. YEO and M. YEUNG (1998). Classification, Simplification and Dynamic Visualization of Scene Transition Graphs for Video Browsing, Proc. SPIE Storage and Retrieval of Image and Video Databases VI, San Jose CA, Feb. 1998., 1998.
• 57: B.-L. YEO and B. LIU (1995). Rapid scene analysis on compressed video. <u>IEEE Transactions on Circuits and Systems for Video Technology</u> 5(6): 533--544.
• 58: M. M. Yeung , B-L. Yeo, Time-Constrained Clustering for Segmentation of Video into Story Unites, Proceedings of the International Conference on Pattern Recognition (ICPR '96) Volume III-Volume 7276, p.375, August 25-29, 1996
• 59: M. YEUNG and B.-L. YEO (1997). Video visualization for compact presentation and fast browsing of pictorial content. <u>IEEE Transactions on Circuits and Systems for Video Technology</u> 7(5): 771--785.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Theory and methods; User-centered design
      H.5.5 Sound and Music Computing

  J. Computer Applications
  J.5 ARTS AND HUMANITIES
      Subjects: Fine arts

General Terms:
Algorithms, Design, Documentation, Human Factors, Theory, Verification

Keywords:
art, context, experiential models, hyper-mediation, implications, semantic nets, wordNet