What's next?: emergent storytelling from video collection

ABSTRACT

In the world of visual storytelling, narrative development relies on a particular temporal ordering of shots and sequences and scenes. Rarely is this ordering cast in stone. Rather, the particular ordering of a story reflects a myriad of interdependent decisions about the interplay of structure, narrative arc and character development. For storytellers, particularly those developing their narratives from large documentary archives, it would be helpful to have a visualization system partnered with them to present suggestions for the most compelling story path.

We present Storied Navigation, a video editing system that helps authors compose a sequence of scenes that tell a story, by selecting from a corpus of annotated clips. The clips are annotated in unrestricted natural language. Authors can also type a story in unrestricted English, and the system finds possibilities for clips that best match high-level elements of the story. Beyond simple keyword matching, these elements can include the characters, emotions, themes, and story structure. Authors can also interactively replace existing scenes or predict the next scene to continue a story, based on these characteristics. Storied Navigation gives the author the feel of brainstorming about the story rather than simply editing the media.

REFERENCES

• 1: Preetha Appan , Hari Sundaram , David Birchfield, Communicating everyday experiences, Proceedings of the 1st ACM workshop on Story representation, mechanism and context, October 15-15, 2004, New York, NY, USA [doi>10.1145/1026633.1026638]
• 2: Jackie Assa , Yaron Caspi , Daniel Cohen-Or, Action synopsis: pose selection and illustration, ACM SIGGRAPH 2005 Papers, July 31-August 04, 2005, Los Angeles, California
• 3: BANGSØ, O., JENSEN, O. G., JENSEN, F. V., ANDERSEN, P. B., AND KOCKA, T. Non-linear interactive storytelling using object-oriented Bayesian networks. Proc of CGAIDE 2004.
• 4: Barbara A. Barry , Glorianna Davenport, Mindful documentary, Massachusetts Institute of Technology, Cambridge, MA, 2005
• 5: Stefano Bocconi , Frank Nack , Lynda Hardman, Vox populi: a tool for automatically generating video documentaries, Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, September 06-09, 2005, Salzburg, Austria [doi>10.1145/1083356.1083427]
• 6: CAVAZZA M. AND PIZZI. D. Narratology for Interactive Storytelling: a Critical Introduction. Proc of TIDSE 2006.
• 7: G. Davenport , B. Barry , A. Kelliher , P. Nemirovsky, Media Fabric — A Process-Oriented Approach to Media Creation and Exchange, BT Technology Journal, v.22 n.4, p.160-170, October 2004 [doi>10.1023/B:BTTJ.0000047595.31067.f1]
• 8: G. Davenport , M. Murtaugh, Automatist storyteller systems and the shifting sands of story, IBM Systems Journal, v.36 n.3, p.446-456, 1997
• 9: Dan B Goldman , Brian Curless , David Salesin , Steven M. Seitz, Schematic storyboarding for video visualization and editing, ACM SIGGRAPH 2006 Papers, July 30-August 03, 2006, Boston, Massachusetts
• 10: FELLBAUM, C. (Ed). WordNet: An Electronic Lexical Database. MIT Press. 1998.
• 11: FIGA, E. AND TARAU, P. The VISTA Project: An Agent Architecture for Virtual Interactive Storytelling. Proc of TIDSE 2003.
• 12: GOGUEN, J. AND HARRELL, F. Foundations for Active Multimedia Narrative: Semiotic spaces and structural blending. In Interaction Studies: Social Behaviour and Communication in Biological and Artificial Systems. 2004.
• 13: Byunghee Jung , Junehwa Song , Yoonjoon Lee, A narrative-based abstraction framework for story-oriented video, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.2, p.11-es, May 2007 [doi>10.1145/1230812.1230817]
• 14: Henry Lieberman , Hugo Liu, Adaptive Linking between Text and Photos Using Common Sense Reasoning, Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, p.2-11, May 29-31, 2002
• 15: Hugo Liu , Henry Lieberman , Ted Selker, A model of textual affect sensing using real-world knowledge, Proceedings of the 8th international conference on Intelligent user interfaces, January 12-15, 2003, Miami, Florida, USA [doi>10.1145/604045.604067]
• 16: H. Liu , P. Singh, ConceptNet — A Practical Commonsense Reasoning Tool-Kit, BT Technology Journal, v.22 n.4, p.211-226, October 2004 [doi>10.1023/B:BTTJ.0000047600.45421.6d]
• 17: MEHRABIAN, A. Manual for a comprehensive system of measures of emotional states: The PAD Model. Available from Albert Mehrabian, 1130 Alta Mesa Road, Monterey, CA, USA 93940. 1998.
• 18: Marvin Minsky, The society of mind, Simon & Schuster, Inc., New York, NY, 1986
• 19: OSSENBRUGGEN, J. v. AND HARDMAN, L. Semantic Timeline Interfaces for Annotated Multimedia Assets. Proc. of EWIMT 2005.
• 20: PROPP, V. Morphology of the Folktale. Texas Press. 1968.
• 21: ROSSI, F. http://www.documentarydoctor.com
• 22: Roger C. Schank, Tell Me a Story: A New Look at Real and Arfificial Memory: A New Look at Real and Artificial MEM, Simon & Schuster Trade, 1990
• 23: Push Singh , Thomas Lin , Erik T. Mueller , Grace Lim , Travell Perkins , Wan Li Zhu, Open Mind Common Sense: Knowledge Acquisition from the General Public, On the Move to Meaningful Internet Systems, 2002 - DOA/CoopIS/ODBASE 2002 Confederated International Conferences DOA, CoopIS and ODBASE 2002, p.1223-1237, October 30-November 01, 2002
• 24: Xiaofeng Tong , Qingshan Liu , Lingyu Duan , Hanqing Lu , Changsheng Xu , Qi Tian, A unified framework for semantic shot representation of sports video, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101848]
• 25: Ba Tu Truong , Svetha Venkatesh, Video abstraction: A systematic review and classification, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.3 n.1, p.3-es, February 2007 [doi>10.1145/1198302.1198305]
• 26: TUFFIELD, M. M., MILLARD, D. E., AND SHADBOLT, N. R. Ontological Approaches to Modeling Narrative. Proc. of AKT DTA Symposium 2006.
• 27: VAUCELLE, C. AND DAVENPORT, G. An open-ended tool to compose movies for cross-cultural digital storytelling: Textable Movie. Proc. of ICHIM 2004.
• 28: BROOKS, K. M. Agent Stories: Authoring Computational Cinematic Stories. PhD Thesis, MIT Media Lab. 1999.
• 29: 29. MATEAS, M. and STERN, A. Natural Language Understanding in Façade: Surface-text Processing. Proc of TIDSE, Darmstadt, Germany, June 2004

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Theory and methods

General Terms:
Algorithms, Design

Keywords:
emergent storytelling, interactive storytelling, storied navigation