A video retrieval and sequencing system

ABSTRACT

Video is an effective medium for capturing the events in the real world around us, and a vast amount of video materials exists, covering a wide range of applications. However, widespread use of video in computer applications is often impeded by the lack of effective tools to manage video information systematically. This article discusses the design and implementation of a frame-based video retrieval and sequencing system (VRSS). The system is designed to support the entire process of video information management: segmenting, indexing, retrieving, and sequencing of video data. A semiautomatic tool is developed to divide video sequences into meaningful shots. Each video shot is logged using text descriptions, audio dialogue, and cinematic attributes. A two-layered, concept-based model is used as the basis for accurately retrieving relevant video shots based on users' free-text queries. A cinematic, rule-based, virtual editing tool is also developed to sequence the video shots retrieved for presentation within a specified time constraint. The system has been tested on a video documentary on the NUS (National University of Singapore) engineering faculty. The results of video retrieval experiments are encouraging.

REFERENCES

• 1: AUGUIERRE-SMITH, T. G. AND PINCEVER, N.C. 1991. Parsing movies in context. In Proceedzngs of USENIX (Summer 1991). USENIX Assoc., Berkeley, Calif., 157 168.
• 2: ARNHEiM, R. 1933. Film Faber and Faber Ltd., London, U.K.
• 3: BALAZS, B. 1952. Theory of the Ftlm. Dennis Dobson Ltd., London, U K.
• 4: Richard Kuehn Belew, Adaptive information retrieval: machine learning in associative networks (connectionist, free-text, browsing, feedback), 1986
• 5: BLOCH, G.R. 1988 From concepts to film sequences In Procee&ngs of RIAO '88. 761-767.
• 6: BORDWELL, D. 1991. Making Meamng Harvard University Press, Cambridge, Mass.
• 7: BRAUN, T.L. 1971. Document vector modification. In The SMARTRetr~eval System, G. Salton, Ed. Prentice-Hall, N.J, chapt. 24.
• 8: BRUCKMAN, A. S. 1991. The electronic scrapbook: Towards an intelligent home video editing system. Master's dissertation, MIT, Cambridge, Mass
• 9: CHUA, T.S. 1992. A dynamic hypertext system. J. Singapore Nat. Acad ScL 20, I 6.
• 10: Tat-Seng Chua , Elaine Pui-Man Lai, Supporting composition in a hypermedia environment, Hypermedia, v.3 n.3, p.207-238, 1991
• 11: CHUA, T. S, PUNG, H. K., LU, G. J., AND JONG, H.S. 1994. A concept-based image retrieval system. In Procee&ngs of the 27th An~ual Hawat~ International Con/~rence on Systems Sctences (Jan. 4 7). 590-598.
• 12: Jeff Conklin, Hypertext: an introduction and survey, Computer, v.20 n.9, p.17-41, Sept. 1987 [doi>10.1109/MC.1987.1663693]
• 13: W. B. Croft , R. H. Thompson, I3R: a new approach to the design of document retrieval systems, Journal of the American Society for Information Science, v.38 n.6, p.389-404, Nov. 1987 [doi>10.1002/(SICI)1097-4571(198711)38:6<389::AID-ASI1>3.0.CO;2-4]
• 14: Glorianna Davenport , Thomas Aguirre Smith , Natalio Pincever, Cinematic Primitives for Multimedia, IEEE Computer Graphics and Applications, v.11 n.4, p.67-74, July 1991 [doi>10.1109/38.126883]
• 15: DYER, M.R. 1966. Film: A Montage of Theorzes E. P. Dutton, New York.
• 16: Mark,E. Frisse, Searching for information in a hypertext medical handbook, Communications of the ACM, v.31 n.7, p.880-886, July 1988 [doi>10.1145/48511.48518]
• 17: G~ANNETTI, L. 1987. Understanding Mot,~es. Prentice-Hall, Englewood Cliffs, N.J
• 18: Simon Gibbs, Composite multimedia and active objects, Conference proceedings on Object-oriented programming systems, languages, and applications, p.97-112, October 06-11, 1991, Phoenix, Arizona, United States
• 19: Bernard J. Haan , Paul Kahn , Victor A. Riley , James H. Coombs , Norman K. Meyrowitz, IRIS hypermedia services, Communications of the ACM, v.35 n.1, p.36-51, Jan. 1992 [doi>10.1145/129617.129618]
• 20: KINDEM, G A. 1987. The Mowng Image. Scott, Foresman, & Co., Glenview, Ill.
• 21: D. Krawczak , P. Smith , S. Shute, EP-X: a demonstration of semantically based search of bibliographic databases, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.263-271, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42035]
• 22: K. L. Kwok, A neural network for probabilistic information retrieval, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.21-30, June 25-28, 1989, Cambridge, Massachusetts, United States
• 23: Andrew Lippman, Movie-maps: An application of the optical videodisc to computer graphics, Proceedings of the 7th annual conference on Computer graphics and interactive techniques, p.32-42, July 14-18, 1980, Seattle, Washington, United States
• 24: Loo, P L. 1991. Starship Manual, version 2.0. ISS Int. Publication. Inst. of Systems Science, Singapore.
• 25: Dario Lucarella, A model for hypertext-based information retrieval, Hypertext: concepts, systems and applications, Cambridge University Press, New York, NY, 1992
• 26: Wendy E. Mackay , Glorianna Davenport, Virtual video editing in interactive multimedia applications, Communications of the ACM, v.32 n.7, p.802-810, July 1989 [doi>10.1145/65445.65447]
• 27: MmLERSON, G. 1985. The Technique of Television Productton. Focal Press, London, U.K.
• 28: M~NSKY, M.A. 1977 A framework for representing knowledge. In The Psychology of Computer Vision, P. Winston, Ed. McGraw-Hill, New York, 211-277.
• 29: Akio Nagasaka , Yuzuru Tanaka, Automatic Video Indexing and Full-Video Search for Object Appearances, Proceedings of the IFIP TC2/WG 2.6 Second Working Conference on Visual Database Systems II, p.113-127, September 30-October 03, 1991
• 30: NOREAUL% T., KOLL, M., AND MCGmL, M. J. 1977. Automatic ranked output from Boolean searches in SIRE. J. Amer. Soc. Inf. Scl. 28, 333-339.
• 31: Alex Pentland , Rosalind Picard , Glorianna Davenport , Ken Haase, Video and Image Semantics: Advanced Tools for Telecommunications, IEEE MultiMedia, v.1 n.2, p.73-75, June 1994
• 32: Rosanne Price , Tat-seng Chua , Suliman Al-Hawamdeh, Applying relevance feedback to a photo archival system, Journal of Information Science, v.18 n.3, p.203-215, 1992 [doi>10.1177/016555159201800305]
• 33: RUAN, L. Q. 1992. An intelligent interactive video system. Master's thesis, Inst. of Systems Science, National Univ. of Singapore, Singapore.
• 34: R~JAN, L. Q., CHUA, T. S., ANn TEH, H.H. 1992a. A frame-based interactive video retrieval system. In Proceedings of the 2nd International Computer Science Conference, Data and Knowledge Engineering: Theory and Apphcattons (Hong Kong, Dec.). IEEE, New York, 23-29.
• 35: RuAN, L. G., SMOLIAR, S. W., AND KANKANHALLI, A. 1992b. An analysis of low-resolution segmentation techniques for animate video. In the 2nd International Conference on Automation, Robotics and Computer Vtston (Singapore, Sept.). CV-16.3.1-5.
• 36: B. Rubin , G. Davenport, Structured content modeling for cinematic information, ACM SIGCHI Bulletin, v.21 n.2, p.78-89, Oct. 1989 [doi>10.1145/70609.70620]
• 37: SACK, W. AND DAVIS, M. 1994. IDIC: Assembling video sequences for story plans and content annotation. In Proceedings of the International Conference on Multimedta Computing and Systems (May). 14-19.
• 38: Gerard Salton , Christopher Buckley, Term-weighting approaches in automatic text retrieval, Information Processing and Management: an International Journal, v.24 n.5, p.513-523, 1988 [doi>10.1016/0306-4573(88)90021-0]
• 39: SALTON, G. AND BUCKLEY, C. 1990. Improving retrieval performance by relevance feedback J. Amer. Soc. Inf. Sci. 41, 4, 288-297.
• 40: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 41: SCI=IANK, R. C. AND ABELSON, R. P. 1977. Scrtpts, Plans, Goals and Understanding--An Inquiry into Human Knowledge Structures. Lawrence Erlbaum Assoc., Hillsdale, N.J.
• 42: Daniel Schwabe , Bruno Feijó , Werther G. Krause, Intelligent hypertext for normative knowledge in engineering, Hypertext: concepts, systems and applications, Cambridge University Press, New York, NY, 1992
• 43: Scott M. Stevens, Intelligent interactive video simulation of a code inspection, Communications of the ACM, v.32 n.7, p.832-843, July 1989 [doi>10.1145/65445.65450]
• 44: R. Tong , L. Appelbaum , V. Askman , J. Cunningham, Conceptual information retrieval using RUBRIC, Proceedings of the 10th annual international ACM SIGIR conference on Research and development in information retrieval, p.247-253, June 03-05, 1987, New Orleans, Louisiana, United States [doi>10.1145/42005.42033]
• 45: Yoshinobu Tonomura, Video handling based on structured information for hypermedia systems, International conference on Multimedia information systems '91, p.333-344, November 1991, Jurong, Singapore
• 46: Kristina Hooper Woolsey, Multimedia Scouting, IEEE Computer Graphics and Applications, v.11 n.4, p.26-38, July 1991 [doi>10.1109/38.126878]
• 47: ZHANG, J. J., SMOLIER, S. W., AND TAN, Y.H. 1993. Towards automating content-based video indexing and retrieval. In Proceedings of the 1st International Conference on Multtmedta Modeling, T.-S. Chua and T. L. Kunii, Eds. World Scientific, River Edge, N.J., 193-206.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Experimentation

Keywords:
cinematic rules, frame-based modeling, multimedia, video retrieval, virtual editing