Motion recovery for video content classification

ABSTRACT

Like other types of digital information, video sequences must be classified based on the semantics of their contents. A more-precise and completer extraction of semantic information will result in a more-effective classification. The most-discernible difference between still images and moving pictures stems from movements and variations. Thus, to go from the realm of still-image repositories to video databases, we must be able to deal with motion. Particularly, we need the ability to classify objects appearing in a video sequence based on their characteristics and features such as shape or color, as well as their movements. By describing the movements that we derive from the process of motion analysis, we introduce a dual hierarchy consisting of spatial and temporal parts for video sequence representation. This gives us the flexibility to examine arbitrary sequences of frames at various levels of abstraction and to retrieve the associated temporal information (say, object trajectories) in addition to the spatial representation. Our algorithm for motion detection uses the motion compensation component of the MPEG video-encoding scheme and then computes trajectories for objects of interest. The specification of a language for retrieval of video based on the spatial as well as motion characteristics is presented.

REFERENCES

• 1: Mark Charles Allmen, Image sequence description using spatiotemporal flow curves: toward motion-based recognition, University of Wisconsin at Madison, Madison, WI, 1992
• 2: F. Arman , R. Depommier , A. Hsu , M.-Y. Chiu, Content-based browsing of video sequences, Proceedings of the second ACM international conference on Multimedia, p.97-103, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192630]
• 3: BOBICK, A.F. 1993. Representational frames in video annotation. In Proceedzngs of the 27th Annual Conference on Signals. Systems and Computers. IEEE Computer Society Press, Los Alamitos, Cahf.
• 4: BUCHANAN, M. C. AND ZELLWEGER, P.T. 1993. Automatically generating consistent schedules for multimedia documents. Multzrned~a Syst. J. 1, 2 (Sept.).
• 5: DAVIS, M. 1993. Media streams: An icomc visual language for video annotation. In Proceedings of the IEEE Symposzum on Wsual Languages (Bergen, Norway). IEEE, New York, 196 202.
• 6: Nevenka Dimitrova, Content classification and retrieval of digital video on motion recovery, Arizona State University, Tempe, AZ, 1995
• 7: N. Dimitrova , F. Golshani, Rx for semantic video database retrieval, Proceedings of the second ACM international conference on Multimedia, p.219-226, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192659]
• 8: DUDA, R. O. AND HART, P.E. 1973 Pattern Classification and Scene Analys~s. John Wiley and Sons, New York.
• 9: G. Farin, Curves and surfaces for computer aided geometric design, Academic Press Professional, Inc., San Diego, CA, 1990
• 10: Borivoje Furht, Multimedia Systems and Techniques, Kluwer Academic Publishers, Norwell, MA, 1996
• 11: GODD^RD, N. 1992. The perception of articulated motion: Recognizing moving light displays. Ph.D. thesis, Univ. of Rochester, Rochester, N.Y.
• 12: GOLSHANI, F. AND DIMITROVA, N. 1994. Retrieval and delivery of information in multimedia database systems. Inf. Softw Tech. 36, 4 (May), 235 242.
• 13: W. I. Grosky , R. Mehrotra, Guest Editors' Introduction: Image Database Management, Computer, v.22 n.12, p.7-8, December 1989
• 14: William I. Grosky, Multimedia Information Systems, IEEE MultiMedia, v.1 n.1, p.12-24, March 1994 [doi>10.1109/93.295262]
• 15: Amarnath Gupta , Terry E. Weymouth , Ramesh Jain, Semantic Queries in Image Databases, Proceedings of the IFIP TC2/WG 2.6 Second Working Conference on Visual Database Systems II, p.201-215, September 30-October 03, 1991
• 16: Amarnath Gupta , Terry E. Weymouth , Ramesh Jain, Semantic Queries with Pictures: The VIMSYS Model, Proceedings of the 17th International Conference on Very Large Data Bases, p.69-79, September 03-06, 1991
• 17: A. Hampapur , T. Weymouth , R. Jain, Digital video segmentation, Proceedings of the second ACM international conference on Multimedia, p.357-364, October 15-20, 1994, San Francisco, California, United States [doi>10.1145/192593.192699]
• 18: HORN, B. K. AND SCHUNCK, B. G. 1981. Determining optical flow. Artzfi Intell. 17, 1-3, 185-203.
• 19: JOHANSSON, G. 1976. Spatio-temporal differentiation and lntegTation in visual motion perception. Psychol. Res. 38, 4, 379 393.
• 20: Dieter Koller , Joseph Weber , Jitendra Malik, Robust Multiple Car Tracking with Occlusion Reasoning, University of California at Berkeley, Berkeley, CA, 1993
• 21: Hiroaki Kubota , Yasukazu Okamoto , Hiroshi Mizoguchi , Yoshinori Kuno, Vision processor system for moving-object analysis, Machine Vision and Applications, v.7 n.1, p.37-43, Winter 1993 [doi>10.1007/BF01212415]
• 22: Didier Le Gall, MPEG: a video compression standard for multimedia applications, Communications of the ACM, v.34 n.4, p.46-58, April 1991 [doi>10.1145/103085.103090]
• 23: T. D. C. Little , G. Ahanger , R. J. Folz , J. F. Gibbon , F. W. Reeve , D. H. Schelleng , D. Venkatesh, A digital on-demand video service supporting content-based queries, Proceedings of the first ACM international conference on Multimedia, p.427-436, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.168450]
• 24: T. D. C. Little , A. Ghafoor , C. Y. R. Chen , C. S. Chang , P. B. Berra, Multimedia synchronization, Data Engineering, v.14 n.3, p.26-35, Sept. 1991
• 25: Phillip E. Mattison, Practical Digital Video with Programming Examples in C, John Wiley & Sons, Inc., New York, NY, 1994
• 26: MICHAEL, N. 1994. VEENA--a visual query language. M.S. thesis, Arizona State Univ., Tempe, Ariz.
• 27: Akio Nagasaka , Yuzuru Tanaka, Automatic Video Indexing and Full-Video Search for Object Appearances, Proceedings of the IFIP TC2/WG 2.6 Second Working Conference on Visual Database Systems II, p.113-127, September 30-October 03, 1991
• 28: John K. Ousterhout, Tcl and the Tk toolkit, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1994
• 29: Kiyotaka Otsuji , Yoshinobu Tonomura, Projection detecting filter for video cut detection, Proceedings of the first ACM international conference on Multimedia, p.251-257, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166295]
• 30: Ketan Patel , Brian C. Smith , Lawrence A. Rowe, Performance of a software MPEG video decoder, Proceedings of the first ACM international conference on Multimedia, p.75-82, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166274]
• 31: RASURE, J., ARGIRO, D., SAUER, T., AND WILLIAMS, C. 1990. Visual language and software development environment for image processing. Int. J. Imaging Syst. Tech. 2, 2, 183-199.
• 32: K. Rohr, Towards model-based recognition of human movements in image sequences, CVGIP: Image Understanding, v.59 n.1, p.94-115, Jan. 1994 [doi>10.1006/ciun.1994.1006]
• 33: Lawrence A. Rowe , Brian C. Smith, A Continuous Media Player, Proceedings of the Third International Workshop on Network and Operating System Support for Digital Audio and Video, p.376-386, November 12-13, 1992
• 34: RowE, L. A., BORECZKY, J. S., AND EADS, C.A. 1994. Indexes for user access to large video databases. In Proceedings of SPIE IS and Symposium on Storage and Retmeval for Image and Video Databases (San Jose, Calif.). SPIE.
• 35: Robert J. Schalkoff, Digital image processing and computer vision: an introduction to theory and implementations, John Wiley & Sons, Inc., New York, NY, 1989
• 36: SWANBERG, D., SHU, C.-F., AND JAIN, R. 1993. Knowledge guided parsing in video databases. In Image and Video Processtng Conference; Symposium on Electromc hnagtng: Sczence and Technology. Vol. 1908. IS & T/SPIE, 13-24.
• 37: Laura Teodosio , Walter Bender, Salient video stills: content and context preserved, Proceedings of the first ACM international conference on Multimedia, p.39-46, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.166270]
• 38: Laura A. Teodosio , Michael Mills, Panoramic overviews for navigating real-world scenes, Proceedings of the first ACM international conference on Multimedia, p.359-364, August 02-06, 1993, Anaheim, California, United States [doi>10.1145/166266.168422]
• 39: WEISS, R. 1994. Content-based access to algebraic video. Tech. Rep., Massachusetts Inst. of Technology, Cambridge, Mass.
• 40: ZttANG, H., GONG, Y., SMOLIAR, S., AND TAN, S.Y. 1994. Automatic parsing of news video. In Proceedings of the Internattonal Conference on Multimedia Computing and Systems (Boston, Mass.). IEEE Computer Society Press. Los Alarnltos, Ca}i~, 45-54.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion

General Terms:
Algorithms, Design

Keywords:
MPEG compressed video analysis, content-based retrieval of video, motion recovery, video databases, video retrieval