Semantic video search using natural language queries

ABSTRACT

Recent advances in computer vision and artificial intelligence algorithms have allowed automatic extraction of metadata from video. This metadata can be represented by using the RDF/OWL ontology which can encode scene objects and their relationships in an unambiguous and well-formed manner. The encoded data can be queried using SPARQL. However, SPARQL has a steep learning curve and cannot be directly utilized by a general user for video content search. In this paper, we propose a method to bridge this gap by automatically translating user provided natural language query into an ontology-based SPARQL query for semantic video search. The proposed method consists of three major steps. First, semantically labeled training corpus of natural language query sentences is used for learning the Semantic Stochastic Context Free Grammar (SSCFG). Second, given a user provided natural language query sentence, we use the Earley-Stolcke parsing algorithm to determine the maximum likelihood semantic parsing of the query sentence. This parsing infers the semantic meaning for each word in the query sentence from which the SPARQL query is constructed. Third, the SPARQL query is executed to retrieve relevant video segments from the RDF-OWL video content database. The method is evaluated by running natural language queries on surveillance videos from maritime and land-based domains, though the framework itself is general and extensible to search videos from other domains.

REFERENCES

• 1: NIST, TRECVID, http://www-nlpir.nist.gov/projects/trecvid/
• 2: R.X. Gao, T.F Wu, N. Sang, and S.C. Zhu "Bayesian Inference for Layer Representation with Mixed Markov Random Field" EMMCVPR, Springer LNCS 4679, Ezhou, China, Aug 2007.
• 3: M. Lee, A. Hakeem, N. Haering., S.C. Zhu. 'SAVE: A Framework for Semantic Annotation of Visual Events', CVPR Workshop, 2008.
• 4: Klein, D, Manning, C.D, "Accurate Unlexicalized Parsing". In ACL 423--430, 2003.
• 5: Fellbaum C., Wordnet: An Electronic Lexical Database, MIT Press.
• 6: D. Moore and I. Essa, "Recognizing Multitasked Activities using Stochastic Context-Free Grammar" CVPR 2001.
• 7: J.C. Earley, An Efficient Context-Free Parsing Algorithm. PhD thesis, Carnegie-Mellon University, 1968.
• 8: http://www.w3.org/TR/rdf-sparql-query/
• 9: R. Nevatia, J. Hobbs, B. Bolles, "An Ontology for Video Event Representation", WS on Event Detection & Recognition, 2004.
• 10: Androutsopoulos, I., Ritchie, G., Thanisch, P., "Natural Language Interfaces to Databases -- An Introduction". Natural Language Engineering. vol.1(1), pp.29--81, 1995.
• 11: Androutsopoulos, I., Ritchie, G., Thanisch, P., "An Efficient and Portable Natural Language Query Interface for Relational Databases". In: IEAAI&ES, pp.327--330, 1993.
• 12: Popescu, A.M., Etzioni, O., Kautz, H.A., "Towards a Theory of Natural Language Interfaces to Databases". In: IUI, 2003.
• 13: C. Wang, M. Xiong, Q. Zhou, Y. Yu., "PANTO: A Portable Natural Language Interface to Ontologies". In ECSW, 2007.
• 14: Kaufmann, E., Bernstein, A., Zumstein, R., "Querix: A Natural Language Interface to Query Ontologies Based on Clarification Dialogs". In: Intl. Semantic Web Conference, pp.980--981, 2006.
• 15: H Wang, K Zhang, Q Liu, D Tran, Y Yu., "Q2Semantic: A Lightweight Keyword Interface to Semantic Search". ICSW, 2008.
• 16: S.C. Zhu, D.B. Mumford, "Quest for a stochastic grammar of images", In FTCGV, 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Video analysis

General Terms:
Algorithms, Experimentation, Performance

Keywords:
SPARQL, SSCFG, semantic video retrieval, video search