Finding question-answer pairs from online forums

ABSTRACT

Online forums contain a huge amount of valuable user generated content. In this paper we address the problem of extracting question-answer pairs from forums. Question-answer pairs extracted from forums can be used to help Question Answering services (e.g. Yahoo! Answers) among other applications. We propose a sequential patterns based classification method to detect questions in a forum thread, and a graph based propagation method to detect answers for questions in the same thread. Experimental results show that our techniques are very promising.

REFERENCES

• 1: Adam Berger , Rich Caruana , David Cohn , Dayne Freitag , Vibhu Mittal, Bridging the lexical chasm: statistical approaches to answer-finding, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.192-199, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345576]
• 2: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, p.107-117, April 1998, Brisbane, Australia
• 3: H. T. Dang, J. Lin, and D. Kelly. Overview of the trec 2007 question answering track. In Proc. of TREC, 2007.
• 4: S. Ding, G. Cong, C.-Y. Lin, and X. Zhu. Using conditional random fields to extract contexts and answers of questions from online forums. In ACL, 2008.
• 5: Donghui Feng , Erin Shaw , Jihie Kim , Eduard Hovy, An intelligent discussion-bot for answering student queries in threaded discussions, Proceedings of the 11th international conference on Intelligent user interfaces, January 29-February 01, 2006, Sydney, Australia [doi>10.1145/1111449.1111488]
• 6: Sanda Harabagiu , Andrew Hickl, Methods for using textual entailment in open-domain question answering, Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, p.905-912, July 17-18, 2006, Sydney, Australia [doi>10.3115/1220175.1220289]
• 7: J. Huang, M. Zhou, and D. Yang. Extracting chatbot knowledge from online discussion forums. In Proc. of IJCAI, 2007.
• 8: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee, Finding similar questions in large question and answer archives, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099572]
• 9: Valentin Jijkoun , Maarten de Rijke, Retrieving answers from frequently asked questions pages on the web, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099571]
• 10: Nitin Jindal , Bing Liu, Identifying comparative sentences in text documents, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148215]
• 11: Jeongwoo Ko , Eric Nyberg , Luo Si, A probabilistic graphical model for joint answer ranking in question answering, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277801]
• 12: Oren Kurland , Lillian Lee, PageRank without hyperlinks: structural re-ranking using links induced by language models, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil [doi>10.1145/1076034.1076087]
• 13: John Lafferty , Chengxiang Zhai, Document language models, query models, and risk minimization for information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.111-119, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383970]
• 14: W. G. Lehnert. The process of question answering: A computer simulation of cognition. Lawrence Erlbaum Associates, 1978.
• 15: Jahna Otterbacher , Güneş Erkan , Dragomir R. Radev, Using random walks for question-focused sentence retrieval, Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, p.915-922, October 06-08, 2005, Vancouver, British Columbia, Canada [doi>10.3115/1220575.1220690]
• 16: Jian Pei , Jiawei Han , Behzad Mortazavi-Asl , Helen Pinto , Qiming Chen , Umeshwar Dayal , Meichun Hsu, PrefixSpan: Mining Sequential Patterns by Prefix-Projected Growth, Proceedings of the 17th International Conference on Data Engineering, p.215-224, April 02-06, 2001
• 17: S. Riezler, A. Vasserman, I. Tsochantaridis, V. Mittal, and Y. Liu. Statistical machine translation for query expansion in answer retrieval. In ACL, 2007.
• 18: Lokesh Shrestha , Kathleen McKeown, Detection of question-answer pairs in email conversations, Proceedings of the 20th international conference on Computational Linguistics, p.889-es, August 23-27, 2004, Geneva, Switzerland [doi>10.3115/1220355.1220483]
• 19: G. Sun, G. Cong, X. Liu, C.-Y. Lin, and M. Zhou. Mining sequential patterns and tree patterns to detect erroneous sentences. In AAAI, 2007.
• 20: E. M. Voorhees and D. M. Tice. The trec-8 question answering track evaluation. In TREC, 1999.
• 21: Klaus Zechner, Automatic generation of concise summaries of spoken dialogues in unrestricted domains, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.199-207, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383989]
• 22: D. Zhang and W. S. Lee. A language modeling approach to passage question answering. In Proc. of TREC, 2003.
• 23: Liang Zhou , Eduard Hovy, Digesting virtual "geek" culture: the summarization of technical internet relay chats, Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, p.298-305, June 25-30, 2005, Ann Arbor, Michigan [doi>10.3115/1219840.1219877]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.7 Natural Language Processing

General Terms:
Algorithms, Experimentation

Keywords:
classification, graph based ranking, information extraction, labeled sequential patterns, question answering