Retrieving answers from frequently asked questions pages on the web

ABSTRACT

We address the task of answering natural language questions by using the large number of Frequently Asked Questions (FAQ) pages available on the web. The task involves three steps: (1) fetching FAQ pages from the web; (2) automatic extraction of question/answer (Q/A) pairs from the collected pages; and (3) answering users' questions by retrieving appropriate Q/A pairs. We discuss our solutions for each of the three tasks, and give detailed evaluation results on a collected corpus of about 3.6Gb of text data (293K pages, 2.8M Q/A pairs), with real users' questions sampled from a web search engine log. Specifically, we propose simple but effective methods for Q/A extraction and investigate task-specific retrieval models for answering questions. Our best model finds answers for 36% of the test questions in the top 20 results. Our overall conclusion is that FAQ pages on the web provide an excellent resource for addressing real users' information needs in a highly focused manner.

REFERENCES

• 1: Apache Lucene: A high-performance, full-featured text search engine library. http://lucene.apache.org.
• 2: Eugene Agichtein , Steve Lawrence , Luis Gravano, Learning to find answers to questions on the Web, ACM Transactions on Internet Technology (TOIT), v.4 n.2, p.129-162, May 2004 [doi>10.1145/990301.990303]
• 3: Adam Berger , Rich Caruana , David Cohn , Dayne Freitag , Vibhu Mittal, Bridging the lexical chasm: statistical approaches to answer-finding, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.192-199, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345576]
• 4: R. Burke, K. Hammond, V. Kulyukin, S. Lytinen, N. Tomuro, and S. Schoenberg. Natural language processing in the FAQFinder system: Results and prospects. In Proc. 1997 AAAI Spring Symposium on Natural Language Processing for the World Wide Web, pages 17--26, 1997.
• 5: R. Burke, K. Hammond, V. Kulyukin, S. Lytinen, N. Tomuro, and S. Schoenberg. Question answering from frequently asked question files: Experiences with the FAQFinder system. AI Magazine, 18(2):57--66, 1997.
• 6: David Carmel , Menachem Shtalhaim , Aya Soffer, eResponder: Electronic Question Responder, Proceedings of the 7th International Conference on Cooperative Information Systems, p.150-161, September 06-08, 2000
• 7: Soumen Chakrabarti , Martin van den Berg , Byron Dom, Focused crawling: a new approach to topic-specific Web resource discovery, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1623-1640, May 17, 1999
• 8: W. Daelemans, J. Zavrel, K. Van Der Sloot, and A. Van Den Bosch. TiMBL: Tilburg Memory Based Learner, version 5.0. Tech. Report 03--10, 2003.
• 9: Oren Etzioni , Michael Cafarella , Doug Downey , Stanley Kok , Ana-Maria Popescu , Tal Shaked , Stephen Soderland , Daniel S. Weld , Alexander Yates, Web-scale information extraction in knowitall: (preliminary results), Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988687]
• 10: A. Foster and N. Ford. Serendipity and information seeking: an empirical study. J. Documentation, 59(3):321--340, 2003.
• 11: N. Fuhr, M. Lalmas, S. Malik, and Z. Szlavik, editors. Advances in XML Information Retrieval: Third International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2004), LNCS 3493, Springer, 2005
• 12: Roxana Girju, Automatic detection of causal relations for Question Answering, Proceedings of the ACL 2003 workshop on Multilingual summarization and question answering, p.76-83, July 11, 2003 [doi>10.3115/1119312.1119322]
• 13: B. Katz. Annotating the World Wide Web using natural language. In Proc. RIAO'97, 1997.
• 14: Boris Katz , Sue Felshin , Deniz Yuret , Ali Ibrahim , Jimmy J. Lin , Gregory Marton , Alton Jerome McFarland , Baris Temelkuran, Omnibase: Uniform Access to Heterogeneous Data for Question Answering, Proceedings of the 6th International Conference on Applications of Natural Language to Information Systems-Revised Papers, p.230-234, June 27-28, 2002
• 15: H. Kim and J. Seo. High-performance FAQ retrieval using an automatic clustering method of query logs. Information Processing & Management, in press.
• 16: Leila Kosseim , Stéphane Beauregard , Guy Lapalme, Using information extraction and natural language generation to answer e-mail, Data & Knowledge Engineering, v.38 n.1, p.85-100, 07/01/2001 [doi>10.1016/S0169-023X(01)00018-0]
• 17: Nicholas Kushmerick, Wrapper induction: efficiency and expressiveness, Artificial Intelligence, v.118 n.1-2, p.15-68, April 2000 [doi>10.1016/S0004-3702(99)00100-9]
• 18: Cody C. T. Kwok , Oren Etzioni , Daniel S. Weld, Scaling question answering to the Web, Proceedings of the 10th international conference on World Wide Web, p.150-161, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371973]
• 19: Yu-Sheng Lai , Kuao-Ann Fung , Chung-Hsien Wu, FAQ mining via list detection, proceeding of the 2002 conference on multilingual summarization and question answering, p.1-7, August 31, 2002 [doi>10.3115/1118845.1118849]
• 20: Hanny Yulius Limanto , Nguyen Ngoc Giang , Vo Tan Trung , Jun Zhang , Qi He , Nguyen Quang Huy, An information extraction engine for web discussion forums, Special interest tracks and posters of the 14th international conference on World Wide Web, May 10-14, 2005, Chiba, Japan [doi>10.1145/1062745.1062827]
• 21: C.-Y. Lin, D. Quan, V. Sinha, K. Bakshi, D. Huynh, B. Katz, and D. Karger. What makes a good answer? The role of context in question answering systems. In Proc. INTERACT 2003, 2003.
• 22: S. Lytinen and N. Tomuro. The use of question types to match questions in FAQFinder. In Proc. AAAI-2002 Spring Symposium on Mining Answers from Texts and Knowledge Bases, pages 46--53, 2002.
• 23: S. Lytinen, N. Tomuro, and T. Repede. The use of WordNet sense tagging in FAQFinder. In Proc. AAAI-2000 Workshop on AI and Web Search, Austin, TX, 2000.
• 24: Andrew McCallum , Dayne Freitag , Fernando C. N. Pereira, Maximum Entropy Markov Models for Information Extraction and Segmentation, Proceedings of the Seventeenth International Conference on Machine Learning, p.591-598, June 29-July 02, 2000
• 25: G. Mishne and M. de Rijke. Boosting Web Retrieval through Query Operations. In Proc. ECIR 2005, pages 502--516, 2005.
• 26: M. Porter. An algorithm for suffix stripping. Program, 14 (3):130--137, 1980.
• 27: Dragomir Radev , Weiguo Fan , Hong Qi , Harris Wu , Amardeep Grewal, Probabilistic question answering on the web, Proceedings of the 11th international conference on World Wide Web, May 07-11, 2002, Honolulu, Hawaii, USA [doi>10.1145/511446.511500]
• 28: Ganesh Ramakrishnan , Soumen Chakrabarti , Deepa Paranjpe , Pushpak Bhattacharya, Is question answering an acquired skill?, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988688]
• 29: R. Soricut and E. Brill. Automatic question answering: Beyond the factoid. In Proc. HLT/NAACL, 2004.
• 30: Ellen M. Voorhees, Evaluating answers to definition questions, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL 2003--short papers, p.109-111, May 27-June 01, 2003, Edmonton, Canada [doi>10.3115/1073483.1073520]
• 31: Jiying Wang , Fred H. Lochovsky, Data extraction and label assignment for web databases, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775179]
• 32: Steven D. Whitehead, Auto-FAQ: an experiment in cyberspace leveraging, Computer Networks and ISDN Systems, v.28 n.1-2, p.137-146, Dec. 1995 [doi>10.1016/0169-7552(95)00101-2]
• 33: Ross Wilkinson, Effective retrieval of structured documents, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.311-317, July 03-06, 1994, Dublin, Ireland
• 34: Z. Zheng. AnswerBus question answering system. In Proc. HLT 2002, 2002.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Linguistic processing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms, Experimentation, Measurement, Performance

Keywords:
FAQ retrieval, question answering, questions beyond factoids