Quality-aware collaborative question answering: methods and evaluation

ABSTRACT

Community Question Answering (QA) portals contain questions and answers contributed by hundreds of millions of users. These databases of questions and answers are of great value if they can be used directly to answer questions from any user. In this research, we address this collaborative QA task by drawing knowledge from the crowds in community QA portals such as Yahoo! Answers. Despite their popularity, it is well known that answers in community QA portals have unequal quality. We therefore propose a quality-aware framework to design methods that select answers from a community QA portal considering answer quality in addition to answer relevance. Besides using answer features for determining answer quality, we introduce several other quality-aware QA methods using answer quality derived from the expertise of answerers. Such expertise can be question independent or question dependent. We evaluate our proposed methods using a database of 95K questions and 537K answers obtained from Yahoo! Answers. Our experiments have shown that answer quality can improve QA performance significantly. Furthermore, question dependent expertise based methods are shown to outperform methods using answer features only. It is also found that there are also good answers not among the best answers identified by Yahoo! Answers users.

REFERENCES

• 1: Eugene Agichtein , Carlos Castillo , Debora Donato , Aristides Gionis , Gilad Mishne, Finding high-quality content in social media, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341557]
• 2: Eugene Agichtein , Steve Lawrence , Luis Gravano, Learning to find answers to questions on the Web, ACM Transactions on Internet Technology (TOIT), v.4 n.2, p.129-162, May 2004 [doi>10.1145/990301.990303]
• 3: Krisztian Balog , Leif Azzopardi , Maarten de Rijke, Formal models for expert finding in enterprise corpora, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148181]
• 4: Jiang Bian , Yandong Liu , Eugene Agichtein , Hongyuan Zha, Finding the right facts in the crowd: factoid question answering over social media, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367561]
• 5: Robin D. Burke , Kristian J. Hammond , Vladimir A. Kulyukin , Steven L. Lytinen , N. Tomuro , S. Schoenberg, Question Answering from Frequently Asked Question Files: Experiences with the FAQ Finder System, University of Chicago, Chicago, IL, 1997
• 6: Christopher S. Campbell , Paul P. Maglio , Alex Cozzi , Byron Dom, Expertise identification using email communications, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956965]
• 7: Raymond D'Amore, Expertise community detection, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009089]
• 8: H. T. Dang, J. Lin, and D. Kelly. Overview of the TREC 2006 Question Answering Track. In TREC, 2006.
• 9: Z. Gyöngyi, G. Koutrika, J. Pederson, and H. Garcia-Molina. Questioning Yahoo! Answers. In WWW, 2008.
• 10: Sanda Harabagiu , Finley Lacatusu , Andrew Hickl, Answering complex questions with random walk models, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148211]
• 11: J.-N. Hwang, S.-R. Lay, and A. Lippman. Nonparametric Multivariate Density Estimation: A Comparative Study. IEEE Transactions of Signal Processing, 42(10):2795--2810, 1994.
• 12: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee, Finding similar questions in large question and answer archives, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099572]
• 13: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee , Soyeon Park, A framework to predict the quality of answers with non-textual features, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148212]
• 14: Valentin Jijkoun , Maarten de Rijke, Retrieving answers from frequently asked questions pages on the web, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099571]
• 15: Pawel Jurczyk , Eugene Agichtein, Discovering authorities in question answer communities by using link analysis, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321575]
• 16: Pawel Jurczyk , Eugene Agichtein, Hits on question answer portals: exploration of link analysis for author ranking, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277938]
• 17: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999 [doi>10.1145/324133.324140]
• 18: Adam L. Berger , Vincent J. Della Pietra , Stephen A. Della Pietra, A maximum entropy approach to natural language processing, Computational Linguistics, v.22 n.1, p.39-71, March 1996
• 19: Gang Luo , Chunqiang Tang , Ying-li Tian, Answering relationship queries on the web, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242648]
• 20: D. Moldovan, S. Harabagiu, R. Girju, P. Morarescu, F. Lacatusu, A. Novischi, A. Badulescu, and O. Bolohan. LCC Tools for Question Answering. In TREC, 2002.
• 21: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 22: E. Sneiders. Automated FAQ Answering: Continued Experience with Shallow Language Understanding. In AAAI, 1999.
• 23: I. Soboroff, A. P. de Vries, and N. Craswell. Overview of the TREC 2006 Enterprise Track. In TREC, 2006.
• 24: Qi Su , Dmitry Pavlov , Jyh-Herng Chow , Wendell C. Baker, Internet-scale collection of human-reviewed data, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242604]
• 25: Xiaoyong and W. B. Croft. Finding Experts in Community-Based Question-Answering Services. In CIKM, 2005.
• 26: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems (TOIS), v.22 n.2, p.179-214, April 2004 [doi>10.1145/984321.984322]
• 27: Jun Zhang , Mark S. Ackerman , Lada Adamic, Expertise networks in online communities: structure and algorithms, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242603]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software

General Terms:
Experimentation

Keywords:
answer quality, expertise, question answering