Modeling information-seeker satisfaction in community question answering

ABSTRACT

Question Answering Communities such as Naver, Baidu Knows, and Yahoo! Answers have emerged as popular, and often effective, means of information seeking on the web. By posting questions for other participants to answer, information seekers can obtain specific answers to their questions. Users of CQA portals have already contributed millions of questions, and received hundreds of millions of answers from other participants. However, CQA is not always effective: in some cases, a user may obtain a perfect answer within minutes, and in others it may require hours—and sometimes days—until a satisfactory answer is contributed. We investigate the problem of predicting information seeker satisfaction in collaborative question answering communities, where we attempt to predict whether a question author will be satisfied with the answers submitted by the community participants. We present a general prediction model, and develop a variety of content, structure, and community-focused features for this task. Our experimental results, obtained from a large-scale evaluation over thousands of real questions and user ratings, demonstrate the feasibility of modeling and predicting asker satisfaction. We complement our results with a thorough investigation of the interactions and information seeking patterns in question answering communities that correlate with information seeker satisfaction. We also explore personalized models of asker satisfaction, and show that when sufficient interaction history exists, personalization can significantly improve prediction accuracy over a “one-size-fits-all” model. Our models and predictions could be useful for a variety of applications, such as user intent inference, answer ranking, interface design, and query suggestion and routing.

REFERENCES

• 1: Nitin Agarwal , Huan Liu , Lei Tang , Philip S. Yu, Identifying the influential bloggers in a community, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341559]
• 2: Eugene Agichtein , Eric Brill , Susan Dumais , Robert Ragno, Learning user interaction models for predicting web search result preferences, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148175]
• 3: Eugene Agichtein , Carlos Castillo , Debora Donato , Aristides Gionis , Gilad Mishne, Finding high-quality content in social media, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341557]
• 4: Belkin, N., Oddy, R. N., and Brooks, H. M. 1982. Information retrieval: Part ii. results of a design study. J. Documen. 38, 3, 145--164.
• 5: Belkin, N. J. 1997. User modeling in information retrieval. In Proceedings of the 6th International Conference on User Modelling (UM'97).
• 6: Eric Brill , Susan Dumais , Michele Banko, An analysis of the AskMSR question-answering system, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.257-264, July 06, 2002 [doi>10.3115/1118693.1118726]
• 7: Cutrell, E. and Guan, Z. 2007. Eye tracking in msn search: Investigating snippet length, target position and task types. MSR-TR-2007-01.
• 8: Dang, H. T., Kelly, D., and Lin, J. 2007. Overview of the TREC 2007 question answering track. In Proceedings of the 16th Text Retrieval Conference (TREC).
• 9: Dina Demner-Fushman , Jimmy Lin, Answering Clinical Questions with Knowledge-Based and Statistical Techniques, Computational Linguistics, v.33 n.1, p.63-103, March 2007 [doi>10.1162/coli.2007.33.1.63]
• 10: Downey, D., Dumais, S. T., and Horvitz, E. 2007. Models of searching and browsing: Languages, studies, and applications. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI).
• 11: Freund, Y. and Schapire, R. 1996. Experiments with a new boosting algorithm. In Proceedings of the 13th International Conference on Machine Learning (ICML).
• 12: Harter, S. P. and Hert, C. A. 1997. Evaluation of information retrieval systems: Approaches, issues, and methods. Annual Review of Information Science and Technology (ARIST) 32, 3--94.
• 13: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee, Finding similar questions in large question and answer archives, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099572]
• 14: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee , Soyeon Park, A framework to predict the quality of answers with non-textual features, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148212]
• 15: Thorsten Joachims , Laura Granka , Bing Pan , Helene Hembrooke , Filip Radlinski , Geri Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search, ACM Transactions on Information Systems (TOIS), v.25 n.2, p.7-es, April 2007 [doi>10.1145/1229179.1229181]
• 16: Mei Kobayashi , Koichi Takeda, Information retrieval on the web, ACM Computing Surveys (CSUR), v.32 n.2, p.144-173, June 2000 [doi>10.1145/358923.358934]
• 17: Jimmy Lin , Dina Demner-Fushman, Methods for automatically evaluating answers to complex questions, Information Retrieval, v.9 n.5, p.565-587, November 2006 [doi>10.1007/s10791-006-9003-7]
• 18: Jimmy Lin , Pengyi Zhang, Deconstructing nuggets: the stability and reliability of complex question answering evaluation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277799]
• 19: Yandong Liu , Eugene Agichtein, On the evolution of the yahoo! answers QA community, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390478]
• 20: Liu, Y. and Agichtein, E. 2008b. You've got answers: Towards personalized models for predicting success in community question answering. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL). 97--100.
• 21: Yandong Liu , Jiang Bian , Eugene Agichtein, Predicting information seeker satisfaction in community question answering, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore [doi>10.1145/1390334.1390417]
• 22: John C. Platt, Fast training of support vector machines using sequential minimal optimization, Advances in kernel methods: support vector learning, MIT Press, Cambridge, MA, 1999
• 23: Quinlan, J. 1996. Improved use of continuous attributes in c4.5. In J. Artif. Intell. Resear.
• 24: Daniel E. Rose , Danny Levinson, Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, May 17-20, 2004, New York, NY, USA [doi>10.1145/988672.988675]
• 25: Ian Ruthven , Leif Azzopardi Glasgow , Mark Baillie , Ralf Bierig , Emma Nicol , Simon Sweeney , Murat Yakici, Intra-assessor consistency in question answering, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277879]
• 26: Soricut, R. and Brill, E. 2004. Automatic question answering: Beyond the factoid. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL).
• 27: Qi Su , Dmitry Pavlov , Jyh-Herng Chow , Wendell C. Baker, Internet-scale collection of human-reviewed data, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242604]
• 28: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001
• 29: Voorhees, E. M. 2003. Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text Retrieval Conference (TREC).
• 30: Ryen W. White , Mikhail Bilenko , Silviu Cucerzan, Studying the use of popular destinations to enhance web search interaction, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277771]
• 31: Ryen W. White , Steven M. Drucker, Investigating behavioral variability in web search, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada [doi>10.1145/1242572.1242576]
• 32: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 33: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

General Terms:
Algorithms, Design, Experimentation

Keywords:
Community question answering, information seeker satisfaction