A framework to predict the quality of answers with non-textual features

ABSTRACT

New types of document collections are being developed by various web services. The service providers keep track of non-textual features such as click counts. In this paper, we present a framework to use non-textual features to predict the quality of documents. We also show our quality measure can be successfully incorporated into the language modeling-based retrieval model. We test our approach on a collection of question and answer pairs gathered from a community based question answering service where people ask and answer questions. Experimental results using our quality measure show a significant improvement over our baseline.

REFERENCES

• 1: Adam L. Berger , Vincent J. Della Pietra , Stephen A. Della Pietra, A maximum entropy approach to natural language processing, Computational Linguistics, v.22 n.1, p.39-71, March 1996
• 2: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 3: R. D. Burke, K. J. Hammond, V. A. Kulyukin, S. L. Lytinen, N. Tomuro, and S. Schoenberg. Question answering from frequently asked question files: Experiences with the faq finder system. AI Magazine, 18(2):57--66, 1997.
• 4: D. Harman. Overview of the first text retrieval conference (trec-1). In Proceedings of the First TREC Conference, pages 1--20, 1992.
• 5: J. Hwang, S. Lay, and A. Lippman. Nonparametric multivariate density estimation: A comparative study. IEEE Transactions of Signal Processing, 42(10):2795--2810, 1994.
• 6: Jiwoon Jeon , W. Bruce Croft , Joon Ho Lee, Finding similar questions in large question and answer archives, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099572]
• 7: J. Jeon and R. Manmatha. Using maximum entropy for automatic image annotation. Image and Video Retrieval Third International Conference, CIVR 2004, Proceedings Series: Lecture Notes in Computer Science, 3115:24--32, 2004.
• 8: Valentin Jijkoun , Maarten de Rijke, Retrieving answers from frequently asked questions pages on the web, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099571]
• 9: Harksoo Kim , Jungyun Seo, High-performance FAQ retrieval using an automatic clustering method of query logs, Information Processing and Management: an International Journal, v.42 n.3, p.650-661, May 2006 [doi>10.1016/j.ipm.2005.04.002]
• 10: Jon M. Kleinberg, Authoritative sources in a hyperlinked environment, Journal of the ACM (JACM), v.46 n.5, p.604-632, Sept. 1999 [doi>10.1145/324133.324140]
• 11: Wessel Kraaij , Thijs Westerveld , Djoerd Hiemstra, The Importance of Prior Probabilities for Entry Page Search, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564383]
• 12: Leah S. Larkey, Automatic essay grading using text categorization techniques, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.90-95, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.290965]
• 13: Mario Lenz , André Hübner , Mirjam Kunze, Question Answering with Textual CBR, Proceedings of the Third International Conference on Flexible Query Answering Systems, p.236-247, May 13-15, 1998
• 14: Xiaoyan Li , W. Bruce Croft, Time-based language models, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956951]
• 15: Robert Malouf, A comparison of algorithms for maximum entropy parameter estimation, proceeding of the 6th conference on Natural language learning, p.1-7, August 31, 2002 [doi>10.3115/1118853.1118871]
• 16: K. Nigam, J. Lafferty, and A. McCallum. Using maximum entropy for text classification. In Proceedings of IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61--67, 1999.
• 17: Bo Pang , Lillian Lee , Shivakumar Vaithyanathan, Thumbs up?: sentiment classification using machine learning techniques, Proceedings of the ACL-02 conference on Empirical methods in natural language processing, p.79-86, July 06, 2002 [doi>10.3115/1118693.1118704]
• 18: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 19: E. Sneiders. Automated faq answering: Continued experience with shallow language understanding. In Proceedings for the 1999 AAAI Fall Symposium on Question Answering Systems, 1999.
• 20: Diane M. Strong , Yang W. Lee , Richard Y. Wang, Data quality in context, Communications of the ACM, v.40 n.5, p.103-110, May 1997 [doi>10.1145/253769.253804]
• 21: Chung-Hsien Wu , Jui-Feng Yeh , Ming-Jun Chen, Domain-specific FAQ retrieval using independent aspects, ACM Transactions on Asian Language Information Processing (TALIP), v.4 n.1, p.1-17, March 2005 [doi>10.1145/1066078.1066079]
• 22: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to Ad Hoc information retrieval, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.334-342, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.384019]
• 23: Yun Zhou , W. Bruce Croft, Document quality models for web ad hoc retrieval, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099652]
• 24: Xiaolan Zhu , Susan Gauch, Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.288-295, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345602]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.0 General

General Terms:
Algorithms, Experimentation, Measurement

Keywords:
document quality, information retrieval, language models, maximum entropy