Evaluating high accuracy retrieval techniques

ABSTRACT

Although information retrieval research has always been concerned with improving the effectiveness of search, in some applications, such as information analysis, a more specific requirement exists for high accuracy retrieval. This means that achieving high precision in the top document ranks is paramount. In this paper we present work aimed at achieving high accuracy in ad-hoc document retrieval by incorporating approaches from question answering(QA). We focus on getting the first relevant result as high as possible in the ranked list and argue that traditional precision and recall are not appropriate measures for evaluatin this task. We instead use the mean reciprocal rank(MRR) of the first relevant result. We evaluate three different methods for modifying queries to achieve high accuracy. The experiments done on TREC data provide support for the approach of using MRR and incorporating QA techniques for getting high accuracy in ad-hoc retrieval task.

REFERENCES

• 1: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 2: Sabine Buchholz and Walter Daelemans. Shapaqa: Shallow parsing for question answering on the world wide web. In Proceedings Euroconference Recent Advances in Natural Language Processing(RANLP), pages 47--51, 2001.
• 3: C. L. A. Clarke, G. V. Cormack, D. I. E. Kisman, and T. R. Lynam. Question answering by passage selection(multitext experiments for trec-9). In Proceedings of Text REtrieval Conference, 2000.
• 4: N. Craswell and D. Hawking. Overview of the TREC-2002 Web Track. In Proceedings of the Eleventh Text REtrieval Conference(TREC), November 2002.
• 5: W. Bruce Croft. What do people want from information retrieval? D-Lib Magazine, November 1995.
• 6: S. Cronen-Townsend and W. B. Croft. Quantifying query ambiguity. In Proceedings of HLT, pages 94--98, 2002.
• 7: Steve Cronen-Townsend , Yun Zhou , W. Bruce Croft, Predicting query performance, Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, August 11-15, 2002, Tampere, Finland [doi>10.1145/564376.564429]
• 8: Ralph Grishman, Information Extraction: Techniques and Challenges, International Summer School on Information Extraction: A Multidisciplinary Approach to an Emerging Information Technology, p.10-27, January 1997
• 9: Ulf Hermjakob and Abdessamad Echihabi. Natural language based reformulation resource and web. In Proceedings of the Eleventh Text REtrieval Conference(TREC), November 2002.
• 10: David A. Hull, Stemming algorithms: a case study for detailed evaluation, Journal of the American Society for Information Science, v.47 n.1, p.70-84, Jan. 1996 [doi>10.1002/(SICI)1097-4571(199601)47:1<70::AID-ASI7>3.3.CO;2-Q]
• 11: Aravind K. Joshi , B. Srinivas, Disambiguation of super parts of speech (or supertags): almost parsing, Proceedings of the 15th conference on Computational linguistics, August 05-09, 1994, Kyoto, Japan [doi>10.3115/991886.991912]
• 12: Victor Lavrenko , W. Bruce Croft, Relevance based language models, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.120-127, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383972]
• 13: Steve Lawrence and C. Lee Giles. Searching the Web: General and Scientific Information Access. IEEE Communications Magazine, January 1999.
• 14: Jimmy Lin and Boris Katz. Question answering techniques for the world wide web. In Tutorial presentation at EACL, 2003.
• 15: J. B. Lovins. Development of a stemming algorithm. Mechanical Translation and Computational Linguistics, 11(1-2):22--31, 1968.
• 16: Rila Mandala, Takenobu Tokunaga, and Hozumi Tanaka. The use of WordNet in information retrieval. In Sanda Harabagiu, editor, Use of WordNet in Natural Language Processing Systems: Proceedings of the Conference, pages 31--37. Association for Computational Linguistics, Somerset, New Jersey, 1998.
• 17: Wesley W. Chu Frank Meng. Database query formation from natural language using semantic modeling and statistical keyword meaning disambiguation. Technical Report 990003, 16, 1999.
• 18: Donald Metzler , W. Bruce Croft, Combining the language model and inference network approaches to retrieval, Information Processing and Management: an International Journal, v.40 n.5, p.735-750, September 2004 [doi>10.1016/j.ipm.2004.05.001]
• 19: Donald Metzler , W. Bruce Croft, Analysis of Statistical Question Classification for Fact-Based Questions, Information Retrieval, v.8 n.3, p.481-504, May 2005 [doi>10.1007/s10791-005-6995-3]
• 20: George A. Miller, Richard Beckwith, Christian Fellbaum, Derek Gross, and Katherine Miller. Introduction to WordNet: An On-line Lexical Database. Cognitive Science Laboratory, Princeton University, August 1993.
• 21: Pedersen, Ted, and Rebecca Bruce. Distinguishing word senses in untagged text. In Proceedings of the 2nd Conference on Empirical Methods in NLP(EMNLP-2), Providence, August 1997.
• 22: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 23: Deepak Ravichandran and Eduard Hovy. Learning surface text patterns for a question answering system. In Proceedings of ACL, 2002.
• 24: Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
• 25: Mark Sanderson, Word sense disambiguation and information retrieval, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.142-151, July 03-06, 1994, Dublin, Ireland
• 26: E. Voorhees. The TREC-8 Question Answering Track Report. In Proceedings of the Eighth Text REtrieval Conference(TREC), November 1999.
• 27: E. Voorhees. Overview of the TREC-9 Question Answering Track. In Proceedings of the Ninth Text REtrieval Conference(TREC), November 2000.
• 28: E. Voorhees. Overview of the TREC 2001 Question Answering Track. In Proceedings of the Tenth Text REtrieval Conference(TREC), November 2001.
• 29: E. Voorhees. Overview of the TREC 2002 Question Answering Track. In Proceedings of the Eleventh Text REtrieval Conference(TREC), November 2002.
• 30: E. M. Voorhees. On expanding query vectors with lexically related wordds. Technical report, NIST, 1993.
• 31: Ellen M. Voorhees, Using WordNet to disambiguate word senses for text retrieval, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.171-180, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160715]
• 32: Ellen M. Voorhees, Query expansion using lexical-semantic relations, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.61-69, July 03-06, 1994, Dublin, Ireland
• 33: Jinxi Xu , W. Bruce Croft, Query expansion using local and global document analysis, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.4-11, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243202]
• 34: Yiming Yang , John Wilbur, Using corpus statistics to remove redundant words in text categorization, Journal of the American Society for Information Science, v.47 n.5, p.357-369, May 1996 [doi>10.1002/(SICI)1097-4571(199605)47:5<357::AID-ASI3>3.3.CO;2-0]
• 35: Dell Zhang , Wee Sun Lee, Question classification using support vector machines, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860443]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

General Terms:
Experimentation, Measurement, Performance

Keywords:
ad-hoc retrieval, high accuracy retrieval, question answering