Scaling IR-system evaluation using term relevance sets

ABSTRACT

This paper describes an evaluation method based on Term Relevance Sets Trels that measures an IR system's quality by examining the content of the retrieved results rather than by looking for pre-specified relevant pages. Trels consist of a list of terms believed to be relevant for a particular query as well as a list of irrelevant terms. The proposed method does not involve any document relevance judgments, and as such is not adversely affected by changes to the underlying collection. Therefore, it can better scale to very large, dynamic collections such as the Web. Moreover, this method can evaluate a system's effectiveness on an updatable "live" collection, or on collections derived from different data sources. Our experiments show that the proposed method is very highly correlated with official TREC measures.

REFERENCES

• 1: Steven M. Beitzel , Eric C. Jensen , Abdur Chowdhury , David Grossman, Using titles and category names from editor-driven taxonomies for automatic evaluation, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA [doi>10.1145/956863.956868]
• 2: Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , Geoffrey Zweig, Syntactic clustering of the Web, Selected papers from the sixth international conference on World Wide Web, p.1157-1166, September 1997, Santa Clara, California, United States
• 3: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 4: David Carmel, Einat Amitay, Miki Herscovici, Yoelle S. Maarek, Yael Petruschka, and Aya Soffer. Juru at TREC 10 - Experiments with Index Pruning. In Proceeding of Tenth Text REtrieval Conference(TREC-10). National Institute of Standards and Technology. NIST, 2001.
• 5: Junghoo Cho , Hector Garcia-Molina , Lawrence Page, Efficient crawling through URL ordering, Computer Networks and ISDN Systems, v.30 n.1-7, p.161-172, April 1, 1998
• 6: Cyril W. Cleverdon, The significance of the Cranfield tests on index languages, Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, p.3-12, October 13-16, 1991, Chicago, Illinois, United States [doi>10.1145/122860.122861]
• 7: Ronald Fagin , Ravi Kumar , D. Sivakumar, Comparing top k lists, Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, January 12-14, 2003, Baltimore, Maryland
• 8: Dennis Fetterly , Mark Manasse , Marc Najork , Janet Wiener, A large-scale study of the evolution of web pages, Proceedings of the 12th international conference on World Wide Web, May 20-24, 2003, Budapest, Hungary [doi>10.1145/775152.775246]
• 9: D. Hawking and N. Craswell. Overview of the TREC-2001 Web Track. In E. M. Voorhees and D. K. Harman, editors, Proceedings of the Tenth Text Retrieval Conference(TREC-10). National Institute of Standards and Technology. NIST, 2001.
• 10: David Hawking , Nick Craswell , Paul Thistlewaite , Donna Harman, Results and challenges in Web search evaluation, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.11-16, p.1321-1330, May 17, 1999
• 11: Steve Lawrence and C. Lee Giles. Searching the world wide web. Science, 280, April 1998.
• 12: Y. S. Maarek , F. Z. Smadja, Full text indexing based on lexical relations an application: software libraries, Proceedings of the 12th annual international ACM SIGIR conference on Research and development in information retrieval, p.198-206, June 25-28, 1989, Cambridge, Massachusetts, United States
• 13: Marc Najork , Janet L. Wiener, Breadth-first crawling yields high-quality pages, Proceedings of the 10th international conference on World Wide Web, p.114-118, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.371965]
• 14: Rabia Nuray , Fazli Can, Automatic ranking of retrieval systems in imperfect environments, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860510]
• 15: John O'Connor, Automatic Subject Recognition in Scientific Papers: An Empirical Study, Journal of the ACM (JACM), v.12 n.4, p.490-515, Oct. 1965 [doi>10.1145/321296.321299]
• 16: Ian Soboroff , Charles Nicholas , Patrick Cahan, Ranking retrieval systems without relevance judgments, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.66-73, September 2001, New Orleans, Louisiana, United States [doi>10.1145/383952.383961]
• 17: Ian Soboroff , Ellen Voorhees , Nick Craswell, Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections, ACM SIGIR Forum, v.37 n.2, Fall 2003 [doi>10.1145/959258.959267]
• 18: M. M. Soubbotin. Patterns of potential answer expressions as clues to the right answers. In Proceeding of Tenth Text REtrieval Conference(TREC-10). National Institute of Standards and Technology. NIST, 2001.
• 19: Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, p.355-370, September 03-04, 2001
• 20: Ellen M. Voorhees and Donna Harman. Overview of the eighth text retrieval conference(trec-8). Information Processing and Management, 36(1):3--35, 2000.
• 21: Shengli Wu , Fabio Crestani, Methods for ranking information retrieval systems without relevance judgments, Proceedings of the 2003 ACM symposium on Applied computing, March 09-12, 2003, Melbourne, Florida [doi>10.1145/952532.952693]
• 22: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Measurement, Performance

Keywords:
IR-system evaluation, keywords, trels, web search evaluation