|
 ABSTRACT
Two stages in measurement of techniques for information
retrieval are gathering of documents for relevance assessment and
use of the assessments to numerically evaluate effectiveness. We
consider both of these stages in the context of the TREC
experiments, to determine whether they lead to measurements that
are trustworthy and fair. Our detailed empirical investigation of
the TREC results shows that the measured relative performance of
systems appears to be reliable, but that recall is overestimated:
it is likely that many relevant documents have not been found. We
propose a new pooling strategy that can significantly in- crease
the number of relevant documents found for given effort, without
compromising fairness.
REFERENCES
Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.
| |
1
|
|
| |
2
|
D. Harman. Overview of the fourth text retrieval conference (TREC-4). In D. Harman, editor, Proc. Text Retrieval Conference (TREC), October 1995.
|
| |
3
|
|
| |
4
|
S.P. Hatter. The Cranfield II relevance assessments: A critical evaluation. Library Quarterly, 41:229-243, 1971.
|
| |
5
|
|
| |
6
|
C. Howson and P. Urbach. Scientific Reasoning: The Bayesian Approach, second edition. Open Court, Chicago Illinois, 1993.
|
| |
7
|
M.E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4(4):343-359, 1969.
|
 |
8
|
|
| |
9
|
|
| |
10
|
|
| |
11
|
D.R. Swanson. Some unexplained aspects of the Cranfield tests of indexing performance factors. Library Quarterly, 41:223-228, 1971.
|
| |
12
|
|
| |
13
|
J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In D. Harman, editor, Proc. Text Retrieval Conference (TREC), pages 385-398, 1994.
|
| |
14
|
E. Voorhees and D. Harman. Overview of the fifth text retrieval conference (TREC-5). In E. Voorhees and D. Harman, editors, Proc. Text Retrieval Conference (TREC), November 1996.
|
| |
15
|
|
| |
16
|
|
CITED BY 90
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Bodo Billerbeck , Falk Scholer , Hugh E. Williams , Justin Zobel, Query expansion using associated queries, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA
|
|
|
|
|
|
|
|
|
|
|
|
Javed A. Aslam , Virgiliu Pavlu , Robert Savell, A unified model for metasearch, pooling, and system evaluation, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Marcos André Gonçalves , Edward A. Fox , Aaron Krowne , Pável Calado , Alberto H. F. Laender , Altigran S. da Silva , Berthier Ribeiro-Neto, The effectiveness of automatically structured queries in digital libraries, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
W. S. Wong , R. W. P. Luk , H. V. Leong , K. S. Ho , D. L. Lee, Re-examining the effects of adding relevance information in a relevance feedback environment, Information Processing and Management: an International Journal, v.44 n.3, p.1086-1116, May, 2008
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Ben Carterette , Virgil Pavlu , Evangelos Kanoulas , Javed A. Aslam , James Allan, Evaluation over thousands of queries, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
|
|
|
Patrick Ruch , Imad Tbahriti , Julien Gobeill , Alan R. Aronson, Argumentative feedback: a linguistically-motivated term expansion for information retrieval, Proceedings of the COLING/ACL on Main conference poster sessions, p.675-682, July 17-18, 2006, Sydney, Australia
|
|
|
|
|
|
|
|
|
K. Y. Lin , S. H. Hsieh , H. P. Tserng , K. W. Chou , H. T. Lin , C. P. Huang , K. F. Tzeng, Enabling the creation of domain-specific reference collections to support text-based information retrieval experiments in the architecture, engineering and construction industries, Advanced Engineering Informatics, v.22 n.3, p.350-361, July, 2008
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Tanuja Bompada , Chi-Chao Chang , John Chen , Ravi Kumar , Rajesh Shenoy, On the robustness of relevance measures with incomplete judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
|
|
|
|
|
|
|
|
|
|
|
|
Abolfazl AleAhmad , Hadi Amiri , Masoud Rahgozar , Farhad Oroumchian, Experiments with English-Persian text retrieval, Proceeding of the 2nd ACM workshop on Improving non english web searching, October 30-30, 2008, Napa Valley, California, USA
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Jianhan Zhu , Jun Wang , Vishwa Vinay , Ingemar J. Cox, Topic (query) selection for IR evaluation, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
|
|
|
|
|
|
Abolfazl AleAhmad , Hadi Amiri , Ehsan Darrudi , Masoud Rahgozar , Farhad Oroumchian, Hamshahri: A standard Persian text collection, Knowledge-Based Systems, v.22 n.5, p.382-387, July, 2009
|
|
|
|
|
|
Javed A. Aslam , Evangelos Kanoulas , Virgil Pavlu , Stefan Savev , Emine Yilmaz, Document selection methodologies for efficient and effective learning-to-rank, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA
|
|
|
|
|
|
|
|
|
|
|
Adobe Acrobat
QuickTime
Windows Media Player
Real Player
Pdf
H.3