Efficient construction of large test collections

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Efficient construction of large test collections

Full text

Pdf (1.02 MB)

Source

Annual ACM Conference on Research and Development in Information Retrieval archive
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval table of contents

Melbourne, Australia

Pages: 282 - 289

Year of Publication: 1998

ISBN:1-58113-015-5

Authors

Gordon V. Cormack	Department of Computer Science, University of Waterloo, Waterloo, Ontario, Canada
Christopher R. Palmer	Department of Computer Science, University of Waterloo, Waterloo, Ontario, Canada
Charles L. A. Clarke	Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada

Sponsors

University of Melbourne : University of Melbourne
SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 9, Downloads (12 Months): 67, Citation Count: 47

Additional Information:

references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/290941.291009 What is a DOI?

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	C. L. A. Clarke and G. V. Cormack. Interactive substring retrival. In D. K. Harman and E. M. Voorhees, editors, information Technology: The Fifth Text REtrieval Conference (TREC-5), Gaithersburg, Maryland, November 1996. National Institute of Standards and Technology (NIST), United States Department of Commerce. Available electronically at http://trec .nist .gov.
	2	C. L. A. Clarke, G. V. Cormack, and F. J. Burkowski. Shortest substring ranking. In D. K. Harman, editor, The Fourth Text REtrieval Conference (TREC-$), pages 295-304, Gaithersburg, Maryland, November 1995. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-238. Available electronically at http:/#tree, hist. gov.
	3	G. V. Cormack, C. L. A. Clarke, C. 1t. Palmer, and S. S.-L. To. Passage based refinement. In Sixth Text REtrieval Conference (TREC-6), Gaithersburg, Maryland, November 1997. National Institute of Standards and Technology (NIST), United States Department of Commerce. Available electronically at http://tree .nisz .gov.
	4	H. Gilbert and K. S. Jones. Statistical bases of relevance assessment for the 'ideal' information retrieval test collection. Technical report, Computer Laboratory, University of Cambridge, 1979. BL R&D Report 5481.
	5	Donna Harman, Overview of the first TREC conference, Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, p.36-47, June 27-July 01, 1993, Pittsburgh, Pennsylvania, United States [doi>10.1145/160688.160692]
	6	D. Harman. Overview of the fourth Text RE- trieval Conference (TREC-4). In D. K. Harman, editor, The Fourth Text REtrieval Conference (TREC- #{), pages 1-23, Gaithersburg, Maryland, November 1995. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-236. Available electronically at http://tree .aist .gov.
	7	D. K. Harman, editor. The First Text REtrieval Conference (TREC-1), Gaithersburg, Maryland, November 1992. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-207.
	8	Donna Harman, Overview of the second text retrieval conference (TREC-2), Proceedings of the second conference on Text retrieval conference, p.271-289, May 1995, Washington, D.C., United States
	9	D. K. Harmon, editor. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, Maryland, November 1994. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-225. Available electronically at http://trec, hist. gov.
	10	D. K. Harmon. Overview of the third Text REtrieval Conference (TREC-3). In D. K. Harmon, editor, Overview of the Third Te#t REtrieval Conference (TREC-3), pages 1-19, Gaithersburg, Maryland, November 1994. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-225. Available electronically at http://trec .hist. gov.
	11	D. K. Harmon, editor. The Fourth Text REtrieval Conference (TRBC-j), Gaithersburg, Maryland, November 1995. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-236. Available electronically at http://trec .nist. gov.
	12	D.K. Harman and E. M. Voorhees, editors. Information Technology: The Fifth Text REtrieval Conference (TREC-5), Gaithersburg, Maryland, November 1996. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-238. Available electronically at http://trec.nist.gov.
	13	D. K. Harman and E. M. Voorhees, editors. The Sixth Text REtrieval Conference (TREC-5), Gaithersburg, Maryland, November 1997. National Institute of Standards and Technology (NIST), United States Department of Commerce. Available electronically at http://tree .n~st. gov.
	14	M. E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Management, 4:343-359, 1966.
	15	E. V. Paul B. Kantor. Report on the TREC-5 confusion track. In D. K. Harmon, editor, Information Technology: The Fifth Text REtrieval Conference (TREC-5), pages 65-74, Gaithersburg, Maryland, November 1996. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-236. Available electronically at http://trec .nist. gov.
	16	S. E. Robertson. The probability ranking principle in it. Journal of Documentation, 33:294-304, 1977.
	17	Gregory Grefenstette, Cross-linguistic information retrieval workshop, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.344, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243323]
	18	K. Sparck Jones and C. J. Van Rijsbergen. Report on the need for and provision of an 'ideal' test collection. Technical report, University Computer Laboratory, Cambridge, 1975.
	19	K. Sparck Jones and C. J. Van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59-72, March 1976.
	20	J. Tague-Sutcliffe and J. Blustein. A statistical analysis of the TREC-3 data. In D. K. Harman, editor, Overview of the Third Text REtrieval Con}erence (TREC-3), pages 385-398, Gaithersburg, Maryland, November 1994. National Institute of Standards and Technology (NIST), United States Department of Commerce. NIST Special Publication 500-225. Available electronically at http://trec.nist, gov.
	21	Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.315-323, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291017]
	22	N. West. Applied Statistics for Marine Affairs Professionals. Praeger, Westport, CT, 1996.
	23	Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

CITED BY 47

	Vo Ngoc Anh , Alistair Moffat, Improved retrieval effectiveness through impact transformation, Australian Computer Science Communications, v.24 n.2, p.41-47, January-February 2002
	Gordon V. Cormack , Ondrej Lhotak , Christopher R. Palmer, Estimating precision by random sampling (poster abstract), Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.273-274, August 15-19, 1999, Berkeley, California, United States
	MetaSpider: meta-searching and categorization on the Web, Journal of the American Society for Information Science and Technology, v.52 n.13, p.1134-1147, November 2001
	Ian Soboroff , Charles Nicholas , Patrick Cahan, Ranking retrieval systems without relevance judgments, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.66-73, September 2001, New Orleans, Louisiana, United States
	Vo Ngoc Anh , Owen de Kretser , Alistair Moffat, Vector-space ranking with effective early termination, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.35-42, September 2001, New Orleans, Louisiana, United States
	Mingfang Wu , Michael Fuller , Ross Wilkinson, The role of a judge in a user based retrieval experiment (poster session), Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.331-333, July 24-28, 2000, Athens, Greece
	Javed A. Aslam , Robert Savell, On the effectiveness of evaluating retrieval systems in the absence of relevance judgments, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada
	Shengli Wu , Fabio Crestani, Methods for ranking information retrieval systems without relevance judgments, Proceedings of the 2003 ACM symposium on Applied computing, March 09-12, 2003, Melbourne, Florida
	Javed A. Aslam , Virgiliu Pavlu , Robert Savell, A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada
	Javed A. Aslam , Virgiliu Pavlu , Robert Savell, A unified model for metasearch, pooling, and system evaluation, Proceedings of the twelfth international conference on Information and knowledge management, November 03-08, 2003, New Orleans, LA, USA
	Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece
	Marcos André Gonçalves , Edward A. Fox , Aaron Krowne , Pável Calado , Alberto H. F. Laender , Altigran S. da Silva , Berthier Ribeiro-Neto, The effectiveness of automatically structured queries in digital libraries, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2004, Tuscon, AZ, USA
	Chris Buckley , Ellen M. Voorhees, Retrieval evaluation with incomplete information, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
	Douglas W. Oard , Dagobert Soergel , David Doermann , Xiaoli Huang , G. Craig Murray , Jianqiang Wang , Bhuvana Ramabhadran , Martin Franz , Samuel Gustman , James Mayfield , Liliya Kharevych , Stephanie Strassel, Building an information retrieval test collection for spontaneous conversational speech, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
	Mark Sanderson , Hideo Joho, Forming test collections with no system pooling, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom
	Jimmy Lin, Evaluation of resources for question answering evaluation, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Ben Carterette , James Allan, Incremental test collections, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany
	Jack G. Conrad , Cindy P. Schriber, Managing déjà vu: Collection building for the identification of nonidentical duplicate documents, Journal of the American Society for Information Science and Technology, v.57 n.7, p.921-932, May 2006
	Jimmy Lin , Boris Katz, Building a reusable test collection for question answering, Journal of the American Society for Information Science and Technology, v.57 n.7, p.851-861, May 2006
	Ben Carterette, Robust test collections for retrieval evaluation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Eric C. Jensen , Steven M. Beitzel , Abdur Chowdhury , Ophir Frieder, Repeatable evaluation of search services in dynamic environments, ACM Transactions on Information Systems (TOIS), v.26 n.1, p.1-es, November 2007
	Mark Baillie , Leif Azzopardi , Ian Ruthven, Evaluating epistemic uncertainty under incomplete assessments, Information Processing and Management: an International Journal, v.44 n.2, p.811-837, March, 2008
	Ben He , Craig Macdonald , Iadh Ounis, Retrieval sensitivity under training using different measures, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	I. E. Kuralenok , I. S. Nekrestyanov, Evaluation of Text Retrieval Systems, Programming and Computing Software, v.28 n.4, p.226-242, July-August 2002
	Emine Yilmaz , Javed A. Aslam, Estimating average precision with incomplete and imperfect judgments, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Chris Buckley , Darrin Dimmick , Ian Soboroff , Ellen Voorhees, Bias and the limits of pooling, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Ben Carterette , James Allan , Ramesh Sitaraman, Minimal test collections for retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Javed A. Aslam , Virgil Pavlu , Emine Yilmaz, A statistical method for system evaluation using incomplete judgments, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Gordon V. Cormack , Thomas R. Lynam, Statistical precision of information retrieval evaluation, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Ian Soboroff, Dynamic test collections: measuring search effectiveness on the live web, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Huyen-Trang Vu , Patrick Gallinari, A machine learning based approach to evaluating retrieval systems, Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.399-406, June 04-09, 2006, New York, New York
	Massimo Melucci, On rank correlation in information retrieval evaluation, ACM SIGIR Forum, v.41 n.1, p.18-33, June 2007
	Anselm Spoerri, Using the structure of overlap between search results to rank retrieval systems without relevance judgments, Information Processing and Management: an International Journal, v.43 n.4, p.1059-1070, July, 2007
	Kuansan Wang , Toby Walker , Zijian Zheng, PSkip: estimating relevance ranking quality from web search clickthrough data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Javed A. Aslam , Emine Yilmaz, Inferring document relevance from incomplete information, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Alistair Moffat , William Webber , Justin Zobel, Strategic system comparisons via targeted relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Gordon V. Cormack , Thomas R. Lynam, Power and bias of subset pooling strategies, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Ben Carterette , James Allan, Semiautomatic evaluation of retrieval systems using document similarities, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal
	Kazuko Kuriyama , Noriko Kando , Toshihiko Nozue , Koji Eguchi, Pooling for a Large-Scale Test Collection: An Analysis of the Search Results from the First NTCIR Workshop, Information Retrieval, v.5 n.1, p.41-59, January 2002
	Tom Rowlands , David Hawking , Ramesh Sankaranarayana, Workload sampling for enterprise search evaluation, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Tetsuya Sakai, Alternatives to Bpref, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Tonya Custis , Khalid Al-Kofahi, A new approach for evaluating query expansion: query-document term mismatch, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Peter Bailey , Nick Craswell , Ian Soboroff , Paul Thomas , Arjen P. de Vries , Emine Yilmaz, Relevance assessment: are judges exchangeable and does it matter, Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, July 20-24, 2008, Singapore, Singapore
	Tetsuya Sakai , Noriko Kando, On information retrieval metrics designed for evaluation with incomplete relevance assessments, Information Retrieval, v.11 n.5, p.447-470, October 2008
	David E. Losada , Leif Azzopardi , Mark Baillie, Revisiting the relationship between document length and relevance, Proceeding of the 17th ACM conference on Information and knowledge management, October 26-30, 2008, Napa Valley, California, USA
	Emine Yilmaz , Javed A. Aslam, Estimating average precision when judgments are incomplete, Knowledge and Information Systems, v.16 n.2, p.173-211, July 2008
	William Webber , Laurence A. F. Park, Score adjustment for correction of pooling bias, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, July 19-23, 2009, Boston, MA, USA