Evaluation of evaluation in information retrieval

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Evaluation of evaluation in information retrieval

Full text

Pdf (1.09 MB)

Source

Annual ACM Conference on Research and Development in Information Retrieval archive
Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval table of contents

Seattle, Washington, United States

Pages: 138 - 146

Year of Publication: 1995

ISBN:0-89791-714-6

Author

Tefko Saracevic

School of Communication, Information and Library Studies, Rutgers University, 4 Huntington St., New Brunswick, NJ

Sponsors

BCS-ISRG : BCS-ISRG
CEPIS : Council of European Professional Informatics Societies
AICA : Assoc Italianai de Calcolo Automatico
BCS-IRSG : BCS/Information Retrieval Specialist Group
German Comp Soc : GI - Gesellshaft for Informatik
IPSJ : Information Processing Society of Japan
SIGIR: ACM Special Interest Group on Information Retrieval
DD :

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 16, Downloads (12 Months): 122, Citation Count: 31

Additional Information:

references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/215206.215351 What is a DOI?

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	David C. Blair , M. E. Maron, An evaluation of retrieval effectiveness for a full-text document-retrieval system, Communications of the ACM, v.28 n.3, p.289-299, March 1985 [doi>10.1145/3166.3197]
	2	David C. Blair , M. E. Maron, Full-text information retrieval: further analysis and clarification, Information Processing and Management: an International Journal, v.26 n.3, p.437-447, 1990 [doi>10.1016/0306-4573(90)90102-8]
	3	C. L. Borgman, All users of information retrieval systems are not created equal: an exploration into individual differences, Information Processing and Management: an International Journal, v.25 n.3, p.237-251, May 1989 [doi>10.1016/0306-4573(89)90042-3]
	4	Bush, V (1945). As we may thhak. Atlantw Monthly, 176, (1), 101-108.
	5	Churchman, C.W. (1968). The ,~vstems approach. New York. Delta.
	6	Cleverdon, C.W., Mills, J. & Keen, E.M. (1966). An mqutl3, m testing ofmfo1~ation retrteval systems. (2 vols.). Cranfield, U.K.: Aslib Cranfield Research Project, College of Aeronautics.
	7	Cooper, W.S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19, ( 1 ), 30-41.
	8	William S. Cooper, The formalism of probability theory in IR: a foundation or an encumbrance?, Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, p.242-247, July 03-06, 1994, Dublin, Ireland
	9	Daniel Crevier, AI: the tumultuous history of the search for artificial intelligence, Basic Books, Inc., New York, NY, 1993
	10	Dalrymple, P.W & Roderer, N. K. (1994). Database access systems. In Williams, M. E. (Ed.)Annual Review of. Informatton Scwnce and Technology,, vol. 29, (pp. 137-178). Medford, NJ: Learned hfformation.
	11	Dervm, B. & Nilan, M.S. (1986). Ilffonnation needs and use. In Williams, M. E. (Ed.) Annual Review of Informanon Scwnce and Technology,, vol. 21, (pp.3-33). White Plains, NY: Knowledge industry.
	12	Fenichel, C.H. (I 98 I). Online searching: measures that discriminate among users with different types of experience. Journal of the American Society for Information Science, 32, (1), 23-32.
	13	Fidel, R. (1991) Searchers' selection of search keys. (3 parts). Journal of the American Soctetyfor Information Science, 42, (7), 490-527.
	14	Donna Harman, Overview of the second text retrieval conference (TREC-2), Information Processing and Management: an International Journal, v.31 n.3, p.271-289, May-June 1995
	15	Haynes, R.B.., McKibbon, K.A., Ryan, N., Fitzgerald, D., & Ramsden, M.F. (1990). Online access to MEDLINE m clinical setting: A study of use and usefuhaess. Annals of Internal Medicine, 112, 78-84.
	16	Peter Ingwersen, Information retrieval interaction, Taylor Graham Publishing, London, UK, 1992
	17	Kent, A., Berry, M., Leuhrs, F.U. & Perry, J.W. (1955). Machhae literature searching VIII. Operational criteria for designing uffonnation retrieval systems. Amerwan Documentanon, 6, (2), 93-101.
	18	Lindberg, D.A~B., Siegel, E.R., Rapp, B.A., Wallhlgford, K.T., Wilson, S.R. (1993). Use of MEDLINE by physicians for clinical problem solving. JA~/A, The Journal of the Amerwan A'ledwal Association, 269, (24), 3124-3129.
	19	Daniel E. Meyer , Den Ruiz, End-user selection of databases—part I: science/technology/medicine, Database, v.13 n.3, p.21-29, Jun. 1990
	20	Mooers, C.N. (1951). Zatoeodmg applied to mechanical organization of knowledge. American Documentation, 2, 20- 32.
	21	Rapp, B.A., Siegel, E. R., Woodsmall, R., Lyon-Hartmann, B. (1990). Evaluating MEDLINE on CD-ROM: Aa~ overview of field tests m library and elhfical settings. Online Review. } 4, (3), 172-186
	22	S. E. Robertson , M. M. Hancock-Beaulieu, On the evaluation of IR systems, Information Processing and Management: an International Journal, v.28 n.4, p.457-466, July-Aug. 1992 [doi>10.1016/0306-4573(92)90004-J]
	23	G. Salton, The SMART Retrieval System—Experiments in Automatic Document Processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1971
	24	Gerard Salton, Another look at automatic text-retrieval systems, Communications of the ACM, v.29 n.7, p.648-656, July 1986 [doi>10.1145/6138.6149]
	25	Gerard Salton, Automatic text processing: the transformation, analysis, and retrieval of information by computer, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1989
	26	Gerard Salton, The state of retrieval system evaluation, Information Processing and Management: an International Journal, v.28 n.4, p.441-449, July-Aug. 1992 [doi>10.1016/0306-4573(92)90002-H]
	27	Saracevie, T. (1975). Relevance' A review of and a framework for the thinking on the notion in hffonnation science. Journal of the American Socw(y,for Informatzon Science, 26, (6), 321- 343.
	28	Saracevie, T., Kantor, P. Chamis, A.Y., & Trivison, D. (1988). A study of information seeking and retrieving. (3 parts). Journal of the Amerwan Socte(yfor Information Scwnce, 39, (3), 161-216.
	29	Sehamber, L. (1994). Relevance and hfformation behavior, in Williams, M. E. (Ed.) Annual Revtew of Information Science and Technology, vol. 29, (pp. 3-48). Medford, NJ: Learned Information.
	30	Karen Sparck Jones, Information Retrieval Experiment, Butterworth-Heinemann, Newton, MA, 1981
	31	Karen Sparck Jones, Reflections on TREC, Information Processing and Management: an International Journal, v.31 n.3, p.291-314, May-June 1995
	32	Amanda Spink, Term relevance feedback and mediated database searching: implications for information retrieval practice and systems design, Information Processing and Management: an International Journal, v.31 n.2, p.161-171, March/April 1995 [doi>10.1016/0306-4573(94)00043-3]
	33	Louise T. Su, Evaluation measures for interactive information retrieval, Information Processing and Management: an International Journal, v.28 n.4, p.503-516, July-Aug. 1992 [doi>10.1016/0306-4573(92)90007-M]
	34	Swanson, D.R. (197 I). Some unexplahaed aspects of the Cralffield test of indexing performance factors. L~bra~y Quarterly, 41, (3), 223-228.
	35	Swets, J.A. (1963). hffonnation retrieval systems. Science, 141, (1), 245-250.
	36	Jean Tague-Sutcliffe, The pragmatics of information retrieval experimentation, revisited, Information Processing and Management: an International Journal, v.28 n.4, p.467-490, July-Aug. 1992 [doi>10.1016/0306-4573(92)90005-K]
	37	W. John Wilbur, An information measure of retrieval performance, Information Systems, v.17 n.4, p.283-298, July 1992 [doi>10.1016/0306-4379(92)90019-J]

CITED BY 31

	Marko Balabanović, An adaptive Web page recommendation service, Proceedings of the first international conference on Autonomous agents, p.378-385, February 05-08, 1997, Marina del Rey, California, United States
	Pia Borlund , Peter Ingwersen, Measures of relative relevance and ranked half-life: performance indicators for interactive IR, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.324-331, August 24-28, 1998, Melbourne, Australia
	Amanda Spink, A user-centered approach to evaluating human interaction with web search engines: an exploratory study, Information Processing and Management: an International Journal, v.38 n.3, p.401-426, May 2002
	Ian Soboroff , Charles Nicholas , Patrick Cahan, Ranking retrieval systems without relevance judgments, Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, p.66-73, September 2001, New Orleans, Louisiana, United States
	Eero Sormunen, A novel method for the evaluation of Boolean query effectiveness across a wide operational range, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.25-32, July 24-28, 2000, Athens, Greece
	Joemon M. Jose , Jonathan Furner , David J. Harper, Spatial querying for image retrieval: a user-oriented evaluation, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.232-240, August 24-28, 1998, Melbourne, Australia
	Gabriella Kazai , Mounia Lalmas , Norbert Fuhr , Norbert Gövert, A report on the first year of the INitiative for the evaluation of XML retrieval (INEX'02), Journal of the American Society for Information Science and Technology, v.55 n.6, p.551-556, April 2004
	Mika Käki, Proportional search interface usability measures, Proceedings of the third Nordic conference on Human-computer interaction, p.365-372, October 23-27, 2004, Tampere, Finland
	Javed A. Aslam , Emine Yilmaz , Virgiliu Pavlu, The maximum entropy method for analyzing retrieval measures, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Ali Shiri , Crawford Revie, Query expansion behavior within a thesaurus-enhanced search environment: A user-centered evaluation, Journal of the American Society for Information Science and Technology, v.57 n.4, p.462-478, February 2006
	Daniela Petrelli , Stephen Levin , Micheline Beaulieu , Mark Sanderson, Which user interaction for cross-language information retrieval? Design issues and reflections, Journal of the American Society for Information Science and Technology, v.57 n.5, p.709-722, March 2006
	Chih-Fong Tsai , Ken McGarry , John Tait, Qualitative evaluation of automatic assignment of keywords to images, Information Processing and Management: an International Journal, v.42 n.1, p.136-154, January 2006
	Ye Diana Wang , Guisseppi Forgionne, A decision-theoretic approach to the evaluation of information retrieval systems, Information Processing and Management: an International Journal, v.42 n.4, p.863-874, July 2006
	Xiangzhu Gao , San Murugesan , Bruce Lo, Multi-Dimensional Evaluation of Information Retrieval Results, Proceedings of the 2004 IEEE/WIC/ACM International Conference on Web Intelligence, p.192-198, September 20-24, 2004
	Mark Baillie , Leif Azzopardi , Ian Ruthven, Evaluating epistemic uncertainty under incomplete assessments, Information Processing and Management: an International Journal, v.44 n.2, p.811-837, March, 2008
	I. E. Kuralenok , I. S. Nekrestyanov, Evaluation of Text Retrieval Systems, Programming and Computing Software, v.28 n.4, p.226-242, July-August 2002
	R. Akavipat , L.-S. Wu , F. Menczer , A.G. Maguitman, Emerging semantic communities in peer web search, Proceedings of the international workshop on Information retrieval in peer-to-peer networks, November 11-11, 2006, Arlington, Virginia, USA
	Yiqun Liu , Yupeng Fu , Min Zhang , Shaoping Ma , Liyun Ru, Automatic search engine performance evaluation with click-through data analysis, Proceedings of the 16th international conference on World Wide Web, May 08-12, 2007, Banff, Alberta, Canada
	Arthur R. Taylor , Colleen Cool , Nicholas J. Belkin , William J. Amadio, Relationships between categories of relevance criteria and stage in task completion, Information Processing and Management: an International Journal, v.43 n.4, p.1071-1084, July, 2007
	Jorge R. Herskovic , M. Sriram Iyengar , Elmer V. Bernstam, Using hit curves to compare search algorithm performance, Journal of Biomedical Informatics, v.40 n.2, p.93-99, April, 2007
	Kuansan Wang , Toby Walker , Zijian Zheng, PSkip: estimating relevance ranking quality from web search clickthrough data, Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, June 28-July 01, 2009, Paris, France
	Alistair Moffat , William Webber , Justin Zobel, Strategic system comparisons via targeted relevance judgments, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands
	Ayşe Göker , Hans Myrhaug, Evaluation of a mobile information system in context, Information Processing and Management: an International Journal, v.44 n.1, p.39-65, January, 2008
	Kalervo Järvelin, An analysis of two approaches in information retrieval: From frameworks to study designs, Journal of the American Society for Information Science and Technology, v.58 n.7, p.971-986, May 2007
	Daqing He , Peter Brusilovsky , Jaewook Ahn , Jonathan Grady , Rosta Farzan , Yefei Peng , Yiming Yang , Monica Rogati, An evaluation of adaptive filtering in the context of realistic task-based information exploration, Information Processing and Management: an International Journal, v.44 n.2, p.511-533, March, 2008
	Yan Qu , George W. Furnas, Model-driven formative evaluation of exploratory search: A study under a sensemaking framework, Information Processing and Management: an International Journal, v.44 n.2, p.534-555, March, 2008
	Daniela Petrelli, On the role of user-centred evaluation in the advancement of interactive information retrieval, Information Processing and Management: an International Journal, v.44 n.1, p.22-38, January, 2008
	Diane Kelly , Chirag Shah , Cassidy R. Sugimoto , Earl W. Bailey , Rachael A. Clemens , Ann K. Irvine , Nicholas A. Johnson , Weimao Ke , Sanghee Oh , Anezka Poljakova , Marcos A. Rodriguez , Megan G. van Noord , Yan Zhang, Effects of performance feedback on users' evaluations of an interactive IR system, Proceedings of the second international symposium on Information interaction in context, October 14-17, 2008, London, United Kingdom
	Ye Diana Wang , Guisseppi Forgionne, Testing a decision-theoretic approach to the evaluation of information retrieval systems, Journal of Information Science, v.34 n.6, p.861-876, December 2008
	Alistair Moffat , Justin Zobel, Rank-biased precision for measurement of retrieval effectiveness, ACM Transactions on Information Systems (TOIS), v.27 n.1, p.1-27, December 2008
	Diane Kelly, Methods for Evaluating Interactive Information Retrieval Systems with Users, Foundations and Trends in Information Retrieval, v.3 n.1—2, p.1-224, January 2009