The overlap problem in content-oriented XML retrieval evaluation

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

The overlap problem in content-oriented XML retrieval evaluation

Full text

Pdf (215 KB)

Source

Annual ACM Conference on Research and Development in Information Retrieval archive
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval table of contents

Sheffield, United Kingdom

SESSION: XML retrieval table of contents

Pages: 72 - 79

Year of Publication: 2004

ISBN:1-58113-881-4

Authors

Gabriella Kazai	Queen Mary University of London, London, UK
Mounia Lalmas	Queen Mary University of London, London, UK
Arjen P. de Vries	CWI, Amsterdam, The Netherlands

Sponsors

ACM: Association for Computing Machinery
SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 5, Downloads (12 Months): 49, Citation Count: 17

Additional Information:

abstract references cited by index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1008992.1009008 What is a DOI?

ABSTRACT

Within the INitiative for the Evaluation of XML Retrieval(INEX) a number of metrics to evaluate the effectiveness of content-oriented XML retrieval approaches were developed. Although these metrics provide a solution towards addressing the problem of overlapping result elements, they do not consider the problem of overlapping reference components within the recall-base, thus leading to skewed effectiveness scores. We propose alternative metrics that aim to provide a solution to both overlap issues.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	H. M. Blanken, T. Grabs, H.-J. Schek, R. Schenkel, and G. Weikum, editors. Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks, volume 2818 of LNCS. Springer, 2003.
	2	J. Clark and S. DeRose. XML Path Language(XPath) version 1.0. W3C Recommendation. http://www.w3.org/TR/xpath. Technical Report REC-xpath-19991116, WWW Consortium, Nov. 1999.
	3	W. Cooper. Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19(1):30--41, 1968.
	4	A. de Vries, G. Kazai, and M. Lalmas. Tolerance to I rrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In Proceedings of the Recherche d'Informations Assistee par Ordinateur(RIAO 2004), Avignon, France, Apr. 2004.
	5	N. Fuhr, M. Lalmas, and S. Malik, editors. Proceedings of the Second Workshop of the INitiative for the Evaluation of XML Retrieval(INEX). Dagstuhl, Germany, December 15--17, 2003, 2004. http://inex.is.informatik.uni-duisburg.de:2003/proceedings.pdf.
	6	N. Gövert, G. Kazai, N. Fuhr, and M. Lalmas. Evaluating the effectiveness of content-oriented XML retrieval Technischer bericht, University of Dortmund, Computer Science 6, 2003.
	7	Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
	8	Gabriella Kazai , Mounia Lalmas , Norbert Fuhr , Norbert Gövert, A report on the first year of the INitiative for the evaluation of XML retrieval (INEX'02), Journal of the American Society for Information Science and Technology, v.55 n.6, p.551-556, April 2004 [doi>10.1002/asi.10386]
	9	Jaana Kekäläinen , Kalervo Järvelin, Using graded relevance assessments in IR evaluation, Journal of the American Society for Information Science and Technology, v.53 n.13, p.1120-1129, November 2002 [doi>10.1002/asi.10137]
	10	Vijay Raghavan , Peter Bollmann , Gwang S. Jung, A critical investigation of recall and precision as measures of retrieval system performance, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.205-229, July 1989 [doi>10.1145/65943.65945]
	11	S. K. M. Wong , Y. Y. Yao, On modeling information retrieval with probabilistic inference, ACM Transactions on Information Systems (TOIS), v.13 n.1, p.38-68, Jan. 1995 [doi>10.1145/195705.195713]

CITED BY 17

	Charles L. A. Clarke , Egidio L. Terra, Approximating the top-m passages in a parallel question answering system, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA
	Charles L. A. Clarke, Controlling overlap in content-oriented XML retrieval, Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, August 15-19, 2005, Salvador, Brazil
	Stamatina Betsi , Mounia Lalmas , Anastasios Tombros , Theodora Tsikrika, User expectations from XML element retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	Thomas Rölleke , Theodora Tsikrika , Gabriella Kazai, A general matrix framework for modelling information retrieval, Information Processing and Management: an International Journal, v.42 n.1, p.4-30, January 2006
	Barbara Hammer-Aebi , Kirstine Wilfred Christensen , Haakon Lund , Birger Larsen, Users, structured documents and overlap: interactive searching of elements and the influence of context on search behaviour, Proceedings of the 1st international conference on Information interaction in context, October 18-20, 2006, Copenhagen, Denmark
	Sujeet Pradhan, An algebraic query model for effective and efficient retrieval of XML fragments, Proceedings of the 32nd international conference on Very large data bases, September 12-15, 2006, Seoul, Korea
	Miro Lehtonen, Preparing heterogeneous XML for full-text search, ACM Transactions on Information Systems (TOIS), v.24 n.4, p.455-474, October 2006
	Benjamin Piwowarski , Georges Dupret, Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM), Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA
	B. Piwowarski , P. Gallinari , G. Dupret, Precision recall with user modeling (PRUM): Application to structured information retrieval, ACM Transactions on Information Systems (TOIS), v.25 n.1, p.1-es, February 2007
	Paul Ogilvie , Mounia Lalmas, Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA
	Gabriella Kazai , Mounia Lalmas, eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval, ACM Transactions on Information Systems (TOIS), v.24 n.4, p.503-542, October 2006
	Sihem Amer-Yahia , Mounia Lalmas, XML search: languages, INEX and scoring, ACM SIGMOD Record, v.35 n.4, p.16-23, December 2006
	Mounia Lalmas , Anastasios Tombros, Evaluating XML retrieval effectiveness at INEX, ACM SIGIR Forum, v.41 n.1, p.40-57, June 2007
	Stephen Robertson , Hugo Zaragoza, On rank-based effectiveness measures and optimization, Information Retrieval, v.10 n.3, p.321-339, June 2007
	Johan List , Vojkan Mihajlovic , Vojkan Mihajlovi , Georgina Ramírez , Arjen P. Vries , Djoerd Hiemstra , Henk Ernst Blok, TIJAH: Embracing IR Methods in XML Databases, Information Retrieval, v.8 n.4, p.547-570, December 2005
	Jovan Pehcevski , James A. Thom , Anne-Marie Vercoustre, Hybrid XML Retrieval: Combining Information Retrieval and a Native XML Database, Information Retrieval, v.8 n.4, p.571-600, December 2005
	Ray R. Larson, A Fusion Approach to XML Structured Document Retrieval, Information Retrieval, v.8 n.4, p.601-629, December 2005