Evaluating XML retrieval effectiveness at INEX

ABSTRACT

The INitiative for the Evaluation of XML retrieval (INEX) was set up in 2002 to establish an infrastructure and provide means, in the form of large test collections and appropriate scoring methods, for evaluating the effectiveness of content-oriented XML retrieval systems. This report provides an overview of the evaluation methodology developed in INEX from 2002 to 2006.

REFERENCES

• 1: C. L. Clarke. Range results in XML retrieval. In INEX 2005 Workshop on Element Retrieval Methodology, pages 4--5, 2005.
• 2: Ludovic Denoyer , Patrick Gallinari, The Wikipedia XML corpus, ACM SIGIR Forum, v.40 n.1, June 2006 [doi>10.1145/1147197.1147210]
• 3: N. Fuhr, N. Gövert, G. Kazai, and M. Lalmas, editors. INitiative for the Evaluation of XML Retrieval (INEX). Proceedings of the First INEX Workshop (INEX 2002), ERCIM Workshop Proceedings, 2003.
• 4: N. Fuhr, M. Lalmas, and S. Malik, editors. INitiative for the Evaluation of XML Retrieval (INEX). Proceedings of the Second INEX Workshop (INEX 2003), 2004.
• 5: Norbert Fuhr , Mounia Lalmas , Saadia Malik , Gabriella Kazai, Advances in XML Information Retrieval and Evaluation: 4th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2005, Dagstuhl ... Papers (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2006
• 6: Norbert Fuhr , Mounia Lalmas , Saadia Malik , Zoltán Szlávik, Advances in XML Information Retrieval: Third International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2004, Dagstuhl Castle, ... 2004 (Lecture Notes in Computer Science), Springer-Verlag New York, Inc., Secaucus, NJ, 2005
• 7: N. Fuhr, M. Lalmas, and A. Trotman, editors. Comparative Evaluation of XML Information Retrieval Systems, 5th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2006), Springer, 2007.
• 8: Norbert Gövert , Norbert Fuhr , Mounia Lalmas , Gabriella Kazai, Evaluating the effectiveness of content-oriented XML retrieval methods, Information Retrieval, v.9 n.6, p.699-722, December 2006 [doi>10.1007/s10791-006-9008-2]
• 9: D. Hawking, E. M. Voorhees, N. Craswell, and P. Bailey. Overview of the TREC-8 Web Track. In TREC, 1999.
• 10: Kalervo Järvelin , Jaana Kekäläinen, Cumulated gain-based evaluation of IR techniques, ACM Transactions on Information Systems (TOIS), v.20 n.4, p.422-446, October 2002 [doi>10.1145/582415.582418]
• 11: Jaap Kamps , Maarten Marx , Maarten de Rijke , Börkur Sigurbjörnsson, Structured queries in XML retrieval, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099559]
• 12: G. Kazai. Choosing an Ideal Recall-Base for the Evaluation of the Focused Task: Sensitivity Analysis of the XCG Evaluation Measures. In Fuhr et al. {7}.
• 13: G. Kazai and M. Lalmas. Notes on What to Measure in INEX. In INEX 2005 Workshop on Element Retrieval Methodology, Glasgow, July 2005.
• 14: Gabriella Kazai , Mounia Lalmas, eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval, ACM Transactions on Information Systems (TOIS), v.24 n.4, p.503-542, October 2006 [doi>10.1145/1185877.1185883]
• 15: G. Kazai and M. Lalmas. INEX 2005 Evaluation Metrics. In Fuhr et al. {5}.
• 16: Gabriella Kazai , Mounia Lalmas , Arjen P. de Vries, The overlap problem in content-oriented XML retrieval evaluation, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009008]
• 17: G. Kazai, S. Masood, and M. Lalmas. A study of the assessment of relevance for the INEX'02 test collection. In ECIR, pages 296--310, 2004.
• 18: Jaana Kekäläinen , Kalervo Järvelin, Using graded relevance assessments in IR evaluation, Journal of the American Society for Information Science and Technology, v.53 n.13, p.1120-1129, November 2002 [doi>10.1002/asi.10137]
• 19: M. Lalmas, G. Kazai, J. Kamps, J. Pehcevski, B. Piwowarski, and S. Robertson. INEX 2006 Evaluation Measures. In Fuhr et al. {7}.
• 20: M. Lalmas and A. Tombros. INEX 2002 - 2006: Understanding XML Retrieval Evaluation. In DELOS Conference on Digital Libraries, Tirrenia, Pisa, Italy, 2007.
• 21: Paul Ogilvie , Mounia Lalmas, Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183631]
• 22: J. Pehcevski and J. A. Thom. Hixeval: Highlighting xml retrieval evaluation. In Advances in XML Information Retrieval and Evaluation: Fourth Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005), 2006.
• 23: Benjamin Piwowarski , Georges Dupret, Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM), Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148218]
• 24: Benjamin Piwowarski , Mounia Lalmas, Providing consistent and exhaustive relevance assessments for XML retrieval evaluation, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA [doi>10.1145/1031171.1031246]
• 25: Vijay Raghavan , Peter Bollmann , Gwang S. Jung, A critical investigation of recall and precision as measures of retrieval system performance, ACM Transactions on Information Systems (TOIS), v.7 n.3, p.205-229, July 1989 [doi>10.1145/65943.65945]
• 26: Anastasios Tombros , Saadia Malik , Birger Larsen, Report on the INEX 2004 interactive track, ACM SIGIR Forum, v.39 n.1, June 2005 [doi>10.1145/1067268.1067283]
• 27: A. Trotman and S. Geva. Passage Retrieval and Other XML-Retrieval Tasks. In SIGIR 2006 Workshop on XML Element Retrieval Methodology, pages 43--50, 2006.
• 28: Andrew Trotman , Mounia Lalmas, Strict and vague interpretation of XML-retrieval queries, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148329]
• 29: Andrew Trotman , Mounia Lalmas, Why structural hints in queries do not help XML-retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148330]
• 30: A. Trotman and B. Sigurbjornsson. Narrowed extended XPATHI (NEXI). In INEX 2004 Workshop Proceedings, pages 16--40, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation

          Nouns: XML