Sound and complete relevance assessment for XML retrieval

ABSTRACT

In information retrieval research, comparing retrieval approaches requires test collections consisting of documents, user requests and relevance assessments. Obtaining relevance assessments that are as sound and complete as possible is crucial for the comparison of retrieval approaches. In XML retrieval, the problem of obtaining sound and complete relevance assessments is further complicated by the structural relationships between retrieval results.

A major difference between XML retrieval and flat document retrieval is that the relevance of elements (the retrievable units) is not independent of that of related elements. This has major consequences for the gathering of relevance assessments. This article describes investigations into the creation of sound and complete relevance assessments for the evaluation of content-oriented XML retrieval as carried out at INEX, the evaluation campaign for XML retrieval. The campaign, now in its seventh year, has had three substantially different approaches to gather assessments and has finally settled on a highlighting method for marking relevant passages within documents—even though the objective is to collect assessments at element level. The different methods of gathering assessments at INEX are discussed and contrasted. The highlighting method is shown to be the most reliable of the methods.

REFERENCES

• 1: Baeza-Yates, R., Fuhr, N., and Maarek, Y. S. (Eds.). 2002. Proceedings of the ACM SIGIR Workshop on XML.
• 2: Blanken, H. M., Grabs, T., Schek, H.-J., Schenkel, R., and Weikum, G. (Eds.). 2003. Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks. Lecture Notes in Computer Science, vol. 2818. Springer, Berlin, Germany.
• 3: Carmel, D., Maarek, Y., and Soffer, A. (Eds.). 2000. Proceedings of the ACM SIGIR Workshop on XML.
• 4: Clarke, C. 2005. Range results in XML retrieval. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 4--5.
• 5: Cleverdon, C. W., Mills, J., and Keen, M. 1966. Factors determining the performance of indexing systems: Cranfield University, Cranfield, Bedforshine, U.K.
• 6: Cormack, G. V., Palmer, C. R., To, S. S. L., and Clarke, C. L. A. 1997. Passage-based refinement (multitext experiements for TREC-6). In Proceedings of the 6th Text REtrieval Conference (TREC-6), 171--186.
• 7: Denoyer, L. and Gallinari, P. 2006. The Wikipedia XML corpus. In Proceedings of the INEX Workshop.
• 8: Fuhr, N., Kamps, J., Lalmas, M., Malik, S., and Trotman, A. 2007. Overview of the INEX 2007 ad hoc track. In Proceedings of INEX. 1--22.
• 9: Harman, D. 1992. Overview of the first text retrieval conference (TREC-1). In Proceedings of the 1st Text REtrieval Conference (TREC-1).
• 10: Harman, D. 1995. The TREC conferences. In Proceedings of the HIM International Conference.
• 11: Jaap Kamps , Marijn Koolen , Mounia Lalmas, Where to start reading a textual XML document?, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277877]
• 12: Kazai, G. and Lalmas, M. 2005. INEX 2005 evaluation metrics. In Proceedings of INEX.
• 13: Kazai, G., Masood, S., and Lalmas, M. 2004. A study of the assessment of relevance for the INEX'02 test collection. In Proceedings of the 26th European Colloquium on Information Retrieval Research (ECIR 2004).
• 14: Jaana Kekäläinen , Kalervo Järvelin, Using graded relevance assessments in IR evaluation, Journal of the American Society for Information Science and Technology, v.53 n.13, p.1120-1129, November 2002 [doi>10.1002/asi.10137]
• 15: Mounia Lalmas , Anastasios Tombros, Evaluating XML retrieval effectiveness at INEX, ACM SIGIR Forum, v.41 n.1, p.40-57, June 2007 [doi>10.1145/1273221.1273225]
• 16: Robert W.P. Luk , H. V. Leong , Tharam S. Dillon , Alvin T.S. Chan , W. Bruce Croft , James Allan, A survey in indexing and searching XML documents, Journal of the American Society for Information Science and Technology, v.53 n.6, p.415-437, May, 2002 [doi>10.1002/asi.10056]
• 17: Malik, S., Kazai, G., Lalmas, M., and Fuhr, N. 2005. Overview of INEX 2005. In Proceedings of the INEX Workshop. 1--15.
• 18: Paul Ogilvie , Mounia Lalmas, Investigating the exhaustivity dimension in content-oriented XML element retrieval evaluation, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183631]
• 19: Pehcevski, J. and Thom, J. A. 2005. HiXEval: Highlighting XML retrieval evaluation. In Proceedings of the INEX Workshop.
• 20: Pehcevski, J., Thom, J. A. and Vercoustre, A.-M. 2005. Users and assessors in the context of INEX: Are relevance dimensions relevant? In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 47--62.
• 21: Benjamin Piwowarski , Mounia Lalmas, Providing consistent and exhaustive relevance assessments for XML retrieval evaluation, Proceedings of the thirteenth ACM international conference on Information and knowledge management, November 08-13, 2004, Washington, D.C., USA [doi>10.1145/1031171.1031246]
• 22: Saracevic, T. 1991. Individual differences in organizing, searching and retrieving information. Proc. Amer. Soc. Inform. Sci. 28, 82--86.
• 23: Tombros, A., Larsen, B., and Malik, S. 2004. The interactive track at INEX 2004. In Proceedings of the INEX Workshop. 410--423.
• 24: Trotman, A. 2005. Wanted: Element retrieval users. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 63--69.
• 25: Trotman, A. and Geva, S. 2006. Passage retrieval and other XML-retrieval tasks. In Proceedings of the SIGIR Workshop on XML Element Retrieval Methodology. 43--50.
• 26: Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, Information Processing and Management: an International Journal, v.36 n.5, p.697-716, Sept. 2000 [doi>10.1016/S0306-4573(00)00010-8]
• 27: Huyen-Trang Vu , Patrick Gallinari, Using RankBoost to compare retrieval systems, Proceedings of the 14th ACM international conference on Information and knowledge management, October 31-November 05, 2005, Bremen, Germany [doi>10.1145/1099554.1099641]
• 28: Woodley, A. and Geva, S. 2005. Fine tuning INEX. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 70--79.
• 29: Justin Zobel, How reliable are the results of large-scale information retrieval experiments?, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.307-314, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291014]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Measurement, Performance, Standardization

Keywords:
INEX, XML, XML retrieval, evaluation, passage retrieval, relevance assessment