Visualizing the problems with the INEX topics

ABSTRACT

Topics form a crucial component of a test collection. We show, through visualization, that the INEX 2008 topics have shortcomings, which questions their validity for evaluating XML retrieval effectiveness.

REFERENCES

• 1: Chris Buckley , Ellen M. Voorhees, Evaluating evaluation measure stability, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.33-40, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345543]
• 2: David Hawking , Nick Craswell , Peter Bailey , Kathleen Griffihs, Measuring Search Engine Quality, Information Retrieval, v.4 n.1, p.33-59, April 2001 [doi>10.1023/A:1011468107287]
• 3: Jaap Kamps , Marijn Koolen , Andrew Trotman, Comparative analysis of clicks and judgments for IR evaluation, Proceedings of the 2009 workshop on Web Search Click Data, p.80-87, February 09-09, 2009, Barcelona, Spain [doi>10.1145/1507509.1507522]
• 4: Lehtonen, M. Designing User Studies for XML Retrieval. SIGIR 2006 Workshop on XML Element Retrieval Methodology, pp28--34.
• 5: Tetsuya Sakai, Evaluating evaluation metrics based on the bootstrap, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148261]
• 6: Andrew Trotman , Mounia Lalmas, Why structural hints in queries do not help XML-retrieval, Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, August 06-11, 2006, Seattle, Washington, USA [doi>10.1145/1148170.1148330]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Experimentation, Human Factors, Reliability, Verification

Keywords:
INEX, IR methodology, XML-IR, element retrieval